b2314ef2ebb056bc54808da3b0ef25cf8fe9efab
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Eric.
4         
5         - work around IE layout bugs
6
7         * hosted/sunspider.css:
8
9 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
10
11         Reviewed by Mark.
12         
13         - fix compare feature
14
15         * hosted/sunspider-results.html:
16
17 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
18
19         Reviewed by Mark.
20         
21         - Change the format of the output URL to be shorter, to meet IE constraints
22
23         * hosted/sunspider-driver.html:
24         * hosted/sunspider-results.html:
25
26 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
27
28         Reviewed by Oliver.
29         
30         - Add some explanatory test and a dumb logo to SunSpider web hosted version
31
32         * hosted/sunspider-driver.html:
33         * hosted/sunspider-results.html:
34         * hosted/sunspider.css:
35         * hosted/sunspider.html:
36
37 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
38
39         Reviewed by Oliver.
40         
41         - Make the web-hosted version of SunSpider look a little bit less drab.
42
43         * hosted/sunspider-driver.html:
44         * hosted/sunspider-results.html:
45         * hosted/sunspider.css: Added.
46         * hosted/sunspider.html:
47         * resources/TEMPLATE.html:
48
49 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
50
51         Reviewed by Sam.
52         
53         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
54         
55         A sample of what new-style results would look like:
56         
57         TEST                   COMPARISON            FROM                 TO             DETAILS
58         ...
59         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
60         ...
61             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
62         
63         * resources/sunspider-compare-results.js:
64
65 2007-11-14  Eric Seidel  <eric@webkit.org>
66
67         Reviewed by Sam.
68
69         * sunspider: add --shark-cache for L2 Cache Miss profiling
70
71 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
72
73         Reviewed by Darin.
74
75         http://bugs.webkit.org/show_bug.cgi?id=15925
76         SunSpider should check for Shark being installed
77
78         * sunspider: Explicitly test that shark command line tool is installed.
79
80 2007-11-04  Darin Adler  <darin@apple.com>
81
82         * hosted/sunspider.html: Tweaked formatting a bit.
83
84 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
85
86         Reviewed by Darin.
87
88         - Add compare mode and a description of the benchmark to the hosted version.
89
90         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
91         * hosted/sunspider.html: Added intro text.
92         * make-hosted: Copy new file.
93         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
94         a function.
95         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
96         expects.
97         * sunspider-compare-results: Use sunspider-standalone-compare.js.
98
99 2007-11-02  Darin Adler  <darin@apple.com>
100
101         * sunspider: Changed "--base" to "--set-baseline".
102
103 2007-11-02  Darin Adler  <darin@apple.com>
104
105         Reviewed by Maciej.
106
107         * sunspider: Added a "--base" option that records the current test as a baseline.
108         Moved the code to find the newest .mshark file into a function (since I needed to use
109         it in the other script and I prefer to copy/paste a function).
110
111         * sunspider-compare-results: Added logic to compare the baseline with the last
112         run when you don't pass any parameters.
113
114 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
115
116         Reviewed by Eric.
117         
118         - Added list of ECMAScript3 features that SunSpider does not cover currently.
119
120         * UNCOVERED: Added.
121
122 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
123
124         Reviewed by Oliver.
125         
126         - Give better error messages when sunspider-compare-results fails
127
128         * sunspider-compare-results:
129
130 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
131
132         Reviewed by Darin.
133         
134         - fix Date tests to work in command-line SpiderMonkey
135
136         * tests/date-format-tofte.js:
137         * tests/date-format-xparb.js:
138
139 2007-10-29  Darin Adler  <darin@apple.com>
140
141         Reviewed by Maciej.
142
143         * sunspider: Added --shark20 option, to run Shark at its highest sample
144         resolution instead of its default.
145
146 2007-10-28  Adam Roben  <aroben@apple.com>
147
148         Fix SunSpider on Windows
149
150         Reviewed by Eric.
151
152         * sunspider: Don't use colons in filenames.
153
154 2007-10-28  Darin Adler  <darin@apple.com>
155
156         Reviewed by Adam.
157
158         * resources/sunspider-analyze-results.js: Tweak the output format a little.
159         Change so that when there's only one run we don't write out confidence
160         intervals at all rather than writing "NaN" over and over again.
161
162         * sunspider: Use the time and date as a suffix on the results file, that way
163         all the results are left behind in the tmp directory. This will make it easy
164         for us to add features that compare past results in the future. Also rename
165         the .mshark files using the same scheme.
166
167         * sunspider-compare-results: Relax the parsing rules so we can parse the
168         results file exactly as it's emitted from sunspider.
169
170 2007-10-25  Adam Roben  <aroben@apple.com>
171
172         Strip carriage returns from results
173
174         Carriage returns were screwing up sunspider-compare-results on
175         Windows.
176
177         Rubberstamped by Oliver.
178
179         * sunspider:
180
181 2007-10-24  Darin Adler  <darin@apple.com>
182
183         Reviewed by Eric.
184
185         * sunspider: Open the profile in shark after completing a run.
186         * sunspider-compare-results: Tiny perl tweaks.
187
188 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
189
190         Reviewed by Darin.
191
192         - Don't hardcode my path to testkjs
193         
194         * sunspider-compare-results:
195
196 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
197
198         Reviewed by Darin.
199         
200         - Add a compare script to compare results, and improve formatting of standard results a bit.
201
202         * resources/sunspider-analyze-results.js:
203         * resources/sunspider-compare-results.js: Added.
204         * sunspider-compare-results: Added.
205
206 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
207
208         Reviewed by Darin.
209
210         * resources/sunspider-analyze-results.js: Do the error range
211         properly, using the t-distribution instead of 1.96 (which was
212         based on the normal distribution).
213         
214         * sunspider: Print results in a way that will be friendlier to the
215         soon-to-come compare mode.
216
217 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
218
219         Reviewed by Review.
220         
221         - add more tests (probably the final set for now)
222         
223         Already balanced. Added date, regexp, control flow, and a few
224         more string and object/array access tests.
225         
226         * tests/LIST:
227         * tests/access-fannkuch.js: Added.
228         * tests/access-nbody.js: Added.
229         * tests/controlflow-recursive.js: Added.
230         * tests/date-format-tofte.js: Added.
231         * tests/date-format-xparb.js: Added.
232         * tests/regexp-dna.js: Added.
233         * tests/string-validate-input.js: Added.
234
235 2007-10-21  Eric Seidel  <eric@webkit.org>
236
237         Reviewed by mjs.
238
239         * sunspider: make --shark only sample from testkjs
240
241 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
242
243         Rubber stamped by Eric.
244         
245         - added crypto tests (already balanced)
246
247         * tests/LIST:
248         * tests/crypto-aes.js: Added.
249         * tests/crypto-md5.js: Added.
250         * tests/crypto-sha1.js: Added.
251
252 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
253
254         Reviewed by Adam.
255         
256         - rebalance new tests
257         
258         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
259         
260         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
261
262         This balances them with the existing tests.
263
264         * tests/access-binary-trees.js:
265         * tests/access-nsieve.js:
266         * tests/bitops-nsieve-bits.js:
267         * tests/math-partial-sums.js:
268         * tests/math-spectral-norm.js:
269         * tests/string-fasta.js:
270
271 2007-10-20  Eric Seidel  <eric@webkit.org>
272
273         Reviewed by darin.
274         
275         Add improved argument handling to sunspider, including
276         --runs=<number>, --tests=<pattern, --shark, and --help
277         Also re-factor code into subroutines
278
279         * sunspider: improved argument handling
280
281 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
282
283         Reviewed by Mark.
284
285         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
286         
287         * TODO:
288         * tests/LIST:
289         * tests/access-binary-trees.js: Added.
290         * tests/access-nsieve.js: Added.
291         * tests/bitops-nsieve-bits.js: Added.
292         * tests/math-partial-sums.js: Added.
293         * tests/math-spectral-norm.js: Added.
294         * tests/string-fasta.js: Added.
295
296 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
297
298         Reviewed by Darin.
299
300         * TODO: Updated to note areas that are now well-covered.
301         
302         - Change the tests and the driver to avoid leaving large object
303         graphs hanging around, since that throws off the subsequent tests.
304         
305         * resources/sunspider-standalone-driver.js:
306         * tests/3d-cube.js:
307         * tests/3d-morph.js:
308         * tests/3d-raytrace.js:
309         * tests/bitops-bitwise-and.js:
310         * tests/string-base64.js:
311         * tests/string-tagcloud.js:
312
313 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
314
315         Reviewed by Nikolas.
316         
317         - Rebalance test complexity.
318         
319         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
320         
321         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
322
323         However, for some tests, IE7 was a huge outlier compared to other
324         major browsers. For those tests, I used Firefox 2.0.0.8 on the
325         same OS and hardware as a normalization basis.
326         
327         * tests/3d-morph.js:
328         * tests/3d-raytrace.js:
329         * tests/bitops-3bit-bits-in-byte.js:
330         * tests/bitops-bits-in-byte.js:
331         * tests/bitops-bitwise-and.js:
332         * tests/math-cordic.js:
333         * tests/string-base64.js:
334         * tests/string-tagcloud.js:
335         * tests/string-unpack-code.js:
336
337 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
338
339         Reviewed by Oliver.
340         
341         - Fix the browser-hosted driver to be compatible with Safari 2
342
343         * hosted/sunspider-driver.html:
344
345 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
346
347         Rubber stamped by Hyatt.
348         
349         - Add BSD license notice to cordic test, the original author said to do so.
350
351         * tests/math-cordic.js:
352
353 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
354
355         Reviewed by Eric.
356         
357         - Support Windows and Windows browsers.
358
359         * hosted/sunspider-results.html: Display results correctly in IE.
360         * sunspider: tweaks to work on Windows, and with spaces in the path.
361
362 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
363
364         Rubber stamped by Oliver.
365
366         * tests/string-tagcloud.js: Remove stray print()
367
368 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
369
370         Reviewed by Oliver.
371         
372         Add some string test cases.
373
374         * tests/string-base64.js: Added. Does base64 encoding and decoding.
375         * tests/string-tagcloud.js: Added. Parses a JSON string and
376         generates tagcloud HTML markup based on the data.        
377         * tests/string-unpack-code.js: Added. Unpack compressed versions of
378         four of the most popular JavaScript libraries.
379         * tests/LIST: Added the new tests.
380
381 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
382
383         Reviewed by Anders.
384
385         * TODO: Updated with info on how to do the stats properly.
386
387         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
388         avoid overly slowing down the benchmark.
389
390         Added some 3D-related test cases.
391         
392         * tests/3d-cube.js: Added.
393         * tests/3d-morph.js: Added.
394         * tests/3d-raytrace.js: Added.
395         * tests/LIST:
396
397 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
398
399         Reviewed by Darin.
400
401         - New JavaScript benchmark
402         http://bugs.webkit.org/show_bug.cgi?id=15515
403         
404         A start on a new JavaScript benchmark, with standalone and
405         browser-hosted drivers.
406         
407         The test content is still incomplete and the drivers could use
408         some more features but there is enough here to make a good start.
409         
410         The drivers compute a 95% confidence interval on the mean for the
411         whole test, each category, and each individual test to make it
412         easier to tell whether differences are statistically
413         significant. The confidence interval can be narrowed by running
414         the test more times.
415         
416         * TODO: Added. Remaining things that need to be done.
417         * sunspider: Added. Perl script that acts as the standalone test driver.
418         * resources/sunspider-standalone-driver.js: Added. JavaScript part
419         of standalone test driver.
420         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
421         analysis code.
422         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
423         * make-hosted: Added. Script to generate browser-hosted tests.
424         * hosted/sunspider.html: Added. Start page for browser-hosted test.
425         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
426         * hosted/sunspider-results.html: Added. Results page for browser-hosted
427         tests.
428         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
429         tests.
430         * tests/LIST: Added. List of tests to use.
431         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
432         * tests/bitops-bits-in-byte.js: Added. Ditto.
433         * tests/bitops-bitwise-and.js: Added. Ditto.
434         * tests/math-cordic.js: Added. Ditto.
435