2008-07-01 Alex Mathews <possessedpenguinbob@gmail.com>
[WebKit-https.git] / SunSpider / ChangeLog
1 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
2
3         Reviewed by Timothy.
4
5         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
6
7         * sunspider:
8         * tests/LIST-SQUIRRELFISH: Removed.
9
10 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
11
12         Rubber-stamped by Mark Rowe.
13
14         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
15
16         * sunspider:
17
18 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
19
20         Rubber stamped by Oliver.
21         
22         - add newly working tests to squirrelfish list
23
24         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
25
26 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
27
28         Rubber stamped by Oliver.
29         
30         - move ubench tests properly.
31
32         * tests/ubench: Added.
33         * tests/ubench/function-closure.js: Added.
34         * tests/ubench/function-empty.js: Added.
35         * tests/ubench/function-missing-args.js: Added.
36         * tests/ubench/function-sum.js: Added.
37         * tests/ubench/loop-empty-resolve.js: Added.
38         * tests/ubench/loop-empty.js: Added.
39         * tests/ubench/loop-sum.js: Added.
40
41 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
42
43         Rubber stamped by Oliver.
44
45         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
46
47 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
48
49         Reviewed by Oliver.
50
51         - make --squirrelfish mode use the real harness, but a reduced set of tests
52
53         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
54
55         * sunspider:
56         * tests/LIST-SQUIRRELFISH:
57         * tests/LIST-UBENCH: Added.
58         * tests/squirrelfish: Moved to tests/ubench
59
60 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
61
62         Reviewed by Maciej Stachowiak.
63         
64         Removed an unused test.
65
66         * tests/squirrelfish/loop-resolve.js: Removed.
67
68 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
69
70         Reviewed by Oliver.
71         
72         Add newly runnable tests to --squirrelfish mode.
73
74         * tests/LIST-SQUIRRELFISH:
75
76 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
77
78         Reviewed by Maciej Stachowiak.
79         
80         A little more love for --squirrelfish mode:
81         
82         Fixed a misplaced sort. Results properly sort now.
83         
84         Added a test list just for squirrelfish, and updated it to include
85         all currently passing SunSpider tests.
86         
87         Fixed the pruning regexp to match 3d-morph.js.
88
89         * sunspider:
90
91 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
92
93         Reviewed by Oliver Hunt.
94         
95         --squirrelfish mode: pared down tests for squirrelfish to chew on.
96
97         * sunspider:
98         * tests/LIST:
99         * tests/squirrelfish: Added.
100         * tests/squirrelfish/function-closure.js: Added.
101         * tests/squirrelfish/function-empty.js: Added.
102         * tests/squirrelfish/function-missing-args.js: Added.
103         * tests/squirrelfish/function-sum.js: Added.
104         * tests/squirrelfish/loop-empty-resolve.js: Added.
105         * tests/squirrelfish/loop-empty.js: Added.
106         * tests/squirrelfish/loop-resolve.js: Added.
107         * tests/squirrelfish/loop-sum.js: Added.
108
109 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
110
111         Reviewed by Geoff.
112
113         * sunspider: Pause update daemon on Darwin to make results reliable
114         again (workaround for <rdar://problem/5811127>).
115
116 2008-02-07  Eric Seidel  <eric@webkit.org>
117
118         Reviewed by Sam.
119
120         * sunspider: add an error message when ./tests/LIST is not found
121
122 2007-12-29  Alp Toker  <alp@atoker.com>
123
124         Fix typo.
125
126         * hosted/sunspider.html:
127
128 2007-12-29  Alp Toker  <alp@atoker.com>
129
130         Reviewed by Maciej.
131
132         Provide the current URL in the SunSpider results page. Useful for
133         copying benchmark results out of browsers without a location bar or
134         where the location bar can't handle long URLs.
135
136         * hosted/sunspider-results.html:
137
138 2007-12-20  Eric Seidel  <eric@webkit.org>
139
140         Reviewed by mjs.
141
142         * sunspider: increase default number of runs for more accurate numbers
143
144 2007-12-18  Mark Rowe  <mrowe@apple.com>
145
146         Fix typo.
147
148         * hosted/sunspider.html:
149
150 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
151
152         Reviewed by Eric.
153         
154         - give an extra digit of precision for small differences.
155
156         * resources/sunspider-compare-results.js:
157
158 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
159
160         Reviewed by Eric.
161         
162         - work around IE layout bugs
163
164         * hosted/sunspider.css:
165
166 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
167
168         Reviewed by Mark.
169         
170         - fix compare feature
171
172         * hosted/sunspider-results.html:
173
174 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
175
176         Reviewed by Mark.
177         
178         - Change the format of the output URL to be shorter, to meet IE constraints
179
180         * hosted/sunspider-driver.html:
181         * hosted/sunspider-results.html:
182
183 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
184
185         Reviewed by Oliver.
186         
187         - Add some explanatory test and a dumb logo to SunSpider web hosted version
188
189         * hosted/sunspider-driver.html:
190         * hosted/sunspider-results.html:
191         * hosted/sunspider.css:
192         * hosted/sunspider.html:
193
194 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
195
196         Reviewed by Oliver.
197         
198         - Make the web-hosted version of SunSpider look a little bit less drab.
199
200         * hosted/sunspider-driver.html:
201         * hosted/sunspider-results.html:
202         * hosted/sunspider.css: Added.
203         * hosted/sunspider.html:
204         * resources/TEMPLATE.html:
205
206 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
207
208         Reviewed by Sam.
209         
210         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
211         
212         A sample of what new-style results would look like:
213         
214         TEST                   COMPARISON            FROM                 TO             DETAILS
215         ...
216         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
217         ...
218             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
219         
220         * resources/sunspider-compare-results.js:
221
222 2007-11-14  Eric Seidel  <eric@webkit.org>
223
224         Reviewed by Sam.
225
226         * sunspider: add --shark-cache for L2 Cache Miss profiling
227
228 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
229
230         Reviewed by Darin.
231
232         http://bugs.webkit.org/show_bug.cgi?id=15925
233         SunSpider should check for Shark being installed
234
235         * sunspider: Explicitly test that shark command line tool is installed.
236
237 2007-11-04  Darin Adler  <darin@apple.com>
238
239         * hosted/sunspider.html: Tweaked formatting a bit.
240
241 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
242
243         Reviewed by Darin.
244
245         - Add compare mode and a description of the benchmark to the hosted version.
246
247         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
248         * hosted/sunspider.html: Added intro text.
249         * make-hosted: Copy new file.
250         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
251         a function.
252         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
253         expects.
254         * sunspider-compare-results: Use sunspider-standalone-compare.js.
255
256 2007-11-02  Darin Adler  <darin@apple.com>
257
258         * sunspider: Changed "--base" to "--set-baseline".
259
260 2007-11-02  Darin Adler  <darin@apple.com>
261
262         Reviewed by Maciej.
263
264         * sunspider: Added a "--base" option that records the current test as a baseline.
265         Moved the code to find the newest .mshark file into a function (since I needed to use
266         it in the other script and I prefer to copy/paste a function).
267
268         * sunspider-compare-results: Added logic to compare the baseline with the last
269         run when you don't pass any parameters.
270
271 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
272
273         Reviewed by Eric.
274         
275         - Added list of ECMAScript3 features that SunSpider does not cover currently.
276
277         * UNCOVERED: Added.
278
279 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
280
281         Reviewed by Oliver.
282         
283         - Give better error messages when sunspider-compare-results fails
284
285         * sunspider-compare-results:
286
287 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
288
289         Reviewed by Darin.
290         
291         - fix Date tests to work in command-line SpiderMonkey
292
293         * tests/date-format-tofte.js:
294         * tests/date-format-xparb.js:
295
296 2007-10-29  Darin Adler  <darin@apple.com>
297
298         Reviewed by Maciej.
299
300         * sunspider: Added --shark20 option, to run Shark at its highest sample
301         resolution instead of its default.
302
303 2007-10-28  Adam Roben  <aroben@apple.com>
304
305         Fix SunSpider on Windows
306
307         Reviewed by Eric.
308
309         * sunspider: Don't use colons in filenames.
310
311 2007-10-28  Darin Adler  <darin@apple.com>
312
313         Reviewed by Adam.
314
315         * resources/sunspider-analyze-results.js: Tweak the output format a little.
316         Change so that when there's only one run we don't write out confidence
317         intervals at all rather than writing "NaN" over and over again.
318
319         * sunspider: Use the time and date as a suffix on the results file, that way
320         all the results are left behind in the tmp directory. This will make it easy
321         for us to add features that compare past results in the future. Also rename
322         the .mshark files using the same scheme.
323
324         * sunspider-compare-results: Relax the parsing rules so we can parse the
325         results file exactly as it's emitted from sunspider.
326
327 2007-10-25  Adam Roben  <aroben@apple.com>
328
329         Strip carriage returns from results
330
331         Carriage returns were screwing up sunspider-compare-results on
332         Windows.
333
334         Rubberstamped by Oliver.
335
336         * sunspider:
337
338 2007-10-24  Darin Adler  <darin@apple.com>
339
340         Reviewed by Eric.
341
342         * sunspider: Open the profile in shark after completing a run.
343         * sunspider-compare-results: Tiny perl tweaks.
344
345 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
346
347         Reviewed by Darin.
348
349         - Don't hardcode my path to testkjs
350         
351         * sunspider-compare-results:
352
353 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
354
355         Reviewed by Darin.
356         
357         - Add a compare script to compare results, and improve formatting of standard results a bit.
358
359         * resources/sunspider-analyze-results.js:
360         * resources/sunspider-compare-results.js: Added.
361         * sunspider-compare-results: Added.
362
363 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
364
365         Reviewed by Darin.
366
367         * resources/sunspider-analyze-results.js: Do the error range
368         properly, using the t-distribution instead of 1.96 (which was
369         based on the normal distribution).
370         
371         * sunspider: Print results in a way that will be friendlier to the
372         soon-to-come compare mode.
373
374 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
375
376         Reviewed by Review.
377         
378         - add more tests (probably the final set for now)
379         
380         Already balanced. Added date, regexp, control flow, and a few
381         more string and object/array access tests.
382         
383         * tests/LIST:
384         * tests/access-fannkuch.js: Added.
385         * tests/access-nbody.js: Added.
386         * tests/controlflow-recursive.js: Added.
387         * tests/date-format-tofte.js: Added.
388         * tests/date-format-xparb.js: Added.
389         * tests/regexp-dna.js: Added.
390         * tests/string-validate-input.js: Added.
391
392 2007-10-21  Eric Seidel  <eric@webkit.org>
393
394         Reviewed by mjs.
395
396         * sunspider: make --shark only sample from testkjs
397
398 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
399
400         Rubber stamped by Eric.
401         
402         - added crypto tests (already balanced)
403
404         * tests/LIST:
405         * tests/crypto-aes.js: Added.
406         * tests/crypto-md5.js: Added.
407         * tests/crypto-sha1.js: Added.
408
409 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
410
411         Reviewed by Adam.
412         
413         - rebalance new tests
414         
415         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
416         
417         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
418
419         This balances them with the existing tests.
420
421         * tests/access-binary-trees.js:
422         * tests/access-nsieve.js:
423         * tests/bitops-nsieve-bits.js:
424         * tests/math-partial-sums.js:
425         * tests/math-spectral-norm.js:
426         * tests/string-fasta.js:
427
428 2007-10-20  Eric Seidel  <eric@webkit.org>
429
430         Reviewed by darin.
431         
432         Add improved argument handling to sunspider, including
433         --runs=<number>, --tests=<pattern, --shark, and --help
434         Also re-factor code into subroutines
435
436         * sunspider: improved argument handling
437
438 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
439
440         Reviewed by Mark.
441
442         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
443         
444         * TODO:
445         * tests/LIST:
446         * tests/access-binary-trees.js: Added.
447         * tests/access-nsieve.js: Added.
448         * tests/bitops-nsieve-bits.js: Added.
449         * tests/math-partial-sums.js: Added.
450         * tests/math-spectral-norm.js: Added.
451         * tests/string-fasta.js: Added.
452
453 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
454
455         Reviewed by Darin.
456
457         * TODO: Updated to note areas that are now well-covered.
458         
459         - Change the tests and the driver to avoid leaving large object
460         graphs hanging around, since that throws off the subsequent tests.
461         
462         * resources/sunspider-standalone-driver.js:
463         * tests/3d-cube.js:
464         * tests/3d-morph.js:
465         * tests/3d-raytrace.js:
466         * tests/bitops-bitwise-and.js:
467         * tests/string-base64.js:
468         * tests/string-tagcloud.js:
469
470 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
471
472         Reviewed by Nikolas.
473         
474         - Rebalance test complexity.
475         
476         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
477         
478         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
479
480         However, for some tests, IE7 was a huge outlier compared to other
481         major browsers. For those tests, I used Firefox 2.0.0.8 on the
482         same OS and hardware as a normalization basis.
483         
484         * tests/3d-morph.js:
485         * tests/3d-raytrace.js:
486         * tests/bitops-3bit-bits-in-byte.js:
487         * tests/bitops-bits-in-byte.js:
488         * tests/bitops-bitwise-and.js:
489         * tests/math-cordic.js:
490         * tests/string-base64.js:
491         * tests/string-tagcloud.js:
492         * tests/string-unpack-code.js:
493
494 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
495
496         Reviewed by Oliver.
497         
498         - Fix the browser-hosted driver to be compatible with Safari 2
499
500         * hosted/sunspider-driver.html:
501
502 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
503
504         Rubber stamped by Hyatt.
505         
506         - Add BSD license notice to cordic test, the original author said to do so.
507
508         * tests/math-cordic.js:
509
510 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
511
512         Reviewed by Eric.
513         
514         - Support Windows and Windows browsers.
515
516         * hosted/sunspider-results.html: Display results correctly in IE.
517         * sunspider: tweaks to work on Windows, and with spaces in the path.
518
519 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
520
521         Rubber stamped by Oliver.
522
523         * tests/string-tagcloud.js: Remove stray print()
524
525 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
526
527         Reviewed by Oliver.
528         
529         Add some string test cases.
530
531         * tests/string-base64.js: Added. Does base64 encoding and decoding.
532         * tests/string-tagcloud.js: Added. Parses a JSON string and
533         generates tagcloud HTML markup based on the data.        
534         * tests/string-unpack-code.js: Added. Unpack compressed versions of
535         four of the most popular JavaScript libraries.
536         * tests/LIST: Added the new tests.
537
538 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
539
540         Reviewed by Anders.
541
542         * TODO: Updated with info on how to do the stats properly.
543
544         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
545         avoid overly slowing down the benchmark.
546
547         Added some 3D-related test cases.
548         
549         * tests/3d-cube.js: Added.
550         * tests/3d-morph.js: Added.
551         * tests/3d-raytrace.js: Added.
552         * tests/LIST:
553
554 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
555
556         Reviewed by Darin.
557
558         - New JavaScript benchmark
559         http://bugs.webkit.org/show_bug.cgi?id=15515
560         
561         A start on a new JavaScript benchmark, with standalone and
562         browser-hosted drivers.
563         
564         The test content is still incomplete and the drivers could use
565         some more features but there is enough here to make a good start.
566         
567         The drivers compute a 95% confidence interval on the mean for the
568         whole test, each category, and each individual test to make it
569         easier to tell whether differences are statistically
570         significant. The confidence interval can be narrowed by running
571         the test more times.
572         
573         * TODO: Added. Remaining things that need to be done.
574         * sunspider: Added. Perl script that acts as the standalone test driver.
575         * resources/sunspider-standalone-driver.js: Added. JavaScript part
576         of standalone test driver.
577         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
578         analysis code.
579         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
580         * make-hosted: Added. Script to generate browser-hosted tests.
581         * hosted/sunspider.html: Added. Start page for browser-hosted test.
582         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
583         * hosted/sunspider-results.html: Added. Results page for browser-hosted
584         tests.
585         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
586         tests.
587         * tests/LIST: Added. List of tests to use.
588         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
589         * tests/bitops-bits-in-byte.js: Added. Ditto.
590         * tests/bitops-bitwise-and.js: Added. Ditto.
591         * tests/math-cordic.js: Added. Ditto.
592