2009-02-12 Simon Fraser <simon.fraser@apple.com>
[WebKit-https.git] / SunSpider / ChangeLog
1 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
2
3         Fix a bug where when V8's or UBENCH's test suite list file is missing
4         the error message is suggesting that SunSpider's list file is missing.
5
6         Reviewed by Sam Weinig.
7
8         * sunspider:
9         (loadTestsList): Fix the error message to use the right list name.
10
11 2008-10-07  David Hyatt  <hyatt@apple.com>
12
13         https://bugs.webkit.org/show_bug.cgi?id=18989
14
15         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
16         we want all browsers to be doing the same thing on the test.
17
18         Reviewed by Maciej
19
20         * tests/regexp-dna.js:
21
22 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
23
24         Rubber-stamped by Maciej Stachowiak.
25
26         The bleeding edge branch of V8 now supports passing fileanmes after
27         '-f', so the SunSpider script should be changed to do this again.
28         This allows all 3 major JavaScript engines to be used with the
29         unmodified SunSpider script.
30
31         * sunspider:
32
33 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
34
35         Reviewed by Maciej Stachowiak.
36
37         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
38         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
39         correct the punctuation in its documentation.
40
41         * sunspider:
42
43 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
44
45         Reviewed by Oliver Hunt.
46
47         Bug 20616: Incorporate V8 benchmarks in testing
48         <https://bugs.webkit.org/show_bug.cgi?id=20616>
49
50         This is a first attempt at running the V8 benchmarks within SunSpider,
51         with the --v8 option. The number of iterations is fixed for each
52         benchmark, and was chosen to make each of the benchmarks run between
53         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
54
55         * sunspider:
56         * tests/LIST-V8: Added.
57         * tests/v8-crypto.js: Added.
58         * tests/v8-deltablue.js: Added.
59         * tests/v8-earley-boyer.js: Added.
60         * tests/v8-raytrace.js: Added.
61         * tests/v8-richards.js: Added.
62
63 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
64
65         Not reviewed.
66
67         Remove a local change that was accidentally committed in r36034.
68
69         * resources/sunspider-standalone-driver.js:
70
71 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
72
73         Rubber-stamped by Oliver Hunt.
74
75         Make SunSpider work with JavaScript shells that don't take filenames
76         after a '-f' argument.
77
78         * sunspider:
79
80 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
81
82         Reviewed by Timothy.
83
84         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
85
86         * sunspider:
87         * tests/LIST-SQUIRRELFISH: Removed.
88
89 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
90
91         Rubber-stamped by Mark Rowe.
92
93         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
94
95         * sunspider:
96
97 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
98
99         Rubber stamped by Oliver.
100         
101         - add newly working tests to squirrelfish list
102
103         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
104
105 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
106
107         Rubber stamped by Oliver.
108         
109         - move ubench tests properly.
110
111         * tests/ubench: Added.
112         * tests/ubench/function-closure.js: Added.
113         * tests/ubench/function-empty.js: Added.
114         * tests/ubench/function-missing-args.js: Added.
115         * tests/ubench/function-sum.js: Added.
116         * tests/ubench/loop-empty-resolve.js: Added.
117         * tests/ubench/loop-empty.js: Added.
118         * tests/ubench/loop-sum.js: Added.
119
120 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
121
122         Rubber stamped by Oliver.
123
124         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
125
126 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
127
128         Reviewed by Oliver.
129
130         - make --squirrelfish mode use the real harness, but a reduced set of tests
131
132         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
133
134         * sunspider:
135         * tests/LIST-SQUIRRELFISH:
136         * tests/LIST-UBENCH: Added.
137         * tests/squirrelfish: Moved to tests/ubench
138
139 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
140
141         Reviewed by Maciej Stachowiak.
142         
143         Removed an unused test.
144
145         * tests/squirrelfish/loop-resolve.js: Removed.
146
147 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
148
149         Reviewed by Oliver.
150         
151         Add newly runnable tests to --squirrelfish mode.
152
153         * tests/LIST-SQUIRRELFISH:
154
155 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
156
157         Reviewed by Maciej Stachowiak.
158         
159         A little more love for --squirrelfish mode:
160         
161         Fixed a misplaced sort. Results properly sort now.
162         
163         Added a test list just for squirrelfish, and updated it to include
164         all currently passing SunSpider tests.
165         
166         Fixed the pruning regexp to match 3d-morph.js.
167
168         * sunspider:
169
170 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
171
172         Reviewed by Oliver Hunt.
173         
174         --squirrelfish mode: pared down tests for squirrelfish to chew on.
175
176         * sunspider:
177         * tests/LIST:
178         * tests/squirrelfish: Added.
179         * tests/squirrelfish/function-closure.js: Added.
180         * tests/squirrelfish/function-empty.js: Added.
181         * tests/squirrelfish/function-missing-args.js: Added.
182         * tests/squirrelfish/function-sum.js: Added.
183         * tests/squirrelfish/loop-empty-resolve.js: Added.
184         * tests/squirrelfish/loop-empty.js: Added.
185         * tests/squirrelfish/loop-resolve.js: Added.
186         * tests/squirrelfish/loop-sum.js: Added.
187
188 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
189
190         Reviewed by Geoff.
191
192         * sunspider: Pause update daemon on Darwin to make results reliable
193         again (workaround for <rdar://problem/5811127>).
194
195 2008-02-07  Eric Seidel  <eric@webkit.org>
196
197         Reviewed by Sam.
198
199         * sunspider: add an error message when ./tests/LIST is not found
200
201 2007-12-29  Alp Toker  <alp@atoker.com>
202
203         Fix typo.
204
205         * hosted/sunspider.html:
206
207 2007-12-29  Alp Toker  <alp@atoker.com>
208
209         Reviewed by Maciej.
210
211         Provide the current URL in the SunSpider results page. Useful for
212         copying benchmark results out of browsers without a location bar or
213         where the location bar can't handle long URLs.
214
215         * hosted/sunspider-results.html:
216
217 2007-12-20  Eric Seidel  <eric@webkit.org>
218
219         Reviewed by mjs.
220
221         * sunspider: increase default number of runs for more accurate numbers
222
223 2007-12-18  Mark Rowe  <mrowe@apple.com>
224
225         Fix typo.
226
227         * hosted/sunspider.html:
228
229 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
230
231         Reviewed by Eric.
232         
233         - give an extra digit of precision for small differences.
234
235         * resources/sunspider-compare-results.js:
236
237 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
238
239         Reviewed by Eric.
240         
241         - work around IE layout bugs
242
243         * hosted/sunspider.css:
244
245 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
246
247         Reviewed by Mark.
248         
249         - fix compare feature
250
251         * hosted/sunspider-results.html:
252
253 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
254
255         Reviewed by Mark.
256         
257         - Change the format of the output URL to be shorter, to meet IE constraints
258
259         * hosted/sunspider-driver.html:
260         * hosted/sunspider-results.html:
261
262 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
263
264         Reviewed by Oliver.
265         
266         - Add some explanatory test and a dumb logo to SunSpider web hosted version
267
268         * hosted/sunspider-driver.html:
269         * hosted/sunspider-results.html:
270         * hosted/sunspider.css:
271         * hosted/sunspider.html:
272
273 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
274
275         Reviewed by Oliver.
276         
277         - Make the web-hosted version of SunSpider look a little bit less drab.
278
279         * hosted/sunspider-driver.html:
280         * hosted/sunspider-results.html:
281         * hosted/sunspider.css: Added.
282         * hosted/sunspider.html:
283         * resources/TEMPLATE.html:
284
285 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
286
287         Reviewed by Sam.
288         
289         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
290         
291         A sample of what new-style results would look like:
292         
293         TEST                   COMPARISON            FROM                 TO             DETAILS
294         ...
295         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
296         ...
297             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
298         
299         * resources/sunspider-compare-results.js:
300
301 2007-11-14  Eric Seidel  <eric@webkit.org>
302
303         Reviewed by Sam.
304
305         * sunspider: add --shark-cache for L2 Cache Miss profiling
306
307 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
308
309         Reviewed by Darin Adler.
310
311         http://bugs.webkit.org/show_bug.cgi?id=15925
312         SunSpider should check for Shark being installed
313
314         * sunspider: Explicitly test that shark command line tool is installed.
315
316 2007-11-04  Darin Adler  <darin@apple.com>
317
318         * hosted/sunspider.html: Tweaked formatting a bit.
319
320 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
321
322         Reviewed by Darin Adler.
323
324         - Add compare mode and a description of the benchmark to the hosted version.
325
326         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
327         * hosted/sunspider.html: Added intro text.
328         * make-hosted: Copy new file.
329         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
330         a function.
331         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
332         expects.
333         * sunspider-compare-results: Use sunspider-standalone-compare.js.
334
335 2007-11-02  Darin Adler  <darin@apple.com>
336
337         * sunspider: Changed "--base" to "--set-baseline".
338
339 2007-11-02  Darin Adler  <darin@apple.com>
340
341         Reviewed by Maciej.
342
343         * sunspider: Added a "--base" option that records the current test as a baseline.
344         Moved the code to find the newest .mshark file into a function (since I needed to use
345         it in the other script and I prefer to copy/paste a function).
346
347         * sunspider-compare-results: Added logic to compare the baseline with the last
348         run when you don't pass any parameters.
349
350 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
351
352         Reviewed by Eric.
353         
354         - Added list of ECMAScript3 features that SunSpider does not cover currently.
355
356         * UNCOVERED: Added.
357
358 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
359
360         Reviewed by Oliver.
361         
362         - Give better error messages when sunspider-compare-results fails
363
364         * sunspider-compare-results:
365
366 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
367
368         Reviewed by Darin Adler.
369         
370         - fix Date tests to work in command-line SpiderMonkey
371
372         * tests/date-format-tofte.js:
373         * tests/date-format-xparb.js:
374
375 2007-10-29  Darin Adler  <darin@apple.com>
376
377         Reviewed by Maciej.
378
379         * sunspider: Added --shark20 option, to run Shark at its highest sample
380         resolution instead of its default.
381
382 2007-10-28  Adam Roben  <aroben@apple.com>
383
384         Fix SunSpider on Windows
385
386         Reviewed by Eric.
387
388         * sunspider: Don't use colons in filenames.
389
390 2007-10-28  Darin Adler  <darin@apple.com>
391
392         Reviewed by Adam.
393
394         * resources/sunspider-analyze-results.js: Tweak the output format a little.
395         Change so that when there's only one run we don't write out confidence
396         intervals at all rather than writing "NaN" over and over again.
397
398         * sunspider: Use the time and date as a suffix on the results file, that way
399         all the results are left behind in the tmp directory. This will make it easy
400         for us to add features that compare past results in the future. Also rename
401         the .mshark files using the same scheme.
402
403         * sunspider-compare-results: Relax the parsing rules so we can parse the
404         results file exactly as it's emitted from sunspider.
405
406 2007-10-25  Adam Roben  <aroben@apple.com>
407
408         Strip carriage returns from results
409
410         Carriage returns were screwing up sunspider-compare-results on
411         Windows.
412
413         Rubberstamped by Oliver.
414
415         * sunspider:
416
417 2007-10-24  Darin Adler  <darin@apple.com>
418
419         Reviewed by Eric.
420
421         * sunspider: Open the profile in shark after completing a run.
422         * sunspider-compare-results: Tiny perl tweaks.
423
424 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
425
426         Reviewed by Darin Adler.
427
428         - Don't hardcode my path to testkjs
429         
430         * sunspider-compare-results:
431
432 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
433
434         Reviewed by Darin Adler.
435         
436         - Add a compare script to compare results, and improve formatting of standard results a bit.
437
438         * resources/sunspider-analyze-results.js:
439         * resources/sunspider-compare-results.js: Added.
440         * sunspider-compare-results: Added.
441
442 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
443
444         Reviewed by Darin Adler.
445
446         * resources/sunspider-analyze-results.js: Do the error range
447         properly, using the t-distribution instead of 1.96 (which was
448         based on the normal distribution).
449         
450         * sunspider: Print results in a way that will be friendlier to the
451         soon-to-come compare mode.
452
453 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
454
455         Reviewed by Review.
456         
457         - add more tests (probably the final set for now)
458         
459         Already balanced. Added date, regexp, control flow, and a few
460         more string and object/array access tests.
461         
462         * tests/LIST:
463         * tests/access-fannkuch.js: Added.
464         * tests/access-nbody.js: Added.
465         * tests/controlflow-recursive.js: Added.
466         * tests/date-format-tofte.js: Added.
467         * tests/date-format-xparb.js: Added.
468         * tests/regexp-dna.js: Added.
469         * tests/string-validate-input.js: Added.
470
471 2007-10-21  Eric Seidel  <eric@webkit.org>
472
473         Reviewed by mjs.
474
475         * sunspider: make --shark only sample from testkjs
476
477 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
478
479         Rubber stamped by Eric.
480         
481         - added crypto tests (already balanced)
482
483         * tests/LIST:
484         * tests/crypto-aes.js: Added.
485         * tests/crypto-md5.js: Added.
486         * tests/crypto-sha1.js: Added.
487
488 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
489
490         Reviewed by Adam.
491         
492         - rebalance new tests
493         
494         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
495         
496         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
497
498         This balances them with the existing tests.
499
500         * tests/access-binary-trees.js:
501         * tests/access-nsieve.js:
502         * tests/bitops-nsieve-bits.js:
503         * tests/math-partial-sums.js:
504         * tests/math-spectral-norm.js:
505         * tests/string-fasta.js:
506
507 2007-10-20  Eric Seidel  <eric@webkit.org>
508
509         Reviewed by darin.
510         
511         Add improved argument handling to sunspider, including
512         --runs=<number>, --tests=<pattern, --shark, and --help
513         Also re-factor code into subroutines
514
515         * sunspider: improved argument handling
516
517 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
518
519         Reviewed by Mark.
520
521         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
522         
523         * TODO:
524         * tests/LIST:
525         * tests/access-binary-trees.js: Added.
526         * tests/access-nsieve.js: Added.
527         * tests/bitops-nsieve-bits.js: Added.
528         * tests/math-partial-sums.js: Added.
529         * tests/math-spectral-norm.js: Added.
530         * tests/string-fasta.js: Added.
531
532 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
533
534         Reviewed by Darin Adler.
535
536         * TODO: Updated to note areas that are now well-covered.
537         
538         - Change the tests and the driver to avoid leaving large object
539         graphs hanging around, since that throws off the subsequent tests.
540         
541         * resources/sunspider-standalone-driver.js:
542         * tests/3d-cube.js:
543         * tests/3d-morph.js:
544         * tests/3d-raytrace.js:
545         * tests/bitops-bitwise-and.js:
546         * tests/string-base64.js:
547         * tests/string-tagcloud.js:
548
549 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
550
551         Reviewed by Nikolas.
552         
553         - Rebalance test complexity.
554         
555         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
556         
557         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
558
559         However, for some tests, IE7 was a huge outlier compared to other
560         major browsers. For those tests, I used Firefox 2.0.0.8 on the
561         same OS and hardware as a normalization basis.
562         
563         * tests/3d-morph.js:
564         * tests/3d-raytrace.js:
565         * tests/bitops-3bit-bits-in-byte.js:
566         * tests/bitops-bits-in-byte.js:
567         * tests/bitops-bitwise-and.js:
568         * tests/math-cordic.js:
569         * tests/string-base64.js:
570         * tests/string-tagcloud.js:
571         * tests/string-unpack-code.js:
572
573 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
574
575         Reviewed by Oliver.
576         
577         - Fix the browser-hosted driver to be compatible with Safari 2
578
579         * hosted/sunspider-driver.html:
580
581 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
582
583         Rubber stamped by Hyatt.
584         
585         - Add BSD license notice to cordic test, the original author said to do so.
586
587         * tests/math-cordic.js:
588
589 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
590
591         Reviewed by Eric.
592         
593         - Support Windows and Windows browsers.
594
595         * hosted/sunspider-results.html: Display results correctly in IE.
596         * sunspider: tweaks to work on Windows, and with spaces in the path.
597
598 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
599
600         Rubber stamped by Oliver.
601
602         * tests/string-tagcloud.js: Remove stray print()
603
604 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
605
606         Reviewed by Oliver.
607         
608         Add some string test cases.
609
610         * tests/string-base64.js: Added. Does base64 encoding and decoding.
611         * tests/string-tagcloud.js: Added. Parses a JSON string and
612         generates tagcloud HTML markup based on the data.        
613         * tests/string-unpack-code.js: Added. Unpack compressed versions of
614         four of the most popular JavaScript libraries.
615         * tests/LIST: Added the new tests.
616
617 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
618
619         Reviewed by Anders.
620
621         * TODO: Updated with info on how to do the stats properly.
622
623         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
624         avoid overly slowing down the benchmark.
625
626         Added some 3D-related test cases.
627         
628         * tests/3d-cube.js: Added.
629         * tests/3d-morph.js: Added.
630         * tests/3d-raytrace.js: Added.
631         * tests/LIST:
632
633 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
634
635         Reviewed by Darin Adler.
636
637         - New JavaScript benchmark
638         http://bugs.webkit.org/show_bug.cgi?id=15515
639         
640         A start on a new JavaScript benchmark, with standalone and
641         browser-hosted drivers.
642         
643         The test content is still incomplete and the drivers could use
644         some more features but there is enough here to make a good start.
645         
646         The drivers compute a 95% confidence interval on the mean for the
647         whole test, each category, and each individual test to make it
648         easier to tell whether differences are statistically
649         significant. The confidence interval can be narrowed by running
650         the test more times.
651         
652         * TODO: Added. Remaining things that need to be done.
653         * sunspider: Added. Perl script that acts as the standalone test driver.
654         * resources/sunspider-standalone-driver.js: Added. JavaScript part
655         of standalone test driver.
656         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
657         analysis code.
658         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
659         * make-hosted: Added. Script to generate browser-hosted tests.
660         * hosted/sunspider.html: Added. Start page for browser-hosted test.
661         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
662         * hosted/sunspider-results.html: Added. Results page for browser-hosted
663         tests.
664         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
665         tests.
666         * tests/LIST: Added. List of tests to use.
667         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
668         * tests/bitops-bits-in-byte.js: Added. Ditto.
669         * tests/bitops-bitwise-and.js: Added. Ditto.
670         * tests/math-cordic.js: Added. Ditto.
671