2009-04-09 Jian Li <jianli@chromium.org>
[WebKit-https.git] / SunSpider / ChangeLog
1 2009-03-04  Oliver Hunt  <oliver@apple.com>
2
3         Added a few more tests to ubench
4
5         RS=Gavin Barraclough
6
7         Adding a couple of test to cover too few and too many arguments.  This
8         is needed to prevent regressions in all call paths from my current refactoring.
9
10         * tests/LIST-UBENCH:
11         * tests/ubench/function-correct-args.js: Added.
12         (f):
13         * tests/ubench/function-excess-args.js: Added.
14         (f):
15
16 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
17
18         Fix a bug where when V8's or UBENCH's test suite list file is missing
19         the error message is suggesting that SunSpider's list file is missing.
20
21         Reviewed by Sam Weinig.
22
23         * sunspider:
24         (loadTestsList): Fix the error message to use the right list name.
25
26 2008-10-07  David Hyatt  <hyatt@apple.com>
27
28         https://bugs.webkit.org/show_bug.cgi?id=18989
29
30         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
31         we want all browsers to be doing the same thing on the test.
32
33         Reviewed by Maciej
34
35         * tests/regexp-dna.js:
36
37 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
38
39         Rubber-stamped by Maciej Stachowiak.
40
41         The bleeding edge branch of V8 now supports passing fileanmes after
42         '-f', so the SunSpider script should be changed to do this again.
43         This allows all 3 major JavaScript engines to be used with the
44         unmodified SunSpider script.
45
46         * sunspider:
47
48 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
49
50         Reviewed by Maciej Stachowiak.
51
52         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
53         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
54         correct the punctuation in its documentation.
55
56         * sunspider:
57
58 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
59
60         Reviewed by Oliver Hunt.
61
62         Bug 20616: Incorporate V8 benchmarks in testing
63         <https://bugs.webkit.org/show_bug.cgi?id=20616>
64
65         This is a first attempt at running the V8 benchmarks within SunSpider,
66         with the --v8 option. The number of iterations is fixed for each
67         benchmark, and was chosen to make each of the benchmarks run between
68         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
69
70         * sunspider:
71         * tests/LIST-V8: Added.
72         * tests/v8-crypto.js: Added.
73         * tests/v8-deltablue.js: Added.
74         * tests/v8-earley-boyer.js: Added.
75         * tests/v8-raytrace.js: Added.
76         * tests/v8-richards.js: Added.
77
78 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
79
80         Not reviewed.
81
82         Remove a local change that was accidentally committed in r36034.
83
84         * resources/sunspider-standalone-driver.js:
85
86 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
87
88         Rubber-stamped by Oliver Hunt.
89
90         Make SunSpider work with JavaScript shells that don't take filenames
91         after a '-f' argument.
92
93         * sunspider:
94
95 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
96
97         Reviewed by Timothy.
98
99         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
100
101         * sunspider:
102         * tests/LIST-SQUIRRELFISH: Removed.
103
104 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
105
106         Rubber-stamped by Mark Rowe.
107
108         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
109
110         * sunspider:
111
112 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
113
114         Rubber stamped by Oliver.
115         
116         - add newly working tests to squirrelfish list
117
118         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
119
120 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
121
122         Rubber stamped by Oliver.
123         
124         - move ubench tests properly.
125
126         * tests/ubench: Added.
127         * tests/ubench/function-closure.js: Added.
128         * tests/ubench/function-empty.js: Added.
129         * tests/ubench/function-missing-args.js: Added.
130         * tests/ubench/function-sum.js: Added.
131         * tests/ubench/loop-empty-resolve.js: Added.
132         * tests/ubench/loop-empty.js: Added.
133         * tests/ubench/loop-sum.js: Added.
134
135 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
136
137         Rubber stamped by Oliver.
138
139         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
140
141 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
142
143         Reviewed by Oliver.
144
145         - make --squirrelfish mode use the real harness, but a reduced set of tests
146
147         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
148
149         * sunspider:
150         * tests/LIST-SQUIRRELFISH:
151         * tests/LIST-UBENCH: Added.
152         * tests/squirrelfish: Moved to tests/ubench
153
154 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
155
156         Reviewed by Maciej Stachowiak.
157         
158         Removed an unused test.
159
160         * tests/squirrelfish/loop-resolve.js: Removed.
161
162 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
163
164         Reviewed by Oliver.
165         
166         Add newly runnable tests to --squirrelfish mode.
167
168         * tests/LIST-SQUIRRELFISH:
169
170 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
171
172         Reviewed by Maciej Stachowiak.
173         
174         A little more love for --squirrelfish mode:
175         
176         Fixed a misplaced sort. Results properly sort now.
177         
178         Added a test list just for squirrelfish, and updated it to include
179         all currently passing SunSpider tests.
180         
181         Fixed the pruning regexp to match 3d-morph.js.
182
183         * sunspider:
184
185 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
186
187         Reviewed by Oliver Hunt.
188         
189         --squirrelfish mode: pared down tests for squirrelfish to chew on.
190
191         * sunspider:
192         * tests/LIST:
193         * tests/squirrelfish: Added.
194         * tests/squirrelfish/function-closure.js: Added.
195         * tests/squirrelfish/function-empty.js: Added.
196         * tests/squirrelfish/function-missing-args.js: Added.
197         * tests/squirrelfish/function-sum.js: Added.
198         * tests/squirrelfish/loop-empty-resolve.js: Added.
199         * tests/squirrelfish/loop-empty.js: Added.
200         * tests/squirrelfish/loop-resolve.js: Added.
201         * tests/squirrelfish/loop-sum.js: Added.
202
203 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
204
205         Reviewed by Geoff.
206
207         * sunspider: Pause update daemon on Darwin to make results reliable
208         again (workaround for <rdar://problem/5811127>).
209
210 2008-02-07  Eric Seidel  <eric@webkit.org>
211
212         Reviewed by Sam.
213
214         * sunspider: add an error message when ./tests/LIST is not found
215
216 2007-12-29  Alp Toker  <alp@atoker.com>
217
218         Fix typo.
219
220         * hosted/sunspider.html:
221
222 2007-12-29  Alp Toker  <alp@atoker.com>
223
224         Reviewed by Maciej.
225
226         Provide the current URL in the SunSpider results page. Useful for
227         copying benchmark results out of browsers without a location bar or
228         where the location bar can't handle long URLs.
229
230         * hosted/sunspider-results.html:
231
232 2007-12-20  Eric Seidel  <eric@webkit.org>
233
234         Reviewed by mjs.
235
236         * sunspider: increase default number of runs for more accurate numbers
237
238 2007-12-18  Mark Rowe  <mrowe@apple.com>
239
240         Fix typo.
241
242         * hosted/sunspider.html:
243
244 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
245
246         Reviewed by Eric.
247         
248         - give an extra digit of precision for small differences.
249
250         * resources/sunspider-compare-results.js:
251
252 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
253
254         Reviewed by Eric.
255         
256         - work around IE layout bugs
257
258         * hosted/sunspider.css:
259
260 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
261
262         Reviewed by Mark.
263         
264         - fix compare feature
265
266         * hosted/sunspider-results.html:
267
268 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
269
270         Reviewed by Mark.
271         
272         - Change the format of the output URL to be shorter, to meet IE constraints
273
274         * hosted/sunspider-driver.html:
275         * hosted/sunspider-results.html:
276
277 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
278
279         Reviewed by Oliver.
280         
281         - Add some explanatory test and a dumb logo to SunSpider web hosted version
282
283         * hosted/sunspider-driver.html:
284         * hosted/sunspider-results.html:
285         * hosted/sunspider.css:
286         * hosted/sunspider.html:
287
288 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
289
290         Reviewed by Oliver.
291         
292         - Make the web-hosted version of SunSpider look a little bit less drab.
293
294         * hosted/sunspider-driver.html:
295         * hosted/sunspider-results.html:
296         * hosted/sunspider.css: Added.
297         * hosted/sunspider.html:
298         * resources/TEMPLATE.html:
299
300 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
301
302         Reviewed by Sam.
303         
304         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
305         
306         A sample of what new-style results would look like:
307         
308         TEST                   COMPARISON            FROM                 TO             DETAILS
309         ...
310         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
311         ...
312             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
313         
314         * resources/sunspider-compare-results.js:
315
316 2007-11-14  Eric Seidel  <eric@webkit.org>
317
318         Reviewed by Sam.
319
320         * sunspider: add --shark-cache for L2 Cache Miss profiling
321
322 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
323
324         Reviewed by Darin Adler.
325
326         http://bugs.webkit.org/show_bug.cgi?id=15925
327         SunSpider should check for Shark being installed
328
329         * sunspider: Explicitly test that shark command line tool is installed.
330
331 2007-11-04  Darin Adler  <darin@apple.com>
332
333         * hosted/sunspider.html: Tweaked formatting a bit.
334
335 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
336
337         Reviewed by Darin Adler.
338
339         - Add compare mode and a description of the benchmark to the hosted version.
340
341         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
342         * hosted/sunspider.html: Added intro text.
343         * make-hosted: Copy new file.
344         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
345         a function.
346         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
347         expects.
348         * sunspider-compare-results: Use sunspider-standalone-compare.js.
349
350 2007-11-02  Darin Adler  <darin@apple.com>
351
352         * sunspider: Changed "--base" to "--set-baseline".
353
354 2007-11-02  Darin Adler  <darin@apple.com>
355
356         Reviewed by Maciej.
357
358         * sunspider: Added a "--base" option that records the current test as a baseline.
359         Moved the code to find the newest .mshark file into a function (since I needed to use
360         it in the other script and I prefer to copy/paste a function).
361
362         * sunspider-compare-results: Added logic to compare the baseline with the last
363         run when you don't pass any parameters.
364
365 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
366
367         Reviewed by Eric.
368         
369         - Added list of ECMAScript3 features that SunSpider does not cover currently.
370
371         * UNCOVERED: Added.
372
373 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
374
375         Reviewed by Oliver.
376         
377         - Give better error messages when sunspider-compare-results fails
378
379         * sunspider-compare-results:
380
381 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
382
383         Reviewed by Darin Adler.
384         
385         - fix Date tests to work in command-line SpiderMonkey
386
387         * tests/date-format-tofte.js:
388         * tests/date-format-xparb.js:
389
390 2007-10-29  Darin Adler  <darin@apple.com>
391
392         Reviewed by Maciej.
393
394         * sunspider: Added --shark20 option, to run Shark at its highest sample
395         resolution instead of its default.
396
397 2007-10-28  Adam Roben  <aroben@apple.com>
398
399         Fix SunSpider on Windows
400
401         Reviewed by Eric.
402
403         * sunspider: Don't use colons in filenames.
404
405 2007-10-28  Darin Adler  <darin@apple.com>
406
407         Reviewed by Adam.
408
409         * resources/sunspider-analyze-results.js: Tweak the output format a little.
410         Change so that when there's only one run we don't write out confidence
411         intervals at all rather than writing "NaN" over and over again.
412
413         * sunspider: Use the time and date as a suffix on the results file, that way
414         all the results are left behind in the tmp directory. This will make it easy
415         for us to add features that compare past results in the future. Also rename
416         the .mshark files using the same scheme.
417
418         * sunspider-compare-results: Relax the parsing rules so we can parse the
419         results file exactly as it's emitted from sunspider.
420
421 2007-10-25  Adam Roben  <aroben@apple.com>
422
423         Strip carriage returns from results
424
425         Carriage returns were screwing up sunspider-compare-results on
426         Windows.
427
428         Rubberstamped by Oliver.
429
430         * sunspider:
431
432 2007-10-24  Darin Adler  <darin@apple.com>
433
434         Reviewed by Eric.
435
436         * sunspider: Open the profile in shark after completing a run.
437         * sunspider-compare-results: Tiny perl tweaks.
438
439 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
440
441         Reviewed by Darin Adler.
442
443         - Don't hardcode my path to testkjs
444         
445         * sunspider-compare-results:
446
447 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
448
449         Reviewed by Darin Adler.
450         
451         - Add a compare script to compare results, and improve formatting of standard results a bit.
452
453         * resources/sunspider-analyze-results.js:
454         * resources/sunspider-compare-results.js: Added.
455         * sunspider-compare-results: Added.
456
457 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
458
459         Reviewed by Darin Adler.
460
461         * resources/sunspider-analyze-results.js: Do the error range
462         properly, using the t-distribution instead of 1.96 (which was
463         based on the normal distribution).
464         
465         * sunspider: Print results in a way that will be friendlier to the
466         soon-to-come compare mode.
467
468 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
469
470         Reviewed by Review.
471         
472         - add more tests (probably the final set for now)
473         
474         Already balanced. Added date, regexp, control flow, and a few
475         more string and object/array access tests.
476         
477         * tests/LIST:
478         * tests/access-fannkuch.js: Added.
479         * tests/access-nbody.js: Added.
480         * tests/controlflow-recursive.js: Added.
481         * tests/date-format-tofte.js: Added.
482         * tests/date-format-xparb.js: Added.
483         * tests/regexp-dna.js: Added.
484         * tests/string-validate-input.js: Added.
485
486 2007-10-21  Eric Seidel  <eric@webkit.org>
487
488         Reviewed by mjs.
489
490         * sunspider: make --shark only sample from testkjs
491
492 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
493
494         Rubber stamped by Eric.
495         
496         - added crypto tests (already balanced)
497
498         * tests/LIST:
499         * tests/crypto-aes.js: Added.
500         * tests/crypto-md5.js: Added.
501         * tests/crypto-sha1.js: Added.
502
503 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
504
505         Reviewed by Adam.
506         
507         - rebalance new tests
508         
509         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
510         
511         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
512
513         This balances them with the existing tests.
514
515         * tests/access-binary-trees.js:
516         * tests/access-nsieve.js:
517         * tests/bitops-nsieve-bits.js:
518         * tests/math-partial-sums.js:
519         * tests/math-spectral-norm.js:
520         * tests/string-fasta.js:
521
522 2007-10-20  Eric Seidel  <eric@webkit.org>
523
524         Reviewed by darin.
525         
526         Add improved argument handling to sunspider, including
527         --runs=<number>, --tests=<pattern, --shark, and --help
528         Also re-factor code into subroutines
529
530         * sunspider: improved argument handling
531
532 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
533
534         Reviewed by Mark.
535
536         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
537         
538         * TODO:
539         * tests/LIST:
540         * tests/access-binary-trees.js: Added.
541         * tests/access-nsieve.js: Added.
542         * tests/bitops-nsieve-bits.js: Added.
543         * tests/math-partial-sums.js: Added.
544         * tests/math-spectral-norm.js: Added.
545         * tests/string-fasta.js: Added.
546
547 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
548
549         Reviewed by Darin Adler.
550
551         * TODO: Updated to note areas that are now well-covered.
552         
553         - Change the tests and the driver to avoid leaving large object
554         graphs hanging around, since that throws off the subsequent tests.
555         
556         * resources/sunspider-standalone-driver.js:
557         * tests/3d-cube.js:
558         * tests/3d-morph.js:
559         * tests/3d-raytrace.js:
560         * tests/bitops-bitwise-and.js:
561         * tests/string-base64.js:
562         * tests/string-tagcloud.js:
563
564 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
565
566         Reviewed by Nikolas.
567         
568         - Rebalance test complexity.
569         
570         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
571         
572         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
573
574         However, for some tests, IE7 was a huge outlier compared to other
575         major browsers. For those tests, I used Firefox 2.0.0.8 on the
576         same OS and hardware as a normalization basis.
577         
578         * tests/3d-morph.js:
579         * tests/3d-raytrace.js:
580         * tests/bitops-3bit-bits-in-byte.js:
581         * tests/bitops-bits-in-byte.js:
582         * tests/bitops-bitwise-and.js:
583         * tests/math-cordic.js:
584         * tests/string-base64.js:
585         * tests/string-tagcloud.js:
586         * tests/string-unpack-code.js:
587
588 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
589
590         Reviewed by Oliver.
591         
592         - Fix the browser-hosted driver to be compatible with Safari 2
593
594         * hosted/sunspider-driver.html:
595
596 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
597
598         Rubber stamped by Hyatt.
599         
600         - Add BSD license notice to cordic test, the original author said to do so.
601
602         * tests/math-cordic.js:
603
604 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
605
606         Reviewed by Eric.
607         
608         - Support Windows and Windows browsers.
609
610         * hosted/sunspider-results.html: Display results correctly in IE.
611         * sunspider: tweaks to work on Windows, and with spaces in the path.
612
613 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
614
615         Rubber stamped by Oliver.
616
617         * tests/string-tagcloud.js: Remove stray print()
618
619 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
620
621         Reviewed by Oliver.
622         
623         Add some string test cases.
624
625         * tests/string-base64.js: Added. Does base64 encoding and decoding.
626         * tests/string-tagcloud.js: Added. Parses a JSON string and
627         generates tagcloud HTML markup based on the data.        
628         * tests/string-unpack-code.js: Added. Unpack compressed versions of
629         four of the most popular JavaScript libraries.
630         * tests/LIST: Added the new tests.
631
632 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
633
634         Reviewed by Anders.
635
636         * TODO: Updated with info on how to do the stats properly.
637
638         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
639         avoid overly slowing down the benchmark.
640
641         Added some 3D-related test cases.
642         
643         * tests/3d-cube.js: Added.
644         * tests/3d-morph.js: Added.
645         * tests/3d-raytrace.js: Added.
646         * tests/LIST:
647
648 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
649
650         Reviewed by Darin Adler.
651
652         - New JavaScript benchmark
653         http://bugs.webkit.org/show_bug.cgi?id=15515
654         
655         A start on a new JavaScript benchmark, with standalone and
656         browser-hosted drivers.
657         
658         The test content is still incomplete and the drivers could use
659         some more features but there is enough here to make a good start.
660         
661         The drivers compute a 95% confidence interval on the mean for the
662         whole test, each category, and each individual test to make it
663         easier to tell whether differences are statistically
664         significant. The confidence interval can be narrowed by running
665         the test more times.
666         
667         * TODO: Added. Remaining things that need to be done.
668         * sunspider: Added. Perl script that acts as the standalone test driver.
669         * resources/sunspider-standalone-driver.js: Added. JavaScript part
670         of standalone test driver.
671         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
672         analysis code.
673         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
674         * make-hosted: Added. Script to generate browser-hosted tests.
675         * hosted/sunspider.html: Added. Start page for browser-hosted test.
676         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
677         * hosted/sunspider-results.html: Added. Results page for browser-hosted
678         tests.
679         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
680         tests.
681         * tests/LIST: Added. List of tests to use.
682         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
683         * tests/bitops-bits-in-byte.js: Added. Ditto.
684         * tests/bitops-bitwise-and.js: Added. Ditto.
685         * tests/math-cordic.js: Added. Ditto.
686