Browser-hosted version of SunSpider runs tests twice as many times as advertised
[WebKit.git] / SunSpider / ChangeLog
1 2010-05-17  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Mark Rowe.
4
5         Browser-hosted version of SunSpider runs tests twice as many times as advertised
6         https://bugs.webkit.org/show_bug.cgi?id=39201
7
8         * make-hosted: Make sure to reset test list after every test suite.
9
10 2010-05-07  Adam Roben  <aroben@apple.com>
11
12         Fix typo in sunspider-compare-results.js that caused incorrect
13         two-sample t statistics to be computed
14
15         Fixes <http://webkit.org/b/38768> SunSpider computes incorrect
16         two-sample t statistics
17
18         Reviewed by Darin Adler.
19
20         * resources/sunspider-compare-results.js:
21         (sunspiderCompareResults.resultLine): Fixed the calculation of the t
22         variable to match the definition of a two-sample t statistic. Looks
23         like this was just a simple typo.
24
25 2010-04-05  Darin Adler  <darin@apple.com>
26
27         Updated the ignore property for changes to the contents here.
28
29         * .: Modified property svn:ignore.
30
31 2010-02-20  Maciej Stachowiak  <mjs@apple.com>
32
33         Reviewed by Adam Barth.
34
35         Add sunspider-0.9.1 to the site in preparation for announcing it.
36         https://bugs.webkit.org/show_bug.cgi?id=35206
37
38         * hosted/sunspider.css: Remove tabs, which I noticed where present while
39         copying to the WebKitSite directory.
40
41 2010-01-11  Oliver Hunt  <oliver@apple.com>
42
43         Reviewed by Darin Adler.
44
45         Sunspider parse-only tests don't run
46         https://bugs.webkit.org/show_bug.cgi?id=33489
47
48         Fix parse-only test list
49
50         * tests/parse-only/LIST:
51
52 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
53
54         Reviewed by Adele Peterson.
55
56         Make SunSpider version more prominent in the title
57         https://bugs.webkit.org/show_bug.cgi?id=32574
58
59         * hosted/sunspider.html:
60         * hosted/versions.html:
61         * resources/driver-TEMPLATE.html:
62         * resources/results-TEMPLATE.html:
63
64 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
65
66         Reviewed by Darin Adler.
67
68         Make SunSpider refuse to compare results across content versions
69         https://bugs.webkit.org/show_bug.cgi?id=32573
70         
71         The results URL now includes a version indicator, and compare mode
72         will refuse to compare between versions.
73
74         * resources/driver-TEMPLATE.html:
75         * resources/results-TEMPLATE.html:
76
77 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
78
79         Reviewed by Darin Adler.
80
81         Some Browser-hosted SunSpider files are not valid HTML5
82         https://bugs.webkit.org/show_bug.cgi?id=32536
83         
84         Made various fixes to validate as HTML5 with no warnings (mainly removing bogus
85         close tags and adding a meta charset declaration).
86
87         * hosted/sunspider.html:
88         * hosted/versions.html:
89         * resources/TEMPLATE.html:
90         * resources/driver-TEMPLATE.html:
91         * resources/results-TEMPLATE.html:
92
93 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
94
95         Reviewed by Eric Seidel.
96
97         Make sunspider-0.9.1 the default content set (both command-line and hosted)
98         https://bugs.webkit.org/show_bug.cgi?id=32537
99
100         * hosted/sunspider.html: Change default suite link.
101         * sunspider: Change default suite.
102         * sunspider-compare-results: Likewise.
103
104 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
105
106         Reviewed by Sam Weinig.
107
108         in-browser SunSpider suffers excessive penalty under power management
109         https://bugs.webkit.org/show_bug.cgi?id=32505
110         
111         I have made a few changes to address this:
112         
113         1) Change how browser-hosted SunSpider loads tests - preload the content and write it in with
114         document.write to reduce triggering of progress UI.
115         2) Reduce time between tests to 10ms from 500ms, so that power management doesn't keep the CPU
116         stepped all the way down the whole time.
117         3) Run the test cycle an extra time for warmup before the runs that count.
118
119         * hosted/sunspider-record-result.js: Removed.
120         * make-hosted:
121         * resources/TEMPLATE.html:
122         * resources/driver-TEMPLATE.html:
123
124 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
125
126         Reviewed by Gavin Barraclough.
127
128         SunSpider/tests/string-base64.js does not compute a valid base64 encoded string
129         https://bugs.webkit.org/show_bug.cgi?id=16806
130
131         Based on a patch by Eric Seidel.
132         
133         Fix the base64 computation to actually compute correct results. The impact on runtime of
134         the test is pretty small, but noticeable for some browsers. But at least it's not
135         doing a wrong and meaningless computation any more.
136         
137         * tests/sunspider-0.9.1/string-base64.js:
138         ():
139         (base64ToString):
140
141 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
142
143         Fixing commit error...
144         
145         I accidentally committed my last patch in a form that broke Web-hosted SunSpider. Fixing.
146
147         * resources/driver-TEMPLATE.html:
148
149 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
150
151         Reviewed by Sam Weinig.
152
153         Use JSON.parse instead of eval for Web-hosted SunSpider results processing
154         https://bugs.webkit.org/show_bug.cgi?id=32490
155
156         * hosted/json2.js: Added. Incorporated from json.org
157         * resources/driver-TEMPLATE.html: Fix an HTML validation bug I noticed.
158         * resources/results-TEMPLATE.html: Use JSON.parse instead of eval to process
159         results.
160
161 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
162
163         Reviewed by Oliver Hunt.
164
165         Enable Web-hosted version of SunSpider to handle multiple versions
166         https://bugs.webkit.org/show_bug.cgi?id=32478
167
168         * make-hosted: Now generate the hosted version to be able to run both the
169         0.9 and the 0.9.1 test suites.
170         * hosted: Modified property svn:ignore.
171         * hosted/sunspider-driver.html: Removed.
172         * hosted/sunspider-results.html: Removed.
173         * hosted/sunspider.html:
174         * hosted/versions.html: Added.
175         * resources/TEMPLATE.html:
176         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-driver.html.
177         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-results.html.
178
179 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
180
181         Reviewed by Oliver Hunt.
182
183         Give command-line sunspider the ability to handle multiple suites and versions
184         https://bugs.webkit.org/show_bug.cgi?id=32477
185         
186         Each suite/version's set of tests are now in a separate subdirectory of tests/, with a file named LIST
187         enumerating the tests. Current suites are:
188         
189         sunspider-0.9 - riginal SunSpider content 
190         sunspider-0.9.1 - SunSpider suite with a few tests fixed (and a few more to come)
191         ubench - "ubench" microbenchmark suite
192         v8-v4 - version 4 of the v8 benchmark
193         parse-only - some parser tests
194         
195         Sunspider can now be run with the --suite= parameter to select a
196         suite. The default is --suite=sunspider-0.9.
197
198         * .: Modified property svn:ignore to ignore new results directories.
199         * make-hosted: Generate based on sunspider-0.9 suite.
200         * resources/TEMPLATE.html: Fix an HTML compliance problem.
201         * resources/sunspider-standalone-driver.js: Become aware of suite names.
202         * sunspider: Modified to handle multiple suites as described above.
203         * sunspider-compare-results: ditto
204         * tests/LIST: Moved to tests/sunspider-0.9/ and tests/sunspider-0.9.1/
205         * tests/LIST-PARSE-ONLY: Moved to parse-only/LIST
206         * tests/LIST-UBENCH: Removed.
207         * tests/LIST-V8: Removed.
208         * tests/3d-cube.js: Moved to tests/sunspider-0.9 and tests/sunspider-0.9.1
209         * tests/3d-morph.js: ditto
210         * tests/3d-raytrace.js: ditto
211         * tests/access-binary-trees.js: ditto
212         * tests/access-fannkuch.js: ditto
213         * tests/access-nbody.js: ditto
214         * tests/access-nsieve.js: ditto
215         * tests/bitops-3bit-bits-in-byte.js: ditto
216         * tests/bitops-bits-in-byte.js: ditto
217         * tests/bitops-bitwise-and.js: ditto
218         * tests/bitops-nsieve-bits.js: ditto
219         * tests/controlflow-recursive.js: ditto
220         * tests/crypto-aes.js: ditto
221         * tests/crypto-md5.js: ditto
222         * tests/crypto-sha1.js: ditto
223         * tests/date-format-tofte.js: ditto
224         * tests/date-format-xparb.js: ditto
225         * tests/math-cordic.js: ditto
226         * tests/math-partial-sums.js: ditto
227         * tests/math-spectral-norm.js: ditto
228         * tests/regexp-dna.js: ditto
229         * tests/string-base64.js: ditto
230         * tests/string-fasta.js: ditto
231         * tests/string-tagcloud.js: ditto
232         * tests/string-unpack-code.js: ditto
233         * tests/string-validate-input.js: ditto
234         * tests/sunspider-0.9/: Added
235         * tests/sunspider-0.9/*.js: Moved from one level up
236         * tests/sunspider-0.9.1: Added.
237         * tests/sunspider-0.9.1/*.js: Moved from one level up.
238         * tests/v8-crypto.js: Moved to v8-v4/ directory.
239         * tests/v8-deltablue.js: ditto
240         * tests/v8-earley-boyer.js: ditto
241         * tests/v8-raytrace.js: ditto
242         * tests/v8-regexp.js: ditto
243         * tests/v8-richards.js: ditto
244         * tests/v8-splay.js: ditto
245         * tests/v8-v4: Added.
246
247 2009-09-30  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
248
249         Reviewed by Darin Adler.
250
251         Platform specific null device replaced with a platform independent.
252         https://bugs.webkit.org/show_bug.cgi?id=29544
253
254         * sunspider: Using File::Spec->devnull() instead of /dev/null and NUL.
255
256 2009-09-21  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
257
258         Reviewed by Maciej Stachowiak.
259
260         [Qt] Make sunspider script work on Windows platform.
261         https://bugs.webkit.org/show_bug.cgi?id=29544
262
263         * sunspider:
264         Using platform specific null device instead of /dev/null.
265
266 2009-06-19  Adam Treat  <adam.treat@torchmobile.com>
267
268         Reviewed by Oliver Hunt.
269
270         https://bugs.webkit.org/show_bug.cgi?id=26540
271         Currently the SunSpider test driver lacks an option to run a test suite that
272         will test JavaScriptCore parsing performance only.  This patch adds just such
273         a test suite and option to SunSpider as well as the jsc test shell.  I've included
274         three large javascript source files found in the wild: jquery, mootools and prototype.
275         Combined with the concatenation of all three, these form a new testsuite to measure
276         and test pure JavaScriptCore parsing performance.
277
278         * resources/sunspider-standalone-driver.js:
279           Load the js with parse-only flag if it is part of the parse-only suite.
280         * sunspider:
281           Add the parse-only suite to the list of options.
282         * sunspider-compare-results:
283           Add the parse-only suite to the list of options.
284         * tests/LIST-PARSE-ONLY: Added.
285         * tests/parse-only/concat-jquery-mootools-prototype.js: Added.
286           A very large concatenation of the scripts found below.  This js takes ~30ms to parse on my machine.
287         * tests/parse-only/jquery-1.3.2.js: Added.
288           Latest version from jquery.com.
289         * tests/parse-only/mootools-1.2.2-core-nc.js: Added.
290           Latest version from mootools.net.
291         * tests/parse-only/prototype-1.6.0.3.js: Added.
292           Latest version from prototypejs.org.
293
294 2009-06-13  Steve Falkenburg  <sfalken@apple.com>
295
296         Update V8 benchmark to version 4.
297         
298         Reviewed by Cameron Zwarich.
299
300         * tests/LIST-V8:
301         * tests/v8-crypto.js:
302         * tests/v8-deltablue.js:
303         * tests/v8-earley-boyer.js:
304         * tests/v8-raytrace.js:
305         * tests/v8-regexp.js: Added.
306         * tests/v8-richards.js:
307         * tests/v8-splay.js: Added.
308
309 2009-05-26  Olivier DOLE  <odole@pleyo.com>
310
311         Reviewed by Maciej.
312
313         Store sunspider test suite results in a specific directory for each
314         different type of benchmark (sunspider, V8 or ubench).
315
316         * sunspider:
317         * sunspider-compare-results:
318
319 2009-05-12  Steve Falkenburg  <sfalken@apple.com>
320
321         Build fix.
322
323         * SunSpider.make:
324
325 2009-05-11  Steve Falkenburg  <sfalken@apple.com>
326
327         Include SunSpider tests in production builds.
328         
329         Reviewed by Adam Roben.
330
331         * SunSpider.make: Added.
332         * xcopy.excludes: Added.
333
334 2009-03-04  Oliver Hunt  <oliver@apple.com>
335
336         Added a few more tests to ubench
337
338         RS=Gavin Barraclough
339
340         Adding a couple of test to cover too few and too many arguments.  This
341         is needed to prevent regressions in all call paths from my current refactoring.
342
343         * tests/LIST-UBENCH:
344         * tests/ubench/function-correct-args.js: Added.
345         (f):
346         * tests/ubench/function-excess-args.js: Added.
347         (f):
348
349 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
350
351         Fix a bug where when V8's or UBENCH's test suite list file is missing
352         the error message is suggesting that SunSpider's list file is missing.
353
354         Reviewed by Sam Weinig.
355
356         * sunspider:
357         (loadTestsList): Fix the error message to use the right list name.
358
359 2008-10-07  David Hyatt  <hyatt@apple.com>
360
361         https://bugs.webkit.org/show_bug.cgi?id=18989
362
363         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
364         we want all browsers to be doing the same thing on the test.
365
366         Reviewed by Maciej
367
368         * tests/regexp-dna.js:
369
370 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
371
372         Rubber-stamped by Maciej Stachowiak.
373
374         The bleeding edge branch of V8 now supports passing fileanmes after
375         '-f', so the SunSpider script should be changed to do this again.
376         This allows all 3 major JavaScript engines to be used with the
377         unmodified SunSpider script.
378
379         * sunspider:
380
381 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
382
383         Reviewed by Maciej Stachowiak.
384
385         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
386         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
387         correct the punctuation in its documentation.
388
389         * sunspider:
390
391 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
392
393         Reviewed by Oliver Hunt.
394
395         Bug 20616: Incorporate V8 benchmarks in testing
396         <https://bugs.webkit.org/show_bug.cgi?id=20616>
397
398         This is a first attempt at running the V8 benchmarks within SunSpider,
399         with the --v8 option. The number of iterations is fixed for each
400         benchmark, and was chosen to make each of the benchmarks run between
401         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
402
403         * sunspider:
404         * tests/LIST-V8: Added.
405         * tests/v8-crypto.js: Added.
406         * tests/v8-deltablue.js: Added.
407         * tests/v8-earley-boyer.js: Added.
408         * tests/v8-raytrace.js: Added.
409         * tests/v8-richards.js: Added.
410
411 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
412
413         Not reviewed.
414
415         Remove a local change that was accidentally committed in r36034.
416
417         * resources/sunspider-standalone-driver.js:
418
419 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
420
421         Rubber-stamped by Oliver Hunt.
422
423         Make SunSpider work with JavaScript shells that don't take filenames
424         after a '-f' argument.
425
426         * sunspider:
427
428 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
429
430         Reviewed by Timothy.
431
432         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
433
434         * sunspider:
435         * tests/LIST-SQUIRRELFISH: Removed.
436
437 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
438
439         Rubber-stamped by Mark Rowe.
440
441         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
442
443         * sunspider:
444
445 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
446
447         Rubber stamped by Oliver.
448         
449         - add newly working tests to squirrelfish list
450
451         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
452
453 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
454
455         Rubber stamped by Oliver.
456         
457         - move ubench tests properly.
458
459         * tests/ubench: Added.
460         * tests/ubench/function-closure.js: Added.
461         * tests/ubench/function-empty.js: Added.
462         * tests/ubench/function-missing-args.js: Added.
463         * tests/ubench/function-sum.js: Added.
464         * tests/ubench/loop-empty-resolve.js: Added.
465         * tests/ubench/loop-empty.js: Added.
466         * tests/ubench/loop-sum.js: Added.
467
468 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
469
470         Rubber stamped by Oliver.
471
472         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
473
474 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
475
476         Reviewed by Oliver.
477
478         - make --squirrelfish mode use the real harness, but a reduced set of tests
479
480         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
481
482         * sunspider:
483         * tests/LIST-SQUIRRELFISH:
484         * tests/LIST-UBENCH: Added.
485         * tests/squirrelfish: Moved to tests/ubench
486
487 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
488
489         Reviewed by Maciej Stachowiak.
490         
491         Removed an unused test.
492
493         * tests/squirrelfish/loop-resolve.js: Removed.
494
495 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
496
497         Reviewed by Oliver.
498         
499         Add newly runnable tests to --squirrelfish mode.
500
501         * tests/LIST-SQUIRRELFISH:
502
503 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
504
505         Reviewed by Maciej Stachowiak.
506         
507         A little more love for --squirrelfish mode:
508         
509         Fixed a misplaced sort. Results properly sort now.
510         
511         Added a test list just for squirrelfish, and updated it to include
512         all currently passing SunSpider tests.
513         
514         Fixed the pruning regexp to match 3d-morph.js.
515
516         * sunspider:
517
518 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
519
520         Reviewed by Oliver Hunt.
521         
522         --squirrelfish mode: pared down tests for squirrelfish to chew on.
523
524         * sunspider:
525         * tests/LIST:
526         * tests/squirrelfish: Added.
527         * tests/squirrelfish/function-closure.js: Added.
528         * tests/squirrelfish/function-empty.js: Added.
529         * tests/squirrelfish/function-missing-args.js: Added.
530         * tests/squirrelfish/function-sum.js: Added.
531         * tests/squirrelfish/loop-empty-resolve.js: Added.
532         * tests/squirrelfish/loop-empty.js: Added.
533         * tests/squirrelfish/loop-resolve.js: Added.
534         * tests/squirrelfish/loop-sum.js: Added.
535
536 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
537
538         Reviewed by Geoff.
539
540         * sunspider: Pause update daemon on Darwin to make results reliable
541         again (workaround for <rdar://problem/5811127>).
542
543 2008-02-07  Eric Seidel  <eric@webkit.org>
544
545         Reviewed by Sam.
546
547         * sunspider: add an error message when ./tests/LIST is not found
548
549 2007-12-29  Alp Toker  <alp@atoker.com>
550
551         Fix typo.
552
553         * hosted/sunspider.html:
554
555 2007-12-29  Alp Toker  <alp@atoker.com>
556
557         Reviewed by Maciej.
558
559         Provide the current URL in the SunSpider results page. Useful for
560         copying benchmark results out of browsers without a location bar or
561         where the location bar can't handle long URLs.
562
563         * hosted/sunspider-results.html:
564
565 2007-12-20  Eric Seidel  <eric@webkit.org>
566
567         Reviewed by mjs.
568
569         * sunspider: increase default number of runs for more accurate numbers
570
571 2007-12-18  Mark Rowe  <mrowe@apple.com>
572
573         Fix typo.
574
575         * hosted/sunspider.html:
576
577 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
578
579         Reviewed by Eric.
580         
581         - give an extra digit of precision for small differences.
582
583         * resources/sunspider-compare-results.js:
584
585 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
586
587         Reviewed by Eric.
588         
589         - work around IE layout bugs
590
591         * hosted/sunspider.css:
592
593 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
594
595         Reviewed by Mark.
596         
597         - fix compare feature
598
599         * hosted/sunspider-results.html:
600
601 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
602
603         Reviewed by Mark.
604         
605         - Change the format of the output URL to be shorter, to meet IE constraints
606
607         * hosted/sunspider-driver.html:
608         * hosted/sunspider-results.html:
609
610 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
611
612         Reviewed by Oliver.
613         
614         - Add some explanatory test and a dumb logo to SunSpider web hosted version
615
616         * hosted/sunspider-driver.html:
617         * hosted/sunspider-results.html:
618         * hosted/sunspider.css:
619         * hosted/sunspider.html:
620
621 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
622
623         Reviewed by Oliver.
624         
625         - Make the web-hosted version of SunSpider look a little bit less drab.
626
627         * hosted/sunspider-driver.html:
628         * hosted/sunspider-results.html:
629         * hosted/sunspider.css: Added.
630         * hosted/sunspider.html:
631         * resources/TEMPLATE.html:
632
633 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
634
635         Reviewed by Sam.
636         
637         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
638         
639         A sample of what new-style results would look like:
640         
641         TEST                   COMPARISON            FROM                 TO             DETAILS
642         ...
643         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
644         ...
645             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
646         
647         * resources/sunspider-compare-results.js:
648
649 2007-11-14  Eric Seidel  <eric@webkit.org>
650
651         Reviewed by Sam.
652
653         * sunspider: add --shark-cache for L2 Cache Miss profiling
654
655 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
656
657         Reviewed by Darin Adler.
658
659         http://bugs.webkit.org/show_bug.cgi?id=15925
660         SunSpider should check for Shark being installed
661
662         * sunspider: Explicitly test that shark command line tool is installed.
663
664 2007-11-04  Darin Adler  <darin@apple.com>
665
666         * hosted/sunspider.html: Tweaked formatting a bit.
667
668 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
669
670         Reviewed by Darin Adler.
671
672         - Add compare mode and a description of the benchmark to the hosted version.
673
674         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
675         * hosted/sunspider.html: Added intro text.
676         * make-hosted: Copy new file.
677         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
678         a function.
679         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
680         expects.
681         * sunspider-compare-results: Use sunspider-standalone-compare.js.
682
683 2007-11-02  Darin Adler  <darin@apple.com>
684
685         * sunspider: Changed "--base" to "--set-baseline".
686
687 2007-11-02  Darin Adler  <darin@apple.com>
688
689         Reviewed by Maciej.
690
691         * sunspider: Added a "--base" option that records the current test as a baseline.
692         Moved the code to find the newest .mshark file into a function (since I needed to use
693         it in the other script and I prefer to copy/paste a function).
694
695         * sunspider-compare-results: Added logic to compare the baseline with the last
696         run when you don't pass any parameters.
697
698 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
699
700         Reviewed by Eric.
701         
702         - Added list of ECMAScript3 features that SunSpider does not cover currently.
703
704         * UNCOVERED: Added.
705
706 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
707
708         Reviewed by Oliver.
709         
710         - Give better error messages when sunspider-compare-results fails
711
712         * sunspider-compare-results:
713
714 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
715
716         Reviewed by Darin Adler.
717         
718         - fix Date tests to work in command-line SpiderMonkey
719
720         * tests/date-format-tofte.js:
721         * tests/date-format-xparb.js:
722
723 2007-10-29  Darin Adler  <darin@apple.com>
724
725         Reviewed by Maciej.
726
727         * sunspider: Added --shark20 option, to run Shark at its highest sample
728         resolution instead of its default.
729
730 2007-10-28  Adam Roben  <aroben@apple.com>
731
732         Fix SunSpider on Windows
733
734         Reviewed by Eric.
735
736         * sunspider: Don't use colons in filenames.
737
738 2007-10-28  Darin Adler  <darin@apple.com>
739
740         Reviewed by Adam.
741
742         * resources/sunspider-analyze-results.js: Tweak the output format a little.
743         Change so that when there's only one run we don't write out confidence
744         intervals at all rather than writing "NaN" over and over again.
745
746         * sunspider: Use the time and date as a suffix on the results file, that way
747         all the results are left behind in the tmp directory. This will make it easy
748         for us to add features that compare past results in the future. Also rename
749         the .mshark files using the same scheme.
750
751         * sunspider-compare-results: Relax the parsing rules so we can parse the
752         results file exactly as it's emitted from sunspider.
753
754 2007-10-25  Adam Roben  <aroben@apple.com>
755
756         Strip carriage returns from results
757
758         Carriage returns were screwing up sunspider-compare-results on
759         Windows.
760
761         Rubberstamped by Oliver.
762
763         * sunspider:
764
765 2007-10-24  Darin Adler  <darin@apple.com>
766
767         Reviewed by Eric.
768
769         * sunspider: Open the profile in shark after completing a run.
770         * sunspider-compare-results: Tiny perl tweaks.
771
772 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
773
774         Reviewed by Darin Adler.
775
776         - Don't hardcode my path to testkjs
777         
778         * sunspider-compare-results:
779
780 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
781
782         Reviewed by Darin Adler.
783         
784         - Add a compare script to compare results, and improve formatting of standard results a bit.
785
786         * resources/sunspider-analyze-results.js:
787         * resources/sunspider-compare-results.js: Added.
788         * sunspider-compare-results: Added.
789
790 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
791
792         Reviewed by Darin Adler.
793
794         * resources/sunspider-analyze-results.js: Do the error range
795         properly, using the t-distribution instead of 1.96 (which was
796         based on the normal distribution).
797         
798         * sunspider: Print results in a way that will be friendlier to the
799         soon-to-come compare mode.
800
801 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
802
803         Reviewed by Review.
804         
805         - add more tests (probably the final set for now)
806         
807         Already balanced. Added date, regexp, control flow, and a few
808         more string and object/array access tests.
809         
810         * tests/LIST:
811         * tests/access-fannkuch.js: Added.
812         * tests/access-nbody.js: Added.
813         * tests/controlflow-recursive.js: Added.
814         * tests/date-format-tofte.js: Added.
815         * tests/date-format-xparb.js: Added.
816         * tests/regexp-dna.js: Added.
817         * tests/string-validate-input.js: Added.
818
819 2007-10-21  Eric Seidel  <eric@webkit.org>
820
821         Reviewed by mjs.
822
823         * sunspider: make --shark only sample from testkjs
824
825 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
826
827         Rubber stamped by Eric.
828         
829         - added crypto tests (already balanced)
830
831         * tests/LIST:
832         * tests/crypto-aes.js: Added.
833         * tests/crypto-md5.js: Added.
834         * tests/crypto-sha1.js: Added.
835
836 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
837
838         Reviewed by Adam.
839         
840         - rebalance new tests
841         
842         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
843         
844         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
845
846         This balances them with the existing tests.
847
848         * tests/access-binary-trees.js:
849         * tests/access-nsieve.js:
850         * tests/bitops-nsieve-bits.js:
851         * tests/math-partial-sums.js:
852         * tests/math-spectral-norm.js:
853         * tests/string-fasta.js:
854
855 2007-10-20  Eric Seidel  <eric@webkit.org>
856
857         Reviewed by darin.
858         
859         Add improved argument handling to sunspider, including
860         --runs=<number>, --tests=<pattern, --shark, and --help
861         Also re-factor code into subroutines
862
863         * sunspider: improved argument handling
864
865 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
866
867         Reviewed by Mark.
868
869         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
870         
871         * TODO:
872         * tests/LIST:
873         * tests/access-binary-trees.js: Added.
874         * tests/access-nsieve.js: Added.
875         * tests/bitops-nsieve-bits.js: Added.
876         * tests/math-partial-sums.js: Added.
877         * tests/math-spectral-norm.js: Added.
878         * tests/string-fasta.js: Added.
879
880 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
881
882         Reviewed by Darin Adler.
883
884         * TODO: Updated to note areas that are now well-covered.
885         
886         - Change the tests and the driver to avoid leaving large object
887         graphs hanging around, since that throws off the subsequent tests.
888         
889         * resources/sunspider-standalone-driver.js:
890         * tests/3d-cube.js:
891         * tests/3d-morph.js:
892         * tests/3d-raytrace.js:
893         * tests/bitops-bitwise-and.js:
894         * tests/string-base64.js:
895         * tests/string-tagcloud.js:
896
897 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
898
899         Reviewed by Nikolas.
900         
901         - Rebalance test complexity.
902         
903         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
904         
905         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
906
907         However, for some tests, IE7 was a huge outlier compared to other
908         major browsers. For those tests, I used Firefox 2.0.0.8 on the
909         same OS and hardware as a normalization basis.
910         
911         * tests/3d-morph.js:
912         * tests/3d-raytrace.js:
913         * tests/bitops-3bit-bits-in-byte.js:
914         * tests/bitops-bits-in-byte.js:
915         * tests/bitops-bitwise-and.js:
916         * tests/math-cordic.js:
917         * tests/string-base64.js:
918         * tests/string-tagcloud.js:
919         * tests/string-unpack-code.js:
920
921 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
922
923         Reviewed by Oliver.
924         
925         - Fix the browser-hosted driver to be compatible with Safari 2
926
927         * hosted/sunspider-driver.html:
928
929 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
930
931         Rubber stamped by Hyatt.
932         
933         - Add BSD license notice to cordic test, the original author said to do so.
934
935         * tests/math-cordic.js:
936
937 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
938
939         Reviewed by Eric.
940         
941         - Support Windows and Windows browsers.
942
943         * hosted/sunspider-results.html: Display results correctly in IE.
944         * sunspider: tweaks to work on Windows, and with spaces in the path.
945
946 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
947
948         Rubber stamped by Oliver.
949
950         * tests/string-tagcloud.js: Remove stray print()
951
952 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
953
954         Reviewed by Oliver.
955         
956         Add some string test cases.
957
958         * tests/string-base64.js: Added. Does base64 encoding and decoding.
959         * tests/string-tagcloud.js: Added. Parses a JSON string and
960         generates tagcloud HTML markup based on the data.        
961         * tests/string-unpack-code.js: Added. Unpack compressed versions of
962         four of the most popular JavaScript libraries.
963         * tests/LIST: Added the new tests.
964
965 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
966
967         Reviewed by Anders.
968
969         * TODO: Updated with info on how to do the stats properly.
970
971         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
972         avoid overly slowing down the benchmark.
973
974         Added some 3D-related test cases.
975         
976         * tests/3d-cube.js: Added.
977         * tests/3d-morph.js: Added.
978         * tests/3d-raytrace.js: Added.
979         * tests/LIST:
980
981 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
982
983         Reviewed by Darin Adler.
984
985         - New JavaScript benchmark
986         http://bugs.webkit.org/show_bug.cgi?id=15515
987         
988         A start on a new JavaScript benchmark, with standalone and
989         browser-hosted drivers.
990         
991         The test content is still incomplete and the drivers could use
992         some more features but there is enough here to make a good start.
993         
994         The drivers compute a 95% confidence interval on the mean for the
995         whole test, each category, and each individual test to make it
996         easier to tell whether differences are statistically
997         significant. The confidence interval can be narrowed by running
998         the test more times.
999         
1000         * TODO: Added. Remaining things that need to be done.
1001         * sunspider: Added. Perl script that acts as the standalone test driver.
1002         * resources/sunspider-standalone-driver.js: Added. JavaScript part
1003         of standalone test driver.
1004         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
1005         analysis code.
1006         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
1007         * make-hosted: Added. Script to generate browser-hosted tests.
1008         * hosted/sunspider.html: Added. Start page for browser-hosted test.
1009         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
1010         * hosted/sunspider-results.html: Added. Results page for browser-hosted
1011         tests.
1012         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
1013         tests.
1014         * tests/LIST: Added. List of tests to use.
1015         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
1016         * tests/bitops-bits-in-byte.js: Added. Ditto.
1017         * tests/bitops-bitwise-and.js: Added. Ditto.
1018         * tests/math-cordic.js: Added. Ditto.
1019