2011-06-30 Maciej Stachowiak <mjs@apple.com>
[WebKit-https.git] / PerformanceTests / SunSpider / ChangeLog
1 2011-06-30  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Adam Barth.
4
5         Create sunspider-1.0 directory in preparation for fixing a bunch of bugs
6         https://bugs.webkit.org/show_bug.cgi?id=63782
7
8         * make-hosted: Learn about the new directory.
9         * tests/sunspider-1.0: Copied from PerformanceTests/SunSpider/tests/sunspider-0.9.1.
10
11 2010-11-29  Geoffrey Garen  <ggaren@apple.com>
12
13         Reviewed by Gavin Barraclough.
14
15         Improved accuracy of command-line SunSpider.
16
17         * resources/sunspider-standalone-driver.js: Use the returned number of
18         elapsed milliseconds from "run" and "checkSyntax" instead of measuring
19         ourselves, for slightly more accurate numbers.
20
21 2010-09-16  Gavin Barraclough  <barraclough@apple.com>
22
23         Reviewed by Mark Rowe.
24         https://bugs.webkit.org/show_bug.cgi?id=45924
25
26         Modify the SunSpider harness to allow a path to be passed to the --suite
27         argument, supress the normal prepend-"tests/" behaviour for suite names
28         containing a '/'.
29
30         Also wrap the code in an anonymous function to move variables out of global
31         scope (currently the harness cannot handle running tests the overwrite the
32         variable 'j').
33
34         * resources/sunspider-standalone-driver.js:
35         * sunspider:
36
37 2010-08-27  Michael Saboff  <msaboff@apple.com>
38
39         Reviewed by Stephanie Lewis.
40
41         Added version 5 and version 6 of V8 tests with modifications similar
42         to the version 4 tests so they'll work with the SunSpider test harness.
43         Note that these files are external source and therefore don't conform 
44         to WebKit coding standards.
45         https://bugs.webkit.org/show_bug.cgi?id=44638
46
47         * tests/v8-v5: Added.
48         * tests/v8-v5/LIST: Added.
49         * tests/v8-v5/v8-crypto.js: Added.
50         * tests/v8-v5/v8-deltablue.js: Added.
51         * tests/v8-v5/v8-earley-boyer.js: Added.
52         * tests/v8-v5/v8-raytrace.js: Added.
53         * tests/v8-v5/v8-regexp.js: Added.
54         * tests/v8-v5/v8-richards.js: Added.
55         * tests/v8-v5/v8-splay.js: Added.
56         * tests/v8-v6: Added.
57         * tests/v8-v6/LIST: Added.
58         * tests/v8-v6/v8-crypto.js: Added.
59         * tests/v8-v6/v8-deltablue.js: Added.
60         * tests/v8-v6/v8-earley-boyer.js: Added.
61         * tests/v8-v6/v8-raytrace.js: Added.
62         * tests/v8-v6/v8-regexp.js: Added.
63         * tests/v8-v6/v8-richards.js: Added.
64         * tests/v8-v6/v8-splay.js: Added.
65
66 2010-07-15  Stephanie Lewis  <slewis@apple.com>
67
68         Reviewed by Geoff Garen.
69
70         http://bugs.webkit.org/show_bug.cgi?id=42406
71         Output location of results file.
72
73         * sunspider:
74
75 2010-05-17  Maciej Stachowiak  <mjs@apple.com>
76
77         Reviewed by Mark Rowe.
78
79         Browser-hosted version of SunSpider runs tests twice as many times as advertised
80         https://bugs.webkit.org/show_bug.cgi?id=39201
81
82         * make-hosted: Make sure to reset test list after every test suite.
83
84 2010-05-07  Adam Roben  <aroben@apple.com>
85
86         Fix typo in sunspider-compare-results.js that caused incorrect
87         two-sample t statistics to be computed
88
89         Fixes <http://webkit.org/b/38768> SunSpider computes incorrect
90         two-sample t statistics
91
92         Reviewed by Darin Adler.
93
94         * resources/sunspider-compare-results.js:
95         (sunspiderCompareResults.resultLine): Fixed the calculation of the t
96         variable to match the definition of a two-sample t statistic. Looks
97         like this was just a simple typo.
98
99 2010-04-05  Darin Adler  <darin@apple.com>
100
101         Updated the ignore property for changes to the contents here.
102
103         * .: Modified property svn:ignore.
104
105 2010-02-20  Maciej Stachowiak  <mjs@apple.com>
106
107         Reviewed by Adam Barth.
108
109         Add sunspider-0.9.1 to the site in preparation for announcing it.
110         https://bugs.webkit.org/show_bug.cgi?id=35206
111
112         * hosted/sunspider.css: Remove tabs, which I noticed where present while
113         copying to the WebKitSite directory.
114
115 2010-01-11  Oliver Hunt  <oliver@apple.com>
116
117         Reviewed by Darin Adler.
118
119         Sunspider parse-only tests don't run
120         https://bugs.webkit.org/show_bug.cgi?id=33489
121
122         Fix parse-only test list
123
124         * tests/parse-only/LIST:
125
126 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
127
128         Reviewed by Adele Peterson.
129
130         Make SunSpider version more prominent in the title
131         https://bugs.webkit.org/show_bug.cgi?id=32574
132
133         * hosted/sunspider.html:
134         * hosted/versions.html:
135         * resources/driver-TEMPLATE.html:
136         * resources/results-TEMPLATE.html:
137
138 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
139
140         Reviewed by Darin Adler.
141
142         Make SunSpider refuse to compare results across content versions
143         https://bugs.webkit.org/show_bug.cgi?id=32573
144         
145         The results URL now includes a version indicator, and compare mode
146         will refuse to compare between versions.
147
148         * resources/driver-TEMPLATE.html:
149         * resources/results-TEMPLATE.html:
150
151 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
152
153         Reviewed by Darin Adler.
154
155         Some Browser-hosted SunSpider files are not valid HTML5
156         https://bugs.webkit.org/show_bug.cgi?id=32536
157         
158         Made various fixes to validate as HTML5 with no warnings (mainly removing bogus
159         close tags and adding a meta charset declaration).
160
161         * hosted/sunspider.html:
162         * hosted/versions.html:
163         * resources/TEMPLATE.html:
164         * resources/driver-TEMPLATE.html:
165         * resources/results-TEMPLATE.html:
166
167 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
168
169         Reviewed by Eric Seidel.
170
171         Make sunspider-0.9.1 the default content set (both command-line and hosted)
172         https://bugs.webkit.org/show_bug.cgi?id=32537
173
174         * hosted/sunspider.html: Change default suite link.
175         * sunspider: Change default suite.
176         * sunspider-compare-results: Likewise.
177
178 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
179
180         Reviewed by Sam Weinig.
181
182         in-browser SunSpider suffers excessive penalty under power management
183         https://bugs.webkit.org/show_bug.cgi?id=32505
184         
185         I have made a few changes to address this:
186         
187         1) Change how browser-hosted SunSpider loads tests - preload the content and write it in with
188         document.write to reduce triggering of progress UI.
189         2) Reduce time between tests to 10ms from 500ms, so that power management doesn't keep the CPU
190         stepped all the way down the whole time.
191         3) Run the test cycle an extra time for warmup before the runs that count.
192
193         * hosted/sunspider-record-result.js: Removed.
194         * make-hosted:
195         * resources/TEMPLATE.html:
196         * resources/driver-TEMPLATE.html:
197
198 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
199
200         Reviewed by Gavin Barraclough.
201
202         SunSpider/tests/string-base64.js does not compute a valid base64 encoded string
203         https://bugs.webkit.org/show_bug.cgi?id=16806
204
205         Based on a patch by Eric Seidel.
206         
207         Fix the base64 computation to actually compute correct results. The impact on runtime of
208         the test is pretty small, but noticeable for some browsers. But at least it's not
209         doing a wrong and meaningless computation any more.
210         
211         * tests/sunspider-0.9.1/string-base64.js:
212         ():
213         (base64ToString):
214
215 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
216
217         Fixing commit error...
218         
219         I accidentally committed my last patch in a form that broke Web-hosted SunSpider. Fixing.
220
221         * resources/driver-TEMPLATE.html:
222
223 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
224
225         Reviewed by Sam Weinig.
226
227         Use JSON.parse instead of eval for Web-hosted SunSpider results processing
228         https://bugs.webkit.org/show_bug.cgi?id=32490
229
230         * hosted/json2.js: Added. Incorporated from json.org
231         * resources/driver-TEMPLATE.html: Fix an HTML validation bug I noticed.
232         * resources/results-TEMPLATE.html: Use JSON.parse instead of eval to process
233         results.
234
235 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
236
237         Reviewed by Oliver Hunt.
238
239         Enable Web-hosted version of SunSpider to handle multiple versions
240         https://bugs.webkit.org/show_bug.cgi?id=32478
241
242         * make-hosted: Now generate the hosted version to be able to run both the
243         0.9 and the 0.9.1 test suites.
244         * hosted: Modified property svn:ignore.
245         * hosted/sunspider-driver.html: Removed.
246         * hosted/sunspider-results.html: Removed.
247         * hosted/sunspider.html:
248         * hosted/versions.html: Added.
249         * resources/TEMPLATE.html:
250         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-driver.html.
251         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-results.html.
252
253 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
254
255         Reviewed by Oliver Hunt.
256
257         Give command-line sunspider the ability to handle multiple suites and versions
258         https://bugs.webkit.org/show_bug.cgi?id=32477
259         
260         Each suite/version's set of tests are now in a separate subdirectory of tests/, with a file named LIST
261         enumerating the tests. Current suites are:
262         
263         sunspider-0.9 - riginal SunSpider content 
264         sunspider-0.9.1 - SunSpider suite with a few tests fixed (and a few more to come)
265         ubench - "ubench" microbenchmark suite
266         v8-v4 - version 4 of the v8 benchmark
267         parse-only - some parser tests
268         
269         Sunspider can now be run with the --suite= parameter to select a
270         suite. The default is --suite=sunspider-0.9.
271
272         * .: Modified property svn:ignore to ignore new results directories.
273         * make-hosted: Generate based on sunspider-0.9 suite.
274         * resources/TEMPLATE.html: Fix an HTML compliance problem.
275         * resources/sunspider-standalone-driver.js: Become aware of suite names.
276         * sunspider: Modified to handle multiple suites as described above.
277         * sunspider-compare-results: ditto
278         * tests/LIST: Moved to tests/sunspider-0.9/ and tests/sunspider-0.9.1/
279         * tests/LIST-PARSE-ONLY: Moved to parse-only/LIST
280         * tests/LIST-UBENCH: Removed.
281         * tests/LIST-V8: Removed.
282         * tests/3d-cube.js: Moved to tests/sunspider-0.9 and tests/sunspider-0.9.1
283         * tests/3d-morph.js: ditto
284         * tests/3d-raytrace.js: ditto
285         * tests/access-binary-trees.js: ditto
286         * tests/access-fannkuch.js: ditto
287         * tests/access-nbody.js: ditto
288         * tests/access-nsieve.js: ditto
289         * tests/bitops-3bit-bits-in-byte.js: ditto
290         * tests/bitops-bits-in-byte.js: ditto
291         * tests/bitops-bitwise-and.js: ditto
292         * tests/bitops-nsieve-bits.js: ditto
293         * tests/controlflow-recursive.js: ditto
294         * tests/crypto-aes.js: ditto
295         * tests/crypto-md5.js: ditto
296         * tests/crypto-sha1.js: ditto
297         * tests/date-format-tofte.js: ditto
298         * tests/date-format-xparb.js: ditto
299         * tests/math-cordic.js: ditto
300         * tests/math-partial-sums.js: ditto
301         * tests/math-spectral-norm.js: ditto
302         * tests/regexp-dna.js: ditto
303         * tests/string-base64.js: ditto
304         * tests/string-fasta.js: ditto
305         * tests/string-tagcloud.js: ditto
306         * tests/string-unpack-code.js: ditto
307         * tests/string-validate-input.js: ditto
308         * tests/sunspider-0.9/: Added
309         * tests/sunspider-0.9/*.js: Moved from one level up
310         * tests/sunspider-0.9.1: Added.
311         * tests/sunspider-0.9.1/*.js: Moved from one level up.
312         * tests/v8-crypto.js: Moved to v8-v4/ directory.
313         * tests/v8-deltablue.js: ditto
314         * tests/v8-earley-boyer.js: ditto
315         * tests/v8-raytrace.js: ditto
316         * tests/v8-regexp.js: ditto
317         * tests/v8-richards.js: ditto
318         * tests/v8-splay.js: ditto
319         * tests/v8-v4: Added.
320
321 2009-09-30  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
322
323         Reviewed by Darin Adler.
324
325         Platform specific null device replaced with a platform independent.
326         https://bugs.webkit.org/show_bug.cgi?id=29544
327
328         * sunspider: Using File::Spec->devnull() instead of /dev/null and NUL.
329
330 2009-09-21  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
331
332         Reviewed by Maciej Stachowiak.
333
334         [Qt] Make sunspider script work on Windows platform.
335         https://bugs.webkit.org/show_bug.cgi?id=29544
336
337         * sunspider:
338         Using platform specific null device instead of /dev/null.
339
340 2009-06-19  Adam Treat  <adam.treat@torchmobile.com>
341
342         Reviewed by Oliver Hunt.
343
344         https://bugs.webkit.org/show_bug.cgi?id=26540
345         Currently the SunSpider test driver lacks an option to run a test suite that
346         will test JavaScriptCore parsing performance only.  This patch adds just such
347         a test suite and option to SunSpider as well as the jsc test shell.  I've included
348         three large javascript source files found in the wild: jquery, mootools and prototype.
349         Combined with the concatenation of all three, these form a new testsuite to measure
350         and test pure JavaScriptCore parsing performance.
351
352         * resources/sunspider-standalone-driver.js:
353           Load the js with parse-only flag if it is part of the parse-only suite.
354         * sunspider:
355           Add the parse-only suite to the list of options.
356         * sunspider-compare-results:
357           Add the parse-only suite to the list of options.
358         * tests/LIST-PARSE-ONLY: Added.
359         * tests/parse-only/concat-jquery-mootools-prototype.js: Added.
360           A very large concatenation of the scripts found below.  This js takes ~30ms to parse on my machine.
361         * tests/parse-only/jquery-1.3.2.js: Added.
362           Latest version from jquery.com.
363         * tests/parse-only/mootools-1.2.2-core-nc.js: Added.
364           Latest version from mootools.net.
365         * tests/parse-only/prototype-1.6.0.3.js: Added.
366           Latest version from prototypejs.org.
367
368 2009-06-13  Steve Falkenburg  <sfalken@apple.com>
369
370         Update V8 benchmark to version 4.
371         
372         Reviewed by Cameron Zwarich.
373
374         * tests/LIST-V8:
375         * tests/v8-crypto.js:
376         * tests/v8-deltablue.js:
377         * tests/v8-earley-boyer.js:
378         * tests/v8-raytrace.js:
379         * tests/v8-regexp.js: Added.
380         * tests/v8-richards.js:
381         * tests/v8-splay.js: Added.
382
383 2009-05-26  Olivier DOLE  <odole@pleyo.com>
384
385         Reviewed by Maciej.
386
387         Store sunspider test suite results in a specific directory for each
388         different type of benchmark (sunspider, V8 or ubench).
389
390         * sunspider:
391         * sunspider-compare-results:
392
393 2009-05-12  Steve Falkenburg  <sfalken@apple.com>
394
395         Build fix.
396
397         * SunSpider.make:
398
399 2009-05-11  Steve Falkenburg  <sfalken@apple.com>
400
401         Include SunSpider tests in production builds.
402         
403         Reviewed by Adam Roben.
404
405         * SunSpider.make: Added.
406         * xcopy.excludes: Added.
407
408 2009-03-04  Oliver Hunt  <oliver@apple.com>
409
410         Added a few more tests to ubench
411
412         RS=Gavin Barraclough
413
414         Adding a couple of test to cover too few and too many arguments.  This
415         is needed to prevent regressions in all call paths from my current refactoring.
416
417         * tests/LIST-UBENCH:
418         * tests/ubench/function-correct-args.js: Added.
419         (f):
420         * tests/ubench/function-excess-args.js: Added.
421         (f):
422
423 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
424
425         Fix a bug where when V8's or UBENCH's test suite list file is missing
426         the error message is suggesting that SunSpider's list file is missing.
427
428         Reviewed by Sam Weinig.
429
430         * sunspider:
431         (loadTestsList): Fix the error message to use the right list name.
432
433 2008-10-07  David Hyatt  <hyatt@apple.com>
434
435         https://bugs.webkit.org/show_bug.cgi?id=18989
436
437         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
438         we want all browsers to be doing the same thing on the test.
439
440         Reviewed by Maciej
441
442         * tests/regexp-dna.js:
443
444 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
445
446         Rubber-stamped by Maciej Stachowiak.
447
448         The bleeding edge branch of V8 now supports passing fileanmes after
449         '-f', so the SunSpider script should be changed to do this again.
450         This allows all 3 major JavaScript engines to be used with the
451         unmodified SunSpider script.
452
453         * sunspider:
454
455 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
456
457         Reviewed by Maciej Stachowiak.
458
459         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
460         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
461         correct the punctuation in its documentation.
462
463         * sunspider:
464
465 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
466
467         Reviewed by Oliver Hunt.
468
469         Bug 20616: Incorporate V8 benchmarks in testing
470         <https://bugs.webkit.org/show_bug.cgi?id=20616>
471
472         This is a first attempt at running the V8 benchmarks within SunSpider,
473         with the --v8 option. The number of iterations is fixed for each
474         benchmark, and was chosen to make each of the benchmarks run between
475         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
476
477         * sunspider:
478         * tests/LIST-V8: Added.
479         * tests/v8-crypto.js: Added.
480         * tests/v8-deltablue.js: Added.
481         * tests/v8-earley-boyer.js: Added.
482         * tests/v8-raytrace.js: Added.
483         * tests/v8-richards.js: Added.
484
485 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
486
487         Not reviewed.
488
489         Remove a local change that was accidentally committed in r36034.
490
491         * resources/sunspider-standalone-driver.js:
492
493 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
494
495         Rubber-stamped by Oliver Hunt.
496
497         Make SunSpider work with JavaScript shells that don't take filenames
498         after a '-f' argument.
499
500         * sunspider:
501
502 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
503
504         Reviewed by Timothy.
505
506         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
507
508         * sunspider:
509         * tests/LIST-SQUIRRELFISH: Removed.
510
511 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
512
513         Rubber-stamped by Mark Rowe.
514
515         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
516
517         * sunspider:
518
519 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
520
521         Rubber stamped by Oliver.
522         
523         - add newly working tests to squirrelfish list
524
525         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
526
527 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
528
529         Rubber stamped by Oliver.
530         
531         - move ubench tests properly.
532
533         * tests/ubench: Added.
534         * tests/ubench/function-closure.js: Added.
535         * tests/ubench/function-empty.js: Added.
536         * tests/ubench/function-missing-args.js: Added.
537         * tests/ubench/function-sum.js: Added.
538         * tests/ubench/loop-empty-resolve.js: Added.
539         * tests/ubench/loop-empty.js: Added.
540         * tests/ubench/loop-sum.js: Added.
541
542 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
543
544         Rubber stamped by Oliver.
545
546         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
547
548 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
549
550         Reviewed by Oliver.
551
552         - make --squirrelfish mode use the real harness, but a reduced set of tests
553
554         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
555
556         * sunspider:
557         * tests/LIST-SQUIRRELFISH:
558         * tests/LIST-UBENCH: Added.
559         * tests/squirrelfish: Moved to tests/ubench
560
561 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
562
563         Reviewed by Maciej Stachowiak.
564         
565         Removed an unused test.
566
567         * tests/squirrelfish/loop-resolve.js: Removed.
568
569 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
570
571         Reviewed by Oliver.
572         
573         Add newly runnable tests to --squirrelfish mode.
574
575         * tests/LIST-SQUIRRELFISH:
576
577 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
578
579         Reviewed by Maciej Stachowiak.
580         
581         A little more love for --squirrelfish mode:
582         
583         Fixed a misplaced sort. Results properly sort now.
584         
585         Added a test list just for squirrelfish, and updated it to include
586         all currently passing SunSpider tests.
587         
588         Fixed the pruning regexp to match 3d-morph.js.
589
590         * sunspider:
591
592 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
593
594         Reviewed by Oliver Hunt.
595         
596         --squirrelfish mode: pared down tests for squirrelfish to chew on.
597
598         * sunspider:
599         * tests/LIST:
600         * tests/squirrelfish: Added.
601         * tests/squirrelfish/function-closure.js: Added.
602         * tests/squirrelfish/function-empty.js: Added.
603         * tests/squirrelfish/function-missing-args.js: Added.
604         * tests/squirrelfish/function-sum.js: Added.
605         * tests/squirrelfish/loop-empty-resolve.js: Added.
606         * tests/squirrelfish/loop-empty.js: Added.
607         * tests/squirrelfish/loop-resolve.js: Added.
608         * tests/squirrelfish/loop-sum.js: Added.
609
610 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
611
612         Reviewed by Geoff.
613
614         * sunspider: Pause update daemon on Darwin to make results reliable
615         again (workaround for <rdar://problem/5811127>).
616
617 2008-02-07  Eric Seidel  <eric@webkit.org>
618
619         Reviewed by Sam.
620
621         * sunspider: add an error message when ./tests/LIST is not found
622
623 2007-12-29  Alp Toker  <alp@atoker.com>
624
625         Fix typo.
626
627         * hosted/sunspider.html:
628
629 2007-12-29  Alp Toker  <alp@atoker.com>
630
631         Reviewed by Maciej.
632
633         Provide the current URL in the SunSpider results page. Useful for
634         copying benchmark results out of browsers without a location bar or
635         where the location bar can't handle long URLs.
636
637         * hosted/sunspider-results.html:
638
639 2007-12-20  Eric Seidel  <eric@webkit.org>
640
641         Reviewed by mjs.
642
643         * sunspider: increase default number of runs for more accurate numbers
644
645 2007-12-18  Mark Rowe  <mrowe@apple.com>
646
647         Fix typo.
648
649         * hosted/sunspider.html:
650
651 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
652
653         Reviewed by Eric.
654         
655         - give an extra digit of precision for small differences.
656
657         * resources/sunspider-compare-results.js:
658
659 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
660
661         Reviewed by Eric.
662         
663         - work around IE layout bugs
664
665         * hosted/sunspider.css:
666
667 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
668
669         Reviewed by Mark.
670         
671         - fix compare feature
672
673         * hosted/sunspider-results.html:
674
675 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
676
677         Reviewed by Mark.
678         
679         - Change the format of the output URL to be shorter, to meet IE constraints
680
681         * hosted/sunspider-driver.html:
682         * hosted/sunspider-results.html:
683
684 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
685
686         Reviewed by Oliver.
687         
688         - Add some explanatory test and a dumb logo to SunSpider web hosted version
689
690         * hosted/sunspider-driver.html:
691         * hosted/sunspider-results.html:
692         * hosted/sunspider.css:
693         * hosted/sunspider.html:
694
695 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
696
697         Reviewed by Oliver.
698         
699         - Make the web-hosted version of SunSpider look a little bit less drab.
700
701         * hosted/sunspider-driver.html:
702         * hosted/sunspider-results.html:
703         * hosted/sunspider.css: Added.
704         * hosted/sunspider.html:
705         * resources/TEMPLATE.html:
706
707 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
708
709         Reviewed by Sam.
710         
711         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
712         
713         A sample of what new-style results would look like:
714         
715         TEST                   COMPARISON            FROM                 TO             DETAILS
716         ...
717         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
718         ...
719             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
720         
721         * resources/sunspider-compare-results.js:
722
723 2007-11-14  Eric Seidel  <eric@webkit.org>
724
725         Reviewed by Sam.
726
727         * sunspider: add --shark-cache for L2 Cache Miss profiling
728
729 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
730
731         Reviewed by Darin Adler.
732
733         http://bugs.webkit.org/show_bug.cgi?id=15925
734         SunSpider should check for Shark being installed
735
736         * sunspider: Explicitly test that shark command line tool is installed.
737
738 2007-11-04  Darin Adler  <darin@apple.com>
739
740         * hosted/sunspider.html: Tweaked formatting a bit.
741
742 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
743
744         Reviewed by Darin Adler.
745
746         - Add compare mode and a description of the benchmark to the hosted version.
747
748         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
749         * hosted/sunspider.html: Added intro text.
750         * make-hosted: Copy new file.
751         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
752         a function.
753         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
754         expects.
755         * sunspider-compare-results: Use sunspider-standalone-compare.js.
756
757 2007-11-02  Darin Adler  <darin@apple.com>
758
759         * sunspider: Changed "--base" to "--set-baseline".
760
761 2007-11-02  Darin Adler  <darin@apple.com>
762
763         Reviewed by Maciej.
764
765         * sunspider: Added a "--base" option that records the current test as a baseline.
766         Moved the code to find the newest .mshark file into a function (since I needed to use
767         it in the other script and I prefer to copy/paste a function).
768
769         * sunspider-compare-results: Added logic to compare the baseline with the last
770         run when you don't pass any parameters.
771
772 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
773
774         Reviewed by Eric.
775         
776         - Added list of ECMAScript3 features that SunSpider does not cover currently.
777
778         * UNCOVERED: Added.
779
780 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
781
782         Reviewed by Oliver.
783         
784         - Give better error messages when sunspider-compare-results fails
785
786         * sunspider-compare-results:
787
788 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
789
790         Reviewed by Darin Adler.
791         
792         - fix Date tests to work in command-line SpiderMonkey
793
794         * tests/date-format-tofte.js:
795         * tests/date-format-xparb.js:
796
797 2007-10-29  Darin Adler  <darin@apple.com>
798
799         Reviewed by Maciej.
800
801         * sunspider: Added --shark20 option, to run Shark at its highest sample
802         resolution instead of its default.
803
804 2007-10-28  Adam Roben  <aroben@apple.com>
805
806         Fix SunSpider on Windows
807
808         Reviewed by Eric.
809
810         * sunspider: Don't use colons in filenames.
811
812 2007-10-28  Darin Adler  <darin@apple.com>
813
814         Reviewed by Adam.
815
816         * resources/sunspider-analyze-results.js: Tweak the output format a little.
817         Change so that when there's only one run we don't write out confidence
818         intervals at all rather than writing "NaN" over and over again.
819
820         * sunspider: Use the time and date as a suffix on the results file, that way
821         all the results are left behind in the tmp directory. This will make it easy
822         for us to add features that compare past results in the future. Also rename
823         the .mshark files using the same scheme.
824
825         * sunspider-compare-results: Relax the parsing rules so we can parse the
826         results file exactly as it's emitted from sunspider.
827
828 2007-10-25  Adam Roben  <aroben@apple.com>
829
830         Strip carriage returns from results
831
832         Carriage returns were screwing up sunspider-compare-results on
833         Windows.
834
835         Rubberstamped by Oliver.
836
837         * sunspider:
838
839 2007-10-24  Darin Adler  <darin@apple.com>
840
841         Reviewed by Eric.
842
843         * sunspider: Open the profile in shark after completing a run.
844         * sunspider-compare-results: Tiny perl tweaks.
845
846 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
847
848         Reviewed by Darin Adler.
849
850         - Don't hardcode my path to testkjs
851         
852         * sunspider-compare-results:
853
854 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
855
856         Reviewed by Darin Adler.
857         
858         - Add a compare script to compare results, and improve formatting of standard results a bit.
859
860         * resources/sunspider-analyze-results.js:
861         * resources/sunspider-compare-results.js: Added.
862         * sunspider-compare-results: Added.
863
864 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
865
866         Reviewed by Darin Adler.
867
868         * resources/sunspider-analyze-results.js: Do the error range
869         properly, using the t-distribution instead of 1.96 (which was
870         based on the normal distribution).
871         
872         * sunspider: Print results in a way that will be friendlier to the
873         soon-to-come compare mode.
874
875 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
876
877         Reviewed by Review.
878         
879         - add more tests (probably the final set for now)
880         
881         Already balanced. Added date, regexp, control flow, and a few
882         more string and object/array access tests.
883         
884         * tests/LIST:
885         * tests/access-fannkuch.js: Added.
886         * tests/access-nbody.js: Added.
887         * tests/controlflow-recursive.js: Added.
888         * tests/date-format-tofte.js: Added.
889         * tests/date-format-xparb.js: Added.
890         * tests/regexp-dna.js: Added.
891         * tests/string-validate-input.js: Added.
892
893 2007-10-21  Eric Seidel  <eric@webkit.org>
894
895         Reviewed by mjs.
896
897         * sunspider: make --shark only sample from testkjs
898
899 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
900
901         Rubber stamped by Eric.
902         
903         - added crypto tests (already balanced)
904
905         * tests/LIST:
906         * tests/crypto-aes.js: Added.
907         * tests/crypto-md5.js: Added.
908         * tests/crypto-sha1.js: Added.
909
910 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
911
912         Reviewed by Adam.
913         
914         - rebalance new tests
915         
916         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
917         
918         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
919
920         This balances them with the existing tests.
921
922         * tests/access-binary-trees.js:
923         * tests/access-nsieve.js:
924         * tests/bitops-nsieve-bits.js:
925         * tests/math-partial-sums.js:
926         * tests/math-spectral-norm.js:
927         * tests/string-fasta.js:
928
929 2007-10-20  Eric Seidel  <eric@webkit.org>
930
931         Reviewed by darin.
932         
933         Add improved argument handling to sunspider, including
934         --runs=<number>, --tests=<pattern, --shark, and --help
935         Also re-factor code into subroutines
936
937         * sunspider: improved argument handling
938
939 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
940
941         Reviewed by Mark.
942
943         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
944         
945         * TODO:
946         * tests/LIST:
947         * tests/access-binary-trees.js: Added.
948         * tests/access-nsieve.js: Added.
949         * tests/bitops-nsieve-bits.js: Added.
950         * tests/math-partial-sums.js: Added.
951         * tests/math-spectral-norm.js: Added.
952         * tests/string-fasta.js: Added.
953
954 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
955
956         Reviewed by Darin Adler.
957
958         * TODO: Updated to note areas that are now well-covered.
959         
960         - Change the tests and the driver to avoid leaving large object
961         graphs hanging around, since that throws off the subsequent tests.
962         
963         * resources/sunspider-standalone-driver.js:
964         * tests/3d-cube.js:
965         * tests/3d-morph.js:
966         * tests/3d-raytrace.js:
967         * tests/bitops-bitwise-and.js:
968         * tests/string-base64.js:
969         * tests/string-tagcloud.js:
970
971 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
972
973         Reviewed by Nikolas.
974         
975         - Rebalance test complexity.
976         
977         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
978         
979         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
980
981         However, for some tests, IE7 was a huge outlier compared to other
982         major browsers. For those tests, I used Firefox 2.0.0.8 on the
983         same OS and hardware as a normalization basis.
984         
985         * tests/3d-morph.js:
986         * tests/3d-raytrace.js:
987         * tests/bitops-3bit-bits-in-byte.js:
988         * tests/bitops-bits-in-byte.js:
989         * tests/bitops-bitwise-and.js:
990         * tests/math-cordic.js:
991         * tests/string-base64.js:
992         * tests/string-tagcloud.js:
993         * tests/string-unpack-code.js:
994
995 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
996
997         Reviewed by Oliver.
998         
999         - Fix the browser-hosted driver to be compatible with Safari 2
1000
1001         * hosted/sunspider-driver.html:
1002
1003 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1004
1005         Rubber stamped by Hyatt.
1006         
1007         - Add BSD license notice to cordic test, the original author said to do so.
1008
1009         * tests/math-cordic.js:
1010
1011 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
1012
1013         Reviewed by Eric.
1014         
1015         - Support Windows and Windows browsers.
1016
1017         * hosted/sunspider-results.html: Display results correctly in IE.
1018         * sunspider: tweaks to work on Windows, and with spaces in the path.
1019
1020 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
1021
1022         Rubber stamped by Oliver.
1023
1024         * tests/string-tagcloud.js: Remove stray print()
1025
1026 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
1027
1028         Reviewed by Oliver.
1029         
1030         Add some string test cases.
1031
1032         * tests/string-base64.js: Added. Does base64 encoding and decoding.
1033         * tests/string-tagcloud.js: Added. Parses a JSON string and
1034         generates tagcloud HTML markup based on the data.        
1035         * tests/string-unpack-code.js: Added. Unpack compressed versions of
1036         four of the most popular JavaScript libraries.
1037         * tests/LIST: Added the new tests.
1038
1039 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
1040
1041         Reviewed by Anders.
1042
1043         * TODO: Updated with info on how to do the stats properly.
1044
1045         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
1046         avoid overly slowing down the benchmark.
1047
1048         Added some 3D-related test cases.
1049         
1050         * tests/3d-cube.js: Added.
1051         * tests/3d-morph.js: Added.
1052         * tests/3d-raytrace.js: Added.
1053         * tests/LIST:
1054
1055 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
1056
1057         Reviewed by Darin Adler.
1058
1059         - New JavaScript benchmark
1060         http://bugs.webkit.org/show_bug.cgi?id=15515
1061         
1062         A start on a new JavaScript benchmark, with standalone and
1063         browser-hosted drivers.
1064         
1065         The test content is still incomplete and the drivers could use
1066         some more features but there is enough here to make a good start.
1067         
1068         The drivers compute a 95% confidence interval on the mean for the
1069         whole test, each category, and each individual test to make it
1070         easier to tell whether differences are statistically
1071         significant. The confidence interval can be narrowed by running
1072         the test more times.
1073         
1074         * TODO: Added. Remaining things that need to be done.
1075         * sunspider: Added. Perl script that acts as the standalone test driver.
1076         * resources/sunspider-standalone-driver.js: Added. JavaScript part
1077         of standalone test driver.
1078         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
1079         analysis code.
1080         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
1081         * make-hosted: Added. Script to generate browser-hosted tests.
1082         * hosted/sunspider.html: Added. Start page for browser-hosted test.
1083         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
1084         * hosted/sunspider-results.html: Added. Results page for browser-hosted
1085         tests.
1086         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
1087         tests.
1088         * tests/LIST: Added. List of tests to use.
1089         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
1090         * tests/bitops-bits-in-byte.js: Added. Ditto.
1091         * tests/bitops-bitwise-and.js: Added. Ditto.
1092         * tests/math-cordic.js: Added. Ditto.
1093