Get rid of the 10ms delay between test executions
[WebKit-https.git] / PerformanceTests / SunSpider / ChangeLog
1 2013-04-11  Filip Pizlo  <fpizlo@apple.com>
2
3         Get rid of the 10ms delay between test executions
4         https://bugs.webkit.org/show_bug.cgi?id=114458
5         <rdar://problem/13631112>
6
7         Reviewed by Darin Adler and Geoffrey Garen.
8         
9         More details on this in the bug. Long story short, I don't know why the 10ms delay is
10         there, and it appears to be the root of some noisyness and unreliability in the results.
11         Removing it makes results more consistent, which is a good thing.
12
13         * resources/driver-TEMPLATE.html:
14
15 2013-01-16  Eric Seidel  <eric@webkit.org>
16
17         Remove --shark* support from sunspider/run-sunspider now that Shark is dead
18         https://bugs.webkit.org/show_bug.cgi?id=99512
19
20         Reviewed by Darin Adler.
21
22         I tried to use --instruments, but that also seems broken (in the same way before/after this patch).
23
24         * sunspider:
25         (runTestsOnce):
26
27 2012-08-02  Filip Pizlo  <fpizlo@apple.com>
28
29         Unreviewed, revert debugging stuff I landed in http://trac.webkit.org/changeset/124557
30         by accident.
31
32         * tests/v8-v6/v8-crypto.js:
33         (am1):
34         (am2):
35         (am3):
36         (am4):
37         (bnpCopyTo):
38         (bnpFromString):
39         (bnpClamp):
40
41 2012-03-11  Maciej Stachowiak  <mjs@apple.com>
42
43         Add --instruments option to sunspider to profile with Instruments
44         https://bugs.webkit.org/show_bug.cgi?id=80783
45
46         Reviewed by Filip Pizlo.
47
48         * resources/TimeProfile20us.tracetemplate: Added.
49         * sunspider:
50         (runTestsOnce):
51
52 2011-11-15  Andy Wingo  <wingo@igalia.com>
53
54         Extend sunspider driver to be able to run kraken
55         https://bugs.webkit.org/show_bug.cgi?id=71799
56
57         Reviewed by Filip Pizlo.
58
59         * resources/sunspider-standalone-driver.js: Try to load a -data
60         file.  If that succeeds, we have a kraken-like test, so we time
61         the test using `load'.  Otherwise fall back to using `run'.
62
63 2011-08-11  Mark Rowe  <mrowe@apple.com>
64
65         Fix sunspider-compare-results to default to the same suite version as the sunspider script.
66
67         Rubber-stamped by Maciej Stachowiak.
68
69         * sunspider-compare-results:
70
71 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
72
73         Update SunSpider versioning to 1.0
74         https://bugs.webkit.org/show_bug.cgi?id=63870
75
76         Reviewed by Dan Bernstein.
77
78         * hosted/sunspider.html:
79         * hosted/versions.html:
80         * resources/driver-TEMPLATE.html:
81         * resources/results-TEMPLATE.html:
82         * sunspider:
83
84 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
85
86         Sunspider 0.9.1 harness doesn't actually close() its test documents
87         https://bugs.webkit.org/show_bug.cgi?id=47045
88
89         Reviewed by Daniel Bates.
90
91         * hosted/versions.html:
92         * resources/driver-TEMPLATE.html:
93
94 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
95
96         Not Reviewed. 
97         
98         Fix obviou typo in previous commit.
99
100         * tests/sunspider-1.0/bitops-bitwise-and.js:
101
102 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
103
104         Reviewed by Anders Carlsson.
105
106         SunSpider: all four bitops benchmarks can be replaced with NOP
107         https://bugs.webkit.org/show_bug.cgi?id=38446
108
109         * tests/sunspider-1.0/bitops-3bit-bits-in-byte.js:
110         (TimeFunc): Save the result.
111         * tests/sunspider-1.0/bitops-bits-in-byte.js:
112         (TimeFunc): Save the result.
113         * tests/sunspider-1.0/bitops-bitwise-and.js: More explicitly
114         save the results.
115         * tests/sunspider-1.0/bitops-nsieve-bits.js:
116         (sieve): Save the result.
117
118 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
119
120         SunSpider: string-validate-input.js uses an undeclared variable named 'name', which is a DOM API
121         https://bugs.webkit.org/show_bug.cgi?id=60937
122
123         Reviewed by Dan Bernstein.
124
125         * tests/sunspider-1.0/string-validate-input.js:
126         (doTest): Rename the variable to username
127
128 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
129
130         Reviewed by Darin Adler.
131
132         SunSpider: The main function of math-cordic is dead code
133         https://bugs.webkit.org/show_bug.cgi?id=63863
134         
135         Try to prevent math-cordic from being eliminated as dead code or
136         compiled down to a constant.
137
138         * tests/sunspider-1.0/math-cordic.js:
139         (cordicsincos):
140         (cordic):
141
142 2011-06-30  Maciej Stachowiak  <mjs@apple.com>
143
144         Reviewed by Adam Barth.
145
146         Create sunspider-1.0 directory in preparation for fixing a bunch of bugs
147         https://bugs.webkit.org/show_bug.cgi?id=63782
148
149         * make-hosted: Learn about the new directory.
150         * tests/sunspider-1.0: Copied from PerformanceTests/SunSpider/tests/sunspider-0.9.1.
151
152 2010-11-29  Geoffrey Garen  <ggaren@apple.com>
153
154         Reviewed by Gavin Barraclough.
155
156         Improved accuracy of command-line SunSpider.
157
158         * resources/sunspider-standalone-driver.js: Use the returned number of
159         elapsed milliseconds from "run" and "checkSyntax" instead of measuring
160         ourselves, for slightly more accurate numbers.
161
162 2010-09-16  Gavin Barraclough  <barraclough@apple.com>
163
164         Reviewed by Mark Rowe.
165         https://bugs.webkit.org/show_bug.cgi?id=45924
166
167         Modify the SunSpider harness to allow a path to be passed to the --suite
168         argument, supress the normal prepend-"tests/" behaviour for suite names
169         containing a '/'.
170
171         Also wrap the code in an anonymous function to move variables out of global
172         scope (currently the harness cannot handle running tests the overwrite the
173         variable 'j').
174
175         * resources/sunspider-standalone-driver.js:
176         * sunspider:
177
178 2010-08-27  Michael Saboff  <msaboff@apple.com>
179
180         Reviewed by Stephanie Lewis.
181
182         Added version 5 and version 6 of V8 tests with modifications similar
183         to the version 4 tests so they'll work with the SunSpider test harness.
184         Note that these files are external source and therefore don't conform 
185         to WebKit coding standards.
186         https://bugs.webkit.org/show_bug.cgi?id=44638
187
188         * tests/v8-v5: Added.
189         * tests/v8-v5/LIST: Added.
190         * tests/v8-v5/v8-crypto.js: Added.
191         * tests/v8-v5/v8-deltablue.js: Added.
192         * tests/v8-v5/v8-earley-boyer.js: Added.
193         * tests/v8-v5/v8-raytrace.js: Added.
194         * tests/v8-v5/v8-regexp.js: Added.
195         * tests/v8-v5/v8-richards.js: Added.
196         * tests/v8-v5/v8-splay.js: Added.
197         * tests/v8-v6: Added.
198         * tests/v8-v6/LIST: Added.
199         * tests/v8-v6/v8-crypto.js: Added.
200         * tests/v8-v6/v8-deltablue.js: Added.
201         * tests/v8-v6/v8-earley-boyer.js: Added.
202         * tests/v8-v6/v8-raytrace.js: Added.
203         * tests/v8-v6/v8-regexp.js: Added.
204         * tests/v8-v6/v8-richards.js: Added.
205         * tests/v8-v6/v8-splay.js: Added.
206
207 2010-07-15  Stephanie Lewis  <slewis@apple.com>
208
209         Reviewed by Geoff Garen.
210
211         http://bugs.webkit.org/show_bug.cgi?id=42406
212         Output location of results file.
213
214         * sunspider:
215
216 2010-05-17  Maciej Stachowiak  <mjs@apple.com>
217
218         Reviewed by Mark Rowe.
219
220         Browser-hosted version of SunSpider runs tests twice as many times as advertised
221         https://bugs.webkit.org/show_bug.cgi?id=39201
222
223         * make-hosted: Make sure to reset test list after every test suite.
224
225 2010-05-07  Adam Roben  <aroben@apple.com>
226
227         Fix typo in sunspider-compare-results.js that caused incorrect
228         two-sample t statistics to be computed
229
230         Fixes <http://webkit.org/b/38768> SunSpider computes incorrect
231         two-sample t statistics
232
233         Reviewed by Darin Adler.
234
235         * resources/sunspider-compare-results.js:
236         (sunspiderCompareResults.resultLine): Fixed the calculation of the t
237         variable to match the definition of a two-sample t statistic. Looks
238         like this was just a simple typo.
239
240 2010-04-05  Darin Adler  <darin@apple.com>
241
242         Updated the ignore property for changes to the contents here.
243
244         * .: Modified property svn:ignore.
245
246 2010-02-20  Maciej Stachowiak  <mjs@apple.com>
247
248         Reviewed by Adam Barth.
249
250         Add sunspider-0.9.1 to the site in preparation for announcing it.
251         https://bugs.webkit.org/show_bug.cgi?id=35206
252
253         * hosted/sunspider.css: Remove tabs, which I noticed where present while
254         copying to the WebKitSite directory.
255
256 2010-01-11  Oliver Hunt  <oliver@apple.com>
257
258         Reviewed by Darin Adler.
259
260         Sunspider parse-only tests don't run
261         https://bugs.webkit.org/show_bug.cgi?id=33489
262
263         Fix parse-only test list
264
265         * tests/parse-only/LIST:
266
267 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
268
269         Reviewed by Adele Peterson.
270
271         Make SunSpider version more prominent in the title
272         https://bugs.webkit.org/show_bug.cgi?id=32574
273
274         * hosted/sunspider.html:
275         * hosted/versions.html:
276         * resources/driver-TEMPLATE.html:
277         * resources/results-TEMPLATE.html:
278
279 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
280
281         Reviewed by Darin Adler.
282
283         Make SunSpider refuse to compare results across content versions
284         https://bugs.webkit.org/show_bug.cgi?id=32573
285         
286         The results URL now includes a version indicator, and compare mode
287         will refuse to compare between versions.
288
289         * resources/driver-TEMPLATE.html:
290         * resources/results-TEMPLATE.html:
291
292 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
293
294         Reviewed by Darin Adler.
295
296         Some Browser-hosted SunSpider files are not valid HTML5
297         https://bugs.webkit.org/show_bug.cgi?id=32536
298         
299         Made various fixes to validate as HTML5 with no warnings (mainly removing bogus
300         close tags and adding a meta charset declaration).
301
302         * hosted/sunspider.html:
303         * hosted/versions.html:
304         * resources/TEMPLATE.html:
305         * resources/driver-TEMPLATE.html:
306         * resources/results-TEMPLATE.html:
307
308 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
309
310         Reviewed by Eric Seidel.
311
312         Make sunspider-0.9.1 the default content set (both command-line and hosted)
313         https://bugs.webkit.org/show_bug.cgi?id=32537
314
315         * hosted/sunspider.html: Change default suite link.
316         * sunspider: Change default suite.
317         * sunspider-compare-results: Likewise.
318
319 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
320
321         Reviewed by Sam Weinig.
322
323         in-browser SunSpider suffers excessive penalty under power management
324         https://bugs.webkit.org/show_bug.cgi?id=32505
325         
326         I have made a few changes to address this:
327         
328         1) Change how browser-hosted SunSpider loads tests - preload the content and write it in with
329         document.write to reduce triggering of progress UI.
330         2) Reduce time between tests to 10ms from 500ms, so that power management doesn't keep the CPU
331         stepped all the way down the whole time.
332         3) Run the test cycle an extra time for warmup before the runs that count.
333
334         * hosted/sunspider-record-result.js: Removed.
335         * make-hosted:
336         * resources/TEMPLATE.html:
337         * resources/driver-TEMPLATE.html:
338
339 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
340
341         Reviewed by Gavin Barraclough.
342
343         SunSpider/tests/string-base64.js does not compute a valid base64 encoded string
344         https://bugs.webkit.org/show_bug.cgi?id=16806
345
346         Based on a patch by Eric Seidel.
347         
348         Fix the base64 computation to actually compute correct results. The impact on runtime of
349         the test is pretty small, but noticeable for some browsers. But at least it's not
350         doing a wrong and meaningless computation any more.
351         
352         * tests/sunspider-0.9.1/string-base64.js:
353         ():
354         (base64ToString):
355
356 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
357
358         Fixing commit error...
359         
360         I accidentally committed my last patch in a form that broke Web-hosted SunSpider. Fixing.
361
362         * resources/driver-TEMPLATE.html:
363
364 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
365
366         Reviewed by Sam Weinig.
367
368         Use JSON.parse instead of eval for Web-hosted SunSpider results processing
369         https://bugs.webkit.org/show_bug.cgi?id=32490
370
371         * hosted/json2.js: Added. Incorporated from json.org
372         * resources/driver-TEMPLATE.html: Fix an HTML validation bug I noticed.
373         * resources/results-TEMPLATE.html: Use JSON.parse instead of eval to process
374         results.
375
376 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
377
378         Reviewed by Oliver Hunt.
379
380         Enable Web-hosted version of SunSpider to handle multiple versions
381         https://bugs.webkit.org/show_bug.cgi?id=32478
382
383         * make-hosted: Now generate the hosted version to be able to run both the
384         0.9 and the 0.9.1 test suites.
385         * hosted: Modified property svn:ignore.
386         * hosted/sunspider-driver.html: Removed.
387         * hosted/sunspider-results.html: Removed.
388         * hosted/sunspider.html:
389         * hosted/versions.html: Added.
390         * resources/TEMPLATE.html:
391         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-driver.html.
392         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-results.html.
393
394 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
395
396         Reviewed by Oliver Hunt.
397
398         Give command-line sunspider the ability to handle multiple suites and versions
399         https://bugs.webkit.org/show_bug.cgi?id=32477
400         
401         Each suite/version's set of tests are now in a separate subdirectory of tests/, with a file named LIST
402         enumerating the tests. Current suites are:
403         
404         sunspider-0.9 - riginal SunSpider content 
405         sunspider-0.9.1 - SunSpider suite with a few tests fixed (and a few more to come)
406         ubench - "ubench" microbenchmark suite
407         v8-v4 - version 4 of the v8 benchmark
408         parse-only - some parser tests
409         
410         Sunspider can now be run with the --suite= parameter to select a
411         suite. The default is --suite=sunspider-0.9.
412
413         * .: Modified property svn:ignore to ignore new results directories.
414         * make-hosted: Generate based on sunspider-0.9 suite.
415         * resources/TEMPLATE.html: Fix an HTML compliance problem.
416         * resources/sunspider-standalone-driver.js: Become aware of suite names.
417         * sunspider: Modified to handle multiple suites as described above.
418         * sunspider-compare-results: ditto
419         * tests/LIST: Moved to tests/sunspider-0.9/ and tests/sunspider-0.9.1/
420         * tests/LIST-PARSE-ONLY: Moved to parse-only/LIST
421         * tests/LIST-UBENCH: Removed.
422         * tests/LIST-V8: Removed.
423         * tests/3d-cube.js: Moved to tests/sunspider-0.9 and tests/sunspider-0.9.1
424         * tests/3d-morph.js: ditto
425         * tests/3d-raytrace.js: ditto
426         * tests/access-binary-trees.js: ditto
427         * tests/access-fannkuch.js: ditto
428         * tests/access-nbody.js: ditto
429         * tests/access-nsieve.js: ditto
430         * tests/bitops-3bit-bits-in-byte.js: ditto
431         * tests/bitops-bits-in-byte.js: ditto
432         * tests/bitops-bitwise-and.js: ditto
433         * tests/bitops-nsieve-bits.js: ditto
434         * tests/controlflow-recursive.js: ditto
435         * tests/crypto-aes.js: ditto
436         * tests/crypto-md5.js: ditto
437         * tests/crypto-sha1.js: ditto
438         * tests/date-format-tofte.js: ditto
439         * tests/date-format-xparb.js: ditto
440         * tests/math-cordic.js: ditto
441         * tests/math-partial-sums.js: ditto
442         * tests/math-spectral-norm.js: ditto
443         * tests/regexp-dna.js: ditto
444         * tests/string-base64.js: ditto
445         * tests/string-fasta.js: ditto
446         * tests/string-tagcloud.js: ditto
447         * tests/string-unpack-code.js: ditto
448         * tests/string-validate-input.js: ditto
449         * tests/sunspider-0.9/: Added
450         * tests/sunspider-0.9/*.js: Moved from one level up
451         * tests/sunspider-0.9.1: Added.
452         * tests/sunspider-0.9.1/*.js: Moved from one level up.
453         * tests/v8-crypto.js: Moved to v8-v4/ directory.
454         * tests/v8-deltablue.js: ditto
455         * tests/v8-earley-boyer.js: ditto
456         * tests/v8-raytrace.js: ditto
457         * tests/v8-regexp.js: ditto
458         * tests/v8-richards.js: ditto
459         * tests/v8-splay.js: ditto
460         * tests/v8-v4: Added.
461
462 2009-09-30  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
463
464         Reviewed by Darin Adler.
465
466         Platform specific null device replaced with a platform independent.
467         https://bugs.webkit.org/show_bug.cgi?id=29544
468
469         * sunspider: Using File::Spec->devnull() instead of /dev/null and NUL.
470
471 2009-09-21  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
472
473         Reviewed by Maciej Stachowiak.
474
475         [Qt] Make sunspider script work on Windows platform.
476         https://bugs.webkit.org/show_bug.cgi?id=29544
477
478         * sunspider:
479         Using platform specific null device instead of /dev/null.
480
481 2009-06-19  Adam Treat  <adam.treat@torchmobile.com>
482
483         Reviewed by Oliver Hunt.
484
485         https://bugs.webkit.org/show_bug.cgi?id=26540
486         Currently the SunSpider test driver lacks an option to run a test suite that
487         will test JavaScriptCore parsing performance only.  This patch adds just such
488         a test suite and option to SunSpider as well as the jsc test shell.  I've included
489         three large javascript source files found in the wild: jquery, mootools and prototype.
490         Combined with the concatenation of all three, these form a new testsuite to measure
491         and test pure JavaScriptCore parsing performance.
492
493         * resources/sunspider-standalone-driver.js:
494           Load the js with parse-only flag if it is part of the parse-only suite.
495         * sunspider:
496           Add the parse-only suite to the list of options.
497         * sunspider-compare-results:
498           Add the parse-only suite to the list of options.
499         * tests/LIST-PARSE-ONLY: Added.
500         * tests/parse-only/concat-jquery-mootools-prototype.js: Added.
501           A very large concatenation of the scripts found below.  This js takes ~30ms to parse on my machine.
502         * tests/parse-only/jquery-1.3.2.js: Added.
503           Latest version from jquery.com.
504         * tests/parse-only/mootools-1.2.2-core-nc.js: Added.
505           Latest version from mootools.net.
506         * tests/parse-only/prototype-1.6.0.3.js: Added.
507           Latest version from prototypejs.org.
508
509 2009-06-13  Steve Falkenburg  <sfalken@apple.com>
510
511         Update V8 benchmark to version 4.
512         
513         Reviewed by Cameron Zwarich.
514
515         * tests/LIST-V8:
516         * tests/v8-crypto.js:
517         * tests/v8-deltablue.js:
518         * tests/v8-earley-boyer.js:
519         * tests/v8-raytrace.js:
520         * tests/v8-regexp.js: Added.
521         * tests/v8-richards.js:
522         * tests/v8-splay.js: Added.
523
524 2009-05-26  Olivier DOLE  <odole@pleyo.com>
525
526         Reviewed by Maciej.
527
528         Store sunspider test suite results in a specific directory for each
529         different type of benchmark (sunspider, V8 or ubench).
530
531         * sunspider:
532         * sunspider-compare-results:
533
534 2009-05-12  Steve Falkenburg  <sfalken@apple.com>
535
536         Build fix.
537
538         * SunSpider.make:
539
540 2009-05-11  Steve Falkenburg  <sfalken@apple.com>
541
542         Include SunSpider tests in production builds.
543         
544         Reviewed by Adam Roben.
545
546         * SunSpider.make: Added.
547         * xcopy.excludes: Added.
548
549 2009-03-04  Oliver Hunt  <oliver@apple.com>
550
551         Added a few more tests to ubench
552
553         RS=Gavin Barraclough
554
555         Adding a couple of test to cover too few and too many arguments.  This
556         is needed to prevent regressions in all call paths from my current refactoring.
557
558         * tests/LIST-UBENCH:
559         * tests/ubench/function-correct-args.js: Added.
560         (f):
561         * tests/ubench/function-excess-args.js: Added.
562         (f):
563
564 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
565
566         Fix a bug where when V8's or UBENCH's test suite list file is missing
567         the error message is suggesting that SunSpider's list file is missing.
568
569         Reviewed by Sam Weinig.
570
571         * sunspider:
572         (loadTestsList): Fix the error message to use the right list name.
573
574 2008-10-07  David Hyatt  <hyatt@apple.com>
575
576         https://bugs.webkit.org/show_bug.cgi?id=18989
577
578         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
579         we want all browsers to be doing the same thing on the test.
580
581         Reviewed by Maciej
582
583         * tests/regexp-dna.js:
584
585 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
586
587         Rubber-stamped by Maciej Stachowiak.
588
589         The bleeding edge branch of V8 now supports passing fileanmes after
590         '-f', so the SunSpider script should be changed to do this again.
591         This allows all 3 major JavaScript engines to be used with the
592         unmodified SunSpider script.
593
594         * sunspider:
595
596 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
597
598         Reviewed by Maciej Stachowiak.
599
600         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
601         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
602         correct the punctuation in its documentation.
603
604         * sunspider:
605
606 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
607
608         Reviewed by Oliver Hunt.
609
610         Bug 20616: Incorporate V8 benchmarks in testing
611         <https://bugs.webkit.org/show_bug.cgi?id=20616>
612
613         This is a first attempt at running the V8 benchmarks within SunSpider,
614         with the --v8 option. The number of iterations is fixed for each
615         benchmark, and was chosen to make each of the benchmarks run between
616         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
617
618         * sunspider:
619         * tests/LIST-V8: Added.
620         * tests/v8-crypto.js: Added.
621         * tests/v8-deltablue.js: Added.
622         * tests/v8-earley-boyer.js: Added.
623         * tests/v8-raytrace.js: Added.
624         * tests/v8-richards.js: Added.
625
626 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
627
628         Not reviewed.
629
630         Remove a local change that was accidentally committed in r36034.
631
632         * resources/sunspider-standalone-driver.js:
633
634 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
635
636         Rubber-stamped by Oliver Hunt.
637
638         Make SunSpider work with JavaScript shells that don't take filenames
639         after a '-f' argument.
640
641         * sunspider:
642
643 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
644
645         Reviewed by Timothy.
646
647         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
648
649         * sunspider:
650         * tests/LIST-SQUIRRELFISH: Removed.
651
652 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
653
654         Rubber-stamped by Mark Rowe.
655
656         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
657
658         * sunspider:
659
660 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
661
662         Rubber stamped by Oliver.
663         
664         - add newly working tests to squirrelfish list
665
666         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
667
668 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
669
670         Rubber stamped by Oliver.
671         
672         - move ubench tests properly.
673
674         * tests/ubench: Added.
675         * tests/ubench/function-closure.js: Added.
676         * tests/ubench/function-empty.js: Added.
677         * tests/ubench/function-missing-args.js: Added.
678         * tests/ubench/function-sum.js: Added.
679         * tests/ubench/loop-empty-resolve.js: Added.
680         * tests/ubench/loop-empty.js: Added.
681         * tests/ubench/loop-sum.js: Added.
682
683 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
684
685         Rubber stamped by Oliver.
686
687         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
688
689 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
690
691         Reviewed by Oliver.
692
693         - make --squirrelfish mode use the real harness, but a reduced set of tests
694
695         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
696
697         * sunspider:
698         * tests/LIST-SQUIRRELFISH:
699         * tests/LIST-UBENCH: Added.
700         * tests/squirrelfish: Moved to tests/ubench
701
702 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
703
704         Reviewed by Maciej Stachowiak.
705         
706         Removed an unused test.
707
708         * tests/squirrelfish/loop-resolve.js: Removed.
709
710 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
711
712         Reviewed by Oliver.
713         
714         Add newly runnable tests to --squirrelfish mode.
715
716         * tests/LIST-SQUIRRELFISH:
717
718 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
719
720         Reviewed by Maciej Stachowiak.
721         
722         A little more love for --squirrelfish mode:
723         
724         Fixed a misplaced sort. Results properly sort now.
725         
726         Added a test list just for squirrelfish, and updated it to include
727         all currently passing SunSpider tests.
728         
729         Fixed the pruning regexp to match 3d-morph.js.
730
731         * sunspider:
732
733 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
734
735         Reviewed by Oliver Hunt.
736         
737         --squirrelfish mode: pared down tests for squirrelfish to chew on.
738
739         * sunspider:
740         * tests/LIST:
741         * tests/squirrelfish: Added.
742         * tests/squirrelfish/function-closure.js: Added.
743         * tests/squirrelfish/function-empty.js: Added.
744         * tests/squirrelfish/function-missing-args.js: Added.
745         * tests/squirrelfish/function-sum.js: Added.
746         * tests/squirrelfish/loop-empty-resolve.js: Added.
747         * tests/squirrelfish/loop-empty.js: Added.
748         * tests/squirrelfish/loop-resolve.js: Added.
749         * tests/squirrelfish/loop-sum.js: Added.
750
751 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
752
753         Reviewed by Geoff.
754
755         * sunspider: Pause update daemon on Darwin to make results reliable
756         again (workaround for <rdar://problem/5811127>).
757
758 2008-02-07  Eric Seidel  <eric@webkit.org>
759
760         Reviewed by Sam.
761
762         * sunspider: add an error message when ./tests/LIST is not found
763
764 2007-12-29  Alp Toker  <alp@atoker.com>
765
766         Fix typo.
767
768         * hosted/sunspider.html:
769
770 2007-12-29  Alp Toker  <alp@atoker.com>
771
772         Reviewed by Maciej.
773
774         Provide the current URL in the SunSpider results page. Useful for
775         copying benchmark results out of browsers without a location bar or
776         where the location bar can't handle long URLs.
777
778         * hosted/sunspider-results.html:
779
780 2007-12-20  Eric Seidel  <eric@webkit.org>
781
782         Reviewed by mjs.
783
784         * sunspider: increase default number of runs for more accurate numbers
785
786 2007-12-18  Mark Rowe  <mrowe@apple.com>
787
788         Fix typo.
789
790         * hosted/sunspider.html:
791
792 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
793
794         Reviewed by Eric.
795         
796         - give an extra digit of precision for small differences.
797
798         * resources/sunspider-compare-results.js:
799
800 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
801
802         Reviewed by Eric.
803         
804         - work around IE layout bugs
805
806         * hosted/sunspider.css:
807
808 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
809
810         Reviewed by Mark.
811         
812         - fix compare feature
813
814         * hosted/sunspider-results.html:
815
816 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
817
818         Reviewed by Mark.
819         
820         - Change the format of the output URL to be shorter, to meet IE constraints
821
822         * hosted/sunspider-driver.html:
823         * hosted/sunspider-results.html:
824
825 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
826
827         Reviewed by Oliver.
828         
829         - Add some explanatory test and a dumb logo to SunSpider web hosted version
830
831         * hosted/sunspider-driver.html:
832         * hosted/sunspider-results.html:
833         * hosted/sunspider.css:
834         * hosted/sunspider.html:
835
836 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
837
838         Reviewed by Oliver.
839         
840         - Make the web-hosted version of SunSpider look a little bit less drab.
841
842         * hosted/sunspider-driver.html:
843         * hosted/sunspider-results.html:
844         * hosted/sunspider.css: Added.
845         * hosted/sunspider.html:
846         * resources/TEMPLATE.html:
847
848 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
849
850         Reviewed by Sam.
851         
852         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
853         
854         A sample of what new-style results would look like:
855         
856         TEST                   COMPARISON            FROM                 TO             DETAILS
857         ...
858         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
859         ...
860             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
861         
862         * resources/sunspider-compare-results.js:
863
864 2007-11-14  Eric Seidel  <eric@webkit.org>
865
866         Reviewed by Sam.
867
868         * sunspider: add --shark-cache for L2 Cache Miss profiling
869
870 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
871
872         Reviewed by Darin Adler.
873
874         http://bugs.webkit.org/show_bug.cgi?id=15925
875         SunSpider should check for Shark being installed
876
877         * sunspider: Explicitly test that shark command line tool is installed.
878
879 2007-11-04  Darin Adler  <darin@apple.com>
880
881         * hosted/sunspider.html: Tweaked formatting a bit.
882
883 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
884
885         Reviewed by Darin Adler.
886
887         - Add compare mode and a description of the benchmark to the hosted version.
888
889         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
890         * hosted/sunspider.html: Added intro text.
891         * make-hosted: Copy new file.
892         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
893         a function.
894         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
895         expects.
896         * sunspider-compare-results: Use sunspider-standalone-compare.js.
897
898 2007-11-02  Darin Adler  <darin@apple.com>
899
900         * sunspider: Changed "--base" to "--set-baseline".
901
902 2007-11-02  Darin Adler  <darin@apple.com>
903
904         Reviewed by Maciej.
905
906         * sunspider: Added a "--base" option that records the current test as a baseline.
907         Moved the code to find the newest .mshark file into a function (since I needed to use
908         it in the other script and I prefer to copy/paste a function).
909
910         * sunspider-compare-results: Added logic to compare the baseline with the last
911         run when you don't pass any parameters.
912
913 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
914
915         Reviewed by Eric.
916         
917         - Added list of ECMAScript3 features that SunSpider does not cover currently.
918
919         * UNCOVERED: Added.
920
921 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
922
923         Reviewed by Oliver.
924         
925         - Give better error messages when sunspider-compare-results fails
926
927         * sunspider-compare-results:
928
929 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
930
931         Reviewed by Darin Adler.
932         
933         - fix Date tests to work in command-line SpiderMonkey
934
935         * tests/date-format-tofte.js:
936         * tests/date-format-xparb.js:
937
938 2007-10-29  Darin Adler  <darin@apple.com>
939
940         Reviewed by Maciej.
941
942         * sunspider: Added --shark20 option, to run Shark at its highest sample
943         resolution instead of its default.
944
945 2007-10-28  Adam Roben  <aroben@apple.com>
946
947         Fix SunSpider on Windows
948
949         Reviewed by Eric.
950
951         * sunspider: Don't use colons in filenames.
952
953 2007-10-28  Darin Adler  <darin@apple.com>
954
955         Reviewed by Adam.
956
957         * resources/sunspider-analyze-results.js: Tweak the output format a little.
958         Change so that when there's only one run we don't write out confidence
959         intervals at all rather than writing "NaN" over and over again.
960
961         * sunspider: Use the time and date as a suffix on the results file, that way
962         all the results are left behind in the tmp directory. This will make it easy
963         for us to add features that compare past results in the future. Also rename
964         the .mshark files using the same scheme.
965
966         * sunspider-compare-results: Relax the parsing rules so we can parse the
967         results file exactly as it's emitted from sunspider.
968
969 2007-10-25  Adam Roben  <aroben@apple.com>
970
971         Strip carriage returns from results
972
973         Carriage returns were screwing up sunspider-compare-results on
974         Windows.
975
976         Rubberstamped by Oliver.
977
978         * sunspider:
979
980 2007-10-24  Darin Adler  <darin@apple.com>
981
982         Reviewed by Eric.
983
984         * sunspider: Open the profile in shark after completing a run.
985         * sunspider-compare-results: Tiny perl tweaks.
986
987 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
988
989         Reviewed by Darin Adler.
990
991         - Don't hardcode my path to testkjs
992         
993         * sunspider-compare-results:
994
995 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
996
997         Reviewed by Darin Adler.
998         
999         - Add a compare script to compare results, and improve formatting of standard results a bit.
1000
1001         * resources/sunspider-analyze-results.js:
1002         * resources/sunspider-compare-results.js: Added.
1003         * sunspider-compare-results: Added.
1004
1005 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
1006
1007         Reviewed by Darin Adler.
1008
1009         * resources/sunspider-analyze-results.js: Do the error range
1010         properly, using the t-distribution instead of 1.96 (which was
1011         based on the normal distribution).
1012         
1013         * sunspider: Print results in a way that will be friendlier to the
1014         soon-to-come compare mode.
1015
1016 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
1017
1018         Reviewed by Review.
1019         
1020         - add more tests (probably the final set for now)
1021         
1022         Already balanced. Added date, regexp, control flow, and a few
1023         more string and object/array access tests.
1024         
1025         * tests/LIST:
1026         * tests/access-fannkuch.js: Added.
1027         * tests/access-nbody.js: Added.
1028         * tests/controlflow-recursive.js: Added.
1029         * tests/date-format-tofte.js: Added.
1030         * tests/date-format-xparb.js: Added.
1031         * tests/regexp-dna.js: Added.
1032         * tests/string-validate-input.js: Added.
1033
1034 2007-10-21  Eric Seidel  <eric@webkit.org>
1035
1036         Reviewed by mjs.
1037
1038         * sunspider: make --shark only sample from testkjs
1039
1040 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
1041
1042         Rubber stamped by Eric.
1043         
1044         - added crypto tests (already balanced)
1045
1046         * tests/LIST:
1047         * tests/crypto-aes.js: Added.
1048         * tests/crypto-md5.js: Added.
1049         * tests/crypto-sha1.js: Added.
1050
1051 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
1052
1053         Reviewed by Adam.
1054         
1055         - rebalance new tests
1056         
1057         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
1058         
1059         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
1060
1061         This balances them with the existing tests.
1062
1063         * tests/access-binary-trees.js:
1064         * tests/access-nsieve.js:
1065         * tests/bitops-nsieve-bits.js:
1066         * tests/math-partial-sums.js:
1067         * tests/math-spectral-norm.js:
1068         * tests/string-fasta.js:
1069
1070 2007-10-20  Eric Seidel  <eric@webkit.org>
1071
1072         Reviewed by darin.
1073         
1074         Add improved argument handling to sunspider, including
1075         --runs=<number>, --tests=<pattern, --shark, and --help
1076         Also re-factor code into subroutines
1077
1078         * sunspider: improved argument handling
1079
1080 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1081
1082         Reviewed by Mark.
1083
1084         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
1085         
1086         * TODO:
1087         * tests/LIST:
1088         * tests/access-binary-trees.js: Added.
1089         * tests/access-nsieve.js: Added.
1090         * tests/bitops-nsieve-bits.js: Added.
1091         * tests/math-partial-sums.js: Added.
1092         * tests/math-spectral-norm.js: Added.
1093         * tests/string-fasta.js: Added.
1094
1095 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1096
1097         Reviewed by Darin Adler.
1098
1099         * TODO: Updated to note areas that are now well-covered.
1100         
1101         - Change the tests and the driver to avoid leaving large object
1102         graphs hanging around, since that throws off the subsequent tests.
1103         
1104         * resources/sunspider-standalone-driver.js:
1105         * tests/3d-cube.js:
1106         * tests/3d-morph.js:
1107         * tests/3d-raytrace.js:
1108         * tests/bitops-bitwise-and.js:
1109         * tests/string-base64.js:
1110         * tests/string-tagcloud.js:
1111
1112 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1113
1114         Reviewed by Nikolas.
1115         
1116         - Rebalance test complexity.
1117         
1118         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
1119         
1120         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
1121
1122         However, for some tests, IE7 was a huge outlier compared to other
1123         major browsers. For those tests, I used Firefox 2.0.0.8 on the
1124         same OS and hardware as a normalization basis.
1125         
1126         * tests/3d-morph.js:
1127         * tests/3d-raytrace.js:
1128         * tests/bitops-3bit-bits-in-byte.js:
1129         * tests/bitops-bits-in-byte.js:
1130         * tests/bitops-bitwise-and.js:
1131         * tests/math-cordic.js:
1132         * tests/string-base64.js:
1133         * tests/string-tagcloud.js:
1134         * tests/string-unpack-code.js:
1135
1136 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1137
1138         Reviewed by Oliver.
1139         
1140         - Fix the browser-hosted driver to be compatible with Safari 2
1141
1142         * hosted/sunspider-driver.html:
1143
1144 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1145
1146         Rubber stamped by Hyatt.
1147         
1148         - Add BSD license notice to cordic test, the original author said to do so.
1149
1150         * tests/math-cordic.js:
1151
1152 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
1153
1154         Reviewed by Eric.
1155         
1156         - Support Windows and Windows browsers.
1157
1158         * hosted/sunspider-results.html: Display results correctly in IE.
1159         * sunspider: tweaks to work on Windows, and with spaces in the path.
1160
1161 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
1162
1163         Rubber stamped by Oliver.
1164
1165         * tests/string-tagcloud.js: Remove stray print()
1166
1167 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
1168
1169         Reviewed by Oliver.
1170         
1171         Add some string test cases.
1172
1173         * tests/string-base64.js: Added. Does base64 encoding and decoding.
1174         * tests/string-tagcloud.js: Added. Parses a JSON string and
1175         generates tagcloud HTML markup based on the data.        
1176         * tests/string-unpack-code.js: Added. Unpack compressed versions of
1177         four of the most popular JavaScript libraries.
1178         * tests/LIST: Added the new tests.
1179
1180 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
1181
1182         Reviewed by Anders.
1183
1184         * TODO: Updated with info on how to do the stats properly.
1185
1186         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
1187         avoid overly slowing down the benchmark.
1188
1189         Added some 3D-related test cases.
1190         
1191         * tests/3d-cube.js: Added.
1192         * tests/3d-morph.js: Added.
1193         * tests/3d-raytrace.js: Added.
1194         * tests/LIST:
1195
1196 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
1197
1198         Reviewed by Darin Adler.
1199
1200         - New JavaScript benchmark
1201         http://bugs.webkit.org/show_bug.cgi?id=15515
1202         
1203         A start on a new JavaScript benchmark, with standalone and
1204         browser-hosted drivers.
1205         
1206         The test content is still incomplete and the drivers could use
1207         some more features but there is enough here to make a good start.
1208         
1209         The drivers compute a 95% confidence interval on the mean for the
1210         whole test, each category, and each individual test to make it
1211         easier to tell whether differences are statistically
1212         significant. The confidence interval can be narrowed by running
1213         the test more times.
1214         
1215         * TODO: Added. Remaining things that need to be done.
1216         * sunspider: Added. Perl script that acts as the standalone test driver.
1217         * resources/sunspider-standalone-driver.js: Added. JavaScript part
1218         of standalone test driver.
1219         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
1220         analysis code.
1221         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
1222         * make-hosted: Added. Script to generate browser-hosted tests.
1223         * hosted/sunspider.html: Added. Start page for browser-hosted test.
1224         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
1225         * hosted/sunspider-results.html: Added. Results page for browser-hosted
1226         tests.
1227         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
1228         tests.
1229         * tests/LIST: Added. List of tests to use.
1230         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
1231         * tests/bitops-bits-in-byte.js: Added. Ditto.
1232         * tests/bitops-bitwise-and.js: Added. Ditto.
1233         * tests/math-cordic.js: Added. Ditto.
1234