Whenever it is cheap and non-invasive, SunSpider tests should validate their results...
[WebKit-https.git] / PerformanceTests / SunSpider / ChangeLog
1 2013-04-18  Filip Pizlo  <fpizlo@apple.com>
2
3         Whenever it is cheap and non-invasive, SunSpider tests should validate their results to ensure that the browser runs them correctly
4         https://bugs.webkit.org/show_bug.cgi?id=114852
5
6         Reviewed by Geoffrey Garen.
7         
8         This adds low-overhead checking of the results of each SunSpider tests. This is not
9         meant to be an exhaustive test that everything was executed correctly, but rather a
10         simple sanity check that will catch glaring mistakes. The philosophy here is that we're
11         not trying to prevent all forms of cheating, or that we're trying to prove the
12         browser's correctness. Moreover, these checks are meant to incur minimal overhead;
13         they currently clock in at <2% slow-down in SunSpider 1.0.
14         
15         The test expectations were generated using the LLInt on command-line, and I've verified
16         that Firefox 20, Chrome 26, IE 10, Safari 6, and ToT with all of the JITs agree. Note
17         that some tests cannot be precisely validated because they use Math functions, which
18         ECMAScript chooses to not formally specify - those functions like sin() and friends are
19         allowed to return implementation-dependent results. Also some tests cannot be validated
20         at all because their behavior is either intentionally random or is timezone-dependent.
21         But 23 out of 26 tests now have some kind of validation.
22         
23         I've updated the harnesses to show erroneous runs when displaying results.
24
25         * resources/TEMPLATE.html:
26         * resources/driver-TEMPLATE.html:
27         * resources/sunspider-analyze-results.js:
28         (formatResult):
29         (resultLine):
30         (printOutput):
31         * resources/sunspider-compare-results.js:
32         (.formatMean):
33         (.resultLine):
34         (.printOutput):
35         (sunspiderCompareResults):
36         * resources/sunspider-standalone-driver.js:
37         * tests/sunspider-1.0/3d-cube.js:
38         (Init):
39         * tests/sunspider-1.0/3d-morph.js:
40         * tests/sunspider-1.0/3d-raytrace.js:
41         * tests/sunspider-1.0/access-binary-trees.js:
42         * tests/sunspider-1.0/access-fannkuch.js:
43         * tests/sunspider-1.0/access-nbody.js:
44         * tests/sunspider-1.0/access-nsieve.js:
45         (sieve):
46         * tests/sunspider-1.0/bitops-3bit-bits-in-byte.js:
47         * tests/sunspider-1.0/bitops-bits-in-byte.js:
48         * tests/sunspider-1.0/bitops-bitwise-and.js:
49         * tests/sunspider-1.0/bitops-nsieve-bits.js:
50         * tests/sunspider-1.0/controlflow-recursive.js:
51         * tests/sunspider-1.0/crypto-aes.js:
52         * tests/sunspider-1.0/crypto-md5.js:
53         * tests/sunspider-1.0/crypto-sha1.js:
54         * tests/sunspider-1.0/date-format-tofte.js:
55         * tests/sunspider-1.0/date-format-xparb.js:
56         * tests/sunspider-1.0/math-cordic.js:
57         * tests/sunspider-1.0/math-partial-sums.js:
58         (partial):
59         * tests/sunspider-1.0/math-spectral-norm.js:
60         * tests/sunspider-1.0/regexp-dna.js:
61         * tests/sunspider-1.0/string-base64.js:
62         * tests/sunspider-1.0/string-fasta.js:
63         (fastaRepeat):
64         (fastaRandom):
65         * tests/sunspider-1.0/string-tagcloud.js:
66         * tests/sunspider-1.0/string-unpack-code.js:
67         * tests/sunspider-1.0/string-validate-input.js:
68
69 2013-04-11  Filip Pizlo  <fpizlo@apple.com>
70
71         Get rid of the 10ms delay between test executions
72         https://bugs.webkit.org/show_bug.cgi?id=114458
73         <rdar://problem/13631112>
74
75         Reviewed by Darin Adler and Geoffrey Garen.
76         
77         More details on this in the bug. Long story short, I don't know why the 10ms delay is
78         there, and it appears to be the root of some noisyness and unreliability in the results.
79         Removing it makes results more consistent, which is a good thing.
80
81         * resources/driver-TEMPLATE.html:
82
83 2013-01-16  Eric Seidel  <eric@webkit.org>
84
85         Remove --shark* support from sunspider/run-sunspider now that Shark is dead
86         https://bugs.webkit.org/show_bug.cgi?id=99512
87
88         Reviewed by Darin Adler.
89
90         I tried to use --instruments, but that also seems broken (in the same way before/after this patch).
91
92         * sunspider:
93         (runTestsOnce):
94
95 2012-08-02  Filip Pizlo  <fpizlo@apple.com>
96
97         Unreviewed, revert debugging stuff I landed in http://trac.webkit.org/changeset/124557
98         by accident.
99
100         * tests/v8-v6/v8-crypto.js:
101         (am1):
102         (am2):
103         (am3):
104         (am4):
105         (bnpCopyTo):
106         (bnpFromString):
107         (bnpClamp):
108
109 2012-03-11  Maciej Stachowiak  <mjs@apple.com>
110
111         Add --instruments option to sunspider to profile with Instruments
112         https://bugs.webkit.org/show_bug.cgi?id=80783
113
114         Reviewed by Filip Pizlo.
115
116         * resources/TimeProfile20us.tracetemplate: Added.
117         * sunspider:
118         (runTestsOnce):
119
120 2011-11-15  Andy Wingo  <wingo@igalia.com>
121
122         Extend sunspider driver to be able to run kraken
123         https://bugs.webkit.org/show_bug.cgi?id=71799
124
125         Reviewed by Filip Pizlo.
126
127         * resources/sunspider-standalone-driver.js: Try to load a -data
128         file.  If that succeeds, we have a kraken-like test, so we time
129         the test using `load'.  Otherwise fall back to using `run'.
130
131 2011-08-11  Mark Rowe  <mrowe@apple.com>
132
133         Fix sunspider-compare-results to default to the same suite version as the sunspider script.
134
135         Rubber-stamped by Maciej Stachowiak.
136
137         * sunspider-compare-results:
138
139 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
140
141         Update SunSpider versioning to 1.0
142         https://bugs.webkit.org/show_bug.cgi?id=63870
143
144         Reviewed by Dan Bernstein.
145
146         * hosted/sunspider.html:
147         * hosted/versions.html:
148         * resources/driver-TEMPLATE.html:
149         * resources/results-TEMPLATE.html:
150         * sunspider:
151
152 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
153
154         Sunspider 0.9.1 harness doesn't actually close() its test documents
155         https://bugs.webkit.org/show_bug.cgi?id=47045
156
157         Reviewed by Daniel Bates.
158
159         * hosted/versions.html:
160         * resources/driver-TEMPLATE.html:
161
162 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
163
164         Not Reviewed. 
165         
166         Fix obviou typo in previous commit.
167
168         * tests/sunspider-1.0/bitops-bitwise-and.js:
169
170 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
171
172         Reviewed by Anders Carlsson.
173
174         SunSpider: all four bitops benchmarks can be replaced with NOP
175         https://bugs.webkit.org/show_bug.cgi?id=38446
176
177         * tests/sunspider-1.0/bitops-3bit-bits-in-byte.js:
178         (TimeFunc): Save the result.
179         * tests/sunspider-1.0/bitops-bits-in-byte.js:
180         (TimeFunc): Save the result.
181         * tests/sunspider-1.0/bitops-bitwise-and.js: More explicitly
182         save the results.
183         * tests/sunspider-1.0/bitops-nsieve-bits.js:
184         (sieve): Save the result.
185
186 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
187
188         SunSpider: string-validate-input.js uses an undeclared variable named 'name', which is a DOM API
189         https://bugs.webkit.org/show_bug.cgi?id=60937
190
191         Reviewed by Dan Bernstein.
192
193         * tests/sunspider-1.0/string-validate-input.js:
194         (doTest): Rename the variable to username
195
196 2011-07-02  Maciej Stachowiak  <mjs@apple.com>
197
198         Reviewed by Darin Adler.
199
200         SunSpider: The main function of math-cordic is dead code
201         https://bugs.webkit.org/show_bug.cgi?id=63863
202         
203         Try to prevent math-cordic from being eliminated as dead code or
204         compiled down to a constant.
205
206         * tests/sunspider-1.0/math-cordic.js:
207         (cordicsincos):
208         (cordic):
209
210 2011-06-30  Maciej Stachowiak  <mjs@apple.com>
211
212         Reviewed by Adam Barth.
213
214         Create sunspider-1.0 directory in preparation for fixing a bunch of bugs
215         https://bugs.webkit.org/show_bug.cgi?id=63782
216
217         * make-hosted: Learn about the new directory.
218         * tests/sunspider-1.0: Copied from PerformanceTests/SunSpider/tests/sunspider-0.9.1.
219
220 2010-11-29  Geoffrey Garen  <ggaren@apple.com>
221
222         Reviewed by Gavin Barraclough.
223
224         Improved accuracy of command-line SunSpider.
225
226         * resources/sunspider-standalone-driver.js: Use the returned number of
227         elapsed milliseconds from "run" and "checkSyntax" instead of measuring
228         ourselves, for slightly more accurate numbers.
229
230 2010-09-16  Gavin Barraclough  <barraclough@apple.com>
231
232         Reviewed by Mark Rowe.
233         https://bugs.webkit.org/show_bug.cgi?id=45924
234
235         Modify the SunSpider harness to allow a path to be passed to the --suite
236         argument, supress the normal prepend-"tests/" behaviour for suite names
237         containing a '/'.
238
239         Also wrap the code in an anonymous function to move variables out of global
240         scope (currently the harness cannot handle running tests the overwrite the
241         variable 'j').
242
243         * resources/sunspider-standalone-driver.js:
244         * sunspider:
245
246 2010-08-27  Michael Saboff  <msaboff@apple.com>
247
248         Reviewed by Stephanie Lewis.
249
250         Added version 5 and version 6 of V8 tests with modifications similar
251         to the version 4 tests so they'll work with the SunSpider test harness.
252         Note that these files are external source and therefore don't conform 
253         to WebKit coding standards.
254         https://bugs.webkit.org/show_bug.cgi?id=44638
255
256         * tests/v8-v5: Added.
257         * tests/v8-v5/LIST: Added.
258         * tests/v8-v5/v8-crypto.js: Added.
259         * tests/v8-v5/v8-deltablue.js: Added.
260         * tests/v8-v5/v8-earley-boyer.js: Added.
261         * tests/v8-v5/v8-raytrace.js: Added.
262         * tests/v8-v5/v8-regexp.js: Added.
263         * tests/v8-v5/v8-richards.js: Added.
264         * tests/v8-v5/v8-splay.js: Added.
265         * tests/v8-v6: Added.
266         * tests/v8-v6/LIST: Added.
267         * tests/v8-v6/v8-crypto.js: Added.
268         * tests/v8-v6/v8-deltablue.js: Added.
269         * tests/v8-v6/v8-earley-boyer.js: Added.
270         * tests/v8-v6/v8-raytrace.js: Added.
271         * tests/v8-v6/v8-regexp.js: Added.
272         * tests/v8-v6/v8-richards.js: Added.
273         * tests/v8-v6/v8-splay.js: Added.
274
275 2010-07-15  Stephanie Lewis  <slewis@apple.com>
276
277         Reviewed by Geoff Garen.
278
279         http://bugs.webkit.org/show_bug.cgi?id=42406
280         Output location of results file.
281
282         * sunspider:
283
284 2010-05-17  Maciej Stachowiak  <mjs@apple.com>
285
286         Reviewed by Mark Rowe.
287
288         Browser-hosted version of SunSpider runs tests twice as many times as advertised
289         https://bugs.webkit.org/show_bug.cgi?id=39201
290
291         * make-hosted: Make sure to reset test list after every test suite.
292
293 2010-05-07  Adam Roben  <aroben@apple.com>
294
295         Fix typo in sunspider-compare-results.js that caused incorrect
296         two-sample t statistics to be computed
297
298         Fixes <http://webkit.org/b/38768> SunSpider computes incorrect
299         two-sample t statistics
300
301         Reviewed by Darin Adler.
302
303         * resources/sunspider-compare-results.js:
304         (sunspiderCompareResults.resultLine): Fixed the calculation of the t
305         variable to match the definition of a two-sample t statistic. Looks
306         like this was just a simple typo.
307
308 2010-04-05  Darin Adler  <darin@apple.com>
309
310         Updated the ignore property for changes to the contents here.
311
312         * .: Modified property svn:ignore.
313
314 2010-02-20  Maciej Stachowiak  <mjs@apple.com>
315
316         Reviewed by Adam Barth.
317
318         Add sunspider-0.9.1 to the site in preparation for announcing it.
319         https://bugs.webkit.org/show_bug.cgi?id=35206
320
321         * hosted/sunspider.css: Remove tabs, which I noticed where present while
322         copying to the WebKitSite directory.
323
324 2010-01-11  Oliver Hunt  <oliver@apple.com>
325
326         Reviewed by Darin Adler.
327
328         Sunspider parse-only tests don't run
329         https://bugs.webkit.org/show_bug.cgi?id=33489
330
331         Fix parse-only test list
332
333         * tests/parse-only/LIST:
334
335 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
336
337         Reviewed by Adele Peterson.
338
339         Make SunSpider version more prominent in the title
340         https://bugs.webkit.org/show_bug.cgi?id=32574
341
342         * hosted/sunspider.html:
343         * hosted/versions.html:
344         * resources/driver-TEMPLATE.html:
345         * resources/results-TEMPLATE.html:
346
347 2009-12-15  Maciej Stachowiak  <mjs@apple.com>
348
349         Reviewed by Darin Adler.
350
351         Make SunSpider refuse to compare results across content versions
352         https://bugs.webkit.org/show_bug.cgi?id=32573
353         
354         The results URL now includes a version indicator, and compare mode
355         will refuse to compare between versions.
356
357         * resources/driver-TEMPLATE.html:
358         * resources/results-TEMPLATE.html:
359
360 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
361
362         Reviewed by Darin Adler.
363
364         Some Browser-hosted SunSpider files are not valid HTML5
365         https://bugs.webkit.org/show_bug.cgi?id=32536
366         
367         Made various fixes to validate as HTML5 with no warnings (mainly removing bogus
368         close tags and adding a meta charset declaration).
369
370         * hosted/sunspider.html:
371         * hosted/versions.html:
372         * resources/TEMPLATE.html:
373         * resources/driver-TEMPLATE.html:
374         * resources/results-TEMPLATE.html:
375
376 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
377
378         Reviewed by Eric Seidel.
379
380         Make sunspider-0.9.1 the default content set (both command-line and hosted)
381         https://bugs.webkit.org/show_bug.cgi?id=32537
382
383         * hosted/sunspider.html: Change default suite link.
384         * sunspider: Change default suite.
385         * sunspider-compare-results: Likewise.
386
387 2009-12-14  Maciej Stachowiak  <mjs@apple.com>
388
389         Reviewed by Sam Weinig.
390
391         in-browser SunSpider suffers excessive penalty under power management
392         https://bugs.webkit.org/show_bug.cgi?id=32505
393         
394         I have made a few changes to address this:
395         
396         1) Change how browser-hosted SunSpider loads tests - preload the content and write it in with
397         document.write to reduce triggering of progress UI.
398         2) Reduce time between tests to 10ms from 500ms, so that power management doesn't keep the CPU
399         stepped all the way down the whole time.
400         3) Run the test cycle an extra time for warmup before the runs that count.
401
402         * hosted/sunspider-record-result.js: Removed.
403         * make-hosted:
404         * resources/TEMPLATE.html:
405         * resources/driver-TEMPLATE.html:
406
407 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
408
409         Reviewed by Gavin Barraclough.
410
411         SunSpider/tests/string-base64.js does not compute a valid base64 encoded string
412         https://bugs.webkit.org/show_bug.cgi?id=16806
413
414         Based on a patch by Eric Seidel.
415         
416         Fix the base64 computation to actually compute correct results. The impact on runtime of
417         the test is pretty small, but noticeable for some browsers. But at least it's not
418         doing a wrong and meaningless computation any more.
419         
420         * tests/sunspider-0.9.1/string-base64.js:
421         ():
422         (base64ToString):
423
424 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
425
426         Fixing commit error...
427         
428         I accidentally committed my last patch in a form that broke Web-hosted SunSpider. Fixing.
429
430         * resources/driver-TEMPLATE.html:
431
432 2009-12-13  Maciej Stachowiak  <mjs@apple.com>
433
434         Reviewed by Sam Weinig.
435
436         Use JSON.parse instead of eval for Web-hosted SunSpider results processing
437         https://bugs.webkit.org/show_bug.cgi?id=32490
438
439         * hosted/json2.js: Added. Incorporated from json.org
440         * resources/driver-TEMPLATE.html: Fix an HTML validation bug I noticed.
441         * resources/results-TEMPLATE.html: Use JSON.parse instead of eval to process
442         results.
443
444 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
445
446         Reviewed by Oliver Hunt.
447
448         Enable Web-hosted version of SunSpider to handle multiple versions
449         https://bugs.webkit.org/show_bug.cgi?id=32478
450
451         * make-hosted: Now generate the hosted version to be able to run both the
452         0.9 and the 0.9.1 test suites.
453         * hosted: Modified property svn:ignore.
454         * hosted/sunspider-driver.html: Removed.
455         * hosted/sunspider-results.html: Removed.
456         * hosted/sunspider.html:
457         * hosted/versions.html: Added.
458         * resources/TEMPLATE.html:
459         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-driver.html.
460         * resources/driver-TEMPLATE.html: Copied from hosted/sunspider-results.html.
461
462 2009-12-12  Maciej Stachowiak  <mjs@apple.com>
463
464         Reviewed by Oliver Hunt.
465
466         Give command-line sunspider the ability to handle multiple suites and versions
467         https://bugs.webkit.org/show_bug.cgi?id=32477
468         
469         Each suite/version's set of tests are now in a separate subdirectory of tests/, with a file named LIST
470         enumerating the tests. Current suites are:
471         
472         sunspider-0.9 - riginal SunSpider content 
473         sunspider-0.9.1 - SunSpider suite with a few tests fixed (and a few more to come)
474         ubench - "ubench" microbenchmark suite
475         v8-v4 - version 4 of the v8 benchmark
476         parse-only - some parser tests
477         
478         Sunspider can now be run with the --suite= parameter to select a
479         suite. The default is --suite=sunspider-0.9.
480
481         * .: Modified property svn:ignore to ignore new results directories.
482         * make-hosted: Generate based on sunspider-0.9 suite.
483         * resources/TEMPLATE.html: Fix an HTML compliance problem.
484         * resources/sunspider-standalone-driver.js: Become aware of suite names.
485         * sunspider: Modified to handle multiple suites as described above.
486         * sunspider-compare-results: ditto
487         * tests/LIST: Moved to tests/sunspider-0.9/ and tests/sunspider-0.9.1/
488         * tests/LIST-PARSE-ONLY: Moved to parse-only/LIST
489         * tests/LIST-UBENCH: Removed.
490         * tests/LIST-V8: Removed.
491         * tests/3d-cube.js: Moved to tests/sunspider-0.9 and tests/sunspider-0.9.1
492         * tests/3d-morph.js: ditto
493         * tests/3d-raytrace.js: ditto
494         * tests/access-binary-trees.js: ditto
495         * tests/access-fannkuch.js: ditto
496         * tests/access-nbody.js: ditto
497         * tests/access-nsieve.js: ditto
498         * tests/bitops-3bit-bits-in-byte.js: ditto
499         * tests/bitops-bits-in-byte.js: ditto
500         * tests/bitops-bitwise-and.js: ditto
501         * tests/bitops-nsieve-bits.js: ditto
502         * tests/controlflow-recursive.js: ditto
503         * tests/crypto-aes.js: ditto
504         * tests/crypto-md5.js: ditto
505         * tests/crypto-sha1.js: ditto
506         * tests/date-format-tofte.js: ditto
507         * tests/date-format-xparb.js: ditto
508         * tests/math-cordic.js: ditto
509         * tests/math-partial-sums.js: ditto
510         * tests/math-spectral-norm.js: ditto
511         * tests/regexp-dna.js: ditto
512         * tests/string-base64.js: ditto
513         * tests/string-fasta.js: ditto
514         * tests/string-tagcloud.js: ditto
515         * tests/string-unpack-code.js: ditto
516         * tests/string-validate-input.js: ditto
517         * tests/sunspider-0.9/: Added
518         * tests/sunspider-0.9/*.js: Moved from one level up
519         * tests/sunspider-0.9.1: Added.
520         * tests/sunspider-0.9.1/*.js: Moved from one level up.
521         * tests/v8-crypto.js: Moved to v8-v4/ directory.
522         * tests/v8-deltablue.js: ditto
523         * tests/v8-earley-boyer.js: ditto
524         * tests/v8-raytrace.js: ditto
525         * tests/v8-regexp.js: ditto
526         * tests/v8-richards.js: ditto
527         * tests/v8-splay.js: ditto
528         * tests/v8-v4: Added.
529
530 2009-09-30  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
531
532         Reviewed by Darin Adler.
533
534         Platform specific null device replaced with a platform independent.
535         https://bugs.webkit.org/show_bug.cgi?id=29544
536
537         * sunspider: Using File::Spec->devnull() instead of /dev/null and NUL.
538
539 2009-09-21  Csaba Osztrogonac  <oszi@inf.u-szeged.hu>
540
541         Reviewed by Maciej Stachowiak.
542
543         [Qt] Make sunspider script work on Windows platform.
544         https://bugs.webkit.org/show_bug.cgi?id=29544
545
546         * sunspider:
547         Using platform specific null device instead of /dev/null.
548
549 2009-06-19  Adam Treat  <adam.treat@torchmobile.com>
550
551         Reviewed by Oliver Hunt.
552
553         https://bugs.webkit.org/show_bug.cgi?id=26540
554         Currently the SunSpider test driver lacks an option to run a test suite that
555         will test JavaScriptCore parsing performance only.  This patch adds just such
556         a test suite and option to SunSpider as well as the jsc test shell.  I've included
557         three large javascript source files found in the wild: jquery, mootools and prototype.
558         Combined with the concatenation of all three, these form a new testsuite to measure
559         and test pure JavaScriptCore parsing performance.
560
561         * resources/sunspider-standalone-driver.js:
562           Load the js with parse-only flag if it is part of the parse-only suite.
563         * sunspider:
564           Add the parse-only suite to the list of options.
565         * sunspider-compare-results:
566           Add the parse-only suite to the list of options.
567         * tests/LIST-PARSE-ONLY: Added.
568         * tests/parse-only/concat-jquery-mootools-prototype.js: Added.
569           A very large concatenation of the scripts found below.  This js takes ~30ms to parse on my machine.
570         * tests/parse-only/jquery-1.3.2.js: Added.
571           Latest version from jquery.com.
572         * tests/parse-only/mootools-1.2.2-core-nc.js: Added.
573           Latest version from mootools.net.
574         * tests/parse-only/prototype-1.6.0.3.js: Added.
575           Latest version from prototypejs.org.
576
577 2009-06-13  Steve Falkenburg  <sfalken@apple.com>
578
579         Update V8 benchmark to version 4.
580         
581         Reviewed by Cameron Zwarich.
582
583         * tests/LIST-V8:
584         * tests/v8-crypto.js:
585         * tests/v8-deltablue.js:
586         * tests/v8-earley-boyer.js:
587         * tests/v8-raytrace.js:
588         * tests/v8-regexp.js: Added.
589         * tests/v8-richards.js:
590         * tests/v8-splay.js: Added.
591
592 2009-05-26  Olivier DOLE  <odole@pleyo.com>
593
594         Reviewed by Maciej.
595
596         Store sunspider test suite results in a specific directory for each
597         different type of benchmark (sunspider, V8 or ubench).
598
599         * sunspider:
600         * sunspider-compare-results:
601
602 2009-05-12  Steve Falkenburg  <sfalken@apple.com>
603
604         Build fix.
605
606         * SunSpider.make:
607
608 2009-05-11  Steve Falkenburg  <sfalken@apple.com>
609
610         Include SunSpider tests in production builds.
611         
612         Reviewed by Adam Roben.
613
614         * SunSpider.make: Added.
615         * xcopy.excludes: Added.
616
617 2009-03-04  Oliver Hunt  <oliver@apple.com>
618
619         Added a few more tests to ubench
620
621         RS=Gavin Barraclough
622
623         Adding a couple of test to cover too few and too many arguments.  This
624         is needed to prevent regressions in all call paths from my current refactoring.
625
626         * tests/LIST-UBENCH:
627         * tests/ubench/function-correct-args.js: Added.
628         (f):
629         * tests/ubench/function-excess-args.js: Added.
630         (f):
631
632 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
633
634         Fix a bug where when V8's or UBENCH's test suite list file is missing
635         the error message is suggesting that SunSpider's list file is missing.
636
637         Reviewed by Sam Weinig.
638
639         * sunspider:
640         (loadTestsList): Fix the error message to use the right list name.
641
642 2008-10-07  David Hyatt  <hyatt@apple.com>
643
644         https://bugs.webkit.org/show_bug.cgi?id=18989
645
646         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
647         we want all browsers to be doing the same thing on the test.
648
649         Reviewed by Maciej
650
651         * tests/regexp-dna.js:
652
653 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
654
655         Rubber-stamped by Maciej Stachowiak.
656
657         The bleeding edge branch of V8 now supports passing fileanmes after
658         '-f', so the SunSpider script should be changed to do this again.
659         This allows all 3 major JavaScript engines to be used with the
660         unmodified SunSpider script.
661
662         * sunspider:
663
664 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
665
666         Reviewed by Maciej Stachowiak.
667
668         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
669         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
670         correct the punctuation in its documentation.
671
672         * sunspider:
673
674 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
675
676         Reviewed by Oliver Hunt.
677
678         Bug 20616: Incorporate V8 benchmarks in testing
679         <https://bugs.webkit.org/show_bug.cgi?id=20616>
680
681         This is a first attempt at running the V8 benchmarks within SunSpider,
682         with the --v8 option. The number of iterations is fixed for each
683         benchmark, and was chosen to make each of the benchmarks run between
684         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
685
686         * sunspider:
687         * tests/LIST-V8: Added.
688         * tests/v8-crypto.js: Added.
689         * tests/v8-deltablue.js: Added.
690         * tests/v8-earley-boyer.js: Added.
691         * tests/v8-raytrace.js: Added.
692         * tests/v8-richards.js: Added.
693
694 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
695
696         Not reviewed.
697
698         Remove a local change that was accidentally committed in r36034.
699
700         * resources/sunspider-standalone-driver.js:
701
702 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
703
704         Rubber-stamped by Oliver Hunt.
705
706         Make SunSpider work with JavaScript shells that don't take filenames
707         after a '-f' argument.
708
709         * sunspider:
710
711 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
712
713         Reviewed by Timothy.
714
715         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
716
717         * sunspider:
718         * tests/LIST-SQUIRRELFISH: Removed.
719
720 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
721
722         Rubber-stamped by Mark Rowe.
723
724         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
725
726         * sunspider:
727
728 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
729
730         Rubber stamped by Oliver.
731         
732         - add newly working tests to squirrelfish list
733
734         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
735
736 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
737
738         Rubber stamped by Oliver.
739         
740         - move ubench tests properly.
741
742         * tests/ubench: Added.
743         * tests/ubench/function-closure.js: Added.
744         * tests/ubench/function-empty.js: Added.
745         * tests/ubench/function-missing-args.js: Added.
746         * tests/ubench/function-sum.js: Added.
747         * tests/ubench/loop-empty-resolve.js: Added.
748         * tests/ubench/loop-empty.js: Added.
749         * tests/ubench/loop-sum.js: Added.
750
751 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
752
753         Rubber stamped by Oliver.
754
755         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
756
757 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
758
759         Reviewed by Oliver.
760
761         - make --squirrelfish mode use the real harness, but a reduced set of tests
762
763         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
764
765         * sunspider:
766         * tests/LIST-SQUIRRELFISH:
767         * tests/LIST-UBENCH: Added.
768         * tests/squirrelfish: Moved to tests/ubench
769
770 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
771
772         Reviewed by Maciej Stachowiak.
773         
774         Removed an unused test.
775
776         * tests/squirrelfish/loop-resolve.js: Removed.
777
778 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
779
780         Reviewed by Oliver.
781         
782         Add newly runnable tests to --squirrelfish mode.
783
784         * tests/LIST-SQUIRRELFISH:
785
786 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
787
788         Reviewed by Maciej Stachowiak.
789         
790         A little more love for --squirrelfish mode:
791         
792         Fixed a misplaced sort. Results properly sort now.
793         
794         Added a test list just for squirrelfish, and updated it to include
795         all currently passing SunSpider tests.
796         
797         Fixed the pruning regexp to match 3d-morph.js.
798
799         * sunspider:
800
801 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
802
803         Reviewed by Oliver Hunt.
804         
805         --squirrelfish mode: pared down tests for squirrelfish to chew on.
806
807         * sunspider:
808         * tests/LIST:
809         * tests/squirrelfish: Added.
810         * tests/squirrelfish/function-closure.js: Added.
811         * tests/squirrelfish/function-empty.js: Added.
812         * tests/squirrelfish/function-missing-args.js: Added.
813         * tests/squirrelfish/function-sum.js: Added.
814         * tests/squirrelfish/loop-empty-resolve.js: Added.
815         * tests/squirrelfish/loop-empty.js: Added.
816         * tests/squirrelfish/loop-resolve.js: Added.
817         * tests/squirrelfish/loop-sum.js: Added.
818
819 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
820
821         Reviewed by Geoff.
822
823         * sunspider: Pause update daemon on Darwin to make results reliable
824         again (workaround for <rdar://problem/5811127>).
825
826 2008-02-07  Eric Seidel  <eric@webkit.org>
827
828         Reviewed by Sam.
829
830         * sunspider: add an error message when ./tests/LIST is not found
831
832 2007-12-29  Alp Toker  <alp@atoker.com>
833
834         Fix typo.
835
836         * hosted/sunspider.html:
837
838 2007-12-29  Alp Toker  <alp@atoker.com>
839
840         Reviewed by Maciej.
841
842         Provide the current URL in the SunSpider results page. Useful for
843         copying benchmark results out of browsers without a location bar or
844         where the location bar can't handle long URLs.
845
846         * hosted/sunspider-results.html:
847
848 2007-12-20  Eric Seidel  <eric@webkit.org>
849
850         Reviewed by mjs.
851
852         * sunspider: increase default number of runs for more accurate numbers
853
854 2007-12-18  Mark Rowe  <mrowe@apple.com>
855
856         Fix typo.
857
858         * hosted/sunspider.html:
859
860 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
861
862         Reviewed by Eric.
863         
864         - give an extra digit of precision for small differences.
865
866         * resources/sunspider-compare-results.js:
867
868 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
869
870         Reviewed by Eric.
871         
872         - work around IE layout bugs
873
874         * hosted/sunspider.css:
875
876 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
877
878         Reviewed by Mark.
879         
880         - fix compare feature
881
882         * hosted/sunspider-results.html:
883
884 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
885
886         Reviewed by Mark.
887         
888         - Change the format of the output URL to be shorter, to meet IE constraints
889
890         * hosted/sunspider-driver.html:
891         * hosted/sunspider-results.html:
892
893 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
894
895         Reviewed by Oliver.
896         
897         - Add some explanatory test and a dumb logo to SunSpider web hosted version
898
899         * hosted/sunspider-driver.html:
900         * hosted/sunspider-results.html:
901         * hosted/sunspider.css:
902         * hosted/sunspider.html:
903
904 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
905
906         Reviewed by Oliver.
907         
908         - Make the web-hosted version of SunSpider look a little bit less drab.
909
910         * hosted/sunspider-driver.html:
911         * hosted/sunspider-results.html:
912         * hosted/sunspider.css: Added.
913         * hosted/sunspider.html:
914         * resources/TEMPLATE.html:
915
916 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
917
918         Reviewed by Sam.
919         
920         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
921         
922         A sample of what new-style results would look like:
923         
924         TEST                   COMPARISON            FROM                 TO             DETAILS
925         ...
926         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
927         ...
928             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
929         
930         * resources/sunspider-compare-results.js:
931
932 2007-11-14  Eric Seidel  <eric@webkit.org>
933
934         Reviewed by Sam.
935
936         * sunspider: add --shark-cache for L2 Cache Miss profiling
937
938 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
939
940         Reviewed by Darin Adler.
941
942         http://bugs.webkit.org/show_bug.cgi?id=15925
943         SunSpider should check for Shark being installed
944
945         * sunspider: Explicitly test that shark command line tool is installed.
946
947 2007-11-04  Darin Adler  <darin@apple.com>
948
949         * hosted/sunspider.html: Tweaked formatting a bit.
950
951 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
952
953         Reviewed by Darin Adler.
954
955         - Add compare mode and a description of the benchmark to the hosted version.
956
957         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
958         * hosted/sunspider.html: Added intro text.
959         * make-hosted: Copy new file.
960         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
961         a function.
962         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
963         expects.
964         * sunspider-compare-results: Use sunspider-standalone-compare.js.
965
966 2007-11-02  Darin Adler  <darin@apple.com>
967
968         * sunspider: Changed "--base" to "--set-baseline".
969
970 2007-11-02  Darin Adler  <darin@apple.com>
971
972         Reviewed by Maciej.
973
974         * sunspider: Added a "--base" option that records the current test as a baseline.
975         Moved the code to find the newest .mshark file into a function (since I needed to use
976         it in the other script and I prefer to copy/paste a function).
977
978         * sunspider-compare-results: Added logic to compare the baseline with the last
979         run when you don't pass any parameters.
980
981 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
982
983         Reviewed by Eric.
984         
985         - Added list of ECMAScript3 features that SunSpider does not cover currently.
986
987         * UNCOVERED: Added.
988
989 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
990
991         Reviewed by Oliver.
992         
993         - Give better error messages when sunspider-compare-results fails
994
995         * sunspider-compare-results:
996
997 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
998
999         Reviewed by Darin Adler.
1000         
1001         - fix Date tests to work in command-line SpiderMonkey
1002
1003         * tests/date-format-tofte.js:
1004         * tests/date-format-xparb.js:
1005
1006 2007-10-29  Darin Adler  <darin@apple.com>
1007
1008         Reviewed by Maciej.
1009
1010         * sunspider: Added --shark20 option, to run Shark at its highest sample
1011         resolution instead of its default.
1012
1013 2007-10-28  Adam Roben  <aroben@apple.com>
1014
1015         Fix SunSpider on Windows
1016
1017         Reviewed by Eric.
1018
1019         * sunspider: Don't use colons in filenames.
1020
1021 2007-10-28  Darin Adler  <darin@apple.com>
1022
1023         Reviewed by Adam.
1024
1025         * resources/sunspider-analyze-results.js: Tweak the output format a little.
1026         Change so that when there's only one run we don't write out confidence
1027         intervals at all rather than writing "NaN" over and over again.
1028
1029         * sunspider: Use the time and date as a suffix on the results file, that way
1030         all the results are left behind in the tmp directory. This will make it easy
1031         for us to add features that compare past results in the future. Also rename
1032         the .mshark files using the same scheme.
1033
1034         * sunspider-compare-results: Relax the parsing rules so we can parse the
1035         results file exactly as it's emitted from sunspider.
1036
1037 2007-10-25  Adam Roben  <aroben@apple.com>
1038
1039         Strip carriage returns from results
1040
1041         Carriage returns were screwing up sunspider-compare-results on
1042         Windows.
1043
1044         Rubberstamped by Oliver.
1045
1046         * sunspider:
1047
1048 2007-10-24  Darin Adler  <darin@apple.com>
1049
1050         Reviewed by Eric.
1051
1052         * sunspider: Open the profile in shark after completing a run.
1053         * sunspider-compare-results: Tiny perl tweaks.
1054
1055 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
1056
1057         Reviewed by Darin Adler.
1058
1059         - Don't hardcode my path to testkjs
1060         
1061         * sunspider-compare-results:
1062
1063 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
1064
1065         Reviewed by Darin Adler.
1066         
1067         - Add a compare script to compare results, and improve formatting of standard results a bit.
1068
1069         * resources/sunspider-analyze-results.js:
1070         * resources/sunspider-compare-results.js: Added.
1071         * sunspider-compare-results: Added.
1072
1073 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
1074
1075         Reviewed by Darin Adler.
1076
1077         * resources/sunspider-analyze-results.js: Do the error range
1078         properly, using the t-distribution instead of 1.96 (which was
1079         based on the normal distribution).
1080         
1081         * sunspider: Print results in a way that will be friendlier to the
1082         soon-to-come compare mode.
1083
1084 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
1085
1086         Reviewed by Review.
1087         
1088         - add more tests (probably the final set for now)
1089         
1090         Already balanced. Added date, regexp, control flow, and a few
1091         more string and object/array access tests.
1092         
1093         * tests/LIST:
1094         * tests/access-fannkuch.js: Added.
1095         * tests/access-nbody.js: Added.
1096         * tests/controlflow-recursive.js: Added.
1097         * tests/date-format-tofte.js: Added.
1098         * tests/date-format-xparb.js: Added.
1099         * tests/regexp-dna.js: Added.
1100         * tests/string-validate-input.js: Added.
1101
1102 2007-10-21  Eric Seidel  <eric@webkit.org>
1103
1104         Reviewed by mjs.
1105
1106         * sunspider: make --shark only sample from testkjs
1107
1108 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
1109
1110         Rubber stamped by Eric.
1111         
1112         - added crypto tests (already balanced)
1113
1114         * tests/LIST:
1115         * tests/crypto-aes.js: Added.
1116         * tests/crypto-md5.js: Added.
1117         * tests/crypto-sha1.js: Added.
1118
1119 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
1120
1121         Reviewed by Adam.
1122         
1123         - rebalance new tests
1124         
1125         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
1126         
1127         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
1128
1129         This balances them with the existing tests.
1130
1131         * tests/access-binary-trees.js:
1132         * tests/access-nsieve.js:
1133         * tests/bitops-nsieve-bits.js:
1134         * tests/math-partial-sums.js:
1135         * tests/math-spectral-norm.js:
1136         * tests/string-fasta.js:
1137
1138 2007-10-20  Eric Seidel  <eric@webkit.org>
1139
1140         Reviewed by darin.
1141         
1142         Add improved argument handling to sunspider, including
1143         --runs=<number>, --tests=<pattern, --shark, and --help
1144         Also re-factor code into subroutines
1145
1146         * sunspider: improved argument handling
1147
1148 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1149
1150         Reviewed by Mark.
1151
1152         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
1153         
1154         * TODO:
1155         * tests/LIST:
1156         * tests/access-binary-trees.js: Added.
1157         * tests/access-nsieve.js: Added.
1158         * tests/bitops-nsieve-bits.js: Added.
1159         * tests/math-partial-sums.js: Added.
1160         * tests/math-spectral-norm.js: Added.
1161         * tests/string-fasta.js: Added.
1162
1163 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1164
1165         Reviewed by Darin Adler.
1166
1167         * TODO: Updated to note areas that are now well-covered.
1168         
1169         - Change the tests and the driver to avoid leaving large object
1170         graphs hanging around, since that throws off the subsequent tests.
1171         
1172         * resources/sunspider-standalone-driver.js:
1173         * tests/3d-cube.js:
1174         * tests/3d-morph.js:
1175         * tests/3d-raytrace.js:
1176         * tests/bitops-bitwise-and.js:
1177         * tests/string-base64.js:
1178         * tests/string-tagcloud.js:
1179
1180 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1181
1182         Reviewed by Nikolas.
1183         
1184         - Rebalance test complexity.
1185         
1186         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
1187         
1188         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
1189
1190         However, for some tests, IE7 was a huge outlier compared to other
1191         major browsers. For those tests, I used Firefox 2.0.0.8 on the
1192         same OS and hardware as a normalization basis.
1193         
1194         * tests/3d-morph.js:
1195         * tests/3d-raytrace.js:
1196         * tests/bitops-3bit-bits-in-byte.js:
1197         * tests/bitops-bits-in-byte.js:
1198         * tests/bitops-bitwise-and.js:
1199         * tests/math-cordic.js:
1200         * tests/string-base64.js:
1201         * tests/string-tagcloud.js:
1202         * tests/string-unpack-code.js:
1203
1204 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1205
1206         Reviewed by Oliver.
1207         
1208         - Fix the browser-hosted driver to be compatible with Safari 2
1209
1210         * hosted/sunspider-driver.html:
1211
1212 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
1213
1214         Rubber stamped by Hyatt.
1215         
1216         - Add BSD license notice to cordic test, the original author said to do so.
1217
1218         * tests/math-cordic.js:
1219
1220 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
1221
1222         Reviewed by Eric.
1223         
1224         - Support Windows and Windows browsers.
1225
1226         * hosted/sunspider-results.html: Display results correctly in IE.
1227         * sunspider: tweaks to work on Windows, and with spaces in the path.
1228
1229 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
1230
1231         Rubber stamped by Oliver.
1232
1233         * tests/string-tagcloud.js: Remove stray print()
1234
1235 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
1236
1237         Reviewed by Oliver.
1238         
1239         Add some string test cases.
1240
1241         * tests/string-base64.js: Added. Does base64 encoding and decoding.
1242         * tests/string-tagcloud.js: Added. Parses a JSON string and
1243         generates tagcloud HTML markup based on the data.        
1244         * tests/string-unpack-code.js: Added. Unpack compressed versions of
1245         four of the most popular JavaScript libraries.
1246         * tests/LIST: Added the new tests.
1247
1248 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
1249
1250         Reviewed by Anders.
1251
1252         * TODO: Updated with info on how to do the stats properly.
1253
1254         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
1255         avoid overly slowing down the benchmark.
1256
1257         Added some 3D-related test cases.
1258         
1259         * tests/3d-cube.js: Added.
1260         * tests/3d-morph.js: Added.
1261         * tests/3d-raytrace.js: Added.
1262         * tests/LIST:
1263
1264 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
1265
1266         Reviewed by Darin Adler.
1267
1268         - New JavaScript benchmark
1269         http://bugs.webkit.org/show_bug.cgi?id=15515
1270         
1271         A start on a new JavaScript benchmark, with standalone and
1272         browser-hosted drivers.
1273         
1274         The test content is still incomplete and the drivers could use
1275         some more features but there is enough here to make a good start.
1276         
1277         The drivers compute a 95% confidence interval on the mean for the
1278         whole test, each category, and each individual test to make it
1279         easier to tell whether differences are statistically
1280         significant. The confidence interval can be narrowed by running
1281         the test more times.
1282         
1283         * TODO: Added. Remaining things that need to be done.
1284         * sunspider: Added. Perl script that acts as the standalone test driver.
1285         * resources/sunspider-standalone-driver.js: Added. JavaScript part
1286         of standalone test driver.
1287         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
1288         analysis code.
1289         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
1290         * make-hosted: Added. Script to generate browser-hosted tests.
1291         * hosted/sunspider.html: Added. Start page for browser-hosted test.
1292         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
1293         * hosted/sunspider-results.html: Added. Results page for browser-hosted
1294         tests.
1295         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
1296         tests.
1297         * tests/LIST: Added. List of tests to use.
1298         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
1299         * tests/bitops-bits-in-byte.js: Added. Ditto.
1300         * tests/bitops-bitwise-and.js: Added. Ditto.
1301         * tests/math-cordic.js: Added. Ditto.
1302