Build fix.
[WebKit-https.git] / SunSpider / ChangeLog
1 2009-05-12  Steve Falkenburg  <sfalken@apple.com>
2
3         Build fix.
4
5         * SunSpider.make:
6
7 2009-05-11  Steve Falkenburg  <sfalken@apple.com>
8
9         Include SunSpider tests in production builds.
10         
11         Reviewed by Adam Roben.
12
13         * SunSpider.make: Added.
14         * xcopy.excludes: Added.
15
16 2009-03-04  Oliver Hunt  <oliver@apple.com>
17
18         Added a few more tests to ubench
19
20         RS=Gavin Barraclough
21
22         Adding a couple of test to cover too few and too many arguments.  This
23         is needed to prevent regressions in all call paths from my current refactoring.
24
25         * tests/LIST-UBENCH:
26         * tests/ubench/function-correct-args.js: Added.
27         (f):
28         * tests/ubench/function-excess-args.js: Added.
29         (f):
30
31 2008-10-30  Zoltan Horvath  <Horvath.Zoltan.6@stud.u-szeged.hu>
32
33         Fix a bug where when V8's or UBENCH's test suite list file is missing
34         the error message is suggesting that SunSpider's list file is missing.
35
36         Reviewed by Sam Weinig.
37
38         * sunspider:
39         (loadTestsList): Fix the error message to use the right list name.
40
41 2008-10-07  David Hyatt  <hyatt@apple.com>
42
43         https://bugs.webkit.org/show_bug.cgi?id=18989
44
45         Remove global substitution from SunSpider regexp test, since only Firefox was honoring it and
46         we want all browsers to be doing the same thing on the test.
47
48         Reviewed by Maciej
49
50         * tests/regexp-dna.js:
51
52 2008-09-09  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
53
54         Rubber-stamped by Maciej Stachowiak.
55
56         The bleeding edge branch of V8 now supports passing fileanmes after
57         '-f', so the SunSpider script should be changed to do this again.
58         This allows all 3 major JavaScript engines to be used with the
59         unmodified SunSpider script.
60
61         * sunspider:
62
63 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
64
65         Reviewed by Maciej Stachowiak.
66
67         Add a '--args' option to SunSpider, so that we can pass '-j' to TraceMonkey
68         and '--expose-gc' to V8. Also, rename the '--v8' option to '--v8-suite' and
69         correct the punctuation in its documentation.
70
71         * sunspider:
72
73 2008-09-04  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
74
75         Reviewed by Oliver Hunt.
76
77         Bug 20616: Incorporate V8 benchmarks in testing
78         <https://bugs.webkit.org/show_bug.cgi?id=20616>
79
80         This is a first attempt at running the V8 benchmarks within SunSpider,
81         with the --v8 option. The number of iterations is fixed for each
82         benchmark, and was chosen to make each of the benchmarks run between
83         500 ms and 600 ms in V8 on my machine, a 2.16 GHz MacBook Pro.
84
85         * sunspider:
86         * tests/LIST-V8: Added.
87         * tests/v8-crypto.js: Added.
88         * tests/v8-deltablue.js: Added.
89         * tests/v8-earley-boyer.js: Added.
90         * tests/v8-raytrace.js: Added.
91         * tests/v8-richards.js: Added.
92
93 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
94
95         Not reviewed.
96
97         Remove a local change that was accidentally committed in r36034.
98
99         * resources/sunspider-standalone-driver.js:
100
101 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
102
103         Rubber-stamped by Oliver Hunt.
104
105         Make SunSpider work with JavaScript shells that don't take filenames
106         after a '-f' argument.
107
108         * sunspider:
109
110 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
111
112         Reviewed by Timothy.
113
114         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
115
116         * sunspider:
117         * tests/LIST-SQUIRRELFISH: Removed.
118
119 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
120
121         Rubber-stamped by Mark Rowe.
122
123         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
124
125         * sunspider:
126
127 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
128
129         Rubber stamped by Oliver.
130         
131         - add newly working tests to squirrelfish list
132
133         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
134
135 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
136
137         Rubber stamped by Oliver.
138         
139         - move ubench tests properly.
140
141         * tests/ubench: Added.
142         * tests/ubench/function-closure.js: Added.
143         * tests/ubench/function-empty.js: Added.
144         * tests/ubench/function-missing-args.js: Added.
145         * tests/ubench/function-sum.js: Added.
146         * tests/ubench/loop-empty-resolve.js: Added.
147         * tests/ubench/loop-empty.js: Added.
148         * tests/ubench/loop-sum.js: Added.
149
150 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
151
152         Rubber stamped by Oliver.
153
154         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
155
156 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
157
158         Reviewed by Oliver.
159
160         - make --squirrelfish mode use the real harness, but a reduced set of tests
161
162         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
163
164         * sunspider:
165         * tests/LIST-SQUIRRELFISH:
166         * tests/LIST-UBENCH: Added.
167         * tests/squirrelfish: Moved to tests/ubench
168
169 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
170
171         Reviewed by Maciej Stachowiak.
172         
173         Removed an unused test.
174
175         * tests/squirrelfish/loop-resolve.js: Removed.
176
177 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
178
179         Reviewed by Oliver.
180         
181         Add newly runnable tests to --squirrelfish mode.
182
183         * tests/LIST-SQUIRRELFISH:
184
185 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
186
187         Reviewed by Maciej Stachowiak.
188         
189         A little more love for --squirrelfish mode:
190         
191         Fixed a misplaced sort. Results properly sort now.
192         
193         Added a test list just for squirrelfish, and updated it to include
194         all currently passing SunSpider tests.
195         
196         Fixed the pruning regexp to match 3d-morph.js.
197
198         * sunspider:
199
200 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
201
202         Reviewed by Oliver Hunt.
203         
204         --squirrelfish mode: pared down tests for squirrelfish to chew on.
205
206         * sunspider:
207         * tests/LIST:
208         * tests/squirrelfish: Added.
209         * tests/squirrelfish/function-closure.js: Added.
210         * tests/squirrelfish/function-empty.js: Added.
211         * tests/squirrelfish/function-missing-args.js: Added.
212         * tests/squirrelfish/function-sum.js: Added.
213         * tests/squirrelfish/loop-empty-resolve.js: Added.
214         * tests/squirrelfish/loop-empty.js: Added.
215         * tests/squirrelfish/loop-resolve.js: Added.
216         * tests/squirrelfish/loop-sum.js: Added.
217
218 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
219
220         Reviewed by Geoff.
221
222         * sunspider: Pause update daemon on Darwin to make results reliable
223         again (workaround for <rdar://problem/5811127>).
224
225 2008-02-07  Eric Seidel  <eric@webkit.org>
226
227         Reviewed by Sam.
228
229         * sunspider: add an error message when ./tests/LIST is not found
230
231 2007-12-29  Alp Toker  <alp@atoker.com>
232
233         Fix typo.
234
235         * hosted/sunspider.html:
236
237 2007-12-29  Alp Toker  <alp@atoker.com>
238
239         Reviewed by Maciej.
240
241         Provide the current URL in the SunSpider results page. Useful for
242         copying benchmark results out of browsers without a location bar or
243         where the location bar can't handle long URLs.
244
245         * hosted/sunspider-results.html:
246
247 2007-12-20  Eric Seidel  <eric@webkit.org>
248
249         Reviewed by mjs.
250
251         * sunspider: increase default number of runs for more accurate numbers
252
253 2007-12-18  Mark Rowe  <mrowe@apple.com>
254
255         Fix typo.
256
257         * hosted/sunspider.html:
258
259 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
260
261         Reviewed by Eric.
262         
263         - give an extra digit of precision for small differences.
264
265         * resources/sunspider-compare-results.js:
266
267 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
268
269         Reviewed by Eric.
270         
271         - work around IE layout bugs
272
273         * hosted/sunspider.css:
274
275 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
276
277         Reviewed by Mark.
278         
279         - fix compare feature
280
281         * hosted/sunspider-results.html:
282
283 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
284
285         Reviewed by Mark.
286         
287         - Change the format of the output URL to be shorter, to meet IE constraints
288
289         * hosted/sunspider-driver.html:
290         * hosted/sunspider-results.html:
291
292 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
293
294         Reviewed by Oliver.
295         
296         - Add some explanatory test and a dumb logo to SunSpider web hosted version
297
298         * hosted/sunspider-driver.html:
299         * hosted/sunspider-results.html:
300         * hosted/sunspider.css:
301         * hosted/sunspider.html:
302
303 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
304
305         Reviewed by Oliver.
306         
307         - Make the web-hosted version of SunSpider look a little bit less drab.
308
309         * hosted/sunspider-driver.html:
310         * hosted/sunspider-results.html:
311         * hosted/sunspider.css: Added.
312         * hosted/sunspider.html:
313         * resources/TEMPLATE.html:
314
315 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
316
317         Reviewed by Sam.
318         
319         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
320         
321         A sample of what new-style results would look like:
322         
323         TEST                   COMPARISON            FROM                 TO             DETAILS
324         ...
325         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
326         ...
327             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
328         
329         * resources/sunspider-compare-results.js:
330
331 2007-11-14  Eric Seidel  <eric@webkit.org>
332
333         Reviewed by Sam.
334
335         * sunspider: add --shark-cache for L2 Cache Miss profiling
336
337 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
338
339         Reviewed by Darin Adler.
340
341         http://bugs.webkit.org/show_bug.cgi?id=15925
342         SunSpider should check for Shark being installed
343
344         * sunspider: Explicitly test that shark command line tool is installed.
345
346 2007-11-04  Darin Adler  <darin@apple.com>
347
348         * hosted/sunspider.html: Tweaked formatting a bit.
349
350 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
351
352         Reviewed by Darin Adler.
353
354         - Add compare mode and a description of the benchmark to the hosted version.
355
356         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
357         * hosted/sunspider.html: Added intro text.
358         * make-hosted: Copy new file.
359         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
360         a function.
361         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
362         expects.
363         * sunspider-compare-results: Use sunspider-standalone-compare.js.
364
365 2007-11-02  Darin Adler  <darin@apple.com>
366
367         * sunspider: Changed "--base" to "--set-baseline".
368
369 2007-11-02  Darin Adler  <darin@apple.com>
370
371         Reviewed by Maciej.
372
373         * sunspider: Added a "--base" option that records the current test as a baseline.
374         Moved the code to find the newest .mshark file into a function (since I needed to use
375         it in the other script and I prefer to copy/paste a function).
376
377         * sunspider-compare-results: Added logic to compare the baseline with the last
378         run when you don't pass any parameters.
379
380 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
381
382         Reviewed by Eric.
383         
384         - Added list of ECMAScript3 features that SunSpider does not cover currently.
385
386         * UNCOVERED: Added.
387
388 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
389
390         Reviewed by Oliver.
391         
392         - Give better error messages when sunspider-compare-results fails
393
394         * sunspider-compare-results:
395
396 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
397
398         Reviewed by Darin Adler.
399         
400         - fix Date tests to work in command-line SpiderMonkey
401
402         * tests/date-format-tofte.js:
403         * tests/date-format-xparb.js:
404
405 2007-10-29  Darin Adler  <darin@apple.com>
406
407         Reviewed by Maciej.
408
409         * sunspider: Added --shark20 option, to run Shark at its highest sample
410         resolution instead of its default.
411
412 2007-10-28  Adam Roben  <aroben@apple.com>
413
414         Fix SunSpider on Windows
415
416         Reviewed by Eric.
417
418         * sunspider: Don't use colons in filenames.
419
420 2007-10-28  Darin Adler  <darin@apple.com>
421
422         Reviewed by Adam.
423
424         * resources/sunspider-analyze-results.js: Tweak the output format a little.
425         Change so that when there's only one run we don't write out confidence
426         intervals at all rather than writing "NaN" over and over again.
427
428         * sunspider: Use the time and date as a suffix on the results file, that way
429         all the results are left behind in the tmp directory. This will make it easy
430         for us to add features that compare past results in the future. Also rename
431         the .mshark files using the same scheme.
432
433         * sunspider-compare-results: Relax the parsing rules so we can parse the
434         results file exactly as it's emitted from sunspider.
435
436 2007-10-25  Adam Roben  <aroben@apple.com>
437
438         Strip carriage returns from results
439
440         Carriage returns were screwing up sunspider-compare-results on
441         Windows.
442
443         Rubberstamped by Oliver.
444
445         * sunspider:
446
447 2007-10-24  Darin Adler  <darin@apple.com>
448
449         Reviewed by Eric.
450
451         * sunspider: Open the profile in shark after completing a run.
452         * sunspider-compare-results: Tiny perl tweaks.
453
454 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
455
456         Reviewed by Darin Adler.
457
458         - Don't hardcode my path to testkjs
459         
460         * sunspider-compare-results:
461
462 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
463
464         Reviewed by Darin Adler.
465         
466         - Add a compare script to compare results, and improve formatting of standard results a bit.
467
468         * resources/sunspider-analyze-results.js:
469         * resources/sunspider-compare-results.js: Added.
470         * sunspider-compare-results: Added.
471
472 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
473
474         Reviewed by Darin Adler.
475
476         * resources/sunspider-analyze-results.js: Do the error range
477         properly, using the t-distribution instead of 1.96 (which was
478         based on the normal distribution).
479         
480         * sunspider: Print results in a way that will be friendlier to the
481         soon-to-come compare mode.
482
483 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
484
485         Reviewed by Review.
486         
487         - add more tests (probably the final set for now)
488         
489         Already balanced. Added date, regexp, control flow, and a few
490         more string and object/array access tests.
491         
492         * tests/LIST:
493         * tests/access-fannkuch.js: Added.
494         * tests/access-nbody.js: Added.
495         * tests/controlflow-recursive.js: Added.
496         * tests/date-format-tofte.js: Added.
497         * tests/date-format-xparb.js: Added.
498         * tests/regexp-dna.js: Added.
499         * tests/string-validate-input.js: Added.
500
501 2007-10-21  Eric Seidel  <eric@webkit.org>
502
503         Reviewed by mjs.
504
505         * sunspider: make --shark only sample from testkjs
506
507 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
508
509         Rubber stamped by Eric.
510         
511         - added crypto tests (already balanced)
512
513         * tests/LIST:
514         * tests/crypto-aes.js: Added.
515         * tests/crypto-md5.js: Added.
516         * tests/crypto-sha1.js: Added.
517
518 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
519
520         Reviewed by Adam.
521         
522         - rebalance new tests
523         
524         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
525         
526         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
527
528         This balances them with the existing tests.
529
530         * tests/access-binary-trees.js:
531         * tests/access-nsieve.js:
532         * tests/bitops-nsieve-bits.js:
533         * tests/math-partial-sums.js:
534         * tests/math-spectral-norm.js:
535         * tests/string-fasta.js:
536
537 2007-10-20  Eric Seidel  <eric@webkit.org>
538
539         Reviewed by darin.
540         
541         Add improved argument handling to sunspider, including
542         --runs=<number>, --tests=<pattern, --shark, and --help
543         Also re-factor code into subroutines
544
545         * sunspider: improved argument handling
546
547 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
548
549         Reviewed by Mark.
550
551         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
552         
553         * TODO:
554         * tests/LIST:
555         * tests/access-binary-trees.js: Added.
556         * tests/access-nsieve.js: Added.
557         * tests/bitops-nsieve-bits.js: Added.
558         * tests/math-partial-sums.js: Added.
559         * tests/math-spectral-norm.js: Added.
560         * tests/string-fasta.js: Added.
561
562 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
563
564         Reviewed by Darin Adler.
565
566         * TODO: Updated to note areas that are now well-covered.
567         
568         - Change the tests and the driver to avoid leaving large object
569         graphs hanging around, since that throws off the subsequent tests.
570         
571         * resources/sunspider-standalone-driver.js:
572         * tests/3d-cube.js:
573         * tests/3d-morph.js:
574         * tests/3d-raytrace.js:
575         * tests/bitops-bitwise-and.js:
576         * tests/string-base64.js:
577         * tests/string-tagcloud.js:
578
579 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
580
581         Reviewed by Nikolas.
582         
583         - Rebalance test complexity.
584         
585         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
586         
587         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
588
589         However, for some tests, IE7 was a huge outlier compared to other
590         major browsers. For those tests, I used Firefox 2.0.0.8 on the
591         same OS and hardware as a normalization basis.
592         
593         * tests/3d-morph.js:
594         * tests/3d-raytrace.js:
595         * tests/bitops-3bit-bits-in-byte.js:
596         * tests/bitops-bits-in-byte.js:
597         * tests/bitops-bitwise-and.js:
598         * tests/math-cordic.js:
599         * tests/string-base64.js:
600         * tests/string-tagcloud.js:
601         * tests/string-unpack-code.js:
602
603 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
604
605         Reviewed by Oliver.
606         
607         - Fix the browser-hosted driver to be compatible with Safari 2
608
609         * hosted/sunspider-driver.html:
610
611 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
612
613         Rubber stamped by Hyatt.
614         
615         - Add BSD license notice to cordic test, the original author said to do so.
616
617         * tests/math-cordic.js:
618
619 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
620
621         Reviewed by Eric.
622         
623         - Support Windows and Windows browsers.
624
625         * hosted/sunspider-results.html: Display results correctly in IE.
626         * sunspider: tweaks to work on Windows, and with spaces in the path.
627
628 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
629
630         Rubber stamped by Oliver.
631
632         * tests/string-tagcloud.js: Remove stray print()
633
634 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
635
636         Reviewed by Oliver.
637         
638         Add some string test cases.
639
640         * tests/string-base64.js: Added. Does base64 encoding and decoding.
641         * tests/string-tagcloud.js: Added. Parses a JSON string and
642         generates tagcloud HTML markup based on the data.        
643         * tests/string-unpack-code.js: Added. Unpack compressed versions of
644         four of the most popular JavaScript libraries.
645         * tests/LIST: Added the new tests.
646
647 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
648
649         Reviewed by Anders.
650
651         * TODO: Updated with info on how to do the stats properly.
652
653         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
654         avoid overly slowing down the benchmark.
655
656         Added some 3D-related test cases.
657         
658         * tests/3d-cube.js: Added.
659         * tests/3d-morph.js: Added.
660         * tests/3d-raytrace.js: Added.
661         * tests/LIST:
662
663 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
664
665         Reviewed by Darin Adler.
666
667         - New JavaScript benchmark
668         http://bugs.webkit.org/show_bug.cgi?id=15515
669         
670         A start on a new JavaScript benchmark, with standalone and
671         browser-hosted drivers.
672         
673         The test content is still incomplete and the drivers could use
674         some more features but there is enough here to make a good start.
675         
676         The drivers compute a 95% confidence interval on the mean for the
677         whole test, each category, and each individual test to make it
678         easier to tell whether differences are statistically
679         significant. The confidence interval can be narrowed by running
680         the test more times.
681         
682         * TODO: Added. Remaining things that need to be done.
683         * sunspider: Added. Perl script that acts as the standalone test driver.
684         * resources/sunspider-standalone-driver.js: Added. JavaScript part
685         of standalone test driver.
686         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
687         analysis code.
688         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
689         * make-hosted: Added. Script to generate browser-hosted tests.
690         * hosted/sunspider.html: Added. Start page for browser-hosted test.
691         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
692         * hosted/sunspider-results.html: Added. Results page for browser-hosted
693         tests.
694         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
695         tests.
696         * tests/LIST: Added. List of tests to use.
697         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
698         * tests/bitops-bits-in-byte.js: Added. Ditto.
699         * tests/bitops-bitwise-and.js: Added. Ditto.
700         * tests/math-cordic.js: Added. Ditto.
701