2008-09-02 Cameron Zwarich <cwzwarich@uwaterloo.ca>
[WebKit-https.git] / SunSpider / ChangeLog
1 2008-09-02  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
2
3         Rubber-stamped by Oliver Hunt.
4
5         Make SunSpider work with JavaScript shells that don't take filenames
6         after a '-f' argument.
7
8         * sunspider:
9
10 2008-06-07  Cameron Zwarich  <cwzwarich@uwaterloo.ca>
11
12         Reviewed by Timothy.
13
14         Remove the --squirrelfish option from SunSpider, as it is no longer needed.
15
16         * sunspider:
17         * tests/LIST-SQUIRRELFISH: Removed.
18
19 2008-06-03  Alexey Proskuryakov  <ap@webkit.org>
20
21         Rubber-stamped by Mark Rowe.
22
23         Roll out r31215 - with Mac OS X 10.5.3, there is no need to pause update daemon any more.
24
25         * sunspider:
26
27 2008-04-15  Maciej Stachowiak  <mjs@apple.com>
28
29         Rubber stamped by Oliver.
30         
31         - add newly working tests to squirrelfish list
32
33         * tests/LIST-SQUIRRELFISH: Add 3d-raytrace and string-unpack-code
34
35 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
36
37         Rubber stamped by Oliver.
38         
39         - move ubench tests properly.
40
41         * tests/ubench: Added.
42         * tests/ubench/function-closure.js: Added.
43         * tests/ubench/function-empty.js: Added.
44         * tests/ubench/function-missing-args.js: Added.
45         * tests/ubench/function-sum.js: Added.
46         * tests/ubench/loop-empty-resolve.js: Added.
47         * tests/ubench/loop-empty.js: Added.
48         * tests/ubench/loop-sum.js: Added.
49
50 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
51
52         Rubber stamped by Oliver.
53
54         * sunspider: Don't force --runs=1 in squirrelfish mode any more.
55
56 2008-04-14  Maciej Stachowiak  <mjs@apple.com>
57
58         Reviewed by Oliver.
59
60         - make --squirrelfish mode use the real harness, but a reduced set of tests
61
62         Also add a new ubench mode which runs the older squirrelfish microbenchmarks.
63
64         * sunspider:
65         * tests/LIST-SQUIRRELFISH:
66         * tests/LIST-UBENCH: Added.
67         * tests/squirrelfish: Moved to tests/ubench
68
69 2008-04-14  Geoffrey Garen  <ggaren@apple.com>
70
71         Reviewed by Maciej Stachowiak.
72         
73         Removed an unused test.
74
75         * tests/squirrelfish/loop-resolve.js: Removed.
76
77 2008-04-10  Maciej Stachowiak  <mjs@apple.com>
78
79         Reviewed by Oliver.
80         
81         Add newly runnable tests to --squirrelfish mode.
82
83         * tests/LIST-SQUIRRELFISH:
84
85 2008-04-06  Geoffrey Garen  <ggaren@apple.com>
86
87         Reviewed by Maciej Stachowiak.
88         
89         A little more love for --squirrelfish mode:
90         
91         Fixed a misplaced sort. Results properly sort now.
92         
93         Added a test list just for squirrelfish, and updated it to include
94         all currently passing SunSpider tests.
95         
96         Fixed the pruning regexp to match 3d-morph.js.
97
98         * sunspider:
99
100 2008-03-26  Geoffrey Garen  <ggaren@apple.com>
101
102         Reviewed by Oliver Hunt.
103         
104         --squirrelfish mode: pared down tests for squirrelfish to chew on.
105
106         * sunspider:
107         * tests/LIST:
108         * tests/squirrelfish: Added.
109         * tests/squirrelfish/function-closure.js: Added.
110         * tests/squirrelfish/function-empty.js: Added.
111         * tests/squirrelfish/function-missing-args.js: Added.
112         * tests/squirrelfish/function-sum.js: Added.
113         * tests/squirrelfish/loop-empty-resolve.js: Added.
114         * tests/squirrelfish/loop-empty.js: Added.
115         * tests/squirrelfish/loop-resolve.js: Added.
116         * tests/squirrelfish/loop-sum.js: Added.
117
118 2008-03-21  Alexey Proskuryakov  <ap@webkit.org>
119
120         Reviewed by Geoff.
121
122         * sunspider: Pause update daemon on Darwin to make results reliable
123         again (workaround for <rdar://problem/5811127>).
124
125 2008-02-07  Eric Seidel  <eric@webkit.org>
126
127         Reviewed by Sam.
128
129         * sunspider: add an error message when ./tests/LIST is not found
130
131 2007-12-29  Alp Toker  <alp@atoker.com>
132
133         Fix typo.
134
135         * hosted/sunspider.html:
136
137 2007-12-29  Alp Toker  <alp@atoker.com>
138
139         Reviewed by Maciej.
140
141         Provide the current URL in the SunSpider results page. Useful for
142         copying benchmark results out of browsers without a location bar or
143         where the location bar can't handle long URLs.
144
145         * hosted/sunspider-results.html:
146
147 2007-12-20  Eric Seidel  <eric@webkit.org>
148
149         Reviewed by mjs.
150
151         * sunspider: increase default number of runs for more accurate numbers
152
153 2007-12-18  Mark Rowe  <mrowe@apple.com>
154
155         Fix typo.
156
157         * hosted/sunspider.html:
158
159 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
160
161         Reviewed by Eric.
162         
163         - give an extra digit of precision for small differences.
164
165         * resources/sunspider-compare-results.js:
166
167 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
168
169         Reviewed by Eric.
170         
171         - work around IE layout bugs
172
173         * hosted/sunspider.css:
174
175 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
176
177         Reviewed by Mark.
178         
179         - fix compare feature
180
181         * hosted/sunspider-results.html:
182
183 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
184
185         Reviewed by Mark.
186         
187         - Change the format of the output URL to be shorter, to meet IE constraints
188
189         * hosted/sunspider-driver.html:
190         * hosted/sunspider-results.html:
191
192 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
193
194         Reviewed by Oliver.
195         
196         - Add some explanatory test and a dumb logo to SunSpider web hosted version
197
198         * hosted/sunspider-driver.html:
199         * hosted/sunspider-results.html:
200         * hosted/sunspider.css:
201         * hosted/sunspider.html:
202
203 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
204
205         Reviewed by Oliver.
206         
207         - Make the web-hosted version of SunSpider look a little bit less drab.
208
209         * hosted/sunspider-driver.html:
210         * hosted/sunspider-results.html:
211         * hosted/sunspider.css: Added.
212         * hosted/sunspider.html:
213         * resources/TEMPLATE.html:
214
215 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
216
217         Reviewed by Sam.
218         
219         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
220         
221         A sample of what new-style results would look like:
222         
223         TEST                   COMPARISON            FROM                 TO             DETAILS
224         ...
225         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
226         ...
227             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
228         
229         * resources/sunspider-compare-results.js:
230
231 2007-11-14  Eric Seidel  <eric@webkit.org>
232
233         Reviewed by Sam.
234
235         * sunspider: add --shark-cache for L2 Cache Miss profiling
236
237 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
238
239         Reviewed by Darin.
240
241         http://bugs.webkit.org/show_bug.cgi?id=15925
242         SunSpider should check for Shark being installed
243
244         * sunspider: Explicitly test that shark command line tool is installed.
245
246 2007-11-04  Darin Adler  <darin@apple.com>
247
248         * hosted/sunspider.html: Tweaked formatting a bit.
249
250 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
251
252         Reviewed by Darin.
253
254         - Add compare mode and a description of the benchmark to the hosted version.
255
256         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
257         * hosted/sunspider.html: Added intro text.
258         * make-hosted: Copy new file.
259         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
260         a function.
261         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
262         expects.
263         * sunspider-compare-results: Use sunspider-standalone-compare.js.
264
265 2007-11-02  Darin Adler  <darin@apple.com>
266
267         * sunspider: Changed "--base" to "--set-baseline".
268
269 2007-11-02  Darin Adler  <darin@apple.com>
270
271         Reviewed by Maciej.
272
273         * sunspider: Added a "--base" option that records the current test as a baseline.
274         Moved the code to find the newest .mshark file into a function (since I needed to use
275         it in the other script and I prefer to copy/paste a function).
276
277         * sunspider-compare-results: Added logic to compare the baseline with the last
278         run when you don't pass any parameters.
279
280 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
281
282         Reviewed by Eric.
283         
284         - Added list of ECMAScript3 features that SunSpider does not cover currently.
285
286         * UNCOVERED: Added.
287
288 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
289
290         Reviewed by Oliver.
291         
292         - Give better error messages when sunspider-compare-results fails
293
294         * sunspider-compare-results:
295
296 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
297
298         Reviewed by Darin.
299         
300         - fix Date tests to work in command-line SpiderMonkey
301
302         * tests/date-format-tofte.js:
303         * tests/date-format-xparb.js:
304
305 2007-10-29  Darin Adler  <darin@apple.com>
306
307         Reviewed by Maciej.
308
309         * sunspider: Added --shark20 option, to run Shark at its highest sample
310         resolution instead of its default.
311
312 2007-10-28  Adam Roben  <aroben@apple.com>
313
314         Fix SunSpider on Windows
315
316         Reviewed by Eric.
317
318         * sunspider: Don't use colons in filenames.
319
320 2007-10-28  Darin Adler  <darin@apple.com>
321
322         Reviewed by Adam.
323
324         * resources/sunspider-analyze-results.js: Tweak the output format a little.
325         Change so that when there's only one run we don't write out confidence
326         intervals at all rather than writing "NaN" over and over again.
327
328         * sunspider: Use the time and date as a suffix on the results file, that way
329         all the results are left behind in the tmp directory. This will make it easy
330         for us to add features that compare past results in the future. Also rename
331         the .mshark files using the same scheme.
332
333         * sunspider-compare-results: Relax the parsing rules so we can parse the
334         results file exactly as it's emitted from sunspider.
335
336 2007-10-25  Adam Roben  <aroben@apple.com>
337
338         Strip carriage returns from results
339
340         Carriage returns were screwing up sunspider-compare-results on
341         Windows.
342
343         Rubberstamped by Oliver.
344
345         * sunspider:
346
347 2007-10-24  Darin Adler  <darin@apple.com>
348
349         Reviewed by Eric.
350
351         * sunspider: Open the profile in shark after completing a run.
352         * sunspider-compare-results: Tiny perl tweaks.
353
354 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
355
356         Reviewed by Darin.
357
358         - Don't hardcode my path to testkjs
359         
360         * sunspider-compare-results:
361
362 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
363
364         Reviewed by Darin.
365         
366         - Add a compare script to compare results, and improve formatting of standard results a bit.
367
368         * resources/sunspider-analyze-results.js:
369         * resources/sunspider-compare-results.js: Added.
370         * sunspider-compare-results: Added.
371
372 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
373
374         Reviewed by Darin.
375
376         * resources/sunspider-analyze-results.js: Do the error range
377         properly, using the t-distribution instead of 1.96 (which was
378         based on the normal distribution).
379         
380         * sunspider: Print results in a way that will be friendlier to the
381         soon-to-come compare mode.
382
383 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
384
385         Reviewed by Review.
386         
387         - add more tests (probably the final set for now)
388         
389         Already balanced. Added date, regexp, control flow, and a few
390         more string and object/array access tests.
391         
392         * tests/LIST:
393         * tests/access-fannkuch.js: Added.
394         * tests/access-nbody.js: Added.
395         * tests/controlflow-recursive.js: Added.
396         * tests/date-format-tofte.js: Added.
397         * tests/date-format-xparb.js: Added.
398         * tests/regexp-dna.js: Added.
399         * tests/string-validate-input.js: Added.
400
401 2007-10-21  Eric Seidel  <eric@webkit.org>
402
403         Reviewed by mjs.
404
405         * sunspider: make --shark only sample from testkjs
406
407 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
408
409         Rubber stamped by Eric.
410         
411         - added crypto tests (already balanced)
412
413         * tests/LIST:
414         * tests/crypto-aes.js: Added.
415         * tests/crypto-md5.js: Added.
416         * tests/crypto-sha1.js: Added.
417
418 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
419
420         Reviewed by Adam.
421         
422         - rebalance new tests
423         
424         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
425         
426         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
427
428         This balances them with the existing tests.
429
430         * tests/access-binary-trees.js:
431         * tests/access-nsieve.js:
432         * tests/bitops-nsieve-bits.js:
433         * tests/math-partial-sums.js:
434         * tests/math-spectral-norm.js:
435         * tests/string-fasta.js:
436
437 2007-10-20  Eric Seidel  <eric@webkit.org>
438
439         Reviewed by darin.
440         
441         Add improved argument handling to sunspider, including
442         --runs=<number>, --tests=<pattern, --shark, and --help
443         Also re-factor code into subroutines
444
445         * sunspider: improved argument handling
446
447 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
448
449         Reviewed by Mark.
450
451         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
452         
453         * TODO:
454         * tests/LIST:
455         * tests/access-binary-trees.js: Added.
456         * tests/access-nsieve.js: Added.
457         * tests/bitops-nsieve-bits.js: Added.
458         * tests/math-partial-sums.js: Added.
459         * tests/math-spectral-norm.js: Added.
460         * tests/string-fasta.js: Added.
461
462 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
463
464         Reviewed by Darin.
465
466         * TODO: Updated to note areas that are now well-covered.
467         
468         - Change the tests and the driver to avoid leaving large object
469         graphs hanging around, since that throws off the subsequent tests.
470         
471         * resources/sunspider-standalone-driver.js:
472         * tests/3d-cube.js:
473         * tests/3d-morph.js:
474         * tests/3d-raytrace.js:
475         * tests/bitops-bitwise-and.js:
476         * tests/string-base64.js:
477         * tests/string-tagcloud.js:
478
479 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
480
481         Reviewed by Nikolas.
482         
483         - Rebalance test complexity.
484         
485         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
486         
487         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
488
489         However, for some tests, IE7 was a huge outlier compared to other
490         major browsers. For those tests, I used Firefox 2.0.0.8 on the
491         same OS and hardware as a normalization basis.
492         
493         * tests/3d-morph.js:
494         * tests/3d-raytrace.js:
495         * tests/bitops-3bit-bits-in-byte.js:
496         * tests/bitops-bits-in-byte.js:
497         * tests/bitops-bitwise-and.js:
498         * tests/math-cordic.js:
499         * tests/string-base64.js:
500         * tests/string-tagcloud.js:
501         * tests/string-unpack-code.js:
502
503 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
504
505         Reviewed by Oliver.
506         
507         - Fix the browser-hosted driver to be compatible with Safari 2
508
509         * hosted/sunspider-driver.html:
510
511 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
512
513         Rubber stamped by Hyatt.
514         
515         - Add BSD license notice to cordic test, the original author said to do so.
516
517         * tests/math-cordic.js:
518
519 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
520
521         Reviewed by Eric.
522         
523         - Support Windows and Windows browsers.
524
525         * hosted/sunspider-results.html: Display results correctly in IE.
526         * sunspider: tweaks to work on Windows, and with spaces in the path.
527
528 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
529
530         Rubber stamped by Oliver.
531
532         * tests/string-tagcloud.js: Remove stray print()
533
534 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
535
536         Reviewed by Oliver.
537         
538         Add some string test cases.
539
540         * tests/string-base64.js: Added. Does base64 encoding and decoding.
541         * tests/string-tagcloud.js: Added. Parses a JSON string and
542         generates tagcloud HTML markup based on the data.        
543         * tests/string-unpack-code.js: Added. Unpack compressed versions of
544         four of the most popular JavaScript libraries.
545         * tests/LIST: Added the new tests.
546
547 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
548
549         Reviewed by Anders.
550
551         * TODO: Updated with info on how to do the stats properly.
552
553         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
554         avoid overly slowing down the benchmark.
555
556         Added some 3D-related test cases.
557         
558         * tests/3d-cube.js: Added.
559         * tests/3d-morph.js: Added.
560         * tests/3d-raytrace.js: Added.
561         * tests/LIST:
562
563 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
564
565         Reviewed by Darin.
566
567         - New JavaScript benchmark
568         http://bugs.webkit.org/show_bug.cgi?id=15515
569         
570         A start on a new JavaScript benchmark, with standalone and
571         browser-hosted drivers.
572         
573         The test content is still incomplete and the drivers could use
574         some more features but there is enough here to make a good start.
575         
576         The drivers compute a 95% confidence interval on the mean for the
577         whole test, each category, and each individual test to make it
578         easier to tell whether differences are statistically
579         significant. The confidence interval can be narrowed by running
580         the test more times.
581         
582         * TODO: Added. Remaining things that need to be done.
583         * sunspider: Added. Perl script that acts as the standalone test driver.
584         * resources/sunspider-standalone-driver.js: Added. JavaScript part
585         of standalone test driver.
586         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
587         analysis code.
588         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
589         * make-hosted: Added. Script to generate browser-hosted tests.
590         * hosted/sunspider.html: Added. Start page for browser-hosted test.
591         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
592         * hosted/sunspider-results.html: Added. Results page for browser-hosted
593         tests.
594         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
595         tests.
596         * tests/LIST: Added. List of tests to use.
597         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
598         * tests/bitops-bits-in-byte.js: Added. Ditto.
599         * tests/bitops-bitwise-and.js: Added. Ditto.
600         * tests/math-cordic.js: Added. Ditto.
601