Reviewed by Sam.
[WebKit-https.git] / SunSpider / ChangeLog
1 2008-02-07  Eric Seidel  <eric@webkit.org>
2
3         Reviewed by Sam.
4
5         * sunspider: add an error message when ./tests/LIST is not found
6
7 2007-12-29  Alp Toker  <alp@atoker.com>
8
9         Fix typo.
10
11         * hosted/sunspider.html:
12
13 2007-12-29  Alp Toker  <alp@atoker.com>
14
15         Reviewed by Maciej.
16
17         Provide the current URL in the SunSpider results page. Useful for
18         copying benchmark results out of browsers without a location bar or
19         where the location bar can't handle long URLs.
20
21         * hosted/sunspider-results.html:
22
23 2007-12-20  Eric Seidel  <eric@webkit.org>
24
25         Reviewed by mjs.
26
27         * sunspider: increase default number of runs for more accurate numbers
28
29 2007-12-18  Mark Rowe  <mrowe@apple.com>
30
31         Fix typo.
32
33         * hosted/sunspider.html:
34
35 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
36
37         Reviewed by Eric.
38         
39         - give an extra digit of precision for small differences.
40
41         * resources/sunspider-compare-results.js:
42
43 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
44
45         Reviewed by Eric.
46         
47         - work around IE layout bugs
48
49         * hosted/sunspider.css:
50
51 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
52
53         Reviewed by Mark.
54         
55         - fix compare feature
56
57         * hosted/sunspider-results.html:
58
59 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
60
61         Reviewed by Mark.
62         
63         - Change the format of the output URL to be shorter, to meet IE constraints
64
65         * hosted/sunspider-driver.html:
66         * hosted/sunspider-results.html:
67
68 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
69
70         Reviewed by Oliver.
71         
72         - Add some explanatory test and a dumb logo to SunSpider web hosted version
73
74         * hosted/sunspider-driver.html:
75         * hosted/sunspider-results.html:
76         * hosted/sunspider.css:
77         * hosted/sunspider.html:
78
79 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
80
81         Reviewed by Oliver.
82         
83         - Make the web-hosted version of SunSpider look a little bit less drab.
84
85         * hosted/sunspider-driver.html:
86         * hosted/sunspider-results.html:
87         * hosted/sunspider.css: Added.
88         * hosted/sunspider.html:
89         * resources/TEMPLATE.html:
90
91 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
92
93         Reviewed by Sam.
94         
95         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
96         
97         A sample of what new-style results would look like:
98         
99         TEST                   COMPARISON            FROM                 TO             DETAILS
100         ...
101         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
102         ...
103             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
104         
105         * resources/sunspider-compare-results.js:
106
107 2007-11-14  Eric Seidel  <eric@webkit.org>
108
109         Reviewed by Sam.
110
111         * sunspider: add --shark-cache for L2 Cache Miss profiling
112
113 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
114
115         Reviewed by Darin.
116
117         http://bugs.webkit.org/show_bug.cgi?id=15925
118         SunSpider should check for Shark being installed
119
120         * sunspider: Explicitly test that shark command line tool is installed.
121
122 2007-11-04  Darin Adler  <darin@apple.com>
123
124         * hosted/sunspider.html: Tweaked formatting a bit.
125
126 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
127
128         Reviewed by Darin.
129
130         - Add compare mode and a description of the benchmark to the hosted version.
131
132         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
133         * hosted/sunspider.html: Added intro text.
134         * make-hosted: Copy new file.
135         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
136         a function.
137         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
138         expects.
139         * sunspider-compare-results: Use sunspider-standalone-compare.js.
140
141 2007-11-02  Darin Adler  <darin@apple.com>
142
143         * sunspider: Changed "--base" to "--set-baseline".
144
145 2007-11-02  Darin Adler  <darin@apple.com>
146
147         Reviewed by Maciej.
148
149         * sunspider: Added a "--base" option that records the current test as a baseline.
150         Moved the code to find the newest .mshark file into a function (since I needed to use
151         it in the other script and I prefer to copy/paste a function).
152
153         * sunspider-compare-results: Added logic to compare the baseline with the last
154         run when you don't pass any parameters.
155
156 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
157
158         Reviewed by Eric.
159         
160         - Added list of ECMAScript3 features that SunSpider does not cover currently.
161
162         * UNCOVERED: Added.
163
164 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
165
166         Reviewed by Oliver.
167         
168         - Give better error messages when sunspider-compare-results fails
169
170         * sunspider-compare-results:
171
172 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
173
174         Reviewed by Darin.
175         
176         - fix Date tests to work in command-line SpiderMonkey
177
178         * tests/date-format-tofte.js:
179         * tests/date-format-xparb.js:
180
181 2007-10-29  Darin Adler  <darin@apple.com>
182
183         Reviewed by Maciej.
184
185         * sunspider: Added --shark20 option, to run Shark at its highest sample
186         resolution instead of its default.
187
188 2007-10-28  Adam Roben  <aroben@apple.com>
189
190         Fix SunSpider on Windows
191
192         Reviewed by Eric.
193
194         * sunspider: Don't use colons in filenames.
195
196 2007-10-28  Darin Adler  <darin@apple.com>
197
198         Reviewed by Adam.
199
200         * resources/sunspider-analyze-results.js: Tweak the output format a little.
201         Change so that when there's only one run we don't write out confidence
202         intervals at all rather than writing "NaN" over and over again.
203
204         * sunspider: Use the time and date as a suffix on the results file, that way
205         all the results are left behind in the tmp directory. This will make it easy
206         for us to add features that compare past results in the future. Also rename
207         the .mshark files using the same scheme.
208
209         * sunspider-compare-results: Relax the parsing rules so we can parse the
210         results file exactly as it's emitted from sunspider.
211
212 2007-10-25  Adam Roben  <aroben@apple.com>
213
214         Strip carriage returns from results
215
216         Carriage returns were screwing up sunspider-compare-results on
217         Windows.
218
219         Rubberstamped by Oliver.
220
221         * sunspider:
222
223 2007-10-24  Darin Adler  <darin@apple.com>
224
225         Reviewed by Eric.
226
227         * sunspider: Open the profile in shark after completing a run.
228         * sunspider-compare-results: Tiny perl tweaks.
229
230 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
231
232         Reviewed by Darin.
233
234         - Don't hardcode my path to testkjs
235         
236         * sunspider-compare-results:
237
238 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
239
240         Reviewed by Darin.
241         
242         - Add a compare script to compare results, and improve formatting of standard results a bit.
243
244         * resources/sunspider-analyze-results.js:
245         * resources/sunspider-compare-results.js: Added.
246         * sunspider-compare-results: Added.
247
248 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
249
250         Reviewed by Darin.
251
252         * resources/sunspider-analyze-results.js: Do the error range
253         properly, using the t-distribution instead of 1.96 (which was
254         based on the normal distribution).
255         
256         * sunspider: Print results in a way that will be friendlier to the
257         soon-to-come compare mode.
258
259 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
260
261         Reviewed by Review.
262         
263         - add more tests (probably the final set for now)
264         
265         Already balanced. Added date, regexp, control flow, and a few
266         more string and object/array access tests.
267         
268         * tests/LIST:
269         * tests/access-fannkuch.js: Added.
270         * tests/access-nbody.js: Added.
271         * tests/controlflow-recursive.js: Added.
272         * tests/date-format-tofte.js: Added.
273         * tests/date-format-xparb.js: Added.
274         * tests/regexp-dna.js: Added.
275         * tests/string-validate-input.js: Added.
276
277 2007-10-21  Eric Seidel  <eric@webkit.org>
278
279         Reviewed by mjs.
280
281         * sunspider: make --shark only sample from testkjs
282
283 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
284
285         Rubber stamped by Eric.
286         
287         - added crypto tests (already balanced)
288
289         * tests/LIST:
290         * tests/crypto-aes.js: Added.
291         * tests/crypto-md5.js: Added.
292         * tests/crypto-sha1.js: Added.
293
294 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
295
296         Reviewed by Adam.
297         
298         - rebalance new tests
299         
300         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
301         
302         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
303
304         This balances them with the existing tests.
305
306         * tests/access-binary-trees.js:
307         * tests/access-nsieve.js:
308         * tests/bitops-nsieve-bits.js:
309         * tests/math-partial-sums.js:
310         * tests/math-spectral-norm.js:
311         * tests/string-fasta.js:
312
313 2007-10-20  Eric Seidel  <eric@webkit.org>
314
315         Reviewed by darin.
316         
317         Add improved argument handling to sunspider, including
318         --runs=<number>, --tests=<pattern, --shark, and --help
319         Also re-factor code into subroutines
320
321         * sunspider: improved argument handling
322
323 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
324
325         Reviewed by Mark.
326
327         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
328         
329         * TODO:
330         * tests/LIST:
331         * tests/access-binary-trees.js: Added.
332         * tests/access-nsieve.js: Added.
333         * tests/bitops-nsieve-bits.js: Added.
334         * tests/math-partial-sums.js: Added.
335         * tests/math-spectral-norm.js: Added.
336         * tests/string-fasta.js: Added.
337
338 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
339
340         Reviewed by Darin.
341
342         * TODO: Updated to note areas that are now well-covered.
343         
344         - Change the tests and the driver to avoid leaving large object
345         graphs hanging around, since that throws off the subsequent tests.
346         
347         * resources/sunspider-standalone-driver.js:
348         * tests/3d-cube.js:
349         * tests/3d-morph.js:
350         * tests/3d-raytrace.js:
351         * tests/bitops-bitwise-and.js:
352         * tests/string-base64.js:
353         * tests/string-tagcloud.js:
354
355 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
356
357         Reviewed by Nikolas.
358         
359         - Rebalance test complexity.
360         
361         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
362         
363         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
364
365         However, for some tests, IE7 was a huge outlier compared to other
366         major browsers. For those tests, I used Firefox 2.0.0.8 on the
367         same OS and hardware as a normalization basis.
368         
369         * tests/3d-morph.js:
370         * tests/3d-raytrace.js:
371         * tests/bitops-3bit-bits-in-byte.js:
372         * tests/bitops-bits-in-byte.js:
373         * tests/bitops-bitwise-and.js:
374         * tests/math-cordic.js:
375         * tests/string-base64.js:
376         * tests/string-tagcloud.js:
377         * tests/string-unpack-code.js:
378
379 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
380
381         Reviewed by Oliver.
382         
383         - Fix the browser-hosted driver to be compatible with Safari 2
384
385         * hosted/sunspider-driver.html:
386
387 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
388
389         Rubber stamped by Hyatt.
390         
391         - Add BSD license notice to cordic test, the original author said to do so.
392
393         * tests/math-cordic.js:
394
395 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
396
397         Reviewed by Eric.
398         
399         - Support Windows and Windows browsers.
400
401         * hosted/sunspider-results.html: Display results correctly in IE.
402         * sunspider: tweaks to work on Windows, and with spaces in the path.
403
404 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
405
406         Rubber stamped by Oliver.
407
408         * tests/string-tagcloud.js: Remove stray print()
409
410 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
411
412         Reviewed by Oliver.
413         
414         Add some string test cases.
415
416         * tests/string-base64.js: Added. Does base64 encoding and decoding.
417         * tests/string-tagcloud.js: Added. Parses a JSON string and
418         generates tagcloud HTML markup based on the data.        
419         * tests/string-unpack-code.js: Added. Unpack compressed versions of
420         four of the most popular JavaScript libraries.
421         * tests/LIST: Added the new tests.
422
423 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
424
425         Reviewed by Anders.
426
427         * TODO: Updated with info on how to do the stats properly.
428
429         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
430         avoid overly slowing down the benchmark.
431
432         Added some 3D-related test cases.
433         
434         * tests/3d-cube.js: Added.
435         * tests/3d-morph.js: Added.
436         * tests/3d-raytrace.js: Added.
437         * tests/LIST:
438
439 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
440
441         Reviewed by Darin.
442
443         - New JavaScript benchmark
444         http://bugs.webkit.org/show_bug.cgi?id=15515
445         
446         A start on a new JavaScript benchmark, with standalone and
447         browser-hosted drivers.
448         
449         The test content is still incomplete and the drivers could use
450         some more features but there is enough here to make a good start.
451         
452         The drivers compute a 95% confidence interval on the mean for the
453         whole test, each category, and each individual test to make it
454         easier to tell whether differences are statistically
455         significant. The confidence interval can be narrowed by running
456         the test more times.
457         
458         * TODO: Added. Remaining things that need to be done.
459         * sunspider: Added. Perl script that acts as the standalone test driver.
460         * resources/sunspider-standalone-driver.js: Added. JavaScript part
461         of standalone test driver.
462         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
463         analysis code.
464         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
465         * make-hosted: Added. Script to generate browser-hosted tests.
466         * hosted/sunspider.html: Added. Start page for browser-hosted test.
467         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
468         * hosted/sunspider-results.html: Added. Results page for browser-hosted
469         tests.
470         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
471         tests.
472         * tests/LIST: Added. List of tests to use.
473         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
474         * tests/bitops-bits-in-byte.js: Added. Ditto.
475         * tests/bitops-bitwise-and.js: Added. Ditto.
476         * tests/math-cordic.js: Added. Ditto.
477