960d49158cb827d26174332914aa8b90ac5b332a
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-29  Alp Toker  <alp@atoker.com>
2
3         Reviewed by Maciej.
4
5         Provide the current URL in the SunSpider results page. Useful for
6         copying benchmark results out of browsers without a location bar or
7         where the location bar can't handle long URLs.
8
9         * hosted/sunspider-results.html:
10
11 2007-12-20  Eric Seidel  <eric@webkit.org>
12
13         Reviewed by mjs.
14
15         * sunspider: increase default number of runs for more accurate numbers
16
17 2007-12-18  Mark Rowe  <mrowe@apple.com>
18
19         Fix typo.
20
21         * hosted/sunspider.html:
22
23 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
24
25         Reviewed by Eric.
26         
27         - give an extra digit of precision for small differences.
28
29         * resources/sunspider-compare-results.js:
30
31 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
32
33         Reviewed by Eric.
34         
35         - work around IE layout bugs
36
37         * hosted/sunspider.css:
38
39 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
40
41         Reviewed by Mark.
42         
43         - fix compare feature
44
45         * hosted/sunspider-results.html:
46
47 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
48
49         Reviewed by Mark.
50         
51         - Change the format of the output URL to be shorter, to meet IE constraints
52
53         * hosted/sunspider-driver.html:
54         * hosted/sunspider-results.html:
55
56 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
57
58         Reviewed by Oliver.
59         
60         - Add some explanatory test and a dumb logo to SunSpider web hosted version
61
62         * hosted/sunspider-driver.html:
63         * hosted/sunspider-results.html:
64         * hosted/sunspider.css:
65         * hosted/sunspider.html:
66
67 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
68
69         Reviewed by Oliver.
70         
71         - Make the web-hosted version of SunSpider look a little bit less drab.
72
73         * hosted/sunspider-driver.html:
74         * hosted/sunspider-results.html:
75         * hosted/sunspider.css: Added.
76         * hosted/sunspider.html:
77         * resources/TEMPLATE.html:
78
79 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
80
81         Reviewed by Sam.
82         
83         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
84         
85         A sample of what new-style results would look like:
86         
87         TEST                   COMPARISON            FROM                 TO             DETAILS
88         ...
89         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
90         ...
91             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
92         
93         * resources/sunspider-compare-results.js:
94
95 2007-11-14  Eric Seidel  <eric@webkit.org>
96
97         Reviewed by Sam.
98
99         * sunspider: add --shark-cache for L2 Cache Miss profiling
100
101 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
102
103         Reviewed by Darin.
104
105         http://bugs.webkit.org/show_bug.cgi?id=15925
106         SunSpider should check for Shark being installed
107
108         * sunspider: Explicitly test that shark command line tool is installed.
109
110 2007-11-04  Darin Adler  <darin@apple.com>
111
112         * hosted/sunspider.html: Tweaked formatting a bit.
113
114 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
115
116         Reviewed by Darin.
117
118         - Add compare mode and a description of the benchmark to the hosted version.
119
120         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
121         * hosted/sunspider.html: Added intro text.
122         * make-hosted: Copy new file.
123         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
124         a function.
125         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
126         expects.
127         * sunspider-compare-results: Use sunspider-standalone-compare.js.
128
129 2007-11-02  Darin Adler  <darin@apple.com>
130
131         * sunspider: Changed "--base" to "--set-baseline".
132
133 2007-11-02  Darin Adler  <darin@apple.com>
134
135         Reviewed by Maciej.
136
137         * sunspider: Added a "--base" option that records the current test as a baseline.
138         Moved the code to find the newest .mshark file into a function (since I needed to use
139         it in the other script and I prefer to copy/paste a function).
140
141         * sunspider-compare-results: Added logic to compare the baseline with the last
142         run when you don't pass any parameters.
143
144 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
145
146         Reviewed by Eric.
147         
148         - Added list of ECMAScript3 features that SunSpider does not cover currently.
149
150         * UNCOVERED: Added.
151
152 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
153
154         Reviewed by Oliver.
155         
156         - Give better error messages when sunspider-compare-results fails
157
158         * sunspider-compare-results:
159
160 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
161
162         Reviewed by Darin.
163         
164         - fix Date tests to work in command-line SpiderMonkey
165
166         * tests/date-format-tofte.js:
167         * tests/date-format-xparb.js:
168
169 2007-10-29  Darin Adler  <darin@apple.com>
170
171         Reviewed by Maciej.
172
173         * sunspider: Added --shark20 option, to run Shark at its highest sample
174         resolution instead of its default.
175
176 2007-10-28  Adam Roben  <aroben@apple.com>
177
178         Fix SunSpider on Windows
179
180         Reviewed by Eric.
181
182         * sunspider: Don't use colons in filenames.
183
184 2007-10-28  Darin Adler  <darin@apple.com>
185
186         Reviewed by Adam.
187
188         * resources/sunspider-analyze-results.js: Tweak the output format a little.
189         Change so that when there's only one run we don't write out confidence
190         intervals at all rather than writing "NaN" over and over again.
191
192         * sunspider: Use the time and date as a suffix on the results file, that way
193         all the results are left behind in the tmp directory. This will make it easy
194         for us to add features that compare past results in the future. Also rename
195         the .mshark files using the same scheme.
196
197         * sunspider-compare-results: Relax the parsing rules so we can parse the
198         results file exactly as it's emitted from sunspider.
199
200 2007-10-25  Adam Roben  <aroben@apple.com>
201
202         Strip carriage returns from results
203
204         Carriage returns were screwing up sunspider-compare-results on
205         Windows.
206
207         Rubberstamped by Oliver.
208
209         * sunspider:
210
211 2007-10-24  Darin Adler  <darin@apple.com>
212
213         Reviewed by Eric.
214
215         * sunspider: Open the profile in shark after completing a run.
216         * sunspider-compare-results: Tiny perl tweaks.
217
218 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
219
220         Reviewed by Darin.
221
222         - Don't hardcode my path to testkjs
223         
224         * sunspider-compare-results:
225
226 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
227
228         Reviewed by Darin.
229         
230         - Add a compare script to compare results, and improve formatting of standard results a bit.
231
232         * resources/sunspider-analyze-results.js:
233         * resources/sunspider-compare-results.js: Added.
234         * sunspider-compare-results: Added.
235
236 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
237
238         Reviewed by Darin.
239
240         * resources/sunspider-analyze-results.js: Do the error range
241         properly, using the t-distribution instead of 1.96 (which was
242         based on the normal distribution).
243         
244         * sunspider: Print results in a way that will be friendlier to the
245         soon-to-come compare mode.
246
247 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
248
249         Reviewed by Review.
250         
251         - add more tests (probably the final set for now)
252         
253         Already balanced. Added date, regexp, control flow, and a few
254         more string and object/array access tests.
255         
256         * tests/LIST:
257         * tests/access-fannkuch.js: Added.
258         * tests/access-nbody.js: Added.
259         * tests/controlflow-recursive.js: Added.
260         * tests/date-format-tofte.js: Added.
261         * tests/date-format-xparb.js: Added.
262         * tests/regexp-dna.js: Added.
263         * tests/string-validate-input.js: Added.
264
265 2007-10-21  Eric Seidel  <eric@webkit.org>
266
267         Reviewed by mjs.
268
269         * sunspider: make --shark only sample from testkjs
270
271 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
272
273         Rubber stamped by Eric.
274         
275         - added crypto tests (already balanced)
276
277         * tests/LIST:
278         * tests/crypto-aes.js: Added.
279         * tests/crypto-md5.js: Added.
280         * tests/crypto-sha1.js: Added.
281
282 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
283
284         Reviewed by Adam.
285         
286         - rebalance new tests
287         
288         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
289         
290         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
291
292         This balances them with the existing tests.
293
294         * tests/access-binary-trees.js:
295         * tests/access-nsieve.js:
296         * tests/bitops-nsieve-bits.js:
297         * tests/math-partial-sums.js:
298         * tests/math-spectral-norm.js:
299         * tests/string-fasta.js:
300
301 2007-10-20  Eric Seidel  <eric@webkit.org>
302
303         Reviewed by darin.
304         
305         Add improved argument handling to sunspider, including
306         --runs=<number>, --tests=<pattern, --shark, and --help
307         Also re-factor code into subroutines
308
309         * sunspider: improved argument handling
310
311 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
312
313         Reviewed by Mark.
314
315         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
316         
317         * TODO:
318         * tests/LIST:
319         * tests/access-binary-trees.js: Added.
320         * tests/access-nsieve.js: Added.
321         * tests/bitops-nsieve-bits.js: Added.
322         * tests/math-partial-sums.js: Added.
323         * tests/math-spectral-norm.js: Added.
324         * tests/string-fasta.js: Added.
325
326 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
327
328         Reviewed by Darin.
329
330         * TODO: Updated to note areas that are now well-covered.
331         
332         - Change the tests and the driver to avoid leaving large object
333         graphs hanging around, since that throws off the subsequent tests.
334         
335         * resources/sunspider-standalone-driver.js:
336         * tests/3d-cube.js:
337         * tests/3d-morph.js:
338         * tests/3d-raytrace.js:
339         * tests/bitops-bitwise-and.js:
340         * tests/string-base64.js:
341         * tests/string-tagcloud.js:
342
343 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
344
345         Reviewed by Nikolas.
346         
347         - Rebalance test complexity.
348         
349         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
350         
351         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
352
353         However, for some tests, IE7 was a huge outlier compared to other
354         major browsers. For those tests, I used Firefox 2.0.0.8 on the
355         same OS and hardware as a normalization basis.
356         
357         * tests/3d-morph.js:
358         * tests/3d-raytrace.js:
359         * tests/bitops-3bit-bits-in-byte.js:
360         * tests/bitops-bits-in-byte.js:
361         * tests/bitops-bitwise-and.js:
362         * tests/math-cordic.js:
363         * tests/string-base64.js:
364         * tests/string-tagcloud.js:
365         * tests/string-unpack-code.js:
366
367 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
368
369         Reviewed by Oliver.
370         
371         - Fix the browser-hosted driver to be compatible with Safari 2
372
373         * hosted/sunspider-driver.html:
374
375 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
376
377         Rubber stamped by Hyatt.
378         
379         - Add BSD license notice to cordic test, the original author said to do so.
380
381         * tests/math-cordic.js:
382
383 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
384
385         Reviewed by Eric.
386         
387         - Support Windows and Windows browsers.
388
389         * hosted/sunspider-results.html: Display results correctly in IE.
390         * sunspider: tweaks to work on Windows, and with spaces in the path.
391
392 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
393
394         Rubber stamped by Oliver.
395
396         * tests/string-tagcloud.js: Remove stray print()
397
398 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
399
400         Reviewed by Oliver.
401         
402         Add some string test cases.
403
404         * tests/string-base64.js: Added. Does base64 encoding and decoding.
405         * tests/string-tagcloud.js: Added. Parses a JSON string and
406         generates tagcloud HTML markup based on the data.        
407         * tests/string-unpack-code.js: Added. Unpack compressed versions of
408         four of the most popular JavaScript libraries.
409         * tests/LIST: Added the new tests.
410
411 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
412
413         Reviewed by Anders.
414
415         * TODO: Updated with info on how to do the stats properly.
416
417         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
418         avoid overly slowing down the benchmark.
419
420         Added some 3D-related test cases.
421         
422         * tests/3d-cube.js: Added.
423         * tests/3d-morph.js: Added.
424         * tests/3d-raytrace.js: Added.
425         * tests/LIST:
426
427 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
428
429         Reviewed by Darin.
430
431         - New JavaScript benchmark
432         http://bugs.webkit.org/show_bug.cgi?id=15515
433         
434         A start on a new JavaScript benchmark, with standalone and
435         browser-hosted drivers.
436         
437         The test content is still incomplete and the drivers could use
438         some more features but there is enough here to make a good start.
439         
440         The drivers compute a 95% confidence interval on the mean for the
441         whole test, each category, and each individual test to make it
442         easier to tell whether differences are statistically
443         significant. The confidence interval can be narrowed by running
444         the test more times.
445         
446         * TODO: Added. Remaining things that need to be done.
447         * sunspider: Added. Perl script that acts as the standalone test driver.
448         * resources/sunspider-standalone-driver.js: Added. JavaScript part
449         of standalone test driver.
450         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
451         analysis code.
452         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
453         * make-hosted: Added. Script to generate browser-hosted tests.
454         * hosted/sunspider.html: Added. Start page for browser-hosted test.
455         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
456         * hosted/sunspider-results.html: Added. Results page for browser-hosted
457         tests.
458         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
459         tests.
460         * tests/LIST: Added. List of tests to use.
461         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
462         * tests/bitops-bits-in-byte.js: Added. Ditto.
463         * tests/bitops-bitwise-and.js: Added. Ditto.
464         * tests/math-cordic.js: Added. Ditto.
465