Reviewed by Oliver.
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Oliver.
4         
5         - Add some explanatory test and a dumb logo to SunSpider web hosted version
6
7         * hosted/sunspider-driver.html:
8         * hosted/sunspider-results.html:
9         * hosted/sunspider.css:
10         * hosted/sunspider.html:
11
12 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
13
14         Reviewed by Oliver.
15         
16         - Make the web-hosted version of SunSpider look a little bit less drab.
17
18         * hosted/sunspider-driver.html:
19         * hosted/sunspider-results.html:
20         * hosted/sunspider.css: Added.
21         * hosted/sunspider.html:
22         * resources/TEMPLATE.html:
23
24 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
25
26         Reviewed by Sam.
27         
28         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
29         
30         A sample of what new-style results would look like:
31         
32         TEST                   COMPARISON            FROM                 TO             DETAILS
33         ...
34         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
35         ...
36             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
37         
38         * resources/sunspider-compare-results.js:
39
40 2007-11-14  Eric Seidel  <eric@webkit.org>
41
42         Reviewed by Sam.
43
44         * sunspider: add --shark-cache for L2 Cache Miss profiling
45
46 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
47
48         Reviewed by Darin.
49
50         http://bugs.webkit.org/show_bug.cgi?id=15925
51         SunSpider should check for Shark being installed
52
53         * sunspider: Explicitly test that shark command line tool is installed.
54
55 2007-11-04  Darin Adler  <darin@apple.com>
56
57         * hosted/sunspider.html: Tweaked formatting a bit.
58
59 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
60
61         Reviewed by Darin.
62
63         - Add compare mode and a description of the benchmark to the hosted version.
64
65         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
66         * hosted/sunspider.html: Added intro text.
67         * make-hosted: Copy new file.
68         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
69         a function.
70         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
71         expects.
72         * sunspider-compare-results: Use sunspider-standalone-compare.js.
73
74 2007-11-02  Darin Adler  <darin@apple.com>
75
76         * sunspider: Changed "--base" to "--set-baseline".
77
78 2007-11-02  Darin Adler  <darin@apple.com>
79
80         Reviewed by Maciej.
81
82         * sunspider: Added a "--base" option that records the current test as a baseline.
83         Moved the code to find the newest .mshark file into a function (since I needed to use
84         it in the other script and I prefer to copy/paste a function).
85
86         * sunspider-compare-results: Added logic to compare the baseline with the last
87         run when you don't pass any parameters.
88
89 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
90
91         Reviewed by Eric.
92         
93         - Added list of ECMAScript3 features that SunSpider does not cover currently.
94
95         * UNCOVERED: Added.
96
97 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
98
99         Reviewed by Oliver.
100         
101         - Give better error messages when sunspider-compare-results fails
102
103         * sunspider-compare-results:
104
105 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
106
107         Reviewed by Darin.
108         
109         - fix Date tests to work in command-line SpiderMonkey
110
111         * tests/date-format-tofte.js:
112         * tests/date-format-xparb.js:
113
114 2007-10-29  Darin Adler  <darin@apple.com>
115
116         Reviewed by Maciej.
117
118         * sunspider: Added --shark20 option, to run Shark at its highest sample
119         resolution instead of its default.
120
121 2007-10-28  Adam Roben  <aroben@apple.com>
122
123         Fix SunSpider on Windows
124
125         Reviewed by Eric.
126
127         * sunspider: Don't use colons in filenames.
128
129 2007-10-28  Darin Adler  <darin@apple.com>
130
131         Reviewed by Adam.
132
133         * resources/sunspider-analyze-results.js: Tweak the output format a little.
134         Change so that when there's only one run we don't write out confidence
135         intervals at all rather than writing "NaN" over and over again.
136
137         * sunspider: Use the time and date as a suffix on the results file, that way
138         all the results are left behind in the tmp directory. This will make it easy
139         for us to add features that compare past results in the future. Also rename
140         the .mshark files using the same scheme.
141
142         * sunspider-compare-results: Relax the parsing rules so we can parse the
143         results file exactly as it's emitted from sunspider.
144
145 2007-10-25  Adam Roben  <aroben@apple.com>
146
147         Strip carriage returns from results
148
149         Carriage returns were screwing up sunspider-compare-results on
150         Windows.
151
152         Rubberstamped by Oliver.
153
154         * sunspider:
155
156 2007-10-24  Darin Adler  <darin@apple.com>
157
158         Reviewed by Eric.
159
160         * sunspider: Open the profile in shark after completing a run.
161         * sunspider-compare-results: Tiny perl tweaks.
162
163 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
164
165         Reviewed by Darin.
166
167         - Don't hardcode my path to testkjs
168         
169         * sunspider-compare-results:
170
171 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
172
173         Reviewed by Darin.
174         
175         - Add a compare script to compare results, and improve formatting of standard results a bit.
176
177         * resources/sunspider-analyze-results.js:
178         * resources/sunspider-compare-results.js: Added.
179         * sunspider-compare-results: Added.
180
181 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
182
183         Reviewed by Darin.
184
185         * resources/sunspider-analyze-results.js: Do the error range
186         properly, using the t-distribution instead of 1.96 (which was
187         based on the normal distribution).
188         
189         * sunspider: Print results in a way that will be friendlier to the
190         soon-to-come compare mode.
191
192 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
193
194         Reviewed by Review.
195         
196         - add more tests (probably the final set for now)
197         
198         Already balanced. Added date, regexp, control flow, and a few
199         more string and object/array access tests.
200         
201         * tests/LIST:
202         * tests/access-fannkuch.js: Added.
203         * tests/access-nbody.js: Added.
204         * tests/controlflow-recursive.js: Added.
205         * tests/date-format-tofte.js: Added.
206         * tests/date-format-xparb.js: Added.
207         * tests/regexp-dna.js: Added.
208         * tests/string-validate-input.js: Added.
209
210 2007-10-21  Eric Seidel  <eric@webkit.org>
211
212         Reviewed by mjs.
213
214         * sunspider: make --shark only sample from testkjs
215
216 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
217
218         Rubber stamped by Eric.
219         
220         - added crypto tests (already balanced)
221
222         * tests/LIST:
223         * tests/crypto-aes.js: Added.
224         * tests/crypto-md5.js: Added.
225         * tests/crypto-sha1.js: Added.
226
227 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
228
229         Reviewed by Adam.
230         
231         - rebalance new tests
232         
233         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
234         
235         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
236
237         This balances them with the existing tests.
238
239         * tests/access-binary-trees.js:
240         * tests/access-nsieve.js:
241         * tests/bitops-nsieve-bits.js:
242         * tests/math-partial-sums.js:
243         * tests/math-spectral-norm.js:
244         * tests/string-fasta.js:
245
246 2007-10-20  Eric Seidel  <eric@webkit.org>
247
248         Reviewed by darin.
249         
250         Add improved argument handling to sunspider, including
251         --runs=<number>, --tests=<pattern, --shark, and --help
252         Also re-factor code into subroutines
253
254         * sunspider: improved argument handling
255
256 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
257
258         Reviewed by Mark.
259
260         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
261         
262         * TODO:
263         * tests/LIST:
264         * tests/access-binary-trees.js: Added.
265         * tests/access-nsieve.js: Added.
266         * tests/bitops-nsieve-bits.js: Added.
267         * tests/math-partial-sums.js: Added.
268         * tests/math-spectral-norm.js: Added.
269         * tests/string-fasta.js: Added.
270
271 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
272
273         Reviewed by Darin.
274
275         * TODO: Updated to note areas that are now well-covered.
276         
277         - Change the tests and the driver to avoid leaving large object
278         graphs hanging around, since that throws off the subsequent tests.
279         
280         * resources/sunspider-standalone-driver.js:
281         * tests/3d-cube.js:
282         * tests/3d-morph.js:
283         * tests/3d-raytrace.js:
284         * tests/bitops-bitwise-and.js:
285         * tests/string-base64.js:
286         * tests/string-tagcloud.js:
287
288 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
289
290         Reviewed by Nikolas.
291         
292         - Rebalance test complexity.
293         
294         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
295         
296         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
297
298         However, for some tests, IE7 was a huge outlier compared to other
299         major browsers. For those tests, I used Firefox 2.0.0.8 on the
300         same OS and hardware as a normalization basis.
301         
302         * tests/3d-morph.js:
303         * tests/3d-raytrace.js:
304         * tests/bitops-3bit-bits-in-byte.js:
305         * tests/bitops-bits-in-byte.js:
306         * tests/bitops-bitwise-and.js:
307         * tests/math-cordic.js:
308         * tests/string-base64.js:
309         * tests/string-tagcloud.js:
310         * tests/string-unpack-code.js:
311
312 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
313
314         Reviewed by Oliver.
315         
316         - Fix the browser-hosted driver to be compatible with Safari 2
317
318         * hosted/sunspider-driver.html:
319
320 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
321
322         Rubber stamped by Hyatt.
323         
324         - Add BSD license notice to cordic test, the original author said to do so.
325
326         * tests/math-cordic.js:
327
328 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
329
330         Reviewed by Eric.
331         
332         - Support Windows and Windows browsers.
333
334         * hosted/sunspider-results.html: Display results correctly in IE.
335         * sunspider: tweaks to work on Windows, and with spaces in the path.
336
337 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
338
339         Rubber stamped by Oliver.
340
341         * tests/string-tagcloud.js: Remove stray print()
342
343 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
344
345         Reviewed by Oliver.
346         
347         Add some string test cases.
348
349         * tests/string-base64.js: Added. Does base64 encoding and decoding.
350         * tests/string-tagcloud.js: Added. Parses a JSON string and
351         generates tagcloud HTML markup based on the data.        
352         * tests/string-unpack-code.js: Added. Unpack compressed versions of
353         four of the most popular JavaScript libraries.
354         * tests/LIST: Added the new tests.
355
356 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
357
358         Reviewed by Anders.
359
360         * TODO: Updated with info on how to do the stats properly.
361
362         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
363         avoid overly slowing down the benchmark.
364
365         Added some 3D-related test cases.
366         
367         * tests/3d-cube.js: Added.
368         * tests/3d-morph.js: Added.
369         * tests/3d-raytrace.js: Added.
370         * tests/LIST:
371
372 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
373
374         Reviewed by Darin.
375
376         - New JavaScript benchmark
377         http://bugs.webkit.org/show_bug.cgi?id=15515
378         
379         A start on a new JavaScript benchmark, with standalone and
380         browser-hosted drivers.
381         
382         The test content is still incomplete and the drivers could use
383         some more features but there is enough here to make a good start.
384         
385         The drivers compute a 95% confidence interval on the mean for the
386         whole test, each category, and each individual test to make it
387         easier to tell whether differences are statistically
388         significant. The confidence interval can be narrowed by running
389         the test more times.
390         
391         * TODO: Added. Remaining things that need to be done.
392         * sunspider: Added. Perl script that acts as the standalone test driver.
393         * resources/sunspider-standalone-driver.js: Added. JavaScript part
394         of standalone test driver.
395         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
396         analysis code.
397         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
398         * make-hosted: Added. Script to generate browser-hosted tests.
399         * hosted/sunspider.html: Added. Start page for browser-hosted test.
400         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
401         * hosted/sunspider-results.html: Added. Results page for browser-hosted
402         tests.
403         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
404         tests.
405         * tests/LIST: Added. List of tests to use.
406         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
407         * tests/bitops-bits-in-byte.js: Added. Ditto.
408         * tests/bitops-bitwise-and.js: Added. Ditto.
409         * tests/math-cordic.js: Added. Ditto.
410