f68e4d317fd02762110cd867adaf991fc319e088
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Mark.
4         
5         - Change the format of the output URL to be shorter, to meet IE constraints
6
7         * hosted/sunspider-driver.html:
8         * hosted/sunspider-results.html:
9
10 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
11
12         Reviewed by Oliver.
13         
14         - Add some explanatory test and a dumb logo to SunSpider web hosted version
15
16         * hosted/sunspider-driver.html:
17         * hosted/sunspider-results.html:
18         * hosted/sunspider.css:
19         * hosted/sunspider.html:
20
21 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
22
23         Reviewed by Oliver.
24         
25         - Make the web-hosted version of SunSpider look a little bit less drab.
26
27         * hosted/sunspider-driver.html:
28         * hosted/sunspider-results.html:
29         * hosted/sunspider.css: Added.
30         * hosted/sunspider.html:
31         * resources/TEMPLATE.html:
32
33 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
34
35         Reviewed by Sam.
36         
37         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
38         
39         A sample of what new-style results would look like:
40         
41         TEST                   COMPARISON            FROM                 TO             DETAILS
42         ...
43         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
44         ...
45             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
46         
47         * resources/sunspider-compare-results.js:
48
49 2007-11-14  Eric Seidel  <eric@webkit.org>
50
51         Reviewed by Sam.
52
53         * sunspider: add --shark-cache for L2 Cache Miss profiling
54
55 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
56
57         Reviewed by Darin.
58
59         http://bugs.webkit.org/show_bug.cgi?id=15925
60         SunSpider should check for Shark being installed
61
62         * sunspider: Explicitly test that shark command line tool is installed.
63
64 2007-11-04  Darin Adler  <darin@apple.com>
65
66         * hosted/sunspider.html: Tweaked formatting a bit.
67
68 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
69
70         Reviewed by Darin.
71
72         - Add compare mode and a description of the benchmark to the hosted version.
73
74         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
75         * hosted/sunspider.html: Added intro text.
76         * make-hosted: Copy new file.
77         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
78         a function.
79         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
80         expects.
81         * sunspider-compare-results: Use sunspider-standalone-compare.js.
82
83 2007-11-02  Darin Adler  <darin@apple.com>
84
85         * sunspider: Changed "--base" to "--set-baseline".
86
87 2007-11-02  Darin Adler  <darin@apple.com>
88
89         Reviewed by Maciej.
90
91         * sunspider: Added a "--base" option that records the current test as a baseline.
92         Moved the code to find the newest .mshark file into a function (since I needed to use
93         it in the other script and I prefer to copy/paste a function).
94
95         * sunspider-compare-results: Added logic to compare the baseline with the last
96         run when you don't pass any parameters.
97
98 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
99
100         Reviewed by Eric.
101         
102         - Added list of ECMAScript3 features that SunSpider does not cover currently.
103
104         * UNCOVERED: Added.
105
106 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
107
108         Reviewed by Oliver.
109         
110         - Give better error messages when sunspider-compare-results fails
111
112         * sunspider-compare-results:
113
114 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
115
116         Reviewed by Darin.
117         
118         - fix Date tests to work in command-line SpiderMonkey
119
120         * tests/date-format-tofte.js:
121         * tests/date-format-xparb.js:
122
123 2007-10-29  Darin Adler  <darin@apple.com>
124
125         Reviewed by Maciej.
126
127         * sunspider: Added --shark20 option, to run Shark at its highest sample
128         resolution instead of its default.
129
130 2007-10-28  Adam Roben  <aroben@apple.com>
131
132         Fix SunSpider on Windows
133
134         Reviewed by Eric.
135
136         * sunspider: Don't use colons in filenames.
137
138 2007-10-28  Darin Adler  <darin@apple.com>
139
140         Reviewed by Adam.
141
142         * resources/sunspider-analyze-results.js: Tweak the output format a little.
143         Change so that when there's only one run we don't write out confidence
144         intervals at all rather than writing "NaN" over and over again.
145
146         * sunspider: Use the time and date as a suffix on the results file, that way
147         all the results are left behind in the tmp directory. This will make it easy
148         for us to add features that compare past results in the future. Also rename
149         the .mshark files using the same scheme.
150
151         * sunspider-compare-results: Relax the parsing rules so we can parse the
152         results file exactly as it's emitted from sunspider.
153
154 2007-10-25  Adam Roben  <aroben@apple.com>
155
156         Strip carriage returns from results
157
158         Carriage returns were screwing up sunspider-compare-results on
159         Windows.
160
161         Rubberstamped by Oliver.
162
163         * sunspider:
164
165 2007-10-24  Darin Adler  <darin@apple.com>
166
167         Reviewed by Eric.
168
169         * sunspider: Open the profile in shark after completing a run.
170         * sunspider-compare-results: Tiny perl tweaks.
171
172 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
173
174         Reviewed by Darin.
175
176         - Don't hardcode my path to testkjs
177         
178         * sunspider-compare-results:
179
180 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
181
182         Reviewed by Darin.
183         
184         - Add a compare script to compare results, and improve formatting of standard results a bit.
185
186         * resources/sunspider-analyze-results.js:
187         * resources/sunspider-compare-results.js: Added.
188         * sunspider-compare-results: Added.
189
190 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
191
192         Reviewed by Darin.
193
194         * resources/sunspider-analyze-results.js: Do the error range
195         properly, using the t-distribution instead of 1.96 (which was
196         based on the normal distribution).
197         
198         * sunspider: Print results in a way that will be friendlier to the
199         soon-to-come compare mode.
200
201 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
202
203         Reviewed by Review.
204         
205         - add more tests (probably the final set for now)
206         
207         Already balanced. Added date, regexp, control flow, and a few
208         more string and object/array access tests.
209         
210         * tests/LIST:
211         * tests/access-fannkuch.js: Added.
212         * tests/access-nbody.js: Added.
213         * tests/controlflow-recursive.js: Added.
214         * tests/date-format-tofte.js: Added.
215         * tests/date-format-xparb.js: Added.
216         * tests/regexp-dna.js: Added.
217         * tests/string-validate-input.js: Added.
218
219 2007-10-21  Eric Seidel  <eric@webkit.org>
220
221         Reviewed by mjs.
222
223         * sunspider: make --shark only sample from testkjs
224
225 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
226
227         Rubber stamped by Eric.
228         
229         - added crypto tests (already balanced)
230
231         * tests/LIST:
232         * tests/crypto-aes.js: Added.
233         * tests/crypto-md5.js: Added.
234         * tests/crypto-sha1.js: Added.
235
236 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
237
238         Reviewed by Adam.
239         
240         - rebalance new tests
241         
242         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
243         
244         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
245
246         This balances them with the existing tests.
247
248         * tests/access-binary-trees.js:
249         * tests/access-nsieve.js:
250         * tests/bitops-nsieve-bits.js:
251         * tests/math-partial-sums.js:
252         * tests/math-spectral-norm.js:
253         * tests/string-fasta.js:
254
255 2007-10-20  Eric Seidel  <eric@webkit.org>
256
257         Reviewed by darin.
258         
259         Add improved argument handling to sunspider, including
260         --runs=<number>, --tests=<pattern, --shark, and --help
261         Also re-factor code into subroutines
262
263         * sunspider: improved argument handling
264
265 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
266
267         Reviewed by Mark.
268
269         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
270         
271         * TODO:
272         * tests/LIST:
273         * tests/access-binary-trees.js: Added.
274         * tests/access-nsieve.js: Added.
275         * tests/bitops-nsieve-bits.js: Added.
276         * tests/math-partial-sums.js: Added.
277         * tests/math-spectral-norm.js: Added.
278         * tests/string-fasta.js: Added.
279
280 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
281
282         Reviewed by Darin.
283
284         * TODO: Updated to note areas that are now well-covered.
285         
286         - Change the tests and the driver to avoid leaving large object
287         graphs hanging around, since that throws off the subsequent tests.
288         
289         * resources/sunspider-standalone-driver.js:
290         * tests/3d-cube.js:
291         * tests/3d-morph.js:
292         * tests/3d-raytrace.js:
293         * tests/bitops-bitwise-and.js:
294         * tests/string-base64.js:
295         * tests/string-tagcloud.js:
296
297 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
298
299         Reviewed by Nikolas.
300         
301         - Rebalance test complexity.
302         
303         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
304         
305         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
306
307         However, for some tests, IE7 was a huge outlier compared to other
308         major browsers. For those tests, I used Firefox 2.0.0.8 on the
309         same OS and hardware as a normalization basis.
310         
311         * tests/3d-morph.js:
312         * tests/3d-raytrace.js:
313         * tests/bitops-3bit-bits-in-byte.js:
314         * tests/bitops-bits-in-byte.js:
315         * tests/bitops-bitwise-and.js:
316         * tests/math-cordic.js:
317         * tests/string-base64.js:
318         * tests/string-tagcloud.js:
319         * tests/string-unpack-code.js:
320
321 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
322
323         Reviewed by Oliver.
324         
325         - Fix the browser-hosted driver to be compatible with Safari 2
326
327         * hosted/sunspider-driver.html:
328
329 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
330
331         Rubber stamped by Hyatt.
332         
333         - Add BSD license notice to cordic test, the original author said to do so.
334
335         * tests/math-cordic.js:
336
337 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
338
339         Reviewed by Eric.
340         
341         - Support Windows and Windows browsers.
342
343         * hosted/sunspider-results.html: Display results correctly in IE.
344         * sunspider: tweaks to work on Windows, and with spaces in the path.
345
346 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
347
348         Rubber stamped by Oliver.
349
350         * tests/string-tagcloud.js: Remove stray print()
351
352 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
353
354         Reviewed by Oliver.
355         
356         Add some string test cases.
357
358         * tests/string-base64.js: Added. Does base64 encoding and decoding.
359         * tests/string-tagcloud.js: Added. Parses a JSON string and
360         generates tagcloud HTML markup based on the data.        
361         * tests/string-unpack-code.js: Added. Unpack compressed versions of
362         four of the most popular JavaScript libraries.
363         * tests/LIST: Added the new tests.
364
365 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
366
367         Reviewed by Anders.
368
369         * TODO: Updated with info on how to do the stats properly.
370
371         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
372         avoid overly slowing down the benchmark.
373
374         Added some 3D-related test cases.
375         
376         * tests/3d-cube.js: Added.
377         * tests/3d-morph.js: Added.
378         * tests/3d-raytrace.js: Added.
379         * tests/LIST:
380
381 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
382
383         Reviewed by Darin.
384
385         - New JavaScript benchmark
386         http://bugs.webkit.org/show_bug.cgi?id=15515
387         
388         A start on a new JavaScript benchmark, with standalone and
389         browser-hosted drivers.
390         
391         The test content is still incomplete and the drivers could use
392         some more features but there is enough here to make a good start.
393         
394         The drivers compute a 95% confidence interval on the mean for the
395         whole test, each category, and each individual test to make it
396         easier to tell whether differences are statistically
397         significant. The confidence interval can be narrowed by running
398         the test more times.
399         
400         * TODO: Added. Remaining things that need to be done.
401         * sunspider: Added. Perl script that acts as the standalone test driver.
402         * resources/sunspider-standalone-driver.js: Added. JavaScript part
403         of standalone test driver.
404         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
405         analysis code.
406         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
407         * make-hosted: Added. Script to generate browser-hosted tests.
408         * hosted/sunspider.html: Added. Start page for browser-hosted test.
409         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
410         * hosted/sunspider-results.html: Added. Results page for browser-hosted
411         tests.
412         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
413         tests.
414         * tests/LIST: Added. List of tests to use.
415         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
416         * tests/bitops-bits-in-byte.js: Added. Ditto.
417         * tests/bitops-bitwise-and.js: Added. Ditto.
418         * tests/math-cordic.js: Added. Ditto.
419