6c2ed401e128a5e2ea04a241c64097429b6a74b5
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Oliver.
4         
5         - Make the web-hosted version of SunSpider look a little bit less drab.
6
7         * hosted/sunspider-driver.html:
8         * hosted/sunspider-results.html:
9         * hosted/sunspider.css: Added.
10         * hosted/sunspider.html:
11         * resources/TEMPLATE.html:
12
13 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
14
15         Reviewed by Sam.
16         
17         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
18         
19         A sample of what new-style results would look like:
20         
21         TEST                   COMPARISON            FROM                 TO             DETAILS
22         ...
23         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
24         ...
25             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
26         
27         * resources/sunspider-compare-results.js:
28
29 2007-11-14  Eric Seidel  <eric@webkit.org>
30
31         Reviewed by Sam.
32
33         * sunspider: add --shark-cache for L2 Cache Miss profiling
34
35 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
36
37         Reviewed by Darin.
38
39         http://bugs.webkit.org/show_bug.cgi?id=15925
40         SunSpider should check for Shark being installed
41
42         * sunspider: Explicitly test that shark command line tool is installed.
43
44 2007-11-04  Darin Adler  <darin@apple.com>
45
46         * hosted/sunspider.html: Tweaked formatting a bit.
47
48 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
49
50         Reviewed by Darin.
51
52         - Add compare mode and a description of the benchmark to the hosted version.
53
54         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
55         * hosted/sunspider.html: Added intro text.
56         * make-hosted: Copy new file.
57         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
58         a function.
59         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
60         expects.
61         * sunspider-compare-results: Use sunspider-standalone-compare.js.
62
63 2007-11-02  Darin Adler  <darin@apple.com>
64
65         * sunspider: Changed "--base" to "--set-baseline".
66
67 2007-11-02  Darin Adler  <darin@apple.com>
68
69         Reviewed by Maciej.
70
71         * sunspider: Added a "--base" option that records the current test as a baseline.
72         Moved the code to find the newest .mshark file into a function (since I needed to use
73         it in the other script and I prefer to copy/paste a function).
74
75         * sunspider-compare-results: Added logic to compare the baseline with the last
76         run when you don't pass any parameters.
77
78 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
79
80         Reviewed by Eric.
81         
82         - Added list of ECMAScript3 features that SunSpider does not cover currently.
83
84         * UNCOVERED: Added.
85
86 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
87
88         Reviewed by Oliver.
89         
90         - Give better error messages when sunspider-compare-results fails
91
92         * sunspider-compare-results:
93
94 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
95
96         Reviewed by Darin.
97         
98         - fix Date tests to work in command-line SpiderMonkey
99
100         * tests/date-format-tofte.js:
101         * tests/date-format-xparb.js:
102
103 2007-10-29  Darin Adler  <darin@apple.com>
104
105         Reviewed by Maciej.
106
107         * sunspider: Added --shark20 option, to run Shark at its highest sample
108         resolution instead of its default.
109
110 2007-10-28  Adam Roben  <aroben@apple.com>
111
112         Fix SunSpider on Windows
113
114         Reviewed by Eric.
115
116         * sunspider: Don't use colons in filenames.
117
118 2007-10-28  Darin Adler  <darin@apple.com>
119
120         Reviewed by Adam.
121
122         * resources/sunspider-analyze-results.js: Tweak the output format a little.
123         Change so that when there's only one run we don't write out confidence
124         intervals at all rather than writing "NaN" over and over again.
125
126         * sunspider: Use the time and date as a suffix on the results file, that way
127         all the results are left behind in the tmp directory. This will make it easy
128         for us to add features that compare past results in the future. Also rename
129         the .mshark files using the same scheme.
130
131         * sunspider-compare-results: Relax the parsing rules so we can parse the
132         results file exactly as it's emitted from sunspider.
133
134 2007-10-25  Adam Roben  <aroben@apple.com>
135
136         Strip carriage returns from results
137
138         Carriage returns were screwing up sunspider-compare-results on
139         Windows.
140
141         Rubberstamped by Oliver.
142
143         * sunspider:
144
145 2007-10-24  Darin Adler  <darin@apple.com>
146
147         Reviewed by Eric.
148
149         * sunspider: Open the profile in shark after completing a run.
150         * sunspider-compare-results: Tiny perl tweaks.
151
152 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
153
154         Reviewed by Darin.
155
156         - Don't hardcode my path to testkjs
157         
158         * sunspider-compare-results:
159
160 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
161
162         Reviewed by Darin.
163         
164         - Add a compare script to compare results, and improve formatting of standard results a bit.
165
166         * resources/sunspider-analyze-results.js:
167         * resources/sunspider-compare-results.js: Added.
168         * sunspider-compare-results: Added.
169
170 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
171
172         Reviewed by Darin.
173
174         * resources/sunspider-analyze-results.js: Do the error range
175         properly, using the t-distribution instead of 1.96 (which was
176         based on the normal distribution).
177         
178         * sunspider: Print results in a way that will be friendlier to the
179         soon-to-come compare mode.
180
181 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
182
183         Reviewed by Review.
184         
185         - add more tests (probably the final set for now)
186         
187         Already balanced. Added date, regexp, control flow, and a few
188         more string and object/array access tests.
189         
190         * tests/LIST:
191         * tests/access-fannkuch.js: Added.
192         * tests/access-nbody.js: Added.
193         * tests/controlflow-recursive.js: Added.
194         * tests/date-format-tofte.js: Added.
195         * tests/date-format-xparb.js: Added.
196         * tests/regexp-dna.js: Added.
197         * tests/string-validate-input.js: Added.
198
199 2007-10-21  Eric Seidel  <eric@webkit.org>
200
201         Reviewed by mjs.
202
203         * sunspider: make --shark only sample from testkjs
204
205 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
206
207         Rubber stamped by Eric.
208         
209         - added crypto tests (already balanced)
210
211         * tests/LIST:
212         * tests/crypto-aes.js: Added.
213         * tests/crypto-md5.js: Added.
214         * tests/crypto-sha1.js: Added.
215
216 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
217
218         Reviewed by Adam.
219         
220         - rebalance new tests
221         
222         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
223         
224         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
225
226         This balances them with the existing tests.
227
228         * tests/access-binary-trees.js:
229         * tests/access-nsieve.js:
230         * tests/bitops-nsieve-bits.js:
231         * tests/math-partial-sums.js:
232         * tests/math-spectral-norm.js:
233         * tests/string-fasta.js:
234
235 2007-10-20  Eric Seidel  <eric@webkit.org>
236
237         Reviewed by darin.
238         
239         Add improved argument handling to sunspider, including
240         --runs=<number>, --tests=<pattern, --shark, and --help
241         Also re-factor code into subroutines
242
243         * sunspider: improved argument handling
244
245 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
246
247         Reviewed by Mark.
248
249         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
250         
251         * TODO:
252         * tests/LIST:
253         * tests/access-binary-trees.js: Added.
254         * tests/access-nsieve.js: Added.
255         * tests/bitops-nsieve-bits.js: Added.
256         * tests/math-partial-sums.js: Added.
257         * tests/math-spectral-norm.js: Added.
258         * tests/string-fasta.js: Added.
259
260 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
261
262         Reviewed by Darin.
263
264         * TODO: Updated to note areas that are now well-covered.
265         
266         - Change the tests and the driver to avoid leaving large object
267         graphs hanging around, since that throws off the subsequent tests.
268         
269         * resources/sunspider-standalone-driver.js:
270         * tests/3d-cube.js:
271         * tests/3d-morph.js:
272         * tests/3d-raytrace.js:
273         * tests/bitops-bitwise-and.js:
274         * tests/string-base64.js:
275         * tests/string-tagcloud.js:
276
277 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
278
279         Reviewed by Nikolas.
280         
281         - Rebalance test complexity.
282         
283         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
284         
285         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
286
287         However, for some tests, IE7 was a huge outlier compared to other
288         major browsers. For those tests, I used Firefox 2.0.0.8 on the
289         same OS and hardware as a normalization basis.
290         
291         * tests/3d-morph.js:
292         * tests/3d-raytrace.js:
293         * tests/bitops-3bit-bits-in-byte.js:
294         * tests/bitops-bits-in-byte.js:
295         * tests/bitops-bitwise-and.js:
296         * tests/math-cordic.js:
297         * tests/string-base64.js:
298         * tests/string-tagcloud.js:
299         * tests/string-unpack-code.js:
300
301 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
302
303         Reviewed by Oliver.
304         
305         - Fix the browser-hosted driver to be compatible with Safari 2
306
307         * hosted/sunspider-driver.html:
308
309 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
310
311         Rubber stamped by Hyatt.
312         
313         - Add BSD license notice to cordic test, the original author said to do so.
314
315         * tests/math-cordic.js:
316
317 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
318
319         Reviewed by Eric.
320         
321         - Support Windows and Windows browsers.
322
323         * hosted/sunspider-results.html: Display results correctly in IE.
324         * sunspider: tweaks to work on Windows, and with spaces in the path.
325
326 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
327
328         Rubber stamped by Oliver.
329
330         * tests/string-tagcloud.js: Remove stray print()
331
332 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
333
334         Reviewed by Oliver.
335         
336         Add some string test cases.
337
338         * tests/string-base64.js: Added. Does base64 encoding and decoding.
339         * tests/string-tagcloud.js: Added. Parses a JSON string and
340         generates tagcloud HTML markup based on the data.        
341         * tests/string-unpack-code.js: Added. Unpack compressed versions of
342         four of the most popular JavaScript libraries.
343         * tests/LIST: Added the new tests.
344
345 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
346
347         Reviewed by Anders.
348
349         * TODO: Updated with info on how to do the stats properly.
350
351         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
352         avoid overly slowing down the benchmark.
353
354         Added some 3D-related test cases.
355         
356         * tests/3d-cube.js: Added.
357         * tests/3d-morph.js: Added.
358         * tests/3d-raytrace.js: Added.
359         * tests/LIST:
360
361 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
362
363         Reviewed by Darin.
364
365         - New JavaScript benchmark
366         http://bugs.webkit.org/show_bug.cgi?id=15515
367         
368         A start on a new JavaScript benchmark, with standalone and
369         browser-hosted drivers.
370         
371         The test content is still incomplete and the drivers could use
372         some more features but there is enough here to make a good start.
373         
374         The drivers compute a 95% confidence interval on the mean for the
375         whole test, each category, and each individual test to make it
376         easier to tell whether differences are statistically
377         significant. The confidence interval can be narrowed by running
378         the test more times.
379         
380         * TODO: Added. Remaining things that need to be done.
381         * sunspider: Added. Perl script that acts as the standalone test driver.
382         * resources/sunspider-standalone-driver.js: Added. JavaScript part
383         of standalone test driver.
384         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
385         analysis code.
386         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
387         * make-hosted: Added. Script to generate browser-hosted tests.
388         * hosted/sunspider.html: Added. Start page for browser-hosted test.
389         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
390         * hosted/sunspider-results.html: Added. Results page for browser-hosted
391         tests.
392         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
393         tests.
394         * tests/LIST: Added. List of tests to use.
395         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
396         * tests/bitops-bits-in-byte.js: Added. Ditto.
397         * tests/bitops-bitwise-and.js: Added. Ditto.
398         * tests/math-cordic.js: Added. Ditto.
399