Reviewed by Sam.
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Sam.
4         
5         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
6         
7         A sample of what new-style results would look like:
8         
9         TEST                   COMPARISON            FROM                 TO             DETAILS
10         ...
11         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
12         ...
13             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
14         
15         * resources/sunspider-compare-results.js:
16
17 2007-11-14  Eric Seidel  <eric@webkit.org>
18
19         Reviewed by Sam.
20
21         * sunspider: add --shark-cache for L2 Cache Miss profiling
22
23 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
24
25         Reviewed by Darin.
26
27         http://bugs.webkit.org/show_bug.cgi?id=15925
28         SunSpider should check for Shark being installed
29
30         * sunspider: Explicitly test that shark command line tool is installed.
31
32 2007-11-04  Darin Adler  <darin@apple.com>
33
34         * hosted/sunspider.html: Tweaked formatting a bit.
35
36 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
37
38         Reviewed by Darin.
39
40         - Add compare mode and a description of the benchmark to the hosted version.
41
42         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
43         * hosted/sunspider.html: Added intro text.
44         * make-hosted: Copy new file.
45         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
46         a function.
47         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
48         expects.
49         * sunspider-compare-results: Use sunspider-standalone-compare.js.
50
51 2007-11-02  Darin Adler  <darin@apple.com>
52
53         * sunspider: Changed "--base" to "--set-baseline".
54
55 2007-11-02  Darin Adler  <darin@apple.com>
56
57         Reviewed by Maciej.
58
59         * sunspider: Added a "--base" option that records the current test as a baseline.
60         Moved the code to find the newest .mshark file into a function (since I needed to use
61         it in the other script and I prefer to copy/paste a function).
62
63         * sunspider-compare-results: Added logic to compare the baseline with the last
64         run when you don't pass any parameters.
65
66 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
67
68         Reviewed by Eric.
69         
70         - Added list of ECMAScript3 features that SunSpider does not cover currently.
71
72         * UNCOVERED: Added.
73
74 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
75
76         Reviewed by Oliver.
77         
78         - Give better error messages when sunspider-compare-results fails
79
80         * sunspider-compare-results:
81
82 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
83
84         Reviewed by Darin.
85         
86         - fix Date tests to work in command-line SpiderMonkey
87
88         * tests/date-format-tofte.js:
89         * tests/date-format-xparb.js:
90
91 2007-10-29  Darin Adler  <darin@apple.com>
92
93         Reviewed by Maciej.
94
95         * sunspider: Added --shark20 option, to run Shark at its highest sample
96         resolution instead of its default.
97
98 2007-10-28  Adam Roben  <aroben@apple.com>
99
100         Fix SunSpider on Windows
101
102         Reviewed by Eric.
103
104         * sunspider: Don't use colons in filenames.
105
106 2007-10-28  Darin Adler  <darin@apple.com>
107
108         Reviewed by Adam.
109
110         * resources/sunspider-analyze-results.js: Tweak the output format a little.
111         Change so that when there's only one run we don't write out confidence
112         intervals at all rather than writing "NaN" over and over again.
113
114         * sunspider: Use the time and date as a suffix on the results file, that way
115         all the results are left behind in the tmp directory. This will make it easy
116         for us to add features that compare past results in the future. Also rename
117         the .mshark files using the same scheme.
118
119         * sunspider-compare-results: Relax the parsing rules so we can parse the
120         results file exactly as it's emitted from sunspider.
121
122 2007-10-25  Adam Roben  <aroben@apple.com>
123
124         Strip carriage returns from results
125
126         Carriage returns were screwing up sunspider-compare-results on
127         Windows.
128
129         Rubberstamped by Oliver.
130
131         * sunspider:
132
133 2007-10-24  Darin Adler  <darin@apple.com>
134
135         Reviewed by Eric.
136
137         * sunspider: Open the profile in shark after completing a run.
138         * sunspider-compare-results: Tiny perl tweaks.
139
140 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
141
142         Reviewed by Darin.
143
144         - Don't hardcode my path to testkjs
145         
146         * sunspider-compare-results:
147
148 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
149
150         Reviewed by Darin.
151         
152         - Add a compare script to compare results, and improve formatting of standard results a bit.
153
154         * resources/sunspider-analyze-results.js:
155         * resources/sunspider-compare-results.js: Added.
156         * sunspider-compare-results: Added.
157
158 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
159
160         Reviewed by Darin.
161
162         * resources/sunspider-analyze-results.js: Do the error range
163         properly, using the t-distribution instead of 1.96 (which was
164         based on the normal distribution).
165         
166         * sunspider: Print results in a way that will be friendlier to the
167         soon-to-come compare mode.
168
169 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
170
171         Reviewed by Review.
172         
173         - add more tests (probably the final set for now)
174         
175         Already balanced. Added date, regexp, control flow, and a few
176         more string and object/array access tests.
177         
178         * tests/LIST:
179         * tests/access-fannkuch.js: Added.
180         * tests/access-nbody.js: Added.
181         * tests/controlflow-recursive.js: Added.
182         * tests/date-format-tofte.js: Added.
183         * tests/date-format-xparb.js: Added.
184         * tests/regexp-dna.js: Added.
185         * tests/string-validate-input.js: Added.
186
187 2007-10-21  Eric Seidel  <eric@webkit.org>
188
189         Reviewed by mjs.
190
191         * sunspider: make --shark only sample from testkjs
192
193 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
194
195         Rubber stamped by Eric.
196         
197         - added crypto tests (already balanced)
198
199         * tests/LIST:
200         * tests/crypto-aes.js: Added.
201         * tests/crypto-md5.js: Added.
202         * tests/crypto-sha1.js: Added.
203
204 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
205
206         Reviewed by Adam.
207         
208         - rebalance new tests
209         
210         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
211         
212         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
213
214         This balances them with the existing tests.
215
216         * tests/access-binary-trees.js:
217         * tests/access-nsieve.js:
218         * tests/bitops-nsieve-bits.js:
219         * tests/math-partial-sums.js:
220         * tests/math-spectral-norm.js:
221         * tests/string-fasta.js:
222
223 2007-10-20  Eric Seidel  <eric@webkit.org>
224
225         Reviewed by darin.
226         
227         Add improved argument handling to sunspider, including
228         --runs=<number>, --tests=<pattern, --shark, and --help
229         Also re-factor code into subroutines
230
231         * sunspider: improved argument handling
232
233 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
234
235         Reviewed by Mark.
236
237         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
238         
239         * TODO:
240         * tests/LIST:
241         * tests/access-binary-trees.js: Added.
242         * tests/access-nsieve.js: Added.
243         * tests/bitops-nsieve-bits.js: Added.
244         * tests/math-partial-sums.js: Added.
245         * tests/math-spectral-norm.js: Added.
246         * tests/string-fasta.js: Added.
247
248 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
249
250         Reviewed by Darin.
251
252         * TODO: Updated to note areas that are now well-covered.
253         
254         - Change the tests and the driver to avoid leaving large object
255         graphs hanging around, since that throws off the subsequent tests.
256         
257         * resources/sunspider-standalone-driver.js:
258         * tests/3d-cube.js:
259         * tests/3d-morph.js:
260         * tests/3d-raytrace.js:
261         * tests/bitops-bitwise-and.js:
262         * tests/string-base64.js:
263         * tests/string-tagcloud.js:
264
265 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
266
267         Reviewed by Nikolas.
268         
269         - Rebalance test complexity.
270         
271         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
272         
273         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
274
275         However, for some tests, IE7 was a huge outlier compared to other
276         major browsers. For those tests, I used Firefox 2.0.0.8 on the
277         same OS and hardware as a normalization basis.
278         
279         * tests/3d-morph.js:
280         * tests/3d-raytrace.js:
281         * tests/bitops-3bit-bits-in-byte.js:
282         * tests/bitops-bits-in-byte.js:
283         * tests/bitops-bitwise-and.js:
284         * tests/math-cordic.js:
285         * tests/string-base64.js:
286         * tests/string-tagcloud.js:
287         * tests/string-unpack-code.js:
288
289 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
290
291         Reviewed by Oliver.
292         
293         - Fix the browser-hosted driver to be compatible with Safari 2
294
295         * hosted/sunspider-driver.html:
296
297 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
298
299         Rubber stamped by Hyatt.
300         
301         - Add BSD license notice to cordic test, the original author said to do so.
302
303         * tests/math-cordic.js:
304
305 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
306
307         Reviewed by Eric.
308         
309         - Support Windows and Windows browsers.
310
311         * hosted/sunspider-results.html: Display results correctly in IE.
312         * sunspider: tweaks to work on Windows, and with spaces in the path.
313
314 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
315
316         Rubber stamped by Oliver.
317
318         * tests/string-tagcloud.js: Remove stray print()
319
320 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
321
322         Reviewed by Oliver.
323         
324         Add some string test cases.
325
326         * tests/string-base64.js: Added. Does base64 encoding and decoding.
327         * tests/string-tagcloud.js: Added. Parses a JSON string and
328         generates tagcloud HTML markup based on the data.        
329         * tests/string-unpack-code.js: Added. Unpack compressed versions of
330         four of the most popular JavaScript libraries.
331         * tests/LIST: Added the new tests.
332
333 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
334
335         Reviewed by Anders.
336
337         * TODO: Updated with info on how to do the stats properly.
338
339         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
340         avoid overly slowing down the benchmark.
341
342         Added some 3D-related test cases.
343         
344         * tests/3d-cube.js: Added.
345         * tests/3d-morph.js: Added.
346         * tests/3d-raytrace.js: Added.
347         * tests/LIST:
348
349 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
350
351         Reviewed by Darin.
352
353         - New JavaScript benchmark
354         http://bugs.webkit.org/show_bug.cgi?id=15515
355         
356         A start on a new JavaScript benchmark, with standalone and
357         browser-hosted drivers.
358         
359         The test content is still incomplete and the drivers could use
360         some more features but there is enough here to make a good start.
361         
362         The drivers compute a 95% confidence interval on the mean for the
363         whole test, each category, and each individual test to make it
364         easier to tell whether differences are statistically
365         significant. The confidence interval can be narrowed by running
366         the test more times.
367         
368         * TODO: Added. Remaining things that need to be done.
369         * sunspider: Added. Perl script that acts as the standalone test driver.
370         * resources/sunspider-standalone-driver.js: Added. JavaScript part
371         of standalone test driver.
372         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
373         analysis code.
374         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
375         * make-hosted: Added. Script to generate browser-hosted tests.
376         * hosted/sunspider.html: Added. Start page for browser-hosted test.
377         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
378         * hosted/sunspider-results.html: Added. Results page for browser-hosted
379         tests.
380         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
381         tests.
382         * tests/LIST: Added. List of tests to use.
383         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
384         * tests/bitops-bits-in-byte.js: Added. Ditto.
385         * tests/bitops-bitwise-and.js: Added. Ditto.
386         * tests/math-cordic.js: Added. Ditto.
387