Reviewed by Eric.
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Eric.
4         
5         - give an extra digit of precision for small differences.
6
7         * resources/sunspider-compare-results.js:
8
9 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
10
11         Reviewed by Eric.
12         
13         - work around IE layout bugs
14
15         * hosted/sunspider.css:
16
17 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
18
19         Reviewed by Mark.
20         
21         - fix compare feature
22
23         * hosted/sunspider-results.html:
24
25 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
26
27         Reviewed by Mark.
28         
29         - Change the format of the output URL to be shorter, to meet IE constraints
30
31         * hosted/sunspider-driver.html:
32         * hosted/sunspider-results.html:
33
34 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
35
36         Reviewed by Oliver.
37         
38         - Add some explanatory test and a dumb logo to SunSpider web hosted version
39
40         * hosted/sunspider-driver.html:
41         * hosted/sunspider-results.html:
42         * hosted/sunspider.css:
43         * hosted/sunspider.html:
44
45 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
46
47         Reviewed by Oliver.
48         
49         - Make the web-hosted version of SunSpider look a little bit less drab.
50
51         * hosted/sunspider-driver.html:
52         * hosted/sunspider-results.html:
53         * hosted/sunspider.css: Added.
54         * hosted/sunspider.html:
55         * resources/TEMPLATE.html:
56
57 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
58
59         Reviewed by Sam.
60         
61         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
62         
63         A sample of what new-style results would look like:
64         
65         TEST                   COMPARISON            FROM                 TO             DETAILS
66         ...
67         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
68         ...
69             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
70         
71         * resources/sunspider-compare-results.js:
72
73 2007-11-14  Eric Seidel  <eric@webkit.org>
74
75         Reviewed by Sam.
76
77         * sunspider: add --shark-cache for L2 Cache Miss profiling
78
79 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
80
81         Reviewed by Darin.
82
83         http://bugs.webkit.org/show_bug.cgi?id=15925
84         SunSpider should check for Shark being installed
85
86         * sunspider: Explicitly test that shark command line tool is installed.
87
88 2007-11-04  Darin Adler  <darin@apple.com>
89
90         * hosted/sunspider.html: Tweaked formatting a bit.
91
92 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
93
94         Reviewed by Darin.
95
96         - Add compare mode and a description of the benchmark to the hosted version.
97
98         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
99         * hosted/sunspider.html: Added intro text.
100         * make-hosted: Copy new file.
101         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
102         a function.
103         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
104         expects.
105         * sunspider-compare-results: Use sunspider-standalone-compare.js.
106
107 2007-11-02  Darin Adler  <darin@apple.com>
108
109         * sunspider: Changed "--base" to "--set-baseline".
110
111 2007-11-02  Darin Adler  <darin@apple.com>
112
113         Reviewed by Maciej.
114
115         * sunspider: Added a "--base" option that records the current test as a baseline.
116         Moved the code to find the newest .mshark file into a function (since I needed to use
117         it in the other script and I prefer to copy/paste a function).
118
119         * sunspider-compare-results: Added logic to compare the baseline with the last
120         run when you don't pass any parameters.
121
122 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
123
124         Reviewed by Eric.
125         
126         - Added list of ECMAScript3 features that SunSpider does not cover currently.
127
128         * UNCOVERED: Added.
129
130 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
131
132         Reviewed by Oliver.
133         
134         - Give better error messages when sunspider-compare-results fails
135
136         * sunspider-compare-results:
137
138 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
139
140         Reviewed by Darin.
141         
142         - fix Date tests to work in command-line SpiderMonkey
143
144         * tests/date-format-tofte.js:
145         * tests/date-format-xparb.js:
146
147 2007-10-29  Darin Adler  <darin@apple.com>
148
149         Reviewed by Maciej.
150
151         * sunspider: Added --shark20 option, to run Shark at its highest sample
152         resolution instead of its default.
153
154 2007-10-28  Adam Roben  <aroben@apple.com>
155
156         Fix SunSpider on Windows
157
158         Reviewed by Eric.
159
160         * sunspider: Don't use colons in filenames.
161
162 2007-10-28  Darin Adler  <darin@apple.com>
163
164         Reviewed by Adam.
165
166         * resources/sunspider-analyze-results.js: Tweak the output format a little.
167         Change so that when there's only one run we don't write out confidence
168         intervals at all rather than writing "NaN" over and over again.
169
170         * sunspider: Use the time and date as a suffix on the results file, that way
171         all the results are left behind in the tmp directory. This will make it easy
172         for us to add features that compare past results in the future. Also rename
173         the .mshark files using the same scheme.
174
175         * sunspider-compare-results: Relax the parsing rules so we can parse the
176         results file exactly as it's emitted from sunspider.
177
178 2007-10-25  Adam Roben  <aroben@apple.com>
179
180         Strip carriage returns from results
181
182         Carriage returns were screwing up sunspider-compare-results on
183         Windows.
184
185         Rubberstamped by Oliver.
186
187         * sunspider:
188
189 2007-10-24  Darin Adler  <darin@apple.com>
190
191         Reviewed by Eric.
192
193         * sunspider: Open the profile in shark after completing a run.
194         * sunspider-compare-results: Tiny perl tweaks.
195
196 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
197
198         Reviewed by Darin.
199
200         - Don't hardcode my path to testkjs
201         
202         * sunspider-compare-results:
203
204 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
205
206         Reviewed by Darin.
207         
208         - Add a compare script to compare results, and improve formatting of standard results a bit.
209
210         * resources/sunspider-analyze-results.js:
211         * resources/sunspider-compare-results.js: Added.
212         * sunspider-compare-results: Added.
213
214 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
215
216         Reviewed by Darin.
217
218         * resources/sunspider-analyze-results.js: Do the error range
219         properly, using the t-distribution instead of 1.96 (which was
220         based on the normal distribution).
221         
222         * sunspider: Print results in a way that will be friendlier to the
223         soon-to-come compare mode.
224
225 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
226
227         Reviewed by Review.
228         
229         - add more tests (probably the final set for now)
230         
231         Already balanced. Added date, regexp, control flow, and a few
232         more string and object/array access tests.
233         
234         * tests/LIST:
235         * tests/access-fannkuch.js: Added.
236         * tests/access-nbody.js: Added.
237         * tests/controlflow-recursive.js: Added.
238         * tests/date-format-tofte.js: Added.
239         * tests/date-format-xparb.js: Added.
240         * tests/regexp-dna.js: Added.
241         * tests/string-validate-input.js: Added.
242
243 2007-10-21  Eric Seidel  <eric@webkit.org>
244
245         Reviewed by mjs.
246
247         * sunspider: make --shark only sample from testkjs
248
249 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
250
251         Rubber stamped by Eric.
252         
253         - added crypto tests (already balanced)
254
255         * tests/LIST:
256         * tests/crypto-aes.js: Added.
257         * tests/crypto-md5.js: Added.
258         * tests/crypto-sha1.js: Added.
259
260 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
261
262         Reviewed by Adam.
263         
264         - rebalance new tests
265         
266         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
267         
268         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
269
270         This balances them with the existing tests.
271
272         * tests/access-binary-trees.js:
273         * tests/access-nsieve.js:
274         * tests/bitops-nsieve-bits.js:
275         * tests/math-partial-sums.js:
276         * tests/math-spectral-norm.js:
277         * tests/string-fasta.js:
278
279 2007-10-20  Eric Seidel  <eric@webkit.org>
280
281         Reviewed by darin.
282         
283         Add improved argument handling to sunspider, including
284         --runs=<number>, --tests=<pattern, --shark, and --help
285         Also re-factor code into subroutines
286
287         * sunspider: improved argument handling
288
289 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
290
291         Reviewed by Mark.
292
293         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
294         
295         * TODO:
296         * tests/LIST:
297         * tests/access-binary-trees.js: Added.
298         * tests/access-nsieve.js: Added.
299         * tests/bitops-nsieve-bits.js: Added.
300         * tests/math-partial-sums.js: Added.
301         * tests/math-spectral-norm.js: Added.
302         * tests/string-fasta.js: Added.
303
304 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
305
306         Reviewed by Darin.
307
308         * TODO: Updated to note areas that are now well-covered.
309         
310         - Change the tests and the driver to avoid leaving large object
311         graphs hanging around, since that throws off the subsequent tests.
312         
313         * resources/sunspider-standalone-driver.js:
314         * tests/3d-cube.js:
315         * tests/3d-morph.js:
316         * tests/3d-raytrace.js:
317         * tests/bitops-bitwise-and.js:
318         * tests/string-base64.js:
319         * tests/string-tagcloud.js:
320
321 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
322
323         Reviewed by Nikolas.
324         
325         - Rebalance test complexity.
326         
327         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
328         
329         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
330
331         However, for some tests, IE7 was a huge outlier compared to other
332         major browsers. For those tests, I used Firefox 2.0.0.8 on the
333         same OS and hardware as a normalization basis.
334         
335         * tests/3d-morph.js:
336         * tests/3d-raytrace.js:
337         * tests/bitops-3bit-bits-in-byte.js:
338         * tests/bitops-bits-in-byte.js:
339         * tests/bitops-bitwise-and.js:
340         * tests/math-cordic.js:
341         * tests/string-base64.js:
342         * tests/string-tagcloud.js:
343         * tests/string-unpack-code.js:
344
345 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
346
347         Reviewed by Oliver.
348         
349         - Fix the browser-hosted driver to be compatible with Safari 2
350
351         * hosted/sunspider-driver.html:
352
353 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
354
355         Rubber stamped by Hyatt.
356         
357         - Add BSD license notice to cordic test, the original author said to do so.
358
359         * tests/math-cordic.js:
360
361 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
362
363         Reviewed by Eric.
364         
365         - Support Windows and Windows browsers.
366
367         * hosted/sunspider-results.html: Display results correctly in IE.
368         * sunspider: tweaks to work on Windows, and with spaces in the path.
369
370 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
371
372         Rubber stamped by Oliver.
373
374         * tests/string-tagcloud.js: Remove stray print()
375
376 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
377
378         Reviewed by Oliver.
379         
380         Add some string test cases.
381
382         * tests/string-base64.js: Added. Does base64 encoding and decoding.
383         * tests/string-tagcloud.js: Added. Parses a JSON string and
384         generates tagcloud HTML markup based on the data.        
385         * tests/string-unpack-code.js: Added. Unpack compressed versions of
386         four of the most popular JavaScript libraries.
387         * tests/LIST: Added the new tests.
388
389 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
390
391         Reviewed by Anders.
392
393         * TODO: Updated with info on how to do the stats properly.
394
395         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
396         avoid overly slowing down the benchmark.
397
398         Added some 3D-related test cases.
399         
400         * tests/3d-cube.js: Added.
401         * tests/3d-morph.js: Added.
402         * tests/3d-raytrace.js: Added.
403         * tests/LIST:
404
405 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
406
407         Reviewed by Darin.
408
409         - New JavaScript benchmark
410         http://bugs.webkit.org/show_bug.cgi?id=15515
411         
412         A start on a new JavaScript benchmark, with standalone and
413         browser-hosted drivers.
414         
415         The test content is still incomplete and the drivers could use
416         some more features but there is enough here to make a good start.
417         
418         The drivers compute a 95% confidence interval on the mean for the
419         whole test, each category, and each individual test to make it
420         easier to tell whether differences are statistically
421         significant. The confidence interval can be narrowed by running
422         the test more times.
423         
424         * TODO: Added. Remaining things that need to be done.
425         * sunspider: Added. Perl script that acts as the standalone test driver.
426         * resources/sunspider-standalone-driver.js: Added. JavaScript part
427         of standalone test driver.
428         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
429         analysis code.
430         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
431         * make-hosted: Added. Script to generate browser-hosted tests.
432         * hosted/sunspider.html: Added. Start page for browser-hosted test.
433         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
434         * hosted/sunspider-results.html: Added. Results page for browser-hosted
435         tests.
436         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
437         tests.
438         * tests/LIST: Added. List of tests to use.
439         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
440         * tests/bitops-bits-in-byte.js: Added. Ditto.
441         * tests/bitops-bitwise-and.js: Added. Ditto.
442         * tests/math-cordic.js: Added. Ditto.
443