- updated results (character used for controls changed in r28945)
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-20  Eric Seidel  <eric@webkit.org>
2
3         Reviewed by mjs.
4
5         * sunspider: increase default number of runs for more accurate numbers
6
7 2007-12-18  Mark Rowe  <mrowe@apple.com>
8
9         Fix typo.
10
11         * hosted/sunspider.html:
12
13 2007-12-18  Maciej Stachowiak  <mjs@apple.com>
14
15         Reviewed by Eric.
16         
17         - give an extra digit of precision for small differences.
18
19         * resources/sunspider-compare-results.js:
20
21 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
22
23         Reviewed by Eric.
24         
25         - work around IE layout bugs
26
27         * hosted/sunspider.css:
28
29 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
30
31         Reviewed by Mark.
32         
33         - fix compare feature
34
35         * hosted/sunspider-results.html:
36
37 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
38
39         Reviewed by Mark.
40         
41         - Change the format of the output URL to be shorter, to meet IE constraints
42
43         * hosted/sunspider-driver.html:
44         * hosted/sunspider-results.html:
45
46 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
47
48         Reviewed by Oliver.
49         
50         - Add some explanatory test and a dumb logo to SunSpider web hosted version
51
52         * hosted/sunspider-driver.html:
53         * hosted/sunspider-results.html:
54         * hosted/sunspider.css:
55         * hosted/sunspider.html:
56
57 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
58
59         Reviewed by Oliver.
60         
61         - Make the web-hosted version of SunSpider look a little bit less drab.
62
63         * hosted/sunspider-driver.html:
64         * hosted/sunspider-results.html:
65         * hosted/sunspider.css: Added.
66         * hosted/sunspider.html:
67         * resources/TEMPLATE.html:
68
69 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
70
71         Reviewed by Sam.
72         
73         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
74         
75         A sample of what new-style results would look like:
76         
77         TEST                   COMPARISON            FROM                 TO             DETAILS
78         ...
79         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
80         ...
81             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
82         
83         * resources/sunspider-compare-results.js:
84
85 2007-11-14  Eric Seidel  <eric@webkit.org>
86
87         Reviewed by Sam.
88
89         * sunspider: add --shark-cache for L2 Cache Miss profiling
90
91 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
92
93         Reviewed by Darin.
94
95         http://bugs.webkit.org/show_bug.cgi?id=15925
96         SunSpider should check for Shark being installed
97
98         * sunspider: Explicitly test that shark command line tool is installed.
99
100 2007-11-04  Darin Adler  <darin@apple.com>
101
102         * hosted/sunspider.html: Tweaked formatting a bit.
103
104 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
105
106         Reviewed by Darin.
107
108         - Add compare mode and a description of the benchmark to the hosted version.
109
110         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
111         * hosted/sunspider.html: Added intro text.
112         * make-hosted: Copy new file.
113         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
114         a function.
115         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
116         expects.
117         * sunspider-compare-results: Use sunspider-standalone-compare.js.
118
119 2007-11-02  Darin Adler  <darin@apple.com>
120
121         * sunspider: Changed "--base" to "--set-baseline".
122
123 2007-11-02  Darin Adler  <darin@apple.com>
124
125         Reviewed by Maciej.
126
127         * sunspider: Added a "--base" option that records the current test as a baseline.
128         Moved the code to find the newest .mshark file into a function (since I needed to use
129         it in the other script and I prefer to copy/paste a function).
130
131         * sunspider-compare-results: Added logic to compare the baseline with the last
132         run when you don't pass any parameters.
133
134 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
135
136         Reviewed by Eric.
137         
138         - Added list of ECMAScript3 features that SunSpider does not cover currently.
139
140         * UNCOVERED: Added.
141
142 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
143
144         Reviewed by Oliver.
145         
146         - Give better error messages when sunspider-compare-results fails
147
148         * sunspider-compare-results:
149
150 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
151
152         Reviewed by Darin.
153         
154         - fix Date tests to work in command-line SpiderMonkey
155
156         * tests/date-format-tofte.js:
157         * tests/date-format-xparb.js:
158
159 2007-10-29  Darin Adler  <darin@apple.com>
160
161         Reviewed by Maciej.
162
163         * sunspider: Added --shark20 option, to run Shark at its highest sample
164         resolution instead of its default.
165
166 2007-10-28  Adam Roben  <aroben@apple.com>
167
168         Fix SunSpider on Windows
169
170         Reviewed by Eric.
171
172         * sunspider: Don't use colons in filenames.
173
174 2007-10-28  Darin Adler  <darin@apple.com>
175
176         Reviewed by Adam.
177
178         * resources/sunspider-analyze-results.js: Tweak the output format a little.
179         Change so that when there's only one run we don't write out confidence
180         intervals at all rather than writing "NaN" over and over again.
181
182         * sunspider: Use the time and date as a suffix on the results file, that way
183         all the results are left behind in the tmp directory. This will make it easy
184         for us to add features that compare past results in the future. Also rename
185         the .mshark files using the same scheme.
186
187         * sunspider-compare-results: Relax the parsing rules so we can parse the
188         results file exactly as it's emitted from sunspider.
189
190 2007-10-25  Adam Roben  <aroben@apple.com>
191
192         Strip carriage returns from results
193
194         Carriage returns were screwing up sunspider-compare-results on
195         Windows.
196
197         Rubberstamped by Oliver.
198
199         * sunspider:
200
201 2007-10-24  Darin Adler  <darin@apple.com>
202
203         Reviewed by Eric.
204
205         * sunspider: Open the profile in shark after completing a run.
206         * sunspider-compare-results: Tiny perl tweaks.
207
208 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
209
210         Reviewed by Darin.
211
212         - Don't hardcode my path to testkjs
213         
214         * sunspider-compare-results:
215
216 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
217
218         Reviewed by Darin.
219         
220         - Add a compare script to compare results, and improve formatting of standard results a bit.
221
222         * resources/sunspider-analyze-results.js:
223         * resources/sunspider-compare-results.js: Added.
224         * sunspider-compare-results: Added.
225
226 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
227
228         Reviewed by Darin.
229
230         * resources/sunspider-analyze-results.js: Do the error range
231         properly, using the t-distribution instead of 1.96 (which was
232         based on the normal distribution).
233         
234         * sunspider: Print results in a way that will be friendlier to the
235         soon-to-come compare mode.
236
237 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
238
239         Reviewed by Review.
240         
241         - add more tests (probably the final set for now)
242         
243         Already balanced. Added date, regexp, control flow, and a few
244         more string and object/array access tests.
245         
246         * tests/LIST:
247         * tests/access-fannkuch.js: Added.
248         * tests/access-nbody.js: Added.
249         * tests/controlflow-recursive.js: Added.
250         * tests/date-format-tofte.js: Added.
251         * tests/date-format-xparb.js: Added.
252         * tests/regexp-dna.js: Added.
253         * tests/string-validate-input.js: Added.
254
255 2007-10-21  Eric Seidel  <eric@webkit.org>
256
257         Reviewed by mjs.
258
259         * sunspider: make --shark only sample from testkjs
260
261 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
262
263         Rubber stamped by Eric.
264         
265         - added crypto tests (already balanced)
266
267         * tests/LIST:
268         * tests/crypto-aes.js: Added.
269         * tests/crypto-md5.js: Added.
270         * tests/crypto-sha1.js: Added.
271
272 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
273
274         Reviewed by Adam.
275         
276         - rebalance new tests
277         
278         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
279         
280         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
281
282         This balances them with the existing tests.
283
284         * tests/access-binary-trees.js:
285         * tests/access-nsieve.js:
286         * tests/bitops-nsieve-bits.js:
287         * tests/math-partial-sums.js:
288         * tests/math-spectral-norm.js:
289         * tests/string-fasta.js:
290
291 2007-10-20  Eric Seidel  <eric@webkit.org>
292
293         Reviewed by darin.
294         
295         Add improved argument handling to sunspider, including
296         --runs=<number>, --tests=<pattern, --shark, and --help
297         Also re-factor code into subroutines
298
299         * sunspider: improved argument handling
300
301 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
302
303         Reviewed by Mark.
304
305         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
306         
307         * TODO:
308         * tests/LIST:
309         * tests/access-binary-trees.js: Added.
310         * tests/access-nsieve.js: Added.
311         * tests/bitops-nsieve-bits.js: Added.
312         * tests/math-partial-sums.js: Added.
313         * tests/math-spectral-norm.js: Added.
314         * tests/string-fasta.js: Added.
315
316 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
317
318         Reviewed by Darin.
319
320         * TODO: Updated to note areas that are now well-covered.
321         
322         - Change the tests and the driver to avoid leaving large object
323         graphs hanging around, since that throws off the subsequent tests.
324         
325         * resources/sunspider-standalone-driver.js:
326         * tests/3d-cube.js:
327         * tests/3d-morph.js:
328         * tests/3d-raytrace.js:
329         * tests/bitops-bitwise-and.js:
330         * tests/string-base64.js:
331         * tests/string-tagcloud.js:
332
333 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
334
335         Reviewed by Nikolas.
336         
337         - Rebalance test complexity.
338         
339         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
340         
341         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
342
343         However, for some tests, IE7 was a huge outlier compared to other
344         major browsers. For those tests, I used Firefox 2.0.0.8 on the
345         same OS and hardware as a normalization basis.
346         
347         * tests/3d-morph.js:
348         * tests/3d-raytrace.js:
349         * tests/bitops-3bit-bits-in-byte.js:
350         * tests/bitops-bits-in-byte.js:
351         * tests/bitops-bitwise-and.js:
352         * tests/math-cordic.js:
353         * tests/string-base64.js:
354         * tests/string-tagcloud.js:
355         * tests/string-unpack-code.js:
356
357 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
358
359         Reviewed by Oliver.
360         
361         - Fix the browser-hosted driver to be compatible with Safari 2
362
363         * hosted/sunspider-driver.html:
364
365 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
366
367         Rubber stamped by Hyatt.
368         
369         - Add BSD license notice to cordic test, the original author said to do so.
370
371         * tests/math-cordic.js:
372
373 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
374
375         Reviewed by Eric.
376         
377         - Support Windows and Windows browsers.
378
379         * hosted/sunspider-results.html: Display results correctly in IE.
380         * sunspider: tweaks to work on Windows, and with spaces in the path.
381
382 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
383
384         Rubber stamped by Oliver.
385
386         * tests/string-tagcloud.js: Remove stray print()
387
388 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
389
390         Reviewed by Oliver.
391         
392         Add some string test cases.
393
394         * tests/string-base64.js: Added. Does base64 encoding and decoding.
395         * tests/string-tagcloud.js: Added. Parses a JSON string and
396         generates tagcloud HTML markup based on the data.        
397         * tests/string-unpack-code.js: Added. Unpack compressed versions of
398         four of the most popular JavaScript libraries.
399         * tests/LIST: Added the new tests.
400
401 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
402
403         Reviewed by Anders.
404
405         * TODO: Updated with info on how to do the stats properly.
406
407         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
408         avoid overly slowing down the benchmark.
409
410         Added some 3D-related test cases.
411         
412         * tests/3d-cube.js: Added.
413         * tests/3d-morph.js: Added.
414         * tests/3d-raytrace.js: Added.
415         * tests/LIST:
416
417 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
418
419         Reviewed by Darin.
420
421         - New JavaScript benchmark
422         http://bugs.webkit.org/show_bug.cgi?id=15515
423         
424         A start on a new JavaScript benchmark, with standalone and
425         browser-hosted drivers.
426         
427         The test content is still incomplete and the drivers could use
428         some more features but there is enough here to make a good start.
429         
430         The drivers compute a 95% confidence interval on the mean for the
431         whole test, each category, and each individual test to make it
432         easier to tell whether differences are statistically
433         significant. The confidence interval can be narrowed by running
434         the test more times.
435         
436         * TODO: Added. Remaining things that need to be done.
437         * sunspider: Added. Perl script that acts as the standalone test driver.
438         * resources/sunspider-standalone-driver.js: Added. JavaScript part
439         of standalone test driver.
440         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
441         analysis code.
442         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
443         * make-hosted: Added. Script to generate browser-hosted tests.
444         * hosted/sunspider.html: Added. Start page for browser-hosted test.
445         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
446         * hosted/sunspider-results.html: Added. Results page for browser-hosted
447         tests.
448         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
449         tests.
450         * tests/LIST: Added. List of tests to use.
451         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
452         * tests/bitops-bits-in-byte.js: Added. Ditto.
453         * tests/bitops-bitwise-and.js: Added. Ditto.
454         * tests/math-cordic.js: Added. Ditto.
455