0c268cc9eed22e765fd9a0b30bff29039cf78e75
[WebKit-https.git] / SunSpider / ChangeLog
1 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
2
3         Reviewed by Mark.
4         
5         - fix compare feature
6
7         * hosted/sunspider-results.html:
8
9 2007-12-17  Maciej Stachowiak  <mjs@apple.com>
10
11         Reviewed by Mark.
12         
13         - Change the format of the output URL to be shorter, to meet IE constraints
14
15         * hosted/sunspider-driver.html:
16         * hosted/sunspider-results.html:
17
18 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
19
20         Reviewed by Oliver.
21         
22         - Add some explanatory test and a dumb logo to SunSpider web hosted version
23
24         * hosted/sunspider-driver.html:
25         * hosted/sunspider-results.html:
26         * hosted/sunspider.css:
27         * hosted/sunspider.html:
28
29 2007-12-13  Maciej Stachowiak  <mjs@apple.com>
30
31         Reviewed by Oliver.
32         
33         - Make the web-hosted version of SunSpider look a little bit less drab.
34
35         * hosted/sunspider-driver.html:
36         * hosted/sunspider-results.html:
37         * hosted/sunspider.css: Added.
38         * hosted/sunspider.html:
39         * resources/TEMPLATE.html:
40
41 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
42
43         Reviewed by Sam.
44         
45         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
46         
47         A sample of what new-style results would look like:
48         
49         TEST                   COMPARISON            FROM                 TO             DETAILS
50         ...
51         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
52         ...
53             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
54         
55         * resources/sunspider-compare-results.js:
56
57 2007-11-14  Eric Seidel  <eric@webkit.org>
58
59         Reviewed by Sam.
60
61         * sunspider: add --shark-cache for L2 Cache Miss profiling
62
63 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
64
65         Reviewed by Darin.
66
67         http://bugs.webkit.org/show_bug.cgi?id=15925
68         SunSpider should check for Shark being installed
69
70         * sunspider: Explicitly test that shark command line tool is installed.
71
72 2007-11-04  Darin Adler  <darin@apple.com>
73
74         * hosted/sunspider.html: Tweaked formatting a bit.
75
76 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
77
78         Reviewed by Darin.
79
80         - Add compare mode and a description of the benchmark to the hosted version.
81
82         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
83         * hosted/sunspider.html: Added intro text.
84         * make-hosted: Copy new file.
85         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
86         a function.
87         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
88         expects.
89         * sunspider-compare-results: Use sunspider-standalone-compare.js.
90
91 2007-11-02  Darin Adler  <darin@apple.com>
92
93         * sunspider: Changed "--base" to "--set-baseline".
94
95 2007-11-02  Darin Adler  <darin@apple.com>
96
97         Reviewed by Maciej.
98
99         * sunspider: Added a "--base" option that records the current test as a baseline.
100         Moved the code to find the newest .mshark file into a function (since I needed to use
101         it in the other script and I prefer to copy/paste a function).
102
103         * sunspider-compare-results: Added logic to compare the baseline with the last
104         run when you don't pass any parameters.
105
106 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
107
108         Reviewed by Eric.
109         
110         - Added list of ECMAScript3 features that SunSpider does not cover currently.
111
112         * UNCOVERED: Added.
113
114 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
115
116         Reviewed by Oliver.
117         
118         - Give better error messages when sunspider-compare-results fails
119
120         * sunspider-compare-results:
121
122 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
123
124         Reviewed by Darin.
125         
126         - fix Date tests to work in command-line SpiderMonkey
127
128         * tests/date-format-tofte.js:
129         * tests/date-format-xparb.js:
130
131 2007-10-29  Darin Adler  <darin@apple.com>
132
133         Reviewed by Maciej.
134
135         * sunspider: Added --shark20 option, to run Shark at its highest sample
136         resolution instead of its default.
137
138 2007-10-28  Adam Roben  <aroben@apple.com>
139
140         Fix SunSpider on Windows
141
142         Reviewed by Eric.
143
144         * sunspider: Don't use colons in filenames.
145
146 2007-10-28  Darin Adler  <darin@apple.com>
147
148         Reviewed by Adam.
149
150         * resources/sunspider-analyze-results.js: Tweak the output format a little.
151         Change so that when there's only one run we don't write out confidence
152         intervals at all rather than writing "NaN" over and over again.
153
154         * sunspider: Use the time and date as a suffix on the results file, that way
155         all the results are left behind in the tmp directory. This will make it easy
156         for us to add features that compare past results in the future. Also rename
157         the .mshark files using the same scheme.
158
159         * sunspider-compare-results: Relax the parsing rules so we can parse the
160         results file exactly as it's emitted from sunspider.
161
162 2007-10-25  Adam Roben  <aroben@apple.com>
163
164         Strip carriage returns from results
165
166         Carriage returns were screwing up sunspider-compare-results on
167         Windows.
168
169         Rubberstamped by Oliver.
170
171         * sunspider:
172
173 2007-10-24  Darin Adler  <darin@apple.com>
174
175         Reviewed by Eric.
176
177         * sunspider: Open the profile in shark after completing a run.
178         * sunspider-compare-results: Tiny perl tweaks.
179
180 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
181
182         Reviewed by Darin.
183
184         - Don't hardcode my path to testkjs
185         
186         * sunspider-compare-results:
187
188 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
189
190         Reviewed by Darin.
191         
192         - Add a compare script to compare results, and improve formatting of standard results a bit.
193
194         * resources/sunspider-analyze-results.js:
195         * resources/sunspider-compare-results.js: Added.
196         * sunspider-compare-results: Added.
197
198 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
199
200         Reviewed by Darin.
201
202         * resources/sunspider-analyze-results.js: Do the error range
203         properly, using the t-distribution instead of 1.96 (which was
204         based on the normal distribution).
205         
206         * sunspider: Print results in a way that will be friendlier to the
207         soon-to-come compare mode.
208
209 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
210
211         Reviewed by Review.
212         
213         - add more tests (probably the final set for now)
214         
215         Already balanced. Added date, regexp, control flow, and a few
216         more string and object/array access tests.
217         
218         * tests/LIST:
219         * tests/access-fannkuch.js: Added.
220         * tests/access-nbody.js: Added.
221         * tests/controlflow-recursive.js: Added.
222         * tests/date-format-tofte.js: Added.
223         * tests/date-format-xparb.js: Added.
224         * tests/regexp-dna.js: Added.
225         * tests/string-validate-input.js: Added.
226
227 2007-10-21  Eric Seidel  <eric@webkit.org>
228
229         Reviewed by mjs.
230
231         * sunspider: make --shark only sample from testkjs
232
233 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
234
235         Rubber stamped by Eric.
236         
237         - added crypto tests (already balanced)
238
239         * tests/LIST:
240         * tests/crypto-aes.js: Added.
241         * tests/crypto-md5.js: Added.
242         * tests/crypto-sha1.js: Added.
243
244 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
245
246         Reviewed by Adam.
247         
248         - rebalance new tests
249         
250         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
251         
252         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
253
254         This balances them with the existing tests.
255
256         * tests/access-binary-trees.js:
257         * tests/access-nsieve.js:
258         * tests/bitops-nsieve-bits.js:
259         * tests/math-partial-sums.js:
260         * tests/math-spectral-norm.js:
261         * tests/string-fasta.js:
262
263 2007-10-20  Eric Seidel  <eric@webkit.org>
264
265         Reviewed by darin.
266         
267         Add improved argument handling to sunspider, including
268         --runs=<number>, --tests=<pattern, --shark, and --help
269         Also re-factor code into subroutines
270
271         * sunspider: improved argument handling
272
273 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
274
275         Reviewed by Mark.
276
277         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
278         
279         * TODO:
280         * tests/LIST:
281         * tests/access-binary-trees.js: Added.
282         * tests/access-nsieve.js: Added.
283         * tests/bitops-nsieve-bits.js: Added.
284         * tests/math-partial-sums.js: Added.
285         * tests/math-spectral-norm.js: Added.
286         * tests/string-fasta.js: Added.
287
288 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
289
290         Reviewed by Darin.
291
292         * TODO: Updated to note areas that are now well-covered.
293         
294         - Change the tests and the driver to avoid leaving large object
295         graphs hanging around, since that throws off the subsequent tests.
296         
297         * resources/sunspider-standalone-driver.js:
298         * tests/3d-cube.js:
299         * tests/3d-morph.js:
300         * tests/3d-raytrace.js:
301         * tests/bitops-bitwise-and.js:
302         * tests/string-base64.js:
303         * tests/string-tagcloud.js:
304
305 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
306
307         Reviewed by Nikolas.
308         
309         - Rebalance test complexity.
310         
311         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
312         
313         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
314
315         However, for some tests, IE7 was a huge outlier compared to other
316         major browsers. For those tests, I used Firefox 2.0.0.8 on the
317         same OS and hardware as a normalization basis.
318         
319         * tests/3d-morph.js:
320         * tests/3d-raytrace.js:
321         * tests/bitops-3bit-bits-in-byte.js:
322         * tests/bitops-bits-in-byte.js:
323         * tests/bitops-bitwise-and.js:
324         * tests/math-cordic.js:
325         * tests/string-base64.js:
326         * tests/string-tagcloud.js:
327         * tests/string-unpack-code.js:
328
329 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
330
331         Reviewed by Oliver.
332         
333         - Fix the browser-hosted driver to be compatible with Safari 2
334
335         * hosted/sunspider-driver.html:
336
337 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
338
339         Rubber stamped by Hyatt.
340         
341         - Add BSD license notice to cordic test, the original author said to do so.
342
343         * tests/math-cordic.js:
344
345 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
346
347         Reviewed by Eric.
348         
349         - Support Windows and Windows browsers.
350
351         * hosted/sunspider-results.html: Display results correctly in IE.
352         * sunspider: tweaks to work on Windows, and with spaces in the path.
353
354 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
355
356         Rubber stamped by Oliver.
357
358         * tests/string-tagcloud.js: Remove stray print()
359
360 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
361
362         Reviewed by Oliver.
363         
364         Add some string test cases.
365
366         * tests/string-base64.js: Added. Does base64 encoding and decoding.
367         * tests/string-tagcloud.js: Added. Parses a JSON string and
368         generates tagcloud HTML markup based on the data.        
369         * tests/string-unpack-code.js: Added. Unpack compressed versions of
370         four of the most popular JavaScript libraries.
371         * tests/LIST: Added the new tests.
372
373 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
374
375         Reviewed by Anders.
376
377         * TODO: Updated with info on how to do the stats properly.
378
379         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
380         avoid overly slowing down the benchmark.
381
382         Added some 3D-related test cases.
383         
384         * tests/3d-cube.js: Added.
385         * tests/3d-morph.js: Added.
386         * tests/3d-raytrace.js: Added.
387         * tests/LIST:
388
389 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
390
391         Reviewed by Darin.
392
393         - New JavaScript benchmark
394         http://bugs.webkit.org/show_bug.cgi?id=15515
395         
396         A start on a new JavaScript benchmark, with standalone and
397         browser-hosted drivers.
398         
399         The test content is still incomplete and the drivers could use
400         some more features but there is enough here to make a good start.
401         
402         The drivers compute a 95% confidence interval on the mean for the
403         whole test, each category, and each individual test to make it
404         easier to tell whether differences are statistically
405         significant. The confidence interval can be narrowed by running
406         the test more times.
407         
408         * TODO: Added. Remaining things that need to be done.
409         * sunspider: Added. Perl script that acts as the standalone test driver.
410         * resources/sunspider-standalone-driver.js: Added. JavaScript part
411         of standalone test driver.
412         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
413         analysis code.
414         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
415         * make-hosted: Added. Script to generate browser-hosted tests.
416         * hosted/sunspider.html: Added. Start page for browser-hosted test.
417         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
418         * hosted/sunspider-results.html: Added. Results page for browser-hosted
419         tests.
420         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
421         tests.
422         * tests/LIST: Added. List of tests to use.
423         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
424         * tests/bitops-bits-in-byte.js: Added. Ditto.
425         * tests/bitops-bitwise-and.js: Added. Ditto.
426         * tests/math-cordic.js: Added. Ditto.
427