SunSpider/ChangeLog

   1 2007-12-12  Maciej Stachowiak  <mjs@apple.com>
   2
   3         Reviewed by Sam.
   4
   5         - Change sunspider-compare-results output from showing "% faster" values to "x as fast" values.
   6
   7         A sample of what new-style results would look like:
   8
   9         TEST                   COMPARISON            FROM                 TO             DETAILS
  10         ...
  11         ** TOTAL **:           1.41x as fast     6613.6ms +/- 0.1%   4703.0ms +/- 0.1%     significant
  12         ...
  13             3bit-bits-in-byte: *1.36x as slow*     88.8ms +/- 0.6%    121.1ms +/- 0.6%     significant
  14
  15         * resources/sunspider-compare-results.js:
  16
  17 2007-11-14  Eric Seidel  <eric@webkit.org>
  18
  19         Reviewed by Sam.
  20
  21         * sunspider: add --shark-cache for L2 Cache Miss profiling
  22
  23 2007-11-09  Alexey Proskuryakov  <ap@webkit.org>
  24
  25         Reviewed by Darin.
  26
  27         http://bugs.webkit.org/show_bug.cgi?id=15925
  28         SunSpider should check for Shark being installed
  29
  30         * sunspider: Explicitly test that shark command line tool is installed.
  31
  32 2007-11-04  Darin Adler  <darin@apple.com>
  33
  34         * hosted/sunspider.html: Tweaked formatting a bit.
  35
  36 2007-11-03  Maciej Stachowiak  <mjs@apple.com>
  37
  38         Reviewed by Darin.
  39
  40         - Add compare mode and a description of the benchmark to the hosted version.
  41
  42         * hosted/sunspider-results.html: Add compare mode - you can just paste in another results URL.
  43         * hosted/sunspider.html: Added intro text.
  44         * make-hosted: Copy new file.
  45         * resources/sunspider-compare-results.js: Made more reusable by putting the implementation in
  46         a function.
  47         * resources/sunspider-standalone-compare.js: Added. Calls the function in the way standalone mode
  48         expects.
  49         * sunspider-compare-results: Use sunspider-standalone-compare.js.
  50
  51 2007-11-02  Darin Adler  <darin@apple.com>
  52
  53         * sunspider: Changed "--base" to "--set-baseline".
  54
  55 2007-11-02  Darin Adler  <darin@apple.com>
  56
  57         Reviewed by Maciej.
  58
  59         * sunspider: Added a "--base" option that records the current test as a baseline.
  60         Moved the code to find the newest .mshark file into a function (since I needed to use
  61         it in the other script and I prefer to copy/paste a function).
  62
  63         * sunspider-compare-results: Added logic to compare the baseline with the last
  64         run when you don't pass any parameters.
  65
  66 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
  67
  68         Reviewed by Eric.
  69
  70         - Added list of ECMAScript3 features that SunSpider does not cover currently.
  71
  72         * UNCOVERED: Added.
  73
  74 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
  75
  76         Reviewed by Oliver.
  77
  78         - Give better error messages when sunspider-compare-results fails
  79
  80         * sunspider-compare-results:
  81
  82 2007-10-29  Maciej Stachowiak  <mjs@apple.com>
  83
  84         Reviewed by Darin.
  85
  86         - fix Date tests to work in command-line SpiderMonkey
  87
  88         * tests/date-format-tofte.js:
  89         * tests/date-format-xparb.js:
  90
  91 2007-10-29  Darin Adler  <darin@apple.com>
  92
  93         Reviewed by Maciej.
  94
  95         * sunspider: Added --shark20 option, to run Shark at its highest sample
  96         resolution instead of its default.
  97
  98 2007-10-28  Adam Roben  <aroben@apple.com>
  99
 100         Fix SunSpider on Windows
 101
 102         Reviewed by Eric.
 103
 104         * sunspider: Don't use colons in filenames.
 105
 106 2007-10-28  Darin Adler  <darin@apple.com>
 107
 108         Reviewed by Adam.
 109
 110         * resources/sunspider-analyze-results.js: Tweak the output format a little.
 111         Change so that when there's only one run we don't write out confidence
 112         intervals at all rather than writing "NaN" over and over again.
 113
 114         * sunspider: Use the time and date as a suffix on the results file, that way
 115         all the results are left behind in the tmp directory. This will make it easy
 116         for us to add features that compare past results in the future. Also rename
 117         the .mshark files using the same scheme.
 118
 119         * sunspider-compare-results: Relax the parsing rules so we can parse the
 120         results file exactly as it's emitted from sunspider.
 121
 122 2007-10-25  Adam Roben  <aroben@apple.com>
 123
 124         Strip carriage returns from results
 125
 126         Carriage returns were screwing up sunspider-compare-results on
 127         Windows.
 128
 129         Rubberstamped by Oliver.
 130
 131         * sunspider:
 132
 133 2007-10-24  Darin Adler  <darin@apple.com>
 134
 135         Reviewed by Eric.
 136
 137         * sunspider: Open the profile in shark after completing a run.
 138         * sunspider-compare-results: Tiny perl tweaks.
 139
 140 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
 141
 142         Reviewed by Darin.
 143
 144         - Don't hardcode my path to testkjs
 145
 146         * sunspider-compare-results:
 147
 148 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
 149
 150         Reviewed by Darin.
 151
 152         - Add a compare script to compare results, and improve formatting of standard results a bit.
 153
 154         * resources/sunspider-analyze-results.js:
 155         * resources/sunspider-compare-results.js: Added.
 156         * sunspider-compare-results: Added.
 157
 158 2007-10-23  Maciej Stachowiak  <mjs@apple.com>
 159
 160         Reviewed by Darin.
 161
 162         * resources/sunspider-analyze-results.js: Do the error range
 163         properly, using the t-distribution instead of 1.96 (which was
 164         based on the normal distribution).
 165
 166         * sunspider: Print results in a way that will be friendlier to the
 167         soon-to-come compare mode.
 168
 169 2007-10-22  Maciej Stachowiak  <mjs@apple.com>
 170
 171         Reviewed by Review.
 172
 173         - add more tests (probably the final set for now)
 174
 175         Already balanced. Added date, regexp, control flow, and a few
 176         more string and object/array access tests.
 177
 178         * tests/LIST:
 179         * tests/access-fannkuch.js: Added.
 180         * tests/access-nbody.js: Added.
 181         * tests/controlflow-recursive.js: Added.
 182         * tests/date-format-tofte.js: Added.
 183         * tests/date-format-xparb.js: Added.
 184         * tests/regexp-dna.js: Added.
 185         * tests/string-validate-input.js: Added.
 186
 187 2007-10-21  Eric Seidel  <eric@webkit.org>
 188
 189         Reviewed by mjs.
 190
 191         * sunspider: make --shark only sample from testkjs
 192
 193 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
 194
 195         Rubber stamped by Eric.
 196
 197         - added crypto tests (already balanced)
 198
 199         * tests/LIST:
 200         * tests/crypto-aes.js: Added.
 201         * tests/crypto-md5.js: Added.
 202         * tests/crypto-sha1.js: Added.
 203
 204 2007-10-21  Maciej Stachowiak  <mjs@apple.com>
 205
 206         Reviewed by Adam.
 207
 208         - rebalance new tests
 209
 210         Scaled the new tests so they take 400-600ms on a reasonably normal user setup:
 211
 212         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
 213
 214         This balances them with the existing tests.
 215
 216         * tests/access-binary-trees.js:
 217         * tests/access-nsieve.js:
 218         * tests/bitops-nsieve-bits.js:
 219         * tests/math-partial-sums.js:
 220         * tests/math-spectral-norm.js:
 221         * tests/string-fasta.js:
 222
 223 2007-10-20  Eric Seidel  <eric@webkit.org>
 224
 225         Reviewed by darin.
 226
 227         Add improved argument handling to sunspider, including
 228         --runs=<number>, --tests=<pattern, --shark, and --help
 229         Also re-factor code into subroutines
 230
 231         * sunspider: improved argument handling
 232
 233 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
 234
 235         Reviewed by Mark.
 236
 237         - Add more new tests, mostly from the computer language shootout. Not normalized yet.
 238
 239         * TODO:
 240         * tests/LIST:
 241         * tests/access-binary-trees.js: Added.
 242         * tests/access-nsieve.js: Added.
 243         * tests/bitops-nsieve-bits.js: Added.
 244         * tests/math-partial-sums.js: Added.
 245         * tests/math-spectral-norm.js: Added.
 246         * tests/string-fasta.js: Added.
 247
 248 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
 249
 250         Reviewed by Darin.
 251
 252         * TODO: Updated to note areas that are now well-covered.
 253
 254         - Change the tests and the driver to avoid leaving large object
 255         graphs hanging around, since that throws off the subsequent tests.
 256
 257         * resources/sunspider-standalone-driver.js:
 258         * tests/3d-cube.js:
 259         * tests/3d-morph.js:
 260         * tests/3d-raytrace.js:
 261         * tests/bitops-bitwise-and.js:
 262         * tests/string-base64.js:
 263         * tests/string-tagcloud.js:
 264
 265 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
 266
 267         Reviewed by Nikolas.
 268
 269         - Rebalance test complexity.
 270
 271         Scaled all the tests so they take 400-600ms on a reasonably normal user setup:
 272
 273         Internet Explorer 7, Window XP, Dell Latitude D810 with a 2.13GHz Pentium M
 274
 275         However, for some tests, IE7 was a huge outlier compared to other
 276         major browsers. For those tests, I used Firefox 2.0.0.8 on the
 277         same OS and hardware as a normalization basis.
 278
 279         * tests/3d-morph.js:
 280         * tests/3d-raytrace.js:
 281         * tests/bitops-3bit-bits-in-byte.js:
 282         * tests/bitops-bits-in-byte.js:
 283         * tests/bitops-bitwise-and.js:
 284         * tests/math-cordic.js:
 285         * tests/string-base64.js:
 286         * tests/string-tagcloud.js:
 287         * tests/string-unpack-code.js:
 288
 289 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
 290
 291         Reviewed by Oliver.
 292
 293         - Fix the browser-hosted driver to be compatible with Safari 2
 294
 295         * hosted/sunspider-driver.html:
 296
 297 2007-10-20  Maciej Stachowiak  <mjs@apple.com>
 298
 299         Rubber stamped by Hyatt.
 300
 301         - Add BSD license notice to cordic test, the original author said to do so.
 302
 303         * tests/math-cordic.js:
 304
 305 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
 306
 307         Reviewed by Eric.
 308
 309         - Support Windows and Windows browsers.
 310
 311         * hosted/sunspider-results.html: Display results correctly in IE.
 312         * sunspider: tweaks to work on Windows, and with spaces in the path.
 313
 314 2007-10-18  Maciej Stachowiak  <mjs@apple.com>
 315
 316         Rubber stamped by Oliver.
 317
 318         * tests/string-tagcloud.js: Remove stray print()
 319
 320 2007-10-17  Maciej Stachowiak  <mjs@apple.com>
 321
 322         Reviewed by Oliver.
 323
 324         Add some string test cases.
 325
 326         * tests/string-base64.js: Added. Does base64 encoding and decoding.
 327         * tests/string-tagcloud.js: Added. Parses a JSON string and
 328         generates tagcloud HTML markup based on the data.
 329         * tests/string-unpack-code.js: Added. Unpack compressed versions of
 330         four of the most popular JavaScript libraries.
 331         * tests/LIST: Added the new tests.
 332
 333 2007-10-16  Maciej Stachowiak  <mjs@apple.com>
 334
 335         Reviewed by Anders.
 336
 337         * TODO: Updated with info on how to do the stats properly.
 338
 339         * hosted/sunspider-driver.html: Reduce timeout from 500 to 200 to
 340         avoid overly slowing down the benchmark.
 341
 342         Added some 3D-related test cases.
 343
 344         * tests/3d-cube.js: Added.
 345         * tests/3d-morph.js: Added.
 346         * tests/3d-raytrace.js: Added.
 347         * tests/LIST:
 348
 349 2007-10-14  Maciej Stachowiak  <mjs@apple.com>
 350
 351         Reviewed by Darin.
 352
 353         - New JavaScript benchmark
 354         http://bugs.webkit.org/show_bug.cgi?id=15515
 355
 356         A start on a new JavaScript benchmark, with standalone and
 357         browser-hosted drivers.
 358
 359         The test content is still incomplete and the drivers could use
 360         some more features but there is enough here to make a good start.
 361
 362         The drivers compute a 95% confidence interval on the mean for the
 363         whole test, each category, and each individual test to make it
 364         easier to tell whether differences are statistically
 365         significant. The confidence interval can be narrowed by running
 366         the test more times.
 367
 368         * TODO: Added. Remaining things that need to be done.
 369         * sunspider: Added. Perl script that acts as the standalone test driver.
 370         * resources/sunspider-standalone-driver.js: Added. JavaScript part
 371         of standalone test driver.
 372         * resources/sunspider-analyze-results.js: Added. JavaScript statistical
 373         analysis code.
 374         * resources/TEMPLATE.html: Added. Template for browser-hosted tests.
 375         * make-hosted: Added. Script to generate browser-hosted tests.
 376         * hosted/sunspider.html: Added. Start page for browser-hosted test.
 377         * hosted/sunspider-driver.html: Added. Driver for browser-hosted tests.
 378         * hosted/sunspider-results.html: Added. Results page for browser-hosted
 379         tests.
 380         * hosted/sunspider-record-result.js: Added. Helper file for browser-hosted
 381         tests.
 382         * tests/LIST: Added. List of tests to use.
 383         * tests/bitops-3bit-bits-in-byte.js: Added. Some initial test content.
 384         * tests/bitops-bits-in-byte.js: Added. Ditto.
 385         * tests/bitops-bitwise-and.js: Added. Ditto.
 386         * tests/math-cordic.js: Added. Ditto.
 387