It should be fun and easy to run every possible JavaScript benchmark from the command...
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 30 Sep 2014 21:11:57 +0000 (21:11 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 30 Sep 2014 21:11:57 +0000 (21:11 +0000)
commite53ccba03bbf95560eef05f583feecdc91956945
tree03b4b7e34759faa73d255fb23f3f3efb9fc903f6
parentcb414ee30413f3c5e20d4b9c65219003f13642da
It should be fun and easy to run every possible JavaScript benchmark from the command line
https://bugs.webkit.org/show_bug.cgi?id=137245

Reviewed by Oliver Hunt.

PerformanceTests:

This adds the scaffolding for running Octane version 2 inside run-jsc-benchmarks.
In the future we should just land Octane2 in this directory, and run-jsc-benchmarks
should be changed to point directly at this directory instead of requiring the
Octane path to be configured as part of the configuration file.

* Octane: Added.
* Octane/wrappers: Added.
* Octane/wrappers/jsc-box2d.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-boyer.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-closure.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-decrypt.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-deltablue.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-earley.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-encrypt.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-gbemu.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-jquery.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-mandreel.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-navier-stokes.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-pdfjs.js: Added.
(jscSetUp.PdfJS_window.console.log):
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-raytrace.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-regexp.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-richards.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-splay.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-typescript.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):
* Octane/wrappers/jsc-zlib.js: Added.
(jscSetUp):
(jscTearDown):
(jscRun):

Tools:

We previously had Tools/Scripts/bencher.  Then we stopped adding things to it because we
weren't sure about the licensing of things like Kraken and Octane.  Various people ended up
having their own private scripts for doing benchmark runs, and didn't share them in the open
source community, because of fears about the shady licensing of the benchmarks suites that
they were running. The dominant version of this was "run-jsc-benchmarks", which has a lot of
excellent power - it can run benchmarks through either jsc, DumpRenferTree, or
WebKitTestRunner; it can run tests on any number of remote machines; and it has inside
knowledge about how to run *a lot* of test suites. Many of those test suites are not public,
but some of them are. The non-public tests are exclusively those that were not made by any
WebKit contributor, but which JSC/WebKit devs found useful for testing.

This fixes this weirdness by releasing run-jsc-benchmarks. The paths to the test suites
whose licenses are incompatible with WebKit's (to the extent that they cannot be safely
checked into WebKit svn at all) can be run by passing the path to them via a configuration
file. The default configuration file is ~/.run-jsc-benchmarks. The most important benchmark
suites are Octane version 2 and Kraken version 1.1. We should probably check Octane 2 into
WebKit eventually because it seems that the license is fine. Kraken, on the other hand, will
probably never be checked in because there is no license text anywhere in that benchmark.
A valid ~/.run-jsc-benchmarks file will just be something like:

    {
        "OctanePath": "/path/to/Octane2",
        "KrakenPath": "/path/to/Kraken-1.1/tests/kraken-1.1"
    }

If your ~/.run-jsc-benchmarks file omits the directory for any particular test suite, then
run-jsc-benchmarks will just gracefully avoid running that test suite.

Finally, a word about policy: it is understood that different organizations that do
development on JSC may find themselves having internal benchmarks that they cannot share
because of weird licensing. It happens - usually because the organization doing JSC
development found some test in the wild that is owned by someone else and therefore cannot
be shared. So, we should consider it acceptable to write patches against run-jsc-benchmarks
that add support for some new kind of benchmark suite even if the suite is not made public
as part of the same patch - so long as the patch isn't too invasive. An example of
non-invasiveness is the DSPJS suite, which is implemented using some new classes (like
DSPJSAmmoJSRegularBenchmark) and some calls to otherwise reusable functions (like
emitSelfContainedBenchRunCode). It is obviously super helpful if a benchmark suite can be
completely open-sourced and committed to the WebKit repo - but the reality is that this
can't always be done safely.

* Scripts/bencher: Removed.
* Scripts/run-jsc-benchmarks: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@174123 268f45cc-cd09-0410-ab3c-d52691b4dbfc
21 files changed:
PerformanceTests/ChangeLog
PerformanceTests/Octane/wrappers/jsc-box2d.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-boyer.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-closure.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-decrypt.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-deltablue.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-earley.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-encrypt.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-gbemu.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-jquery.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-mandreel.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-navier-stokes.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-pdfjs.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-raytrace.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-regexp.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-richards.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-splay.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-typescript.js [new file with mode: 0644]
PerformanceTests/Octane/wrappers/jsc-zlib.js [new file with mode: 0644]
Tools/ChangeLog
Tools/Scripts/run-jsc-benchmarks [moved from Tools/Scripts/bencher with 50% similarity]