run-benchmark should support SunSpider, Kraken and Octane
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 11 May 2015 21:40:52 +0000 (21:40 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 11 May 2015 21:40:52 +0000 (21:40 +0000)
https://bugs.webkit.org/show_bug.cgi?id=144840

Reviewed by Darin Adler.

Added the support for SunSpider, Kraken, and Octane. Because of the licensing issues, we can't commit the
Kraken source code into the WebKit repository as done for other benchmarks. Instead, we'll dynamically
download it from the remote server using newly added RemoteZipBenchmarkBuilder. We do the same for Octane
for simplicity. Use newly added --local-copy option to specify the location of a local copy if there is any.

Renamed "original_benchmark in the plan file to "local_copy" and added a new optional "remote_archive" to
specify the URL to a remote ZIP file. This optional field is used by Kraken and Octane benchmark plans.

In addition, generalized the ability to run a "create script" in JetStreamBenchmarkBuilder since it's also
needed for SunSpider and Kraken. This feature has now been folded into GenericBenchmarkBuilder.

* Scripts/webkitpy/benchmark_runner/benchmark_builder/benchmark_builders.json: Added
RemoteZipBenchmarkBuilder for Kraken and removed JetStreamBenchmarkBuilder since GenericBenchmarkBuilder
now has the ability to run an arbitrary "create script".

* Scripts/run-benchmark:
(main): Added an optional argument, --local-copy, to override the location of the benchmark's local copy.
This also overrides the remote archive URL specified in the plan.

* Scripts/webkitpy/benchmark_runner/README.md: Updated the description.

* Scripts/webkitpy/benchmark_runner/benchmark_builder/benchmark_builders.json: Deleted the entry for
JetStreamBenchmarkBuilder.

* Scripts/webkitpy/benchmark_runner/benchmark_builder/generic_benchmark_builder.py:
(GenericBenchmarkBuilder.prepare): Call _fetchRemoteArchive if local_copy is not specified in the plan or
by --local-copy option but remote_archive is specified. Also call self.clean() here instead of relying on
_runCreateScript and _applyPatch to clean after themselves.
(GenericBenchmarkBuilder._runCreateScript): Moved from JetStreamBenchmarkBuilder._runCreateScript since
JetStream, SunSpider, Kraken all use this feature.
(GenericBenchmarkBuilder._copyBenchmarkToTempDir): Use self.name as the destination location instead of
the leaf directory name since the latter is not available when a remote ZIP file is used.
(GenericBenchmarkBuilder._fetchRemoteArchive): Added.
(GenericBenchmarkBuilder._applyPatch): Apply the patch inside destination directory to avoid hard coding
the benchmark name in the patches.

* Scripts/webkitpy/benchmark_runner/benchmark_builder/jetstream_benchmark_builder.py: Removed.

* Scripts/webkitpy/benchmark_runner/benchmark_runner.py:
(BenchmarkRunner.__init__): Set self.planName. Also override 'local_copy' when --local-copy option is used.
(BenchmarkRunner.execute): Exit early if neither local_copy nor remote_archive is specified. Prefix the URL
the browser opens by planName as the plan file no longer contains that.

* Scripts/webkitpy/benchmark_runner/data/patches/JetStream.patch: Changed the path to be relative against
the top directory of JetStream instead of its parent.
* Scripts/webkitpy/benchmark_runner/data/patches/Kraken.patch: Added.
* Scripts/webkitpy/benchmark_runner/data/patches/Octane.patch: Added.
* Scripts/webkitpy/benchmark_runner/data/patches/Speedometer.patch: Changed the path to be relative against
the top directory of JetStream instead of its parent.
* Scripts/webkitpy/benchmark_runner/data/patches/SunSpider.patch: Added.
* Scripts/webkitpy/benchmark_runner/data/plans/jetstream.plan: Uses GenericBenchmarkBuilder and specifies
the script to run. The entry point was changed to the relative path from the top directory of JetStream as
done in JetStream.patch.
* Scripts/webkitpy/benchmark_runner/data/plans/kraken.plan: Added.
* Scripts/webkitpy/benchmark_runner/data/plans/octane.plan: Added.
* Scripts/webkitpy/benchmark_runner/data/plans/speedometer.plan: The entry point was changed to the relative
path from the top directory of JetStream as done in JetStream.patch.
* Scripts/webkitpy/benchmark_runner/data/plans/sunspider.plan: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@184124 268f45cc-cd09-0410-ab3c-d52691b4dbfc

17 files changed:
Tools/ChangeLog
Tools/Scripts/run-benchmark
Tools/Scripts/webkitpy/benchmark_runner/README.md
Tools/Scripts/webkitpy/benchmark_runner/benchmark_builder/benchmark_builders.json
Tools/Scripts/webkitpy/benchmark_runner/benchmark_builder/generic_benchmark_builder.py
Tools/Scripts/webkitpy/benchmark_runner/benchmark_builder/jetstream_benchmark_builder.py [deleted file]
Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner.py
Tools/Scripts/webkitpy/benchmark_runner/data/patches/JetStream.patch
Tools/Scripts/webkitpy/benchmark_runner/data/patches/Kraken.patch [new file with mode: 0644]
Tools/Scripts/webkitpy/benchmark_runner/data/patches/Octane.patch [new file with mode: 0644]
Tools/Scripts/webkitpy/benchmark_runner/data/patches/Speedometer.patch
Tools/Scripts/webkitpy/benchmark_runner/data/patches/SunSpider.patch [new file with mode: 0644]
Tools/Scripts/webkitpy/benchmark_runner/data/plans/jetstream.plan
Tools/Scripts/webkitpy/benchmark_runner/data/plans/kraken.plan [new file with mode: 0644]
Tools/Scripts/webkitpy/benchmark_runner/data/plans/octane.plan [new file with mode: 0644]
Tools/Scripts/webkitpy/benchmark_runner/data/plans/speedometer.plan
Tools/Scripts/webkitpy/benchmark_runner/data/plans/sunspider.plan [new file with mode: 0644]

index 15d8e8dedac517cf89703cb2e0f8ab17f731feea..c84e97f441bff31117649ac08c0d2783d3560dfd 100644 (file)
@@ -1,3 +1,69 @@
+2015-05-11  Ryosuke Niwa  <rniwa@webkit.org>
+
+        run-benchmark should support SunSpider, Kraken and Octane
+        https://bugs.webkit.org/show_bug.cgi?id=144840
+
+        Reviewed by Darin Adler.
+
+        Added the support for SunSpider, Kraken, and Octane. Because of the licensing issues, we can't commit the
+        Kraken source code into the WebKit repository as done for other benchmarks. Instead, we'll dynamically
+        download it from the remote server using newly added RemoteZipBenchmarkBuilder. We do the same for Octane
+        for simplicity. Use newly added --local-copy option to specify the location of a local copy if there is any.
+
+        Renamed "original_benchmark in the plan file to "local_copy" and added a new optional "remote_archive" to
+        specify the URL to a remote ZIP file. This optional field is used by Kraken and Octane benchmark plans.
+
+        In addition, generalized the ability to run a "create script" in JetStreamBenchmarkBuilder since it's also
+        needed for SunSpider and Kraken. This feature has now been folded into GenericBenchmarkBuilder.
+
+        * Scripts/webkitpy/benchmark_runner/benchmark_builder/benchmark_builders.json: Added
+        RemoteZipBenchmarkBuilder for Kraken and removed JetStreamBenchmarkBuilder since GenericBenchmarkBuilder
+        now has the ability to run an arbitrary "create script".
+
+        * Scripts/run-benchmark:
+        (main): Added an optional argument, --local-copy, to override the location of the benchmark's local copy.
+        This also overrides the remote archive URL specified in the plan.
+
+        * Scripts/webkitpy/benchmark_runner/README.md: Updated the description.
+
+        * Scripts/webkitpy/benchmark_runner/benchmark_builder/benchmark_builders.json: Deleted the entry for
+        JetStreamBenchmarkBuilder.
+
+        * Scripts/webkitpy/benchmark_runner/benchmark_builder/generic_benchmark_builder.py: 
+        (GenericBenchmarkBuilder.prepare): Call _fetchRemoteArchive if local_copy is not specified in the plan or
+        by --local-copy option but remote_archive is specified. Also call self.clean() here instead of relying on
+        _runCreateScript and _applyPatch to clean after themselves.
+        (GenericBenchmarkBuilder._runCreateScript): Moved from JetStreamBenchmarkBuilder._runCreateScript since
+        JetStream, SunSpider, Kraken all use this feature.
+        (GenericBenchmarkBuilder._copyBenchmarkToTempDir): Use self.name as the destination location instead of
+        the leaf directory name since the latter is not available when a remote ZIP file is used.
+        (GenericBenchmarkBuilder._fetchRemoteArchive): Added.
+        (GenericBenchmarkBuilder._applyPatch): Apply the patch inside destination directory to avoid hard coding
+        the benchmark name in the patches.
+
+        * Scripts/webkitpy/benchmark_runner/benchmark_builder/jetstream_benchmark_builder.py: Removed.
+
+        * Scripts/webkitpy/benchmark_runner/benchmark_runner.py:
+        (BenchmarkRunner.__init__): Set self.planName. Also override 'local_copy' when --local-copy option is used.
+        (BenchmarkRunner.execute): Exit early if neither local_copy nor remote_archive is specified. Prefix the URL
+        the browser opens by planName as the plan file no longer contains that.
+
+        * Scripts/webkitpy/benchmark_runner/data/patches/JetStream.patch: Changed the path to be relative against
+        the top directory of JetStream instead of its parent.
+        * Scripts/webkitpy/benchmark_runner/data/patches/Kraken.patch: Added.
+        * Scripts/webkitpy/benchmark_runner/data/patches/Octane.patch: Added.
+        * Scripts/webkitpy/benchmark_runner/data/patches/Speedometer.patch: Changed the path to be relative against
+        the top directory of JetStream instead of its parent.
+        * Scripts/webkitpy/benchmark_runner/data/patches/SunSpider.patch: Added.
+        * Scripts/webkitpy/benchmark_runner/data/plans/jetstream.plan: Uses GenericBenchmarkBuilder and specifies
+        the script to run. The entry point was changed to the relative path from the top directory of JetStream as
+        done in JetStream.patch.
+        * Scripts/webkitpy/benchmark_runner/data/plans/kraken.plan: Added.
+        * Scripts/webkitpy/benchmark_runner/data/plans/octane.plan: Added.
+        * Scripts/webkitpy/benchmark_runner/data/plans/speedometer.plan: The entry point was changed to the relative
+        path from the top directory of JetStream as done in JetStream.patch.
+        * Scripts/webkitpy/benchmark_runner/data/plans/sunspider.plan: Added.
+
 2015-05-11  Csaba Osztrogon√°c  <ossy@webkit.org>
 
         Fix run-javascriptcore-tests step on the WinCairo bot
index 6d5f66e359872947b635e55f22be9be59e604644..77fc4724457a1ea69a0b51ab569174a32d334ab9 100755 (executable)
@@ -26,6 +26,7 @@ def main():
     # FIXME: Should we add chrome as an option? Well, chrome uses webkit in iOS.
     parser.add_argument('--browser', dest='browser', required=True, choices=BrowserDriverFactory.available_browsers())
     parser.add_argument('--debug', action='store_true')
+    parser.add_argument('--local-copy', dest='localCopy', help='Path to a local copy of the benchmark. e.g. PerformanceTests/SunSpider/')
     args = parser.parse_args()
 
     if args.debug:
@@ -34,7 +35,7 @@ def main():
     _log.debug('\toutput file name\t: %s' % args.output)
     _log.debug('\tbuild directory\t: %s' % args.buildDir)
     _log.debug('\tplan name\t: %s', args.plan)
-    runner = BenchmarkRunner(args.plan, args.buildDir, args.output, args.platform, args.browser)
+    runner = BenchmarkRunner(args.plan, args.localCopy, args.buildDir, args.output, args.platform, args.browser)
     return runner.execute()
 
 
index 0cad4a198e3e0716034c3e865dda41eba0980047..6b952c85ff4d07cb92c6d59d1addf90dc4ad3871 100644 (file)
@@ -58,7 +58,7 @@ To create a plan, you may refer to Plans/jetstream.plan.
     "timeout" : 600,
     "count": 5,
     "benchmark_builder": "JetStreamBenchmarkBuilder",
-    "original_benchmark": "../../../../PerformanceTests/JetStream",
+    "local_copy": "../../../../PerformanceTests/JetStream",
     "benchmark_patch": "data/patches/JetStream.patch",
     "entry_point": "JetStream/JetStream-1.0.1/index.html",
     "output_file": "jetstream.result"
@@ -69,9 +69,10 @@ Plan is a json-formatted dictionary which contains following keys
 * **timeout**: time limit for **EACH RUN** of the benchmark. This can avoid program getting stuck in the extreme circumstances. The time limit is suggested to be 1.5-2x the time spent in a normal run.
 * **count**: the number of times you want to run benchmark
 * **benchmark_builder**:  builder of the benchmark which is responsible for arranging benchmark before the web server serving the directory. In most case, 'GenericBenchmarkHandler' is sufficient. It copies the benchmark to a temporary directory and applies patch to benchmark. If you have special requirement, you could design your own benchmark handle, just like the 'JetStreamBenchmarkHandle' in this example.
-* **original_benchmark**: path of benchmark, a relative path to the root of this project ('benchmark_runner' directory)
+* **local_copy**: path of benchmark, a relative path to the root of this project ('benchmark_runner' directory)
+* **remote_archive**: (**OPTIONAL**) URL of the remote (http/https) ZIP file that contains the benchmark.
 * **benchmark_path**: (**OPTIONAL**) path of patch, a relative path to the root of this project ('benchmark_runner' directory)
-* **entry_point**: the relative url you want browser to launch (a relative path to the webRoot)
+* **entry_point**: the relative url you want browser to launch (a relative path to the benchmark directory)
 * **output_file**: specify the output file, this can be overwritten by specifying '--output-file' while invoking run-benchmark script
 
 ### How to import a benchmark
index b8fc932e6e9de5a114adf8c0e0a560dc35e41056..1f4bdf3cf8fa022bd6054b1b96a0a64e7d03d31d 100644 (file)
@@ -2,9 +2,5 @@
     "GenericBenchmarkBuilder": {
         "filePath": "benchmark_builder.generic_benchmark_builder",
         "moduleName": "GenericBenchmarkBuilder"
-    },
-    "JetStreamBenchmarkBuilder": {
-        "filePath": "benchmark_builder.jetstream_benchmark_builder",
-        "moduleName": "JetStreamBenchmarkBuilder"
     }
 }
index 6ac26f34aadd3bf6068508f3783af5750692bf55..010cd9964bc911843f9a61d7d519fb0f8a6d7273 100644 (file)
@@ -3,9 +3,11 @@
 import logging
 import tempfile
 import os
+import urllib
 import shutil
 import subprocess
 
+from zipfile import ZipFile
 from webkitpy.benchmark_runner.utils import getPathFromProjectRoot, forceRemove
 
 
@@ -14,27 +16,61 @@ _log = logging.getLogger(__name__)
 
 class GenericBenchmarkBuilder(object):
 
-    def prepare(self, benchmarkPath, patch):
-        self._copyBenchmarkToTempDir(benchmarkPath)
-        return self._applyPatch(patch)
+    def prepare(self, name, benchmarkPath, archiveURL, patch, createScript):
+        self.name = name
+        self.webRoot = tempfile.mkdtemp()
+        self.dest = os.path.join(self.webRoot, self.name)
+        if benchmarkPath:
+            self._copyBenchmarkToTempDir(benchmarkPath)
+        else:
+            assert(archiveURL)
+            self._fetchRemoteArchive(archiveURL)
+
+        _log.info('Copied the benchmark into: %s' % self.dest)
+        try:
+            if createScript:
+                self._runCreateScript(createScript)
+            return self._applyPatch(patch)
+        except:
+            self.clean()
+            raise
+
+    def _runCreateScript(self, createScript):
+        oldWorkingDirectory = os.getcwd()
+        os.chdir(self.dest)
+        _log.debug('Running %s in %s' % (createScript, self.dest))
+        errorCode = subprocess.call(createScript)
+        os.chdir(oldWorkingDirectory)
+        if errorCode:
+            raise Exception('Cannot create the benchmark', errorCode)
 
     def _copyBenchmarkToTempDir(self, benchmarkPath):
-        self.webRoot = tempfile.mkdtemp()
-        _log.debug('Servering at webRoot: %s' % self.webRoot)
-        self.dest = os.path.join(self.webRoot, os.path.split(benchmarkPath)[1])
         shutil.copytree(getPathFromProjectRoot(benchmarkPath), self.dest)
 
+    def _fetchRemoteArchive(self, archiveURL):
+        archivePath = os.path.join(self.webRoot, 'archive.zip')
+        _log.info('Downloading %s to %s' % (archiveURL, archivePath))
+        urllib.urlretrieve(archiveURL, archivePath)
+
+        with ZipFile(archivePath, 'r') as archive:
+            archive.extractall(self.dest)
+
+        unarchivedFiles = filter(lambda name: not name.startswith('.'), os.listdir(self.dest))
+        if len(unarchivedFiles) == 1:
+            firstFile = os.path.join(self.dest, unarchivedFiles[0])
+            if os.path.isdir(firstFile):
+                shutil.move(firstFile, self.webRoot)
+                os.rename(os.path.join(self.webRoot, unarchivedFiles[0]), self.dest)
+
     def _applyPatch(self, patch):
         if not patch:
             return self.webRoot
         oldWorkingDirectory = os.getcwd()
-        os.chdir(self.webRoot)
+        os.chdir(self.dest)
         errorCode = subprocess.call(['patch', '-p1', '-f', '-i', getPathFromProjectRoot(patch)])
         os.chdir(oldWorkingDirectory)
         if errorCode:
-            _log.error('Cannot apply patch, will skip current benchmarkPath')
-            self.clean()
-            return None
+            raise Exception('Cannot apply patch, will skip current benchmarkPath')
         return self.webRoot
 
     def clean(self):
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/benchmark_builder/jetstream_benchmark_builder.py b/Tools/Scripts/webkitpy/benchmark_runner/benchmark_builder/jetstream_benchmark_builder.py
deleted file mode 100644 (file)
index 3d958e3..0000000
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/usr/bin/env python
-
-import logging
-import os
-import subprocess
-
-from generic_benchmark_builder import GenericBenchmarkBuilder
-
-
-_log = logging.getLogger(__name__)
-
-
-class JetStreamBenchmarkBuilder(GenericBenchmarkBuilder):
-
-    def prepare(self, benchmarkPath, patch):
-        super(self.__class__, self)._copyBenchmarkToTempDir(benchmarkPath)
-        self._runCreateScript()
-        return super(self.__class__, self)._applyPatch(patch)
-
-    def _runCreateScript(self):
-        oldWorkingDirectory = os.getcwd()
-        os.chdir(self.dest)
-        _log.debug(self.dest)
-        errorCode = subprocess.call(['ruby', 'create.rb'])
-        os.chdir(oldWorkingDirectory)
-        if errorCode:
-            _log.error('Cannot create JetStream Benchmark')
index 179287d7a20b38304deddc10016cd848ccd641a0..e45091061a217586b8e90f7e83ef69ec63e8089c 100644 (file)
@@ -23,12 +23,15 @@ _log = logging.getLogger(__name__)
 
 class BenchmarkRunner(object):
 
-    def __init__(self, planFile, buildDir, outputFile, platform, browser):
+    def __init__(self, planFile, localCopy, buildDir, outputFile, platform, browser):
         _log.info('Initializing benchmark running')
         try:
             planFile = self._findPlanFile(planFile)
             with open(planFile, 'r') as fp:
+                self.planName = os.path.split(os.path.splitext(planFile)[0])[1]
                 self.plan = json.load(fp)
+                if localCopy:
+                    self.plan['local_copy'] = localCopy
                 self.browserDriver = BrowserDriverFactory.create([platform, browser])
                 self.httpServerDriver = HTTPServerDriverFactory.create([self.plan['http_server_driver']])
                 self.buildDir = os.path.abspath(buildDir) if buildDir else None
@@ -56,12 +59,19 @@ class BenchmarkRunner(object):
         _log.info('Start a new benchmark')
         results = []
         benchmarkBuilder = BenchmarkBuilderFactory.create([self.plan['benchmark_builder']])
-        webRoot = benchmarkBuilder.prepare(self.plan['original_benchmark'], self.plan['benchmark_patch'] if 'benchmark_patch' in self.plan else None)
+
+        if not self.plan.get('local_copy') and not self.plan.get('remote_archive'):
+            _log.error('Either local_copy or remote_archive must be specified in the plan')
+            return 2
+
+        webRoot = benchmarkBuilder.prepare(self.planName, self.plan.get('local_copy'), self.plan.get('remote_archive'),
+            self.plan.get('benchmark_patch'), self.plan.get('create_script'))
         for x in xrange(int(self.plan['count'])):
             _log.info('Start the iteration %d of current benchmark' % (x + 1))
             self.httpServerDriver.serve(webRoot)
             self.browserDriver.prepareEnv()
-            self.browserDriver.launchUrl(urlparse.urljoin(self.httpServerDriver.baseUrl(), self.plan['entry_point']), self.buildDir)
+            url = urlparse.urljoin(self.httpServerDriver.baseUrl(), self.planName + '/' + self.plan['entry_point'])
+            self.browserDriver.launchUrl(url, self.buildDir)
             try:
                 with timeout(self.plan['timeout']):
                     result = json.loads(self.httpServerDriver.fetchResult())
index 04bdbe8c1773c068a862ae8aa3129cb806084f6b..200384e13d6c7ad728cc7f0fd28cbd6fd5712917 100644 (file)
@@ -1,7 +1,7 @@
-diff --git a/JetStream/JetStream-1.0.1/JetStreamDriver.js b/JetStream/JetStream-1.0.1/JetStreamDriver.js
+diff --git a/JetStream-1.0.1/JetStreamDriver.js b/JetStream-1.0.1/JetStreamDriver.js
 index 73ee420..60f587c 100644
---- a/JetStream/JetStream-1.0.1/JetStreamDriver.js
-+++ b/JetStream/JetStream-1.0.1/JetStreamDriver.js
+--- a/JetStream-1.0.1/JetStreamDriver.js
++++ b/JetStream-1.0.1/JetStreamDriver.js
 @@ -448,6 +448,14 @@ var JetStream = (function() {
  
          return rawResults;
@@ -41,10 +41,10 @@ index 73ee420..60f587c 100644
      }
  
      function iterate()
-diff --git a/JetStream/JetStream-1.0.1/index.html b/JetStream/JetStream-1.0.1/index.html
+diff --git a/JetStream-1.0.1/index.html b/JetStream-1.0.1/index.html
 index e27535c..001087d 100644
---- a/JetStream/JetStream-1.0.1/index.html
-+++ b/JetStream/JetStream-1.0.1/index.html
+--- a/JetStream-1.0.1/index.html
++++ b/JetStream-1.0.1/index.html
 @@ -34,8 +34,10 @@
      window.onerror = function() { allIsGood = false; }
  
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/data/patches/Kraken.patch b/Tools/Scripts/webkitpy/benchmark_runner/data/patches/Kraken.patch
new file mode 100644 (file)
index 0000000..899251d
--- /dev/null
@@ -0,0 +1,44 @@
+diff --git a/hosted/kraken-1.1/driver.html b/hosted/kraken-1.1/driver2.html
+index bfb2c94..5ad1704 100644
+--- a/hosted/kraken-1.1/driver.html
++++ b/hosted/kraken-1.1/driver2.html
+@@ -117,20 +117,28 @@ function recordResult(time)
+ function finish()
+ {
+-    var outputString = "{";
+-    outputString += '"v": "kraken-1.1", ';
++    var tests = {};
+     for (var test in output[0]) {
+-        outputString += '"' + test + '":[';
+-        for (var i = 0; i < output.length; i++) {
+-             outputString += output[i][test] + ",";
+-        }
+-        outputString = outputString.substring(0, outputString.length - 1);
+-        outputString += "],";
++        var values = new Array(output.length);
++        for (var i = 0; i < output.length; i++)
++            values[i] = output[i][test];
++        tests[test] = {"metrics": {"current": values}}
+     }
+-    outputString = outputString.substring(0, outputString.length - 1);
+-    outputString += "}";
+-    location = "results.html?" + encodeURI(outputString);
++    var results = JSON.stringify({"Kraken": {"metrics": {"Time": ["Total"]}, "tests": tests}});
++    var xhr = new XMLHttpRequest();
++    xhr.open("POST", "/report");
++    xhr.setRequestHeader("Content-Type", "application/json");
++    xhr.setRequestHeader("Content-Length", results.length);
++    xhr.setRequestHeader("Connection", "close");
++    xhr.onreadystatechange = function() {
++    if(xhr.readyState == XMLHttpRequest.DONE && xhr.status == 200) {
++            closeRequest = new XMLHttpRequest();
++            closeRequest.open("GET", "/shutdown");
++            closeRequest.send();
++        }
++    }
++    xhr.send(results);
+ }
+ </script>
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/data/patches/Octane.patch b/Tools/Scripts/webkitpy/benchmark_runner/data/patches/Octane.patch
new file mode 100644 (file)
index 0000000..2e05e36
--- /dev/null
@@ -0,0 +1,68 @@
+diff --git a/base.js b/base.js
+index 9d6e3de..cddf800
+--- a/base.js
++++ b/base.js
+@@ -167,7 +167,7 @@ BenchmarkSuite.RunSuites = function(runner, skipBenchmarks) {
+     if (runner.NotifyScore) {
+       var score = BenchmarkSuite.GeometricMean(BenchmarkSuite.scores);
+       var formatted = BenchmarkSuite.FormatScore(100 * score);
+-      runner.NotifyScore(formatted);
++      runner.NotifyScore(formatted, 100 * score);
+     }
+   }
+   RunStep();
+@@ -250,7 +250,7 @@ BenchmarkSuite.prototype.NotifyResult = function() {
+   BenchmarkSuite.scores.push(score);
+   if (this.runner.NotifyResult) {
+     var formatted = BenchmarkSuite.FormatScore(100 * score);
+-    this.runner.NotifyResult(this.name, formatted);
++    this.runner.NotifyResult(this.name, formatted, 100 * score);
+   }
+   if (this.reference.length == 2) {
+     var meanLatency = BenchmarkSuite.GeometricMeanLatency(this.results);
+@@ -259,7 +259,7 @@ BenchmarkSuite.prototype.NotifyResult = function() {
+       BenchmarkSuite.scores.push(scoreLatency);
+       if (this.runner.NotifyResult) {
+         var formattedLatency = BenchmarkSuite.FormatScore(100 * scoreLatency)
+-        this.runner.NotifyResult(this.name + "Latency", formattedLatency);
++        this.runner.NotifyResult(this.name + "Latency", formattedLatency, 100 * scoreLatency);
+       }
+     }
+   }
+index 77cbdfb..fe56fc4
+--- a/index.html
++++ b/index.html
+@@ -57,7 +57,9 @@
+     });
+   }
+-  function AddResult(name, result) {
++  var octaneSubtests = {};
++  function AddResult(name, result, rawScore) {
++    octaneSubtests[name] = {"metrics": {"Score": {"current": [rawScore]}}};
+     console.log(name + ': ' + result);
+     var box = document.getElementById("Result-" + name);
+     box.innerHTML = result;
+@@ -75,7 +77,21 @@
+     success = false;
+   }
+-  function AddScore(score) {
++  function AddScore(score, rawScore) {
++    var results = JSON.stringify({"Octane": {"metrics": {"Score": {"current": [rawScore]}}, "tests": octaneSubtests}});
++    var xhr = new XMLHttpRequest();
++    xhr.open("POST", "/report");
++    xhr.setRequestHeader("Content-Type", "application/json");
++    xhr.setRequestHeader("Content-Length", results.length);
++    xhr.setRequestHeader("Connection", "close");
++    xhr.onreadystatechange = function() {
++    if(xhr.readyState == XMLHttpRequest.DONE && xhr.status == 200) {
++            closeRequest = new XMLHttpRequest();
++            closeRequest.open("GET", "/shutdown");
++            closeRequest.send();
++        }
++    }
++    xhr.send(results);
+     var status = document.getElementById("main-banner");
+     if (success) {
+       status.innerHTML = "Octane Score: " + score;
index f4092d59fe606af175d839206a2e9d45d63bb91b..308f9c1b14866ff008c0971d9c20c88aeb992ab9 100644 (file)
@@ -1,7 +1,7 @@
-diff --git a/Speedometer/Full.html b/Speedometer/Full.html
+diff --git a/Full.html b/Full.html
 index 419b9f2..0b04c69 100644
---- a/Speedometer/Full.html
-+++ b/Speedometer/Full.html
+--- a/Full.html
++++ b/Full.html
 @@ -7,7 +7,6 @@
      <script src="resources/main.js" defer></script>
      <script src="resources/benchmark-runner.js" defer></script>
@@ -10,10 +10,10 @@ index 419b9f2..0b04c69 100644
      <script src="resources/tests.js" defer></script>
  </head>
  <body>
-diff --git a/Speedometer/resources/benchmark-report.js b/Speedometer/resources/benchmark-report.js
+diff --git a/resources/benchmark-report.js b/resources/benchmark-report.js
 index c4b4c64..874f597 100644
---- a/Speedometer/resources/benchmark-report.js
-+++ b/Speedometer/resources/benchmark-report.js
+--- a/resources/benchmark-report.js
++++ b/resources/benchmark-report.js
 @@ -1,30 +1,26 @@
  // This file can be customized to report results as needed.
  
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/data/patches/SunSpider.patch b/Tools/Scripts/webkitpy/benchmark_runner/data/patches/SunSpider.patch
new file mode 100644 (file)
index 0000000..c7f7c00
--- /dev/null
@@ -0,0 +1,47 @@
+diff --git a/hosted/sunspider-1.0.1/driver.html b/hosted/sunspider-1.0.1/driver.html
+index 0700bb2..2dde01a 100644
+--- a/hosted/sunspider-1.0.1/driver.html
++++ b/hosted/sunspider-1.0.1/driver.html
+@@ -103,23 +103,28 @@ function recordResult(time)
+ function finish()
+ {
+-    var outputString = "{";
+-    outputString += '"v": "sunspider-1.0.1", ';
++    var tests = {};
+     for (var test in output[0]) {
+-        outputString += '"' + test + '":[';
+-        for (var i = 0; i < output.length; i++) {
+-            var time = output[i][test];
+-            if (time != time)
+-                time = "\"NaN\"";
+-            outputString += time + ",";
+-        }
+-        outputString = outputString.substring(0, outputString.length - 1);
+-        outputString += "],";
++        var values = new Array(output.length);
++        for (var i = 0; i < output.length; i++)
++            values[i] = output[i][test];
++        tests[test] = {"metrics": {"Time": {"current": [values]}}}
+     }
+-    outputString = outputString.substring(0, outputString.length - 1);
+-    outputString += "}";
+-    location = "results.html?" + encodeURI(outputString);
++    var results = JSON.stringify({"SunSpider": {"metrics": {"Time": ["Total"]}, "tests": tests}});
++    var xhr = new XMLHttpRequest();
++    xhr.open("POST", "/report");
++    xhr.setRequestHeader("Content-Type", "application/json");
++    xhr.setRequestHeader("Content-Length", results.length);
++    xhr.setRequestHeader("Connection", "close");
++    xhr.onreadystatechange = function() {
++    if(xhr.readyState == XMLHttpRequest.DONE && xhr.status == 200) {
++            closeRequest = new XMLHttpRequest();
++            closeRequest.open("GET", "/shutdown");
++            closeRequest.send();
++        }
++    }
++    xhr.send(results);
+ }
+ </script>
index b6e7e17b74ab2e2d610283319215d7d5391153c8..e9b422b13874f9e136bd53796d0218d27595b293 100644 (file)
@@ -2,9 +2,10 @@
     "http_server_driver": "SimpleHTTPServerDriver", 
     "timeout" : 600,
     "count": 5,
-    "benchmark_builder": "JetStreamBenchmarkBuilder",
-    "original_benchmark": "../../../../PerformanceTests/JetStream",
+    "benchmark_builder": "GenericBenchmarkBuilder",
+    "remote_archive": "../../../../PerformanceTests/JetStream",
     "benchmark_patch": "data/patches/JetStream.patch",
-    "entry_point": "JetStream/JetStream-1.0.1/index.html",
+    "create_script": ["ruby", "create.rb"],
+    "entry_point": "JetStream-1.0.1/index.html",
     "output_file": "jetstream.result"
 }
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/data/plans/kraken.plan b/Tools/Scripts/webkitpy/benchmark_runner/data/plans/kraken.plan
new file mode 100644 (file)
index 0000000..fd9c9b8
--- /dev/null
@@ -0,0 +1,11 @@
+{
+    "http_server_driver": "SimpleHTTPServerDriver",
+    "timeout": 300,
+    "count": 5,
+    "benchmark_builder": "GenericBenchmarkBuilder",
+    "remote_archive": "http://hg.mozilla.org/projects/kraken/archive/tip.zip",
+    "benchmark_patch": "data/patches/Kraken.patch",
+    "create_script": ["python", "make-hosted.py"],
+    "entry_point": "hosted/kraken-1.1/driver.html",
+    "output_file": "kraken.result"
+}
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/data/plans/octane.plan b/Tools/Scripts/webkitpy/benchmark_runner/data/plans/octane.plan
new file mode 100644 (file)
index 0000000..588871f
--- /dev/null
@@ -0,0 +1,10 @@
+{
+    "http_server_driver": "SimpleHTTPServerDriver",
+    "timeout": 300,
+    "count": 5,
+    "benchmark_builder": "GenericBenchmarkBuilder",
+    "remote_archive": "https://github.com/chromium/octane-benchmark/archive/fab09aef01c2a5560c22cdc1c1a2451c0d0f4cdc.zip",
+    "benchmark_patch": "data/patches/Octane.patch",
+    "entry_point": "index.html?auto=1",
+    "output_file": "octane.result"
+}
index 0214438fc695aadae69ba3d8ff9b86b5391ce8cd..9d840f5e520b35ff7c1d2017565e7f85d8197fd3 100644 (file)
@@ -3,8 +3,8 @@
     "timeout": 300,
     "count": 5,
     "benchmark_builder": "GenericBenchmarkBuilder",
-    "original_benchmark": "../../../../PerformanceTests/Speedometer",
+    "local_copy": "../../../../PerformanceTests/Speedometer",
     "benchmark_patch": "data/patches/Speedometer.patch",
-    "entry_point": "Speedometer/Full.html",
+    "entry_point": "Full.html",
     "output_file": "speedometer.result"
 }
diff --git a/Tools/Scripts/webkitpy/benchmark_runner/data/plans/sunspider.plan b/Tools/Scripts/webkitpy/benchmark_runner/data/plans/sunspider.plan
new file mode 100644 (file)
index 0000000..06bae38
--- /dev/null
@@ -0,0 +1,11 @@
+{
+    "http_server_driver": "SimpleHTTPServerDriver",
+    "timeout": 300,
+    "count": 5,
+    "benchmark_builder": "GenericBenchmarkBuilder",
+    "local_copy": "../../../../PerformanceTests/SunSpider",
+    "benchmark_patch": "data/patches/SunSpider.patch",
+    "create_script": ["perl", "make-hosted"],
+    "entry_point": "hosted/sunspider-1.0.1/driver.html",
+    "output_file": "sunspider.result"
+}