performance tests should be able to measure runs/sec rather than time
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 14 May 2012 05:23:37 +0000 (05:23 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 14 May 2012 05:23:37 +0000 (05:23 +0000)
https://bugs.webkit.org/show_bug.cgi?id=86021

Reviewed by Ojan Vafai.

PerformanceTests:

Add PerfTestRunner.runPerSecond. It uses _runLoop but replaces _runner by _perSecondRunner
to compute runs/s of runFunction.

When _perSecondRunner is called for the first time, i.e. _completedRuns is 0 (notice this is -1
in regular run/_runner), it slowly increases the number of function calls to runFunction between
time measurements in order to discount the time used by new Date() calls themselves until the
total time spent reaches 100 milliseconds.

By default, runPerSecond runs the test for at least 750 milliseconds in each run, and executes
21 runs, yielding the total run time of roughly 18 seconds. This is significantly faster than
most of existing performance tests. Also see http://ejohn.org/blog/accuracy-of-javascript-time/.

Finally, refactored the existing methods of PerfTestRunner to allow "runs/s" unit and share code.

* Layout/flexbox-column-nowrap.html:
* Layout/flexbox-column-wrap.html:
* Layout/flexbox-row-nowrap.html:
* Layout/flexbox-row-wrap.html:
* resources/runner.js:
(PerfTestRunner.computeStatistics): Takes unit.
(PerfTestRunner.logStatistics): Ditto.
(PerfTestRunner._runLoop):
(PerfTestRunner._runner):
(PerfTestRunner.runPerSecond): Added.
(PerfTestRunner._perSecondRunner): Added. Called by _runLoop.
(PerfTestRunner._perSecondRunnerIterator): Added.

Tools:

Allow " runs/s" or " ms" to appear after numerical values in tests.

* Scripts/webkitpy/performance_tests/perftest.py:
(PerfTest):

LayoutTests:

Add tests for PerfTestRunner.runPerSecond.

* fast/harness/perftests/runs-per-second-iterations-expected.txt: Added.
* fast/harness/perftests/runs-per-second-iterations.html: Added.
* fast/harness/perftests/runs-per-second-log-expected.txt: Added.
* fast/harness/perftests/runs-per-second-log.html: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@116916 268f45cc-cd09-0410-ab3c-d52691b4dbfc

13 files changed:
LayoutTests/ChangeLog
LayoutTests/fast/harness/perftests/runs-per-second-iterations-expected.txt [new file with mode: 0644]
LayoutTests/fast/harness/perftests/runs-per-second-iterations.html [new file with mode: 0644]
LayoutTests/fast/harness/perftests/runs-per-second-log-expected.txt [new file with mode: 0644]
LayoutTests/fast/harness/perftests/runs-per-second-log.html [new file with mode: 0644]
PerformanceTests/ChangeLog
PerformanceTests/Layout/flexbox-column-nowrap.html
PerformanceTests/Layout/flexbox-column-wrap.html
PerformanceTests/Layout/flexbox-row-nowrap.html
PerformanceTests/Layout/flexbox-row-wrap.html
PerformanceTests/resources/runner.js
Tools/ChangeLog
Tools/Scripts/webkitpy/performance_tests/perftest.py

index 61590fc31896b671360918fdee5e60cdf9377e45..c4b0b606d26a2a3d436f87f7f47db74f5c6379a1 100644 (file)
@@ -1,3 +1,17 @@
+2012-05-13  Ryosuke Niwa  <rniwa@webkit.org>
+
+        performance tests should be able to measure runs/sec rather than time
+        https://bugs.webkit.org/show_bug.cgi?id=86021
+
+        Reviewed by Ojan Vafai.
+
+        Add tests for PerfTestRunner.runPerSecond.
+
+        * fast/harness/perftests/runs-per-second-iterations-expected.txt: Added.
+        * fast/harness/perftests/runs-per-second-iterations.html: Added.
+        * fast/harness/perftests/runs-per-second-log-expected.txt: Added.
+        * fast/harness/perftests/runs-per-second-log.html: Added.
+
 2012-05-13  Mike Lawther  <mikelawther@chromium.org>
 
         Heap-use-after-free in WTF::HashMap<int, WTF::RefPtr<WebCore::CalculationValue>, WTF::IntHash<unsigned int>, WTF::HashTrait
diff --git a/LayoutTests/fast/harness/perftests/runs-per-second-iterations-expected.txt b/LayoutTests/fast/harness/perftests/runs-per-second-iterations-expected.txt
new file mode 100644 (file)
index 0000000..84103d6
--- /dev/null
@@ -0,0 +1,16 @@
+This test verifies PerfTestRunner.runPerSecond() calls runFunction as many times as expected.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+Returning times: [0, 10, 30, 60, 100, 100, 100, 100, 500]
+PASS callsInIterations[0] is 1
+PASS callsInIterations[1] is 10
+PASS callsInIterations[1] + 5 < callsInIterations[2] is true
+PASS callsInIterations[2] + 10 < callsInIterations[3] is true
+PASS callsInIterations[3] is callsInIterations[4]
+PASS callsInIterations[4] is callsInIterations[5]
+PASS callsInIterations[5] is callsInIterations[6]
+PASS callsInIterations[6] is callsInIterations[7]
+PASS callsInIterations[7] is callsInIterations[8]
+PASS callsInIterations[9] is undefined.
+
diff --git a/LayoutTests/fast/harness/perftests/runs-per-second-iterations.html b/LayoutTests/fast/harness/perftests/runs-per-second-iterations.html
new file mode 100644 (file)
index 0000000..8631e4b
--- /dev/null
@@ -0,0 +1,47 @@
+<!DOCTYPE html>
+<html>
+<body>
+<div id="description"></div>
+<div id="console"></div>
+<script src="../../../fast/js/resources/js-test-pre.js"></script>
+<script src="../../../../PerformanceTests/resources/runner.js"></script>
+<script type="text/javascript">
+
+description("This test verifies PerfTestRunner.runPerSecond() calls runFunction as many times as expected.");
+
+var iteration = 0;
+
+var callsInIterations = [];
+var timesInIterations = [0, 10, 30, 60, 100, 100, 100, 100, 500];
+var logLines = [];
+
+PerfTestRunner.log = function (line) { logLines.push(line); }
+PerfTestRunner._perSecondRunnerIterator = function (callsPerIteration) {
+    callsInIterations[iteration] = callsPerIteration;
+    return timesInIterations[iteration++];
+}
+
+PerfTestRunner.runPerSecond({
+    run: function () { },
+    runCount: 1,
+    timeToRun: 500,
+    done: function () {
+        debug("Returning times: [" + timesInIterations.join(", ") + "]");
+        shouldEvaluateTo("callsInIterations[0]", 1);
+        shouldEvaluateTo("callsInIterations[1]", 10);
+        shouldBeTrue("callsInIterations[1] + 5 < callsInIterations[2]");
+        shouldBeTrue("callsInIterations[2] + 10 < callsInIterations[3]");
+        shouldBe("callsInIterations[3]", "callsInIterations[4]");
+        shouldBe("callsInIterations[4]", "callsInIterations[5]");
+        shouldBe("callsInIterations[5]", "callsInIterations[6]");
+        shouldBe("callsInIterations[6]", "callsInIterations[7]");
+        shouldBe("callsInIterations[7]", "callsInIterations[8]");
+        shouldBeUndefined("callsInIterations[9]");
+    }});
+
+var jsTestIsAsync = true;
+
+</script>
+<script src="../../../fast/js/resources/js-test-post.js"></script>
+</body>
+</html>
diff --git a/LayoutTests/fast/harness/perftests/runs-per-second-log-expected.txt b/LayoutTests/fast/harness/perftests/runs-per-second-log-expected.txt
new file mode 100644 (file)
index 0000000..97e791e
--- /dev/null
@@ -0,0 +1,17 @@
+This test verifies PerfTestRunner.runPerSecond() outputs log as expected.
+
+Running 5 times
+Ignoring warm-up run (0 runs/s)
+1 runs/s
+2 runs/s
+3 runs/s
+4 runs/s
+5 runs/s
+
+
+avg 3 runs/s
+median 3 runs/s
+stdev 1.41 runs/s
+min 1 runs/s
+max 5 runs/s
+
diff --git a/LayoutTests/fast/harness/perftests/runs-per-second-log.html b/LayoutTests/fast/harness/perftests/runs-per-second-log.html
new file mode 100644 (file)
index 0000000..58f355c
--- /dev/null
@@ -0,0 +1,37 @@
+<!DOCTYPE html>
+<html>
+<body>
+<p id="description">This test verifies PerfTestRunner.runPerSecond() outputs log as expected.</p>
+<div id="console"></div>
+<script src="../../../../PerformanceTests/resources/runner.js"></script>
+<script type="text/javascript">
+
+var logLines = [];
+var initial = true;
+var runs = 0;
+
+PerfTestRunner._perSecondRunnerIterator = function (callsPerIteration) {
+    return 1000 / runs;
+}
+
+var printStatistics = PerfTestRunner.printStatistics;
+PerfTestRunner.printStatistics = function (statistics) {
+    statistics.stdev = statistics.stdev.toPrecision(3);
+    return printStatistics.call(PerfTestRunner, statistics);
+}
+
+PerfTestRunner.runPerSecond({
+    setup: function () {
+        if (initial)
+            initial = false;
+        else
+            runs++;
+    },
+    run: function () { },
+    runCount: 5,
+    timeToRun: 500,
+});
+
+</script>
+</body>
+</html>
index b7be623fdcf7afa670b39f048027c289b1253a9f..33c05b05620f377713b11c0f2965ed7bb0bf00ba 100644 (file)
@@ -1,3 +1,37 @@
+2012-05-13  Ryosuke Niwa  <rniwa@webkit.org>
+
+        performance tests should be able to measure runs/sec rather than time
+        https://bugs.webkit.org/show_bug.cgi?id=86021
+
+        Reviewed by Ojan Vafai.
+
+        Add PerfTestRunner.runPerSecond. It uses _runLoop but replaces _runner by _perSecondRunner
+        to compute runs/s of runFunction.
+
+        When _perSecondRunner is called for the first time, i.e. _completedRuns is 0 (notice this is -1
+        in regular run/_runner), it slowly increases the number of function calls to runFunction between
+        time measurements in order to discount the time used by new Date() calls themselves until the
+        total time spent reaches 100 milliseconds.
+
+        By default, runPerSecond runs the test for at least 750 milliseconds in each run, and executes
+        21 runs, yielding the total run time of roughly 18 seconds. This is significantly faster than
+        most of existing performance tests. Also see http://ejohn.org/blog/accuracy-of-javascript-time/.
+
+        Finally, refactored the existing methods of PerfTestRunner to allow "runs/s" unit and share code.
+
+        * Layout/flexbox-column-nowrap.html:
+        * Layout/flexbox-column-wrap.html:
+        * Layout/flexbox-row-nowrap.html:
+        * Layout/flexbox-row-wrap.html:
+        * resources/runner.js:
+        (PerfTestRunner.computeStatistics): Takes unit.
+        (PerfTestRunner.logStatistics): Ditto.
+        (PerfTestRunner._runLoop):
+        (PerfTestRunner._runner):
+        (PerfTestRunner.runPerSecond): Added.
+        (PerfTestRunner._perSecondRunner): Added. Called by _runLoop.
+        (PerfTestRunner._perSecondRunnerIterator): Added.
+
 2012-05-09  Tony Chang  <tony@chromium.org>
 
         add some basic perf-o-matic tests for flexbox
index 6c523eed5350c03d93b71d1454f9339ab9a209b6..bc03ff14344a95c50f3fa8ac86fb1ab21ce1ef05 100644 (file)
@@ -28,8 +28,8 @@ function runTest()
     document.body.clientHeight;
 }
 
-PerfTestRunner.run(runTest, 180, 20, function() {
+PerfTestRunner.runPerSecond({run: runTest, done: function() {
     document.getElementById("flexbox").style.display = 'none';
-});
+}});
 </script>
 </html>
index 78c6b6acc8fa4c5dca5552d1db19941f4c1d56a8..b1287aaee9df574f391313262d624f6ebecff698 100644 (file)
@@ -28,8 +28,8 @@ function runTest()
     document.body.clientHeight;
 }
 
-PerfTestRunner.run(runTest, 180, 20, function() {
+PerfTestRunner.runPerSecond({run: runTest, done:function() {
     document.getElementById("flexbox").style.display = 'none';
-});
+}});
 </script>
 </html>
index b772dada20d0f1a26c4740c9187482b9b7d528c1..ddb15168385815cce0980c98a65968a55abfddc4 100644 (file)
@@ -28,8 +28,8 @@ function runTest()
     document.body.clientHeight;
 }
 
-PerfTestRunner.run(runTest, 100, 20, function() {
+PerfTestRunner.runPerSecond({run:runTest, done:function() {
     document.getElementById("flexbox").style.display = 'none';
-});
+}});
 </script>
 </html>
index 8eca1053e53e2382b6bb3122d456f3e73815f91b..f57e7edc696e291366b3ff068f0e3cd7c58882d3 100644 (file)
@@ -28,8 +28,8 @@ function runTest()
     document.body.clientHeight;
 }
 
-PerfTestRunner.run(runTest, 100, 20, function() {
+PerfTestRunner.runPerSecond({run: runTest, done:function() {
     document.getElementById("flexbox").style.display = 'none';
-});
+}});
 </script>
 </html>
index e14f9662dd57b492606b4cd174d35569a3829ccd..4087eef925784f39f51512163ca67c2011edd001 100644 (file)
@@ -49,7 +49,7 @@ PerfTestRunner.loadFile = function (path) {
     return xhr.responseText;
 }
 
-PerfTestRunner.computeStatistics = function (times) {
+PerfTestRunner.computeStatistics = function (times, unit) {
     var data = times.slice();
 
     // Add values from the smallest to the largest to avoid the loss of significance
@@ -76,14 +76,14 @@ PerfTestRunner.computeStatistics = function (times) {
     }
     result.variance = squareSum / data.length;
     result.stdev = Math.sqrt(result.variance);
-    result.unit = "ms";
+    result.unit = unit || "ms";
 
     return result;
 }
 
 PerfTestRunner.logStatistics = function (times) {
     this.log("");
-    var statistics = this.computeStatistics(times);
+    var statistics = this.computeStatistics(times, this.unit);
     this.printStatistics(statistics);
 }
 
@@ -117,7 +117,7 @@ PerfTestRunner._runLoop = function () {
         this.gc();
         window.setTimeout(function () { PerfTestRunner._runner(); }, 0);
     } else {
-        this.logStatistics(this._times);
+        this.logStatistics(this._results);
         this._doneFunction();
         if (window.layoutTestController)
             layoutTestController.notifyDone();
@@ -140,13 +140,27 @@ PerfTestRunner._runner = function () {
     // Assume totalTime can never be zero when _runFunction returns a number.
     var time = totalTime ? totalTime : Date.now() - start;
 
+    this.ignoreWarmUpAndLog(time);
+    this._runLoop();
+}
+
+PerfTestRunner.ignoreWarmUpAndLog = function (result) {
     this._completedRuns++;
+
+    var labeledResult = result + " " + this.unit;
     if (this._completedRuns <= 0)
-        this.log("Ignoring warm-up run (" + time + ")");
+        this.log("Ignoring warm-up run (" + labeledResult + ")");
     else {
-        this._times.push(time);
-        this.log(time);
+        this._results.push(result);
+        this.log(labeledResult);
     }
+}
+
+PerfTestRunner.initAndStartLoop = function() {
+    this._completedRuns = -1;
+    this.customRunFunction = null;
+    this._results = [];
+    this.log("Running " + this._runCount + " times");
     this._runLoop();
 }
 
@@ -155,14 +169,49 @@ PerfTestRunner.run = function (runFunction, loopsPerRun, runCount, doneFunction)
     this._loopsPerRun = loopsPerRun || 10;
     this._runCount = runCount || 20;
     this._doneFunction = doneFunction || function () {};
-    this._completedRuns = -1;
-    this.customRunFunction = null;
-    this._times = [];
+    this.unit = 'ms';
+    this.initAndStartLoop();
+}
 
-    this.log("Running " + this._runCount + " times");
+PerfTestRunner.runPerSecond = function (test) {
+    this._doneFunction = function () { if (test.done) test.done(); };
+    this._runCount = test.runCount || 20;
+    this._callsPerIteration = 1;
+    this.unit = 'runs/s';
+
+    this._test = test;
+    this._runner = this._perSecondRunner;
+    this.initAndStartLoop();
+}
+
+PerfTestRunner._perSecondRunner = function () {
+    var timeToRun = this._test.timeToRun || 750;
+    var totalTime = 0;
+    var i = 0;
+    var callsPerIteration = this._callsPerIteration;
+
+    if (this._test.setup)
+        this._test.setup();
+
+    while (totalTime < timeToRun) {
+        totalTime += this._perSecondRunnerIterator(callsPerIteration);
+        i += callsPerIteration;
+        if (this._completedRuns <= 0 && totalTime < 100)
+            callsPerIteration = Math.max(10, 2 * callsPerIteration);
+    }
+    this._callsPerIteration = callsPerIteration;
+
+    this.ignoreWarmUpAndLog(i * 1000 / totalTime);
     this._runLoop();
 }
 
+PerfTestRunner._perSecondRunnerIterator = function (callsPerIteration) {
+    var startTime = Date.now();
+    for (var i = 0; i < callsPerIteration; i++)
+        this._test.run();
+    return Date.now() - startTime;
+}
+
 if (window.layoutTestController) {
     layoutTestController.waitUntilDone();
     layoutTestController.dumpAsText();
index a71475db0ca653ed0ae7d247c7771cc37d1e1c74..1600b538a149a67ab76147ce409c0b731325031c 100644 (file)
@@ -1,3 +1,15 @@
+2012-05-13  Ryosuke Niwa  <rniwa@webkit.org>
+
+        performance tests should be able to measure runs/sec rather than time
+        https://bugs.webkit.org/show_bug.cgi?id=86021
+
+        Reviewed by Ojan Vafai.
+
+        Allow " runs/s" or " ms" to appear after numerical values in tests.
+
+        * Scripts/webkitpy/performance_tests/perftest.py:
+        (PerfTest):
+
 2012-05-12  Tim Horton  <timothy_horton@apple.com>
 
         run-safari and co. should support --guard-malloc command line argument
index 509dd1d2b3eec59f795df3cdf82338e9b8f30868..c2f3eba86a1fc52aef6ccc2cd5ab697b2bc0ae7f 100644 (file)
@@ -74,7 +74,7 @@ class PerfTest(object):
         re.compile(r'^Running \d+ times$'),
         re.compile(r'^Ignoring warm-up '),
         re.compile(r'^Info:'),
-        re.compile(r'^\d+(.\d+)?$'),
+        re.compile(r'^\d+(.\d+)?(\s*(runs\/s|ms))?$'),
         # Following are for handle existing test like Dromaeo
         re.compile(re.escape("""main frame - has 1 onunload handler(s)""")),
         re.compile(re.escape("""frame "<!--framePath //<!--frame0-->-->" - has 1 onunload handler(s)""")),