Update how the benchmark is run
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 9 Feb 2016 03:25:39 +0000 (03:25 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 9 Feb 2016 03:25:39 +0000 (03:25 +0000)
https://bugs.webkit.org/show_bug.cgi?id=153960

Provisionally reviewed by Said Abou-Hallawa.

Introduce the notion of a Controller. It is responsible for recording, updating,
and processing the statistics and complexity of the benchmark. This allows
plugging in different Controllers.

This strips most of the functionality from Animator and BenchmarkState, so fold
what's left into Benchmark. Now, Benchmarks only own a stage and a controller, but
are responsible for driving the animation loop.

Rewrite Animator._shouldRequestAnotherFrame into two different Controllers. One
maintains a fixed complexity, and the other adapts the complexity to meet a
fixed FPS.

Fix the Kalman estimator to be modeled on a scalar variable with no model.

* Animometer/tests/resources/main.js: Remove BenchmarkState and Animator, and
replace it with a Controller. Add a FixedController and refactor the previous controller
to an AdaptiveController.

(Controller): Controllers own the estimator and the sampler. When a new frame is
displayed, the animation loop calls update(). The estimator and sampler record
stats, then tune. Samplers can track multiple series of data. The basic controller
tracks timestamp, complexity, and estimated frame rate.
        The Kalman estimation is based on the frame length rather than the frame
rate. Because FPS is inverse proportional to frame length, in the case where the measured
frame length is very small, the FPS ends up being a wildly large number (in the order of
600-1000 "FPS"), and it pulls the estimator up drastically enough that it takes a while
for it to settle back down. Using frame length reduces the impact of these spikes.
        Converging the estimation takes enough time to avoid initializing it immediately
when the benchmark starts. Instead, the benchmark runs for a brief period of time (100ms)
before running it in earnest. Allow controllers an opportunity to set the complexity
before starting recording.
        When the benchmark is complete, the controller has an opportunity to process
the samples. The default implementation calculates the raw FPS based on the time
difference of the samples, and calculates the complexity score. This is moved from
Benchmark.processSamples.

(Controller): Initialize timestamps. These are at first relative to the start of the
benchmark, but are offset by the absolute start time during start(). By default maintain
3 data series, but subclasses can override.
(start): Calls recordFirstSample() for subclasses to override if needed.
(recordFirstSample): For basic controller, start sampling at the beginning.
(update): Update the frame length estimator and sample.
(shouldStop): Checks that the time is before _endTimestamp.
(results): Returns the processed samples.
(processSamples): Iterate through the sample data and collate them. Include scores.

(FixedComplexityController): Controller that tunes the stage to the desired complexity
prior to starting, and keeps it at that complexity.

(AdaptiveController): Have the estimator estimate the interval frame rate instead of the
raw frame rate.
        The previous version of this controller ignored the frame that came after the
adjustment. The raw FPS show that whatever noise the scene change adds is negligible
compared to the noise of the system overall. Stop ignoring that frame and include all
frames in the measurements.

(Benchmark): Remove dependency on animator, and instantiate a runner based on what is
selected. Most of the loop's functionality is in Controller, so remove here.
(Benchmark.run): Remove start() since it is only called from run(), and fold it in here.
(Benchmark._animateLoop): Fold in from Animator.animateLoop. Let the benchmark run for
a brief period before calling Controller.start().

* Animometer/tests/resources/math.js: Fix the Kalman estimator. The filter estimates
a scalar variable, and makes basic assumptions regarding the model. As a result
none of the linear algebra classes are needed, so remove Matrix, Vector3, and Matrix3.
(SimpleKalmanEstimator): Calculate the gain based on the provided process and
measurement errors.
(KalmanEstimator): Deleted.
(IdentityEstimator): Deleted.
(PIDController): Refactor to use the Utilities.createClass() helper.

The Kalman filter algorithm is explained here http://greg.czerniak.info/guides/kalman1/.
The state, represented by a scalar, is the estimated frame length. There is no user
transition of the state, and the state is the same as the measurement. With this model,
the estimation error converges, so calculate the gain ahead of time.

* Animometer/developer.html: Remove fixed-after-warmup since it is not useful.
Replace the option to toggle the estimator, and make it possible to customize the
estimator's error parameters. Show raw FPS by default, and remove interval FPS,
which will be shown instead of the filtered raw FPS.
* Animometer/resources/debug-runner/animometer.css: Put the header behind the graph.
Remove #intervalFPS rules; move the color to #filteredFPS.
* Animometer/resources/debug-runner/graph.js:
(updateGraphData): Update the hr style to force the layout to be calculated
correctly. Change the tick format to be in terms of seconds, since the timestamps
are in milliseconds. Remove interval data.
* Animometer/resources/runner/animometer.js:
(window.benchmarkController.startBenchmark): Set Kalman parameters.
* Animometer/resources/runner/benchmark-runner.js:
(_runBenchmarkAndRecordResults): When a benchmark completes, expect it to return
the final data, rather than passing a sampler from the controller. This avoids
needing to expose the sampler variable in the benchmark.
* Animometer/tests/resources/sampler.js:
(process): Move the setting of the target frame rate to AdaptiveController.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@196288 268f45cc-cd09-0410-ab3c-d52691b4dbfc

PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/extensions.js
PerformanceTests/Animometer/resources/runner/animometer.js
PerformanceTests/Animometer/resources/runner/benchmark-runner.js
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/Animometer/tests/resources/math.js
PerformanceTests/Animometer/tests/resources/sampler.js
PerformanceTests/ChangeLog

index 9fa3a0a..c58428b 100644 (file)
@@ -44,8 +44,7 @@
                     <li>
                         <h3>Adjusting the test complexity:</h3>
                         <ul>
-                            <li><label><input name="adjustment" type="radio" value="fixed"> Keep constant</label></li>
-                            <li><label><input name="adjustment" type="radio" value="fixed-after-warmup"> Keep constant after warmup to target FPS</label></li>
+                            <li><label><input name="adjustment" type="radio" value="fixed"> Keep at a fixed complexity</label></li>
                             <li><label><input name="adjustment" type="radio" value="adaptive" checked> Maintain target FPS</label></li>
                         </ul>
                     </li>
                         <label>Target frame rate: <input type="number" id="frame-rate" value="50"> FPS</label>
                     </li>
                     <li>
-                        <label><input type="checkbox" id="estimated-frame-rate" checked> Filter frame rate calculation</label>
+                        <h3>Kalman filter estimated error:</h3>
+                        <ul>
+                            <li><label>Process error (Q): <input type="number" id="kalman-process-error" value="1"></label></li>
+                            <li><label>Measurement error (R): <input type="number" id="kalman-measurement-error" value="4"></label></li>
+                        </ul>
                     </li>
                     </ul>
                     </form>
                         <li><label><input type="checkbox" name="averages" checked> Averages</label></li>
                         <li><label><input type="checkbox" name="complexity" checked> Complexity</label>
                             <span class="complexity"></span></li>
-                        <li><label><input type="checkbox" name="rawFPS"> Raw FPS</label>
+                        <li><label><input type="checkbox" name="rawFPS" checked> Raw FPS</label>
                             <span class="rawFPS"></span></li>
                         <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
                             <span class="filteredFPS"></span></li>
-                        <li><label><input type="checkbox" name="intervalFPS"> Average FPS per sample interval</label>
-                            <span class="intervalFPS"></span></li>
                     </ul>
                 </form>
             </nav>
index b5470ed..1f0317d 100644 (file)
@@ -349,6 +349,7 @@ label.tree-label {
 #test-graph-data {
     flex: 1 1 auto;
     align-self: stretch;
+    z-index: 1;
 }
 
 #test-graph nav {
@@ -441,12 +442,12 @@ label.tree-label {
 }
 
 #filteredFPS path {
-    stroke: rgba(250, 73, 37, .7);
-    stroke-width: 2px;
+    stroke: hsla(30, 96%, 56%, .7);
+    stroke-width: 1px;
 }
 
 #filteredFPS circle {
-    fill: rgb(250, 73, 37);
+    fill: hsl(30, 96%, 56%);
 }
 
 #rawFPS path {
@@ -457,12 +458,3 @@ label.tree-label {
 #rawFPS circle {
     fill: rgb(250, 73, 37);
 }
-
-#intervalFPS path {
-    stroke: hsla(30, 96%, 56%, .7);
-    stroke-width: 1px;
-}
-
-#intervalFPS circle {
-    fill: hsl(30, 96%, 56%);
-}
index 4077593..e518563 100644 (file)
@@ -1,9 +1,13 @@
 Utilities.extendObject(window.benchmarkController, {
+    layoutCounter: 0,
+
     updateGraphData: function(graphData)
     {
         var element = document.getElementById("test-graph-data");
         element.innerHTML = "";
-        var margins = new Insets(10, 30, 30, 40);
+        document.querySelector("hr").style.width = this.layoutCounter++ + "px";
+
+        var margins = new Insets(30, 30, 30, 40);
         var size = Point.elementClientSize(element).subtract(margins.size);
 
         var svg = d3.select("#test-graph-data").append("svg")
@@ -29,7 +33,8 @@ Utilities.extendObject(window.benchmarkController, {
         // Axes
         var xAxis = d3.svg.axis()
                 .scale(x)
-                .orient("bottom");
+                .orient("bottom")
+                .tickFormat(function(d) { return (d/1000).toFixed(0); });
         var yAxisLeft = d3.svg.axis()
                 .scale(yLeft)
                 .orient("left");
@@ -127,9 +132,6 @@ Utilities.extendObject(window.benchmarkController, {
         var filteredData = graphData.samples.filter(function (sample) {
             return "smoothedFPS" in sample;
         });
-        var intervalData = graphData.samples.filter(function (sample) {
-            return "intervalFPS" in sample;
-        });
 
         function addData(name, data, yCoordinateCallback, pointRadius, omitLine) {
             var svgGroup = svg.append("g").attr("id", name);
@@ -156,7 +158,6 @@ Utilities.extendObject(window.benchmarkController, {
         addData("complexity", allData, function(d) { return yLeft(d.complexity); }, 2);
         addData("rawFPS", allData, function(d) { return yRight(d.fps); }, 1);
         addData("filteredFPS", filteredData, function(d) { return yRight(d.smoothedFPS); }, 2);
-        addData("intervalFPS", intervalData, function(d) { return yRight(d.intervalFPS); }, 2);
 
         // Area to handle mouse events
         var area = svg.append("rect")
@@ -167,7 +168,7 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("height", size.y);
 
         var timeBisect = d3.bisector(function(d) { return d.time; }).right;
-        var statsToHighlight = ["complexity", "rawFPS", "filteredFPS", "intervalFPS"];
+        var statsToHighlight = ["complexity", "rawFPS", "filteredFPS"];
         area.on("mouseover", function() {
             document.getElementById("cursor").classList.remove("hidden");
             document.querySelector("#test-graph nav").classList.remove("hide-data");
@@ -189,7 +190,7 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("x2", cursor_x)
                 .attr("y2", yRight(cursor_y));
 
-            document.querySelector("#test-graph nav .time").textContent = data.time.toFixed(4) + "s (" + index + ")";
+            document.querySelector("#test-graph nav .time").textContent = (data.time / 1000).toFixed(4) + "s (" + index + ")";
             statsToHighlight.forEach(function(name) {
                 var element = document.querySelector("#test-graph nav ." + name);
                 var content = "";
@@ -209,12 +210,6 @@ Utilities.extendObject(window.benchmarkController, {
                         data_y = yRight(data.smoothedFPS);
                     }
                     break;
-                case "intervalFPS":
-                    if ("intervalFPS" in data) {
-                        content = data.intervalFPS.toFixed(2);
-                        data_y = yRight(data.intervalFPS);
-                    }
-                    break;
                 }
 
                 element.textContent = content;
@@ -250,6 +245,5 @@ Utilities.extendObject(window.benchmarkController, {
         showOrHideNodes(form["complexity"].checked, "#complexity");
         showOrHideNodes(form["rawFPS"].checked, "#rawFPS");
         showOrHideNodes(form["filteredFPS"].checked, "#filteredFPS");
-        showOrHideNodes(form["intervalFPS"].checked, "#intervalFPS");
     }
 });
index 54865ee..6189fb5 100644 (file)
@@ -173,7 +173,7 @@ Point = Utilities.createClass(
     {
         return "x = " + this.x + ", y = " + this.y;
     },
-    
+
     add: function(other)
     {
         if(isNaN(other.x))
index 41fa976..1f291bd 100644 (file)
@@ -307,7 +307,8 @@ window.benchmarkController = {
             "display": "minimal",
             "adjustment": "adaptive",
             "frame-rate": 50,
-            "estimated-frame-rate": true
+            "kalman-process-error": 1,
+            "kalman-measurement-error": 4
         };
         this._startBenchmark(Suites, options, "test-container");
     },
index f62860e..21bc008 100644 (file)
@@ -94,9 +94,9 @@ BenchmarkRunner = Utilities.createClass(
         Utilities.extendObject(options, contentWindow.Utilities.parseParameters());
 
         var benchmark = new contentWindow.benchmarkClass(options);
-        benchmark.run().then(function(sampler) {
+        benchmark.run().then(function(results) {
             var samplers = self._suitesSamplers[suite.name] || {};
-            samplers[test.name] = sampler.process(options);
+            samplers[test.name] = results;
             self._suitesSamplers[suite.name] = samplers;
 
             if (self._client && self._client.didRunTest)
index 69e19ef..b74a314 100644 (file)
-function BenchmarkState(testInterval)
-{
-    this._currentTimeOffset = 0;
-    this._stageInterval = testInterval / BenchmarkState.stages.FINISHED;
-}
-
-// The enum values and the messages should be in the same order
-BenchmarkState.stages = {
-    WARMING: 0,
-    SAMPLING: 1,
-    FINISHED: 2,
-}
-
-BenchmarkState.prototype =
-{
-    _timeOffset: function(stage)
-    {
-        return stage * this._stageInterval;
+
+Controller = Utilities.createClass(
+    function(testLength, benchmark, seriesCount)
+    {
+        // Initialize timestamps relative to the start of the benchmark
+        // In start() the timestamps are offset by the start timestamp
+        this._startTimestamp = 0;
+        this._endTimestamp = testLength;
+        // Default data series: timestamp, complexity, estimatedFrameLength
+        this._sampler = new Sampler(seriesCount || 3, 60 * testLength, this);
+        this._estimator = new SimpleKalmanEstimator(benchmark.options["kalman-process-error"], benchmark.options["kalman-measurement-error"]);
+
+        this.initialComplexity = 0;
+    }, {
+
+    start: function(stage, startTimestamp)
+    {
+        this._startTimestamp = startTimestamp;
+        this._endTimestamp += startTimestamp;
+        this.recordFirstSample(stage, startTimestamp);
+    },
+
+    recordFirstSample: function(stage, startTimestamp)
+    {
+        this._sampler.record(startTimestamp, stage.complexity(), -1);
+        this._sampler.mark(Strings.json.samplingTimeOffset, { time: 0 });
+    },
+
+    update: function(stage, timestamp)
+    {
+        this._estimator.sample(timestamp - this._sampler.samples[0][this._sampler.sampleCount - 1]);
+        this._sampler.record(timestamp, stage.complexity(), 1000 / this._estimator.estimate);
     },
 
-    _message: function(stage, timeOffset)
+    shouldStop: function(timestamp)
     {
-        if (stage == BenchmarkState.stages.FINISHED)
-            return BenchmarkState.stages.messages[stage];
+        return timestamp > this._endTimestamp;
+    },
 
-        return BenchmarkState.stages.messages[stage] + "... ("
-            + Math.floor((timeOffset - this._timeOffset(stage)) / 1000) + "/"
-            + Math.floor((this._timeOffset(stage + 1) - this._timeOffset(stage)) / 1000) + ")";
+    results: function()
+    {
+        return this._sampler.process();
     },
 
-    update: function(currentTimeOffset)
+    processSamples: function(results)
+    {
+        var complexityExperiment = new Experiment;
+        var smoothedFPSExperiment = new Experiment;
+
+        var samples = this._sampler.samples;
+
+        var samplingIndex = 0;
+        var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
+        if (samplingMark) {
+            samplingIndex = samplingMark.index;
+            results[Strings.json.samplingTimeOffset] = samplingMark.time;
+        }
+
+        results[Strings.json.samples] = samples[0].map(function(timestamp, i) {
+            var result = {
+                // Represent time in seconds
+                time: timestamp - this._startTimestamp,
+                complexity: samples[1][i]
+            };
+
+            // time offsets represented as FPS
+            if (i == 0)
+                result.fps = 60;
+            else
+                result.fps = 1000 / (timestamp - samples[0][i - 1]);
+
+            if (samples[2][i] != -1)
+                result.smoothedFPS = samples[2][i];
+
+            // Don't start adding data to the experiments until we reach the sampling timestamp
+            if (i >= samplingIndex) {
+                complexityExperiment.sample(result.complexity);
+                if (result.smoothedFPS && result.smoothedFPS != -1)
+                    smoothedFPSExperiment.sample(result.smoothedFPS);
+            }
+
+            return result;
+        }, this);
+
+        results[Strings.json.score] = complexityExperiment.score(Experiment.defaults.CONCERN);
+
+        var complexityResults = {};
+        results[Strings.json.experiments.complexity] = complexityResults;
+        complexityResults[Strings.json.measurements.average] = complexityExperiment.mean();
+        complexityResults[Strings.json.measurements.concern] = complexityExperiment.concern(Experiment.defaults.CONCERN);
+        complexityResults[Strings.json.measurements.stdev] = complexityExperiment.standardDeviation();
+        complexityResults[Strings.json.measurements.percent] = complexityExperiment.percentage();
+
+        var smoothedFPSResults = {};
+        results[Strings.json.experiments.frameRate] = smoothedFPSResults;
+        smoothedFPSResults[Strings.json.measurements.average] = smoothedFPSExperiment.mean();
+        smoothedFPSResults[Strings.json.measurements.concern] = smoothedFPSExperiment.concern(Experiment.defaults.CONCERN);
+        smoothedFPSResults[Strings.json.measurements.stdev] = smoothedFPSExperiment.standardDeviation();
+        smoothedFPSResults[Strings.json.measurements.percent] = smoothedFPSExperiment.percentage();
+    }
+});
+
+FixedComplexityController = Utilities.createSubclass(Controller,
+    function(testInterval, benchmark)
+    {
+        Controller.call(this, testInterval, benchmark);
+        this.initialComplexity = benchmark.options["complexity"];
+    }
+);
+
+AdaptiveController = Utilities.createSubclass(Controller,
+    function(testInterval, benchmark)
+    {
+        // Data series: timestamp, complexity, estimatedIntervalFrameLength
+        Controller.call(this, testInterval, benchmark);
+
+        // All tests start at 0, so we expect to see 60 fps quickly.
+        this._samplingTimestamp = testInterval / 2;
+        this._startedSampling = false;
+        this._targetFrameRate = benchmark.options["frame-rate"];
+        this._pid = new PIDController(this._targetFrameRate);
+
+        this._intervalFrameCount = 0;
+        this._numberOfFramesToMeasurePerInterval = 4;
+    }, {
+
+    start: function(stage, startTimestamp)
     {
-        this._currentTimeOffset = currentTimeOffset;
+        Controller.prototype.start.call(this, stage, startTimestamp);
+
+        this._samplingTimestamp += startTimestamp;
+        this._intervalTimestamp = startTimestamp;
     },
 
-    samplingTimeOffset: function()
+    recordFirstSample: function(stage, startTimestamp)
     {
-        return this._timeOffset(BenchmarkState.stages.SAMPLING);
+        this._sampler.record(startTimestamp, stage.complexity(), -1);
     },
 
-    currentStage: function()
+    update: function(stage, timestamp)
     {
-        for (var stage = BenchmarkState.stages.WARMING; stage < BenchmarkState.stages.FINISHED; ++stage) {
-            if (this._currentTimeOffset < this._timeOffset(stage + 1))
-                return stage;
+        if (!this._startedSampling && timestamp > this._samplingTimestamp) {
+            this._startedSampling = true;
+            this._sampler.mark(Strings.json.samplingTimeOffset, {
+                time: this._samplingTimestamp - this._startTimestamp
+            });
+        }
+
+        // Start the work for the next frame.
+        ++this._intervalFrameCount;
+
+        if (this._intervalFrameCount < this._numberOfFramesToMeasurePerInterval) {
+            this._sampler.record(timestamp, stage.complexity(), -1);
+            return;
         }
-        return BenchmarkState.stages.FINISHED;
+
+        // Adjust the test to reach the desired FPS.
+        var intervalLength = timestamp - this._intervalTimestamp;
+        this._estimator.sample(intervalLength / this._numberOfFramesToMeasurePerInterval);
+        var intervalEstimatedFrameRate = 1000 / this._estimator.estimate;
+        var tuneValue = -this._pid.tune(timestamp - this._startTimestamp, intervalLength, intervalEstimatedFrameRate);
+        tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
+        stage.tune(tuneValue);
+
+        this._sampler.record(timestamp, stage.complexity(), intervalEstimatedFrameRate);
+
+        // Start the next interval.
+        this._intervalFrameCount = 0;
+        this._intervalTimestamp = timestamp;
+    },
+
+    processSamples: function(results)
+    {
+        Controller.prototype.processSamples.call(this, results);
+        results[Strings.json.targetFPS] = this._targetFrameRate;
     }
-}
+});
 
 Stage = Utilities.createClass(
     function()
@@ -180,100 +308,26 @@ Rotater = Utilities.createClass(
     }
 });
 
-Animator = Utilities.createClass(
-    function()
-    {
-        this._intervalFrameCount = 0;
-        this._numberOfFramesToMeasurePerInterval = 3;
-        this._referenceTime = 0;
-        this._currentTimeOffset = 0;
-    }, {
-
-    initialize: function(benchmark)
-    {
-        this._benchmark = benchmark;
-
-        // Use Kalman filter to get a more non-fluctuating frame rate.
-        if (benchmark.options["estimated-frame-rate"])
-            this._estimator = new KalmanEstimator(60);
-        else
-            this._estimator = new IdentityEstimator;
-    },
-
-    get benchmark()
-    {
-        return this._benchmark;
-    },
-
-    _intervalTimeDelta: function()
-    {
-        return this._currentTimeOffset - this._startTimeOffset;
-    },
-
-    _shouldRequestAnotherFrame: function()
-    {
-        // Cadence is number of frames to measure, then one more frame to adjust the scene, and drop
-        var currentTime = performance.now();
-
-        if (!this._referenceTime)
-            this._referenceTime = currentTime;
-
-        this._currentTimeOffset = currentTime - this._referenceTime;
-
-        if (!this._intervalFrameCount)
-            this._startTimeOffset = this._currentTimeOffset;
-
-        // Start the work for the next frame.
-        ++this._intervalFrameCount;
-
-        // Drop _dropFrameCount frames and measure the average of _measureFrameCount frames.
-        if (this._intervalFrameCount <= this._numberOfFramesToMeasurePerInterval) {
-            this._benchmark.record(this._currentTimeOffset, -1, -1);
-            return true;
-        }
-
-        // Get the average FPS of _measureFrameCount frames over intervalTimeDelta.
-        var intervalTimeDelta = this._intervalTimeDelta();
-        var intervalFrameRate = 1000 / (intervalTimeDelta / this._numberOfFramesToMeasurePerInterval);
-        var estimatedIntervalFrameRate = this._estimator.estimate(intervalFrameRate);
-        // Record the complexity of the frame we just rendered. The next frame's time respresents the adjusted
-        // complexity
-        this._benchmark.record(this._currentTimeOffset, estimatedIntervalFrameRate, intervalFrameRate);
-
-        // Adjust the test to reach the desired FPS.
-        var shouldContinueRunning = this._benchmark.update(this._currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
-
-        // Start the next drop/measure cycle.
-        this._intervalFrameCount = 0;
-
-        // If result is false, no more requestAnimationFrame() will be invoked.
-        return shouldContinueRunning;
-    },
-
-    animateLoop: function()
-    {
-        if (this._shouldRequestAnotherFrame()) {
-            this._benchmark.stage.animate(this._intervalTimeDelta());
-            requestAnimationFrame(this.animateLoop.bind(this));
-        }
-    }
-});
-
 Benchmark = Utilities.createClass(
     function(stage, options)
     {
         this._options = options;
+        this._animateLoop = this._animateLoop.bind(this);
 
         this._stage = stage;
         this._stage.initialize(this);
-        this._animator = new Animator();
-        this._animator.initialize(this);
 
-        this._recordInterval = 200;
-        this._isSampling = false;
-        this._controller = new PIDController(this._options["frame-rate"]);
-        this._sampler = new Sampler(4, 60 * this._options["test-interval"], this);
-        this._state = new BenchmarkState(this._options["test-interval"] * 1000);
+        var testIntervalMilliseconds = options["test-interval"] * 1000;
+        switch (options["adjustment"])
+        {
+        case "fixed":
+            this._controller = new FixedComplexityController(testIntervalMilliseconds, this);
+            break;
+        case "adaptive":
+        default:
+            this._controller = new AdaptiveController(testIntervalMilliseconds, this);
+            break;
+        }
     }, {
 
     get options()
@@ -286,84 +340,19 @@ Benchmark = Utilities.createClass(
         return this._stage;
     },
 
-    get animator()
-    {
-        return this._animator;
-    },
-
-    // Called from the load event listener or from this.run().
-    start: function()
-    {
-        this._animator.animateLoop();
-    },
-
-    // Called from the animator to adjust the complexity of the test.
-    update: function(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate)
-    {
-        this._state.update(currentTimeOffset);
-
-        var stage = this._state.currentStage();
-        if (stage == BenchmarkState.stages.FINISHED) {
-            this._stage.clear();
-            return false;
-        }
-
-        if (stage == BenchmarkState.stages.SAMPLING && !this._isSampling) {
-            this._sampler.mark(Strings.json.samplingTimeOffset, {
-                time: this._state.samplingTimeOffset() / 1000
-            });
-            this._isSampling = true;
-        }
-
-        var tuneValue = 0;
-        if (this._options["adjustment"] == "fixed") {
-            if (this._options["complexity"]) {
-                // this._stage.tune(0) returns the current complexity of the test.
-                tuneValue = this._options["complexity"] - this._stage.tune(0);
-            }
-        }
-        else if (!(this._isSampling && this._options["adjustment"] == "fixed-after-warmup")) {
-            // The relationship between frameRate and test complexity is inverse-proportional so we
-            // need to use the negative of PIDController.tune() to change the complexity of the test.
-            tuneValue = -this._controller.tune(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
-            tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
-        }
-
-        this._stage.tune(tuneValue);
-
-        if (typeof this._recordTimeOffset == "undefined")
-            this._recordTimeOffset = currentTimeOffset;
-
-        var stage = this._state.currentStage();
-        if (stage != BenchmarkState.stages.FINISHED && currentTimeOffset < this._recordTimeOffset + this._recordInterval)
-            return true;
-
-        this._recordTimeOffset = currentTimeOffset;
-        return true;
-    },
-
-    record: function(currentTimeOffset, estimatedFrameRate, intervalFrameRate)
-    {
-        // If the frame rate is -1 it means we are still recording for this sample
-        this._sampler.record(currentTimeOffset, this.stage.complexity(), estimatedFrameRate, intervalFrameRate);
-    },
-
     run: function()
     {
         return this.waitUntilReady().then(function() {
-            this.start();
-            var promise = new SimplePromise;
-            var resolveWhenFinished = function() {
-                if (typeof this._state != "undefined" && (this._state.currentStage() == BenchmarkState.stages.FINISHED))
-                    return promise.resolve(this._sampler);
-                setTimeout(resolveWhenFinished, 50);
-            }.bind(this);
-
-            resolveWhenFinished();
-            return promise;
+            this._finishPromise = new SimplePromise;
+            this._previousTimestamp = performance.now();
+            this._didWarmUp = false;
+            this._stage.tune(this._controller.initialComplexity - this._stage.complexity());
+            this._animateLoop();
+            return this._finishPromise;
         }.bind(this));
     },
 
+    // Subclasses should override this if they have setup to do prior to commencing.
     waitUntilReady: function()
     {
         var promise = new SimplePromise;
@@ -371,55 +360,30 @@ Benchmark = Utilities.createClass(
         return promise;
     },
 
-    processSamples: function(results)
+    _animateLoop: function()
     {
-        var complexity = new Experiment;
-        var smoothedFPS = new Experiment;
-        var samplingIndex = 0;
+        this._currentTimestamp = performance.now();
 
-        var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
-        if (samplingMark) {
-            samplingIndex = samplingMark.index;
-            results[Strings.json.samplingTimeOffset] = samplingMark.time;
-        }
-
-        results[Strings.json.samples] = this._sampler.samples[0].map(function(d, i) {
-            var result = {
-                // time offsets represented as seconds
-                time: d/1000,
-                complexity: this._sampler.samples[1][i]
-            };
-
-            // time offsets represented as FPS
-            if (i == 0)
-                result.fps = 60;
-            else
-                result.fps = 1000 / (d - this._sampler.samples[0][i - 1]);
-
-            var smoothedFPSresult = this._sampler.samples[2][i];
-            if (smoothedFPSresult != -1) {
-                result.smoothedFPS = smoothedFPSresult;
-                result.intervalFPS = this._sampler.samples[3][i];
+        if (!this._didWarmUp) {
+            if (this._currentTimestamp - this._previousTimestamp >= 100) {
+                this._didWarmUp = true;
+                this._controller.start(this._stage, this._currentTimestamp);
+                this._previousTimestamp = this._currentTimestamp;
             }
 
-            if (i >= samplingIndex) {
-                complexity.sample(result.complexity);
-                if (smoothedFPSresult != -1) {
-                    smoothedFPS.sample(smoothedFPSresult);
-                }
-            }
+            this._stage.animate(0);
+            requestAnimationFrame(this._animateLoop);
+            return;
+        }
 
-            return result;
-        }, this);
+        this._controller.update(this._stage, this._currentTimestamp);
+        if (this._controller.shouldStop(this._currentTimestamp)) {
+            this._finishPromise.resolve(this._controller.results());
+            return;
+        }
 
-        results[Strings.json.score] = complexity.score(Experiment.defaults.CONCERN);
-        [complexity, smoothedFPS].forEach(function(experiment, index) {
-            var jsonExperiment = !index ? Strings.json.experiments.complexity : Strings.json.experiments.frameRate;
-            results[jsonExperiment] = {};
-            results[jsonExperiment][Strings.json.measurements.average] = experiment.mean();
-            results[jsonExperiment][Strings.json.measurements.concern] = experiment.concern(Experiment.defaults.CONCERN);
-            results[jsonExperiment][Strings.json.measurements.stdev] = experiment.standardDeviation();
-            results[jsonExperiment][Strings.json.measurements.percent] = experiment.percentage();
-        });
+        this._stage.animate(this._currentTimestamp - this._previousTimestamp);
+        this._previousTimestamp = this._currentTimestamp;
+        requestAnimationFrame(this._animateLoop);
     }
 });
index 677a20b..bcbe902 100644 (file)
-var Matrix =
-{
-    init: function(m, n, v)
-    {
-        return Array(m * n).fill(v);
-    },
-
-    zeros: function(m, n)
-    {
-        return Matrix.init(m, n, 0);
-    },
-
-    ones: function(m, n)
-    {
-        return Matrix.init(m, n, 1);
-    },
-
-    identity: function(n)
-    {
-        var out = new Matrix.zeros(n, n);
-        for (var i = 0; i < n; ++i)
-            out[i * n + i] = 1;
-        return out;
-    },
-
-    str: function(A, n, m)
-    {
-        var out = (m > 1 && n > 1 ? "Matrix[" + n + ", " + m : "Vector[" + m * n) + "] = [";
-        for (var i = 0; i < n * m; ++i) {
-            out += A[i];
-            if (i < n * m - 1)
-                out += ", ";
-        }
-        return out + "]";
-    },
-
-    pos: function(m, i, j)
-    {
-        return m * i + j;
-    },
-
-    add: function(A, B, n, m)
-    {
-        var out = Matrix.zeros(n, m);
-        for (var i = 0; i < n * m; ++i)
-            out[i] = A[i] + B[i];
-        return out;
-    },
-
-    subtract: function(A, B, n, m)
-    {
-        var out = Matrix.zeros(n, m);
-        for (var i = 0; i < n * m; ++i)
-            out[i] = A[i] - B[i];
-        return out;
-    },
-
-    scale: function(s, A, n, m)
-    {
-        var out = Matrix.zeros(n, m);
-        for (var i = 0; i < n * m; ++i)
-            out[i] = s * A[i];
-        return out;
-    },
+SimpleKalmanEstimator = Utilities.createClass(
+    function(processError, measurementError) {
+        this._initialized = false;
 
-    transpose: function(A, n, m)
-    {
-        var out = Matrix.zeros(m, n);
-        for (var i = 0; i < n; ++i) {
-            for (var j = 0; j < m; ++j)
-                out[Matrix.pos(n, i, j)] = A[Matrix.pos(m, j, i)];
-        }
-        return out;
-    },
+        var error = .5 * (Math.sqrt(processError * processError + 4 * processError * measurementError) - processError);
+        this._gain = error / (error + measurementError);
+    }, {
 
-    multiply: function(A, B, n, m, p)
+    sample: function(newMeasurement)
     {
-        var out = Matrix.zeros(n, p);
-        for (var i = 0; i < n; ++i) {
-            for (var j = 0; j < p; ++j) {
-                for (var k = 0; k < m; ++k) {
-                    out[Matrix.pos(p, i, j)] += A[Matrix.pos(m, i, k)] * B[Matrix.pos(p, k, j)];
-                }
-            }
+        if (!this._initialized) {
+            this._initialized = true;
+            this._estimatedMeasurement = newMeasurement;
+            return;
         }
-        return out;
-    }
-}
-
-var Vector3 =
-{
-    zeros: function()
-    {
-        return Matrix.zeros(1, 3);
-    },
-
-    ones: function()
-    {
-        return Matrix.ones(1, 3);
-    },
-
-    str: function(v)
-    {
-        return Matrix.str(v, 1, 3);
-    },
-
-    add: function(v, w)
-    {
-        return Matrix.add(v, w, 1, 3);
-    },
-
-    subtract: function(v, w)
-    {
-        return Matrix.subtract(v, w, 1, 3);
-    },
-
-    scale: function(s, v)
-    {
-        return Matrix.scale(s, v, 1, 3);
-    },
 
-    multiplyMatrix3: function(v, A)
-    {
-        return Matrix.multiply(v, A, 1, 3, 3);
+        this._estimatedMeasurement = this._estimatedMeasurement + this._gain * (newMeasurement - this._estimatedMeasurement);
     },
 
-    multiplyVector3: function(v, w)
+    get estimate()
     {
-        var out = 0;
-        for (var i = 0; i < 3; ++i)
-            out += v[i] * w[i];
-        return out;
+        return this._estimatedMeasurement;
     }
-}
-
-var Matrix3 =
-{
-    zeros: function()
-    {
-        return Matrix.zeros(3, 3);
-    },
-
-    identity: function()
-    {
-        return Matrix.identity(3, 3);
-    },
-
-    str: function(A)
-    {
-        return Matrix.str(A, 3, 3);
-    },
+});
 
-    pos: function(i, j)
+PIDController = Utilities.createClass(
+    function(ysp)
     {
-        return Matrix.pos(3, i, j);
-    },
-
-    add: function(A, B)
-    {
-        return Matrix.add(A, B, 3, 3);
-    },
-
-    subtract: function(A, B)
-    {
-        return Matrix.subtract(A, B, 3, 3);
-    },
-
-    scale: function(s, A)
-    {
-        return Matrix.scale(s, A, 3, 3);
-    },
+        this._ysp = ysp;
+        this._out = 0;
 
-    transpose: function(A)
-    {
-        return Matrix.transpose(A, 3, 3);
-    },
+        this._Kp = 0;
+        this._stage = PIDController.stages.WARMING;
 
-    multiplyMatrix3: function(A, B)
-    {
-        return Matrix.multiply(A, B, 3, 3, 3);
-    },
+        this._eold = 0;
+        this._I = 0;
+    }, {
 
-    multiplyVector3: function(A, v)
-    {
-        return Matrix.multiply(A, v, 3, 3, 1);
-    }
-}
-
-function PIDController(ysp)
-{
-    this._ysp = ysp;
-    this._out = 0;
-
-    this._Kp = 0;
-    this._stage = PIDController.stages.WARMING;
-
-    this._eold = 0;
-    this._I = 0;
-}
-
-// This enum will be used to tell whether the system output (or the controller input)
-// is moving towards the set-point or away from it.
-PIDController.yPositions = {
-    BEFORE_SETPOINT: 0,
-    AFTER_SETPOINT: 1
-}
-
-// The Ziegler–Nichols method for is used tuning the PID controller. The workflow of
-// the tuning is split into four stages. The first two stages determine the values
-// of the PID controller gains. During these two stages we return the proportional
-// term only. The third stage is used to determine the min-max values of the
-// saturation actuator. In the last stage back-calculation and tracking are applied
-// to avoid integrator windup. During the last two stages, we return a PID control
-// value.
-PIDController.stages = {
-    WARMING: 0,         // Increase the value of the Kp until the system output reaches ysp.
-    OVERSHOOT: 1,       // Measure the oscillation period and the overshoot value
-    UNDERSHOOT: 2,      // Return PID value and measure the undershoot value
-    SATURATE: 3         // Return PID value and apply back-calculation and tracking.
-}
-
-PIDController.prototype =
-{
     // Determines whether the current y is
     //  before ysp => (below ysp if ysp > y0) || (above ysp if ysp < y0)
     //  after ysp => (above ysp if ysp > y0) || (below ysp if ysp < y0)
@@ -410,7 +225,7 @@ PIDController.prototype =
         return u;
     },
 
-    // Called from the benchmark to tune its test. It uses Ziegler–Nichols method
+    // Called from the benchmark to tune its test. It uses Ziegler-Nichols method
     // to calculate the controller parameters. It then returns a PID tuning value.
     tune: function(t, h, y)
     {
@@ -426,66 +241,27 @@ PIDController.prototype =
         // Apply back-calculation and tracking to avoid integrator windup
         return this._saturate(v, e);
     }
-}
-
-function KalmanEstimator(initX)
-{
-    // Initialize state transition matrix.
-    this._matA = Matrix3.identity();
-    this._matA[Matrix3.pos(0, 2)] = 1;
-
-    // Initialize measurement matrix.
-    this._vecH = Vector3.zeros();
-    this._vecH[0] = 1;
-
-    this._matQ = Matrix3.identity();
-    this._R = 1000;
-
-    // Initial state conditions.
-    this._vecX_est = Vector3.zeros();
-    this._vecX_est[0] = initX;
-    this._matP_est = Matrix3.zeros();
-}
-
-KalmanEstimator.prototype =
-{
-    estimate: function(current)
-    {
-        // Project the state ahead
-        //  X_prd(k) = A * X_est(k-1)
-        var vecX_prd = Matrix3.multiplyVector3(this._matA, this._vecX_est);
-
-        // Project the error covariance ahead
-        //  P_prd(k) = A * P_est(k-1) * A' + Q
-        var matP_prd = Matrix3.add(Matrix3.multiplyMatrix3(Matrix3.multiplyMatrix3(this._matA, this._matP_est), Matrix3.transpose(this._matA)), this._matQ);
-
-        // Compute Kalman gain
-        //  B = H * P_prd(k)';
-        //  S = B * H' + R;
-        //  K(k) = (S \ B)';
-        var vecB = Vector3.multiplyMatrix3(this._vecH, Matrix3.transpose(matP_prd));
-        var S = Vector3.multiplyVector3(vecB, this._vecH) + this._R;
-        var vecGain = Vector3.scale(1/S, vecB);
-
-        // Update the estimate via z(k)
-        //  X_est(k) = x_prd + K(k) * (z(k) - H * X_prd(k));
-        this._vecX_est = Vector3.add(vecX_prd, Vector3.scale(current - Vector3.multiplyVector3(this._vecH, vecX_prd), vecGain));
-
-        // Update the error covariance
-        //  P_est(k) = P_prd(k) - K(k) * H * P_prd(k);
-        this._matP_est = Matrix3.subtract(matP_prd, Matrix3.scale(Vector3.multiplyVector3(vecGain, this._vecH), matP_prd));
-
-        // Compute the estimated measurement.
-        //  y = H * X_est(k);
-        return Vector3.multiplyVector3(this._vecH,  this._vecX_est);
-    }
-}
-
-function IdentityEstimator() {}
-IdentityEstimator.prototype =
-{
-    estimate: function(current)
-    {
-        return current;
+});
+
+Utilities.extendObject(PIDController, {
+    // This enum will be used to tell whether the system output (or the controller input)
+    // is moving towards the set-point or away from it.
+    yPositions: {
+        BEFORE_SETPOINT: 0,
+        AFTER_SETPOINT: 1
+    },
+
+    // The Ziegler-Nichols method for is used tuning the PID controller. The workflow of
+    // the tuning is split into four stages. The first two stages determine the values
+    // of the PID controller gains. During these two stages we return the proportional
+    // term only. The third stage is used to determine the min-max values of the
+    // saturation actuator. In the last stage back-calculation and tracking are applied
+    // to avoid integrator windup. During the last two stages, we return a PID control
+    // value.
+    stages: {
+        WARMING: 0,         // Increase the value of the Kp until the system output reaches ysp.
+        OVERSHOOT: 1,       // Measure the oscillation period and the overshoot value
+        UNDERSHOOT: 2,      // Return PID value and measure the undershoot value
+        SATURATE: 3         // Return PID value and apply back-calculation and tracking.
     }
-};
+});
index 52068ec..e60995f 100644 (file)
@@ -81,13 +81,10 @@ Sampler = Utilities.createClass(
         this.marks[comment] = data;
     },
 
-    process: function(options)
+    process: function()
     {
         var results = {};
 
-        if (options["adjustment"] == "adaptive")
-            results[Strings.json.targetFPS] = +options["frame-rate"];
-
         // Remove unused capacity
         this.samples = this.samples.map(function(array) {
             return array.slice(0, this.sampleCount);
index e8e7c39..ec01957 100644 (file)
@@ -1,3 +1,105 @@
+2016-02-07  Jon Lee  <jonlee@apple.com>
+
+        Update how the benchmark is run
+        https://bugs.webkit.org/show_bug.cgi?id=153960
+
+        Provisionally reviewed by Said Abou-Hallawa.
+
+        Introduce the notion of a Controller. It is responsible for recording, updating,
+        and processing the statistics and complexity of the benchmark. This allows
+        plugging in different Controllers.
+
+        This strips most of the functionality from Animator and BenchmarkState, so fold
+        what's left into Benchmark. Now, Benchmarks only own a stage and a controller, but
+        are responsible for driving the animation loop.
+
+        Rewrite Animator._shouldRequestAnotherFrame into two different Controllers. One
+        maintains a fixed complexity, and the other adapts the complexity to meet a
+        fixed FPS.
+
+        Fix the Kalman estimator to be modeled on a scalar variable with no model.
+
+        * Animometer/tests/resources/main.js: Remove BenchmarkState and Animator, and
+        replace it with a Controller. Add a FixedController and refactor the previous controller
+        to an AdaptiveController.
+
+        (Controller): Controllers own the estimator and the sampler. When a new frame is
+        displayed, the animation loop calls update(). The estimator and sampler record
+        stats, then tune. Samplers can track multiple series of data. The basic controller
+        tracks timestamp, complexity, and estimated frame rate.
+                The Kalman estimation is based on the frame length rather than the frame
+        rate. Because FPS is inverse proportional to frame length, in the case where the measured
+        frame length is very small, the FPS ends up being a wildly large number (in the order of
+        600-1000 "FPS"), and it pulls the estimator up drastically enough that it takes a while
+        for it to settle back down. Using frame length reduces the impact of these spikes.
+                Converging the estimation takes enough time to avoid initializing it immediately
+        when the benchmark starts. Instead, the benchmark runs for a brief period of time (100ms)
+        before running it in earnest. Allow controllers an opportunity to set the complexity
+        before starting recording.
+                When the benchmark is complete, the controller has an opportunity to process
+        the samples. The default implementation calculates the raw FPS based on the time
+        difference of the samples, and calculates the complexity score. This is moved from
+        Benchmark.processSamples.
+
+        (Controller): Initialize timestamps. These are at first relative to the start of the
+        benchmark, but are offset by the absolute start time during start(). By default maintain
+        3 data series, but subclasses can override.
+        (start): Calls recordFirstSample() for subclasses to override if needed.
+        (recordFirstSample): For basic controller, start sampling at the beginning.
+        (update): Update the frame length estimator and sample.
+        (shouldStop): Checks that the time is before _endTimestamp.
+        (results): Returns the processed samples.
+        (processSamples): Iterate through the sample data and collate them. Include scores.
+
+        (FixedComplexityController): Controller that tunes the stage to the desired complexity
+        prior to starting, and keeps it at that complexity.
+
+        (AdaptiveController): Have the estimator estimate the interval frame rate instead of the
+        raw frame rate.
+                The previous version of this controller ignored the frame that came after the
+        adjustment. The raw FPS show that whatever noise the scene change adds is negligible
+        compared to the noise of the system overall. Stop ignoring that frame and include all
+        frames in the measurements.
+
+        (Benchmark): Remove dependency on animator, and instantiate a runner based on what is
+        selected. Most of the loop's functionality is in Controller, so remove here.
+        (Benchmark.run): Remove start() since it is only called from run(), and fold it in here.
+        (Benchmark._animateLoop): Fold in from Animator.animateLoop. Let the benchmark run for
+        a brief period before calling Controller.start().
+
+        * Animometer/tests/resources/math.js: Fix the Kalman estimator. The filter estimates
+        a scalar variable, and makes basic assumptions regarding the model. As a result
+        none of the linear algebra classes are needed, so remove Matrix, Vector3, and Matrix3.
+        (SimpleKalmanEstimator): Calculate the gain based on the provided process and
+        measurement errors.
+        (KalmanEstimator): Deleted.
+        (IdentityEstimator): Deleted.
+        (PIDController): Refactor to use the Utilities.createClass() helper.
+
+        The Kalman filter algorithm is explained here http://greg.czerniak.info/guides/kalman1/.
+        The state, represented by a scalar, is the estimated frame length. There is no user
+        transition of the state, and the state is the same as the measurement. With this model,
+        the estimation error converges, so calculate the gain ahead of time.
+
+        * Animometer/developer.html: Remove fixed-after-warmup since it is not useful.
+        Replace the option to toggle the estimator, and make it possible to customize the
+        estimator's error parameters. Show raw FPS by default, and remove interval FPS,
+        which will be shown instead of the filtered raw FPS.
+        * Animometer/resources/debug-runner/animometer.css: Put the header behind the graph.
+        Remove #intervalFPS rules; move the color to #filteredFPS.
+        * Animometer/resources/debug-runner/graph.js:
+        (updateGraphData): Update the hr style to force the layout to be calculated
+        correctly. Change the tick format to be in terms of seconds, since the timestamps
+        are in milliseconds. Remove interval data.
+        * Animometer/resources/runner/animometer.js:
+        (window.benchmarkController.startBenchmark): Set Kalman parameters.
+        * Animometer/resources/runner/benchmark-runner.js:
+        (_runBenchmarkAndRecordResults): When a benchmark completes, expect it to return
+        the final data, rather than passing a sampler from the controller. This avoids
+        needing to expose the sampler variable in the benchmark.
+        * Animometer/tests/resources/sampler.js:
+        (process): Move the setting of the target frame rate to AdaptiveController.
+
 2016-02-06  Jon Lee  <jonlee@apple.com>
 
         Code clean up: Move Rotater function closer to Stage static methods.