Update how the benchmark is run
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 9 Feb 2016 03:25:39 +0000 (03:25 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 9 Feb 2016 03:25:39 +0000 (03:25 +0000)
https://bugs.webkit.org/show_bug.cgi?id=153960

Provisionally reviewed by Said Abou-Hallawa.

Introduce the notion of a Controller. It is responsible for recording, updating,
and processing the statistics and complexity of the benchmark. This allows
plugging in different Controllers.

This strips most of the functionality from Animator and BenchmarkState, so fold
what's left into Benchmark. Now, Benchmarks only own a stage and a controller, but
are responsible for driving the animation loop.

Rewrite Animator._shouldRequestAnotherFrame into two different Controllers. One
maintains a fixed complexity, and the other adapts the complexity to meet a
fixed FPS.

Fix the Kalman estimator to be modeled on a scalar variable with no model.

* Animometer/tests/resources/main.js: Remove BenchmarkState and Animator, and
replace it with a Controller. Add a FixedController and refactor the previous controller
to an AdaptiveController.

(Controller): Controllers own the estimator and the sampler. When a new frame is
displayed, the animation loop calls update(). The estimator and sampler record
stats, then tune. Samplers can track multiple series of data. The basic controller
tracks timestamp, complexity, and estimated frame rate.
        The Kalman estimation is based on the frame length rather than the frame
rate. Because FPS is inverse proportional to frame length, in the case where the measured
frame length is very small, the FPS ends up being a wildly large number (in the order of
600-1000 "FPS"), and it pulls the estimator up drastically enough that it takes a while
for it to settle back down. Using frame length reduces the impact of these spikes.
        Converging the estimation takes enough time to avoid initializing it immediately
when the benchmark starts. Instead, the benchmark runs for a brief period of time (100ms)
before running it in earnest. Allow controllers an opportunity to set the complexity
before starting recording.
        When the benchmark is complete, the controller has an opportunity to process
the samples. The default implementation calculates the raw FPS based on the time
difference of the samples, and calculates the complexity score. This is moved from
Benchmark.processSamples.

(Controller): Initialize timestamps. These are at first relative to the start of the
benchmark, but are offset by the absolute start time during start(). By default maintain
3 data series, but subclasses can override.
(start): Calls recordFirstSample() for subclasses to override if needed.
(recordFirstSample): For basic controller, start sampling at the beginning.
(update): Update the frame length estimator and sample.
(shouldStop): Checks that the time is before _endTimestamp.
(results): Returns the processed samples.
(processSamples): Iterate through the sample data and collate them. Include scores.

(FixedComplexityController): Controller that tunes the stage to the desired complexity
prior to starting, and keeps it at that complexity.

(AdaptiveController): Have the estimator estimate the interval frame rate instead of the
raw frame rate.
        The previous version of this controller ignored the frame that came after the
adjustment. The raw FPS show that whatever noise the scene change adds is negligible
compared to the noise of the system overall. Stop ignoring that frame and include all
frames in the measurements.

(Benchmark): Remove dependency on animator, and instantiate a runner based on what is
selected. Most of the loop's functionality is in Controller, so remove here.
(Benchmark.run): Remove start() since it is only called from run(), and fold it in here.
(Benchmark._animateLoop): Fold in from Animator.animateLoop. Let the benchmark run for
a brief period before calling Controller.start().

* Animometer/tests/resources/math.js: Fix the Kalman estimator. The filter estimates
a scalar variable, and makes basic assumptions regarding the model. As a result
none of the linear algebra classes are needed, so remove Matrix, Vector3, and Matrix3.
(SimpleKalmanEstimator): Calculate the gain based on the provided process and
measurement errors.
(KalmanEstimator): Deleted.
(IdentityEstimator): Deleted.
(PIDController): Refactor to use the Utilities.createClass() helper.

The Kalman filter algorithm is explained here http://greg.czerniak.info/guides/kalman1/.
The state, represented by a scalar, is the estimated frame length. There is no user
transition of the state, and the state is the same as the measurement. With this model,
the estimation error converges, so calculate the gain ahead of time.

* Animometer/developer.html: Remove fixed-after-warmup since it is not useful.
Replace the option to toggle the estimator, and make it possible to customize the
estimator's error parameters. Show raw FPS by default, and remove interval FPS,
which will be shown instead of the filtered raw FPS.
* Animometer/resources/debug-runner/animometer.css: Put the header behind the graph.
Remove #intervalFPS rules; move the color to #filteredFPS.
* Animometer/resources/debug-runner/graph.js:
(updateGraphData): Update the hr style to force the layout to be calculated
correctly. Change the tick format to be in terms of seconds, since the timestamps
are in milliseconds. Remove interval data.
* Animometer/resources/runner/animometer.js:
(window.benchmarkController.startBenchmark): Set Kalman parameters.
* Animometer/resources/runner/benchmark-runner.js:
(_runBenchmarkAndRecordResults): When a benchmark completes, expect it to return
the final data, rather than passing a sampler from the controller. This avoids
needing to expose the sampler variable in the benchmark.
* Animometer/tests/resources/sampler.js:
(process): Move the setting of the target frame rate to AdaptiveController.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@196288 268f45cc-cd09-0410-ab3c-d52691b4dbfc

PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/extensions.js
PerformanceTests/Animometer/resources/runner/animometer.js
PerformanceTests/Animometer/resources/runner/benchmark-runner.js
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/Animometer/tests/resources/math.js
PerformanceTests/Animometer/tests/resources/sampler.js
PerformanceTests/ChangeLog

index 9fa3a0a58a31975da3727880ada2aa4dadd13a40..c58428bc7e2c17b6e15e1be8dd8a114f5c381152 100644 (file)
@@ -44,8 +44,7 @@
                     <li>
                         <h3>Adjusting the test complexity:</h3>
                         <ul>
-                            <li><label><input name="adjustment" type="radio" value="fixed"> Keep constant</label></li>
-                            <li><label><input name="adjustment" type="radio" value="fixed-after-warmup"> Keep constant after warmup to target FPS</label></li>
+                            <li><label><input name="adjustment" type="radio" value="fixed"> Keep at a fixed complexity</label></li>
                             <li><label><input name="adjustment" type="radio" value="adaptive" checked> Maintain target FPS</label></li>
                         </ul>
                     </li>
                         <label>Target frame rate: <input type="number" id="frame-rate" value="50"> FPS</label>
                     </li>
                     <li>
-                        <label><input type="checkbox" id="estimated-frame-rate" checked> Filter frame rate calculation</label>
+                        <h3>Kalman filter estimated error:</h3>
+                        <ul>
+                            <li><label>Process error (Q): <input type="number" id="kalman-process-error" value="1"></label></li>
+                            <li><label>Measurement error (R): <input type="number" id="kalman-measurement-error" value="4"></label></li>
+                        </ul>
                     </li>
                     </ul>
                     </form>
                         <li><label><input type="checkbox" name="averages" checked> Averages</label></li>
                         <li><label><input type="checkbox" name="complexity" checked> Complexity</label>
                             <span class="complexity"></span></li>
-                        <li><label><input type="checkbox" name="rawFPS"> Raw FPS</label>
+                        <li><label><input type="checkbox" name="rawFPS" checked> Raw FPS</label>
                             <span class="rawFPS"></span></li>
                         <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
                             <span class="filteredFPS"></span></li>
-                        <li><label><input type="checkbox" name="intervalFPS"> Average FPS per sample interval</label>
-                            <span class="intervalFPS"></span></li>
                     </ul>
                 </form>
             </nav>
index b5470ed14079c7a4475f4b60930468be69b889ee..1f0317d560ec9457fa36342bc1881c8983c19092 100644 (file)
@@ -349,6 +349,7 @@ label.tree-label {
 #test-graph-data {
     flex: 1 1 auto;
     align-self: stretch;
+    z-index: 1;
 }
 
 #test-graph nav {
@@ -441,12 +442,12 @@ label.tree-label {
 }
 
 #filteredFPS path {
-    stroke: rgba(250, 73, 37, .7);
-    stroke-width: 2px;
+    stroke: hsla(30, 96%, 56%, .7);
+    stroke-width: 1px;
 }
 
 #filteredFPS circle {
-    fill: rgb(250, 73, 37);
+    fill: hsl(30, 96%, 56%);
 }
 
 #rawFPS path {
@@ -457,12 +458,3 @@ label.tree-label {
 #rawFPS circle {
     fill: rgb(250, 73, 37);
 }
-
-#intervalFPS path {
-    stroke: hsla(30, 96%, 56%, .7);
-    stroke-width: 1px;
-}
-
-#intervalFPS circle {
-    fill: hsl(30, 96%, 56%);
-}
index 4077593939f596ba19690c3000fef87f50f6ca45..e51856300c1060f0299ce0444e1edc320219dff3 100644 (file)
@@ -1,9 +1,13 @@
 Utilities.extendObject(window.benchmarkController, {
+    layoutCounter: 0,
+
     updateGraphData: function(graphData)
     {
         var element = document.getElementById("test-graph-data");
         element.innerHTML = "";
-        var margins = new Insets(10, 30, 30, 40);
+        document.querySelector("hr").style.width = this.layoutCounter++ + "px";
+
+        var margins = new Insets(30, 30, 30, 40);
         var size = Point.elementClientSize(element).subtract(margins.size);
 
         var svg = d3.select("#test-graph-data").append("svg")
@@ -29,7 +33,8 @@ Utilities.extendObject(window.benchmarkController, {
         // Axes
         var xAxis = d3.svg.axis()
                 .scale(x)
-                .orient("bottom");
+                .orient("bottom")
+                .tickFormat(function(d) { return (d/1000).toFixed(0); });
         var yAxisLeft = d3.svg.axis()
                 .scale(yLeft)
                 .orient("left");
@@ -127,9 +132,6 @@ Utilities.extendObject(window.benchmarkController, {
         var filteredData = graphData.samples.filter(function (sample) {
             return "smoothedFPS" in sample;
         });
-        var intervalData = graphData.samples.filter(function (sample) {
-            return "intervalFPS" in sample;
-        });
 
         function addData(name, data, yCoordinateCallback, pointRadius, omitLine) {
             var svgGroup = svg.append("g").attr("id", name);
@@ -156,7 +158,6 @@ Utilities.extendObject(window.benchmarkController, {
         addData("complexity", allData, function(d) { return yLeft(d.complexity); }, 2);
         addData("rawFPS", allData, function(d) { return yRight(d.fps); }, 1);
         addData("filteredFPS", filteredData, function(d) { return yRight(d.smoothedFPS); }, 2);
-        addData("intervalFPS", intervalData, function(d) { return yRight(d.intervalFPS); }, 2);
 
         // Area to handle mouse events
         var area = svg.append("rect")
@@ -167,7 +168,7 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("height", size.y);
 
         var timeBisect = d3.bisector(function(d) { return d.time; }).right;
-        var statsToHighlight = ["complexity", "rawFPS", "filteredFPS", "intervalFPS"];
+        var statsToHighlight = ["complexity", "rawFPS", "filteredFPS"];
         area.on("mouseover", function() {
             document.getElementById("cursor").classList.remove("hidden");
             document.querySelector("#test-graph nav").classList.remove("hide-data");
@@ -189,7 +190,7 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("x2", cursor_x)
                 .attr("y2", yRight(cursor_y));
 
-            document.querySelector("#test-graph nav .time").textContent = data.time.toFixed(4) + "s (" + index + ")";
+            document.querySelector("#test-graph nav .time").textContent = (data.time / 1000).toFixed(4) + "s (" + index + ")";
             statsToHighlight.forEach(function(name) {
                 var element = document.querySelector("#test-graph nav ." + name);
                 var content = "";
@@ -209,12 +210,6 @@ Utilities.extendObject(window.benchmarkController, {
                         data_y = yRight(data.smoothedFPS);
                     }
                     break;
-                case "intervalFPS":
-                    if ("intervalFPS" in data) {
-                        content = data.intervalFPS.toFixed(2);
-                        data_y = yRight(data.intervalFPS);
-                    }
-                    break;
                 }
 
                 element.textContent = content;
@@ -250,6 +245,5 @@ Utilities.extendObject(window.benchmarkController, {
         showOrHideNodes(form["complexity"].checked, "#complexity");
         showOrHideNodes(form["rawFPS"].checked, "#rawFPS");
         showOrHideNodes(form["filteredFPS"].checked, "#filteredFPS");
-        showOrHideNodes(form["intervalFPS"].checked, "#intervalFPS");
     }
 });
index 54865ee9b6ac4534361ff2fe727403febc18ee07..6189fb5114d6b29f982080d8e2818e58a830e510 100644 (file)
@@ -173,7 +173,7 @@ Point = Utilities.createClass(
     {
         return "x = " + this.x + ", y = " + this.y;
     },
-    
+
     add: function(other)
     {
         if(isNaN(other.x))
index 41fa9767f21f8eb0cddcc7956fe309deb0c55a91..1f291bdceee66ac4f8e533fa9be8a14e5db8cd0c 100644 (file)
@@ -307,7 +307,8 @@ window.benchmarkController = {
             "display": "minimal",
             "adjustment": "adaptive",
             "frame-rate": 50,
-            "estimated-frame-rate": true
+            "kalman-process-error": 1,
+            "kalman-measurement-error": 4
         };
         this._startBenchmark(Suites, options, "test-container");
     },
index f62860ebf06039ee003062835fab283d5bd1a775..21bc008d74e248b1ed1428fb2182edf6959fe6c2 100644 (file)
@@ -94,9 +94,9 @@ BenchmarkRunner = Utilities.createClass(
         Utilities.extendObject(options, contentWindow.Utilities.parseParameters());
 
         var benchmark = new contentWindow.benchmarkClass(options);
-        benchmark.run().then(function(sampler) {
+        benchmark.run().then(function(results) {
             var samplers = self._suitesSamplers[suite.name] || {};
-            samplers[test.name] = sampler.process(options);
+            samplers[test.name] = results;
             self._suitesSamplers[suite.name] = samplers;
 
             if (self._client && self._client.didRunTest)
index 69e19ef0b611bf02c15f0ea1f0536ed41c30c60a..b74a314a50706c6c1e8492519b0da190cf744baf 100644 (file)
-function BenchmarkState(testInterval)
-{
-    this._currentTimeOffset = 0;
-    this._stageInterval = testInterval / BenchmarkState.stages.FINISHED;
-}
-
-// The enum values and the messages should be in the same order
-BenchmarkState.stages = {
-    WARMING: 0,
-    SAMPLING: 1,
-    FINISHED: 2,
-}
-
-BenchmarkState.prototype =
-{
-    _timeOffset: function(stage)
-    {
-        return stage * this._stageInterval;
+
+Controller = Utilities.createClass(
+    function(testLength, benchmark, seriesCount)
+    {
+        // Initialize timestamps relative to the start of the benchmark
+        // In start() the timestamps are offset by the start timestamp
+        this._startTimestamp = 0;
+        this._endTimestamp = testLength;
+        // Default data series: timestamp, complexity, estimatedFrameLength
+        this._sampler = new Sampler(seriesCount || 3, 60 * testLength, this);
+        this._estimator = new SimpleKalmanEstimator(benchmark.options["kalman-process-error"], benchmark.options["kalman-measurement-error"]);
+
+        this.initialComplexity = 0;
+    }, {
+
+    start: function(stage, startTimestamp)
+    {
+        this._startTimestamp = startTimestamp;
+        this._endTimestamp += startTimestamp;
+        this.recordFirstSample(stage, startTimestamp);
+    },
+
+    recordFirstSample: function(stage, startTimestamp)
+    {
+        this._sampler.record(startTimestamp, stage.complexity(), -1);
+        this._sampler.mark(Strings.json.samplingTimeOffset, { time: 0 });
+    },
+
+    update: function(stage, timestamp)
+    {
+        this._estimator.sample(timestamp - this._sampler.samples[0][this._sampler.sampleCount - 1]);
+        this._sampler.record(timestamp, stage.complexity(), 1000 / this._estimator.estimate);
     },
 
-    _message: function(stage, timeOffset)
+    shouldStop: function(timestamp)
     {
-        if (stage == BenchmarkState.stages.FINISHED)
-            return BenchmarkState.stages.messages[stage];
+        return timestamp > this._endTimestamp;
+    },
 
-        return BenchmarkState.stages.messages[stage] + "... ("
-            + Math.floor((timeOffset - this._timeOffset(stage)) / 1000) + "/"
-            + Math.floor((this._timeOffset(stage + 1) - this._timeOffset(stage)) / 1000) + ")";
+    results: function()
+    {
+        return this._sampler.process();
     },
 
-    update: function(currentTimeOffset)
+    processSamples: function(results)
+    {
+        var complexityExperiment = new Experiment;
+        var smoothedFPSExperiment = new Experiment;
+
+        var samples = this._sampler.samples;
+
+        var samplingIndex = 0;
+        var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
+        if (samplingMark) {
+            samplingIndex = samplingMark.index;
+            results[Strings.json.samplingTimeOffset] = samplingMark.time;
+        }
+
+        results[Strings.json.samples] = samples[0].map(function(timestamp, i) {
+            var result = {
+                // Represent time in seconds
+                time: timestamp - this._startTimestamp,
+                complexity: samples[1][i]
+            };
+
+            // time offsets represented as FPS
+            if (i == 0)
+                result.fps = 60;
+            else
+                result.fps = 1000 / (timestamp - samples[0][i - 1]);
+
+            if (samples[2][i] != -1)
+                result.smoothedFPS = samples[2][i];
+
+            // Don't start adding data to the experiments until we reach the sampling timestamp
+            if (i >= samplingIndex) {
+                complexityExperiment.sample(result.complexity);
+                if (result.smoothedFPS && result.smoothedFPS != -1)
+                    smoothedFPSExperiment.sample(result.smoothedFPS);
+            }
+
+            return result;
+        }, this);
+
+        results[Strings.json.score] = complexityExperiment.score(Experiment.defaults.CONCERN);
+
+        var complexityResults = {};
+        results[Strings.json.experiments.complexity] = complexityResults;
+        complexityResults[Strings.json.measurements.average] = complexityExperiment.mean();
+        complexityResults[Strings.json.measurements.concern] = complexityExperiment.concern(Experiment.defaults.CONCERN);
+        complexityResults[Strings.json.measurements.stdev] = complexityExperiment.standardDeviation();
+        complexityResults[Strings.json.measurements.percent] = complexityExperiment.percentage();
+
+        var smoothedFPSResults = {};
+        results[Strings.json.experiments.frameRate] = smoothedFPSResults;
+        smoothedFPSResults[Strings.json.measurements.average] = smoothedFPSExperiment.mean();
+        smoothedFPSResults[Strings.json.measurements.concern] = smoothedFPSExperiment.concern(Experiment.defaults.CONCERN);
+        smoothedFPSResults[Strings.json.measurements.stdev] = smoothedFPSExperiment.standardDeviation();
+        smoothedFPSResults[Strings.json.measurements.percent] = smoothedFPSExperiment.percentage();
+    }
+});
+
+FixedComplexityController = Utilities.createSubclass(Controller,
+    function(testInterval, benchmark)
+    {
+        Controller.call(this, testInterval, benchmark);
+        this.initialComplexity = benchmark.options["complexity"];
+    }
+);
+
+AdaptiveController = Utilities.createSubclass(Controller,
+    function(testInterval, benchmark)
+    {
+        // Data series: timestamp, complexity, estimatedIntervalFrameLength
+        Controller.call(this, testInterval, benchmark);
+
+        // All tests start at 0, so we expect to see 60 fps quickly.
+        this._samplingTimestamp = testInterval / 2;
+        this._startedSampling = false;
+        this._targetFrameRate = benchmark.options["frame-rate"];
+        this._pid = new PIDController(this._targetFrameRate);
+
+        this._intervalFrameCount = 0;
+        this._numberOfFramesToMeasurePerInterval = 4;
+    }, {
+
+    start: function(stage, startTimestamp)
     {
-        this._currentTimeOffset = currentTimeOffset;
+        Controller.prototype.start.call(this, stage, startTimestamp);
+
+        this._samplingTimestamp += startTimestamp;
+        this._intervalTimestamp = startTimestamp;
     },
 
-    samplingTimeOffset: function()
+    recordFirstSample: function(stage, startTimestamp)
     {
-        return this._timeOffset(BenchmarkState.stages.SAMPLING);
+        this._sampler.record(startTimestamp, stage.complexity(), -1);
     },
 
-    currentStage: function()
+    update: function(stage, timestamp)
     {
-        for (var stage = BenchmarkState.stages.WARMING; stage < BenchmarkState.stages.FINISHED; ++stage) {
-            if (this._currentTimeOffset < this._timeOffset(stage + 1))
-                return stage;
+        if (!this._startedSampling && timestamp > this._samplingTimestamp) {
+            this._startedSampling = true;
+            this._sampler.mark(Strings.json.samplingTimeOffset, {
+                time: this._samplingTimestamp - this._startTimestamp
+            });
+        }
+
+        // Start the work for the next frame.
+        ++this._intervalFrameCount;
+
+        if (this._intervalFrameCount < this._numberOfFramesToMeasurePerInterval) {
+            this._sampler.record(timestamp, stage.complexity(), -1);
+            return;
         }
-        return BenchmarkState.stages.FINISHED;
+
+        // Adjust the test to reach the desired FPS.
+        var intervalLength = timestamp - this._intervalTimestamp;
+        this._estimator.sample(intervalLength / this._numberOfFramesToMeasurePerInterval);
+        var intervalEstimatedFrameRate = 1000 / this._estimator.estimate;
+        var tuneValue = -this._pid.tune(timestamp - this._startTimestamp, intervalLength, intervalEstimatedFrameRate);
+        tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
+        stage.tune(tuneValue);
+
+        this._sampler.record(timestamp, stage.complexity(), intervalEstimatedFrameRate);
+
+        // Start the next interval.
+        this._intervalFrameCount = 0;
+        this._intervalTimestamp = timestamp;
+    },
+
+    processSamples: function(results)
+    {
+        Controller.prototype.processSamples.call(this, results);
+        results[Strings.json.targetFPS] = this._targetFrameRate;
     }
-}
+});
 
 Stage = Utilities.createClass(
     function()
@@ -180,100 +308,26 @@ Rotater = Utilities.createClass(
     }
 });
 
-Animator = Utilities.createClass(
-    function()
-    {
-        this._intervalFrameCount = 0;
-        this._numberOfFramesToMeasurePerInterval = 3;
-        this._referenceTime = 0;
-        this._currentTimeOffset = 0;
-    }, {
-
-    initialize: function(benchmark)
-    {
-        this._benchmark = benchmark;
-
-        // Use Kalman filter to get a more non-fluctuating frame rate.
-        if (benchmark.options["estimated-frame-rate"])
-            this._estimator = new KalmanEstimator(60);
-        else
-            this._estimator = new IdentityEstimator;
-    },
-
-    get benchmark()
-    {
-        return this._benchmark;
-    },
-
-    _intervalTimeDelta: function()
-    {
-        return this._currentTimeOffset - this._startTimeOffset;
-    },
-
-    _shouldRequestAnotherFrame: function()
-    {
-        // Cadence is number of frames to measure, then one more frame to adjust the scene, and drop
-        var currentTime = performance.now();
-
-        if (!this._referenceTime)
-            this._referenceTime = currentTime;
-
-        this._currentTimeOffset = currentTime - this._referenceTime;
-
-        if (!this._intervalFrameCount)
-            this._startTimeOffset = this._currentTimeOffset;
-
-        // Start the work for the next frame.
-        ++this._intervalFrameCount;
-
-        // Drop _dropFrameCount frames and measure the average of _measureFrameCount frames.
-        if (this._intervalFrameCount <= this._numberOfFramesToMeasurePerInterval) {
-            this._benchmark.record(this._currentTimeOffset, -1, -1);
-            return true;
-        }
-
-        // Get the average FPS of _measureFrameCount frames over intervalTimeDelta.
-        var intervalTimeDelta = this._intervalTimeDelta();
-        var intervalFrameRate = 1000 / (intervalTimeDelta / this._numberOfFramesToMeasurePerInterval);
-        var estimatedIntervalFrameRate = this._estimator.estimate(intervalFrameRate);
-        // Record the complexity of the frame we just rendered. The next frame's time respresents the adjusted
-        // complexity
-        this._benchmark.record(this._currentTimeOffset, estimatedIntervalFrameRate, intervalFrameRate);
-
-        // Adjust the test to reach the desired FPS.
-        var shouldContinueRunning = this._benchmark.update(this._currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
-
-        // Start the next drop/measure cycle.
-        this._intervalFrameCount = 0;
-
-        // If result is false, no more requestAnimationFrame() will be invoked.
-        return shouldContinueRunning;
-    },
-
-    animateLoop: function()
-    {
-        if (this._shouldRequestAnotherFrame()) {
-            this._benchmark.stage.animate(this._intervalTimeDelta());
-            requestAnimationFrame(this.animateLoop.bind(this));
-        }
-    }
-});
-
 Benchmark = Utilities.createClass(
     function(stage, options)
     {
         this._options = options;
+        this._animateLoop = this._animateLoop.bind(this);
 
         this._stage = stage;
         this._stage.initialize(this);
-        this._animator = new Animator();
-        this._animator.initialize(this);
 
-        this._recordInterval = 200;
-        this._isSampling = false;
-        this._controller = new PIDController(this._options["frame-rate"]);
-        this._sampler = new Sampler(4, 60 * this._options["test-interval"], this);
-        this._state = new BenchmarkState(this._options["test-interval"] * 1000);
+        var testIntervalMilliseconds = options["test-interval"] * 1000;
+        switch (options["adjustment"])
+        {
+        case "fixed":
+            this._controller = new FixedComplexityController(testIntervalMilliseconds, this);
+            break;
+        case "adaptive":
+        default:
+            this._controller = new AdaptiveController(testIntervalMilliseconds, this);
+            break;
+        }
     }, {
 
     get options()
@@ -286,84 +340,19 @@ Benchmark = Utilities.createClass(
         return this._stage;
     },
 
-    get animator()
-    {
-        return this._animator;
-    },
-
-    // Called from the load event listener or from this.run().
-    start: function()
-    {
-        this._animator.animateLoop();
-    },
-
-    // Called from the animator to adjust the complexity of the test.
-    update: function(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate)
-    {
-        this._state.update(currentTimeOffset);
-
-        var stage = this._state.currentStage();
-        if (stage == BenchmarkState.stages.FINISHED) {
-            this._stage.clear();
-            return false;
-        }
-
-        if (stage == BenchmarkState.stages.SAMPLING && !this._isSampling) {
-            this._sampler.mark(Strings.json.samplingTimeOffset, {
-                time: this._state.samplingTimeOffset() / 1000
-            });
-            this._isSampling = true;
-        }
-
-        var tuneValue = 0;
-        if (this._options["adjustment"] == "fixed") {
-            if (this._options["complexity"]) {
-                // this._stage.tune(0) returns the current complexity of the test.
-                tuneValue = this._options["complexity"] - this._stage.tune(0);
-            }
-        }
-        else if (!(this._isSampling && this._options["adjustment"] == "fixed-after-warmup")) {
-            // The relationship between frameRate and test complexity is inverse-proportional so we
-            // need to use the negative of PIDController.tune() to change the complexity of the test.
-            tuneValue = -this._controller.tune(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
-            tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
-        }
-
-        this._stage.tune(tuneValue);
-
-        if (typeof this._recordTimeOffset == "undefined")
-            this._recordTimeOffset = currentTimeOffset;
-
-        var stage = this._state.currentStage();
-        if (stage != BenchmarkState.stages.FINISHED && currentTimeOffset < this._recordTimeOffset + this._recordInterval)
-            return true;
-
-        this._recordTimeOffset = currentTimeOffset;
-        return true;
-    },
-
-    record: function(currentTimeOffset, estimatedFrameRate, intervalFrameRate)
-    {
-        // If the frame rate is -1 it means we are still recording for this sample
-        this._sampler.record(currentTimeOffset, this.stage.complexity(), estimatedFrameRate, intervalFrameRate);
-    },
-
     run: function()
     {
         return this.waitUntilReady().then(function() {
-            this.start();
-            var promise = new SimplePromise;
-            var resolveWhenFinished = function() {
-                if (typeof this._state != "undefined" && (this._state.currentStage() == BenchmarkState.stages.FINISHED))
-                    return promise.resolve(this._sampler);
-                setTimeout(resolveWhenFinished, 50);
-            }.bind(this);
-
-            resolveWhenFinished();
-            return promise;
+            this._finishPromise = new SimplePromise;
+            this._previousTimestamp = performance.now();
+            this._didWarmUp = false;
+            this._stage.tune(this._controller.initialComplexity - this._stage.complexity());
+            this._animateLoop();
+            return this._finishPromise;
         }.bind(this));
     },
 
+    // Subclasses should override this if they have setup to do prior to commencing.
     waitUntilReady: function()
     {
         var promise = new SimplePromise;
@@ -371,55 +360,30 @@ Benchmark = Utilities.createClass(
         return promise;
     },
 
-    processSamples: function(results)
+    _animateLoop: function()
     {
-        var complexity = new Experiment;
-        var smoothedFPS = new Experiment;
-        var samplingIndex = 0;
+        this._currentTimestamp = performance.now();
 
-        var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
-        if (samplingMark) {
-            samplingIndex = samplingMark.index;
-            results[Strings.json.samplingTimeOffset] = samplingMark.time;
-        }
-
-        results[Strings.json.samples] = this._sampler.samples[0].map(function(d, i) {
-            var result = {
-                // time offsets represented as seconds
-                time: d/1000,
-                complexity: this._sampler.samples[1][i]
-            };
-
-            // time offsets represented as FPS
-            if (i == 0)
-                result.fps = 60;
-            else
-                result.fps = 1000 / (d - this._sampler.samples[0][i - 1]);
-
-            var smoothedFPSresult = this._sampler.samples[2][i];
-            if (smoothedFPSresult != -1) {
-                result.smoothedFPS = smoothedFPSresult;
-                result.intervalFPS = this._sampler.samples[3][i];
+        if (!this._didWarmUp) {
+            if (this._currentTimestamp - this._previousTimestamp >= 100) {
+                this._didWarmUp = true;
+                this._controller.start(this._stage, this._currentTimestamp);
+                this._previousTimestamp = this._currentTimestamp;
             }
 
-            if (i >= samplingIndex) {
-                complexity.sample(result.complexity);
-                if (smoothedFPSresult != -1) {
-                    smoothedFPS.sample(smoothedFPSresult);
-                }
-            }
+            this._stage.animate(0);
+            requestAnimationFrame(this._animateLoop);
+            return;
+        }
 
-            return result;
-        }, this);
+        this._controller.update(this._stage, this._currentTimestamp);
+        if (this._controller.shouldStop(this._currentTimestamp)) {
+            this._finishPromise.resolve(this._controller.results());
+            return;
+        }
 
-        results[Strings.json.score] = complexity.score(Experiment.defaults.CONCERN);
-        [complexity, smoothedFPS].forEach(function(experiment, index) {
-            var jsonExperiment = !index ? Strings.json.experiments.complexity : Strings.json.experiments.frameRate;
-            results[jsonExperiment] = {};
-            results[jsonExperiment][Strings.json.measurements.average] = experiment.mean();
-            results[jsonExperiment][Strings.json.measurements.concern] = experiment.concern(Experiment.defaults.CONCERN);
-            results[jsonExperiment][Strings.json.measurements.stdev] = experiment.standardDeviation();
-            results[jsonExperiment][Strings.json.measurements.percent] = experiment.percentage();
-        });
+        this._stage.animate(this._currentTimestamp - this._previousTimestamp);
+        this._previousTimestamp = this._currentTimestamp;
+        requestAnimationFrame(this._animateLoop);
     }
 });
index 677a20be26349695f4274a1f04b25d19066fbad0..bcbe902928f2014854f0e8fbde939750eea4328b 100644 (file)
-var Matrix =
-{
-    init: function(m, n, v)
-    {
-        return Array(m * n).fill(v);
-    },
-
-    zeros: function(m, n)
-    {
-        return Matrix.init(m, n, 0);
-    },
-
-    ones: function(m, n)
-    {
-        return Matrix.init(m, n, 1);
-    },
-
-    identity: function(n)
-    {
-        var out = new Matrix.zeros(n, n);
-        for (var i = 0; i < n; ++i)
-            out[i * n + i] = 1;
-        return out;
-    },
-
-    str: function(A, n, m)
-    {
-        var out = (m > 1 && n > 1 ? "Matrix[" + n + ", " + m : "Vector[" + m * n) + "] = [";
-        for (var i = 0; i < n * m; ++i) {
-            out += A[i];
-            if (i < n * m - 1)
-                out += ", ";
-        }
-        return out + "]";
-    },
-
-    pos: function(m, i, j)
-    {
-        return m * i + j;
-    },
-
-    add: function(A, B, n, m)
-    {
-        var out = Matrix.zeros(n, m);
-        for (var i = 0; i < n * m; ++i)
-            out[i] = A[i] + B[i];
-        return out;
-    },
-
-    subtract: function(A, B, n, m)
-    {
-        var out = Matrix.zeros(n, m);
-        for (var i = 0; i < n * m; ++i)
-            out[i] = A[i] - B[i];
-        return out;
-    },
-
-    scale: function(s, A, n, m)
-    {
-        var out = Matrix.zeros(n, m);
-        for (var i = 0; i < n * m; ++i)
-            out[i] = s * A[i];
-        return out;
-    },
+SimpleKalmanEstimator = Utilities.createClass(
+    function(processError, measurementError) {
+        this._initialized = false;
 
-    transpose: function(A, n, m)
-    {
-        var out = Matrix.zeros(m, n);
-        for (var i = 0; i < n; ++i) {
-            for (var j = 0; j < m; ++j)
-                out[Matrix.pos(n, i, j)] = A[Matrix.pos(m, j, i)];
-        }
-        return out;
-    },
+        var error = .5 * (Math.sqrt(processError * processError + 4 * processError * measurementError) - processError);
+        this._gain = error / (error + measurementError);
+    }, {
 
-    multiply: function(A, B, n, m, p)
+    sample: function(newMeasurement)
     {
-        var out = Matrix.zeros(n, p);
-        for (var i = 0; i < n; ++i) {
-            for (var j = 0; j < p; ++j) {
-                for (var k = 0; k < m; ++k) {
-                    out[Matrix.pos(p, i, j)] += A[Matrix.pos(m, i, k)] * B[Matrix.pos(p, k, j)];
-                }
-            }
+        if (!this._initialized) {
+            this._initialized = true;
+            this._estimatedMeasurement = newMeasurement;
+            return;
         }
-        return out;
-    }
-}
-
-var Vector3 =
-{
-    zeros: function()
-    {
-        return Matrix.zeros(1, 3);
-    },
-
-    ones: function()
-    {
-        return Matrix.ones(1, 3);
-    },
-
-    str: function(v)
-    {
-        return Matrix.str(v, 1, 3);
-    },
-
-    add: function(v, w)
-    {
-        return Matrix.add(v, w, 1, 3);
-    },
-
-    subtract: function(v, w)
-    {
-        return Matrix.subtract(v, w, 1, 3);
-    },
-
-    scale: function(s, v)
-    {
-        return Matrix.scale(s, v, 1, 3);
-    },
 
-    multiplyMatrix3: function(v, A)
-    {
-        return Matrix.multiply(v, A, 1, 3, 3);
+        this._estimatedMeasurement = this._estimatedMeasurement + this._gain * (newMeasurement - this._estimatedMeasurement);
     },
 
-    multiplyVector3: function(v, w)
+    get estimate()
     {
-        var out = 0;
-        for (var i = 0; i < 3; ++i)
-            out += v[i] * w[i];
-        return out;
+        return this._estimatedMeasurement;
     }
-}
-
-var Matrix3 =
-{
-    zeros: function()
-    {
-        return Matrix.zeros(3, 3);
-    },
-
-    identity: function()
-    {
-        return Matrix.identity(3, 3);
-    },
-
-    str: function(A)
-    {
-        return Matrix.str(A, 3, 3);
-    },
+});
 
-    pos: function(i, j)
+PIDController = Utilities.createClass(
+    function(ysp)
     {
-        return Matrix.pos(3, i, j);
-    },
-
-    add: function(A, B)
-    {
-        return Matrix.add(A, B, 3, 3);
-    },
-
-    subtract: function(A, B)
-    {
-        return Matrix.subtract(A, B, 3, 3);
-    },
-
-    scale: function(s, A)
-    {
-        return Matrix.scale(s, A, 3, 3);
-    },
+        this._ysp = ysp;
+        this._out = 0;
 
-    transpose: function(A)
-    {
-        return Matrix.transpose(A, 3, 3);
-    },
+        this._Kp = 0;
+        this._stage = PIDController.stages.WARMING;
 
-    multiplyMatrix3: function(A, B)
-    {
-        return Matrix.multiply(A, B, 3, 3, 3);
-    },
+        this._eold = 0;
+        this._I = 0;
+    }, {
 
-    multiplyVector3: function(A, v)
-    {
-        return Matrix.multiply(A, v, 3, 3, 1);
-    }
-}
-
-function PIDController(ysp)
-{
-    this._ysp = ysp;
-    this._out = 0;
-
-    this._Kp = 0;
-    this._stage = PIDController.stages.WARMING;
-
-    this._eold = 0;
-    this._I = 0;
-}
-
-// This enum will be used to tell whether the system output (or the controller input)
-// is moving towards the set-point or away from it.
-PIDController.yPositions = {
-    BEFORE_SETPOINT: 0,
-    AFTER_SETPOINT: 1
-}
-
-// The Ziegler–Nichols method for is used tuning the PID controller. The workflow of
-// the tuning is split into four stages. The first two stages determine the values
-// of the PID controller gains. During these two stages we return the proportional
-// term only. The third stage is used to determine the min-max values of the
-// saturation actuator. In the last stage back-calculation and tracking are applied
-// to avoid integrator windup. During the last two stages, we return a PID control
-// value.
-PIDController.stages = {
-    WARMING: 0,         // Increase the value of the Kp until the system output reaches ysp.
-    OVERSHOOT: 1,       // Measure the oscillation period and the overshoot value
-    UNDERSHOOT: 2,      // Return PID value and measure the undershoot value
-    SATURATE: 3         // Return PID value and apply back-calculation and tracking.
-}
-
-PIDController.prototype =
-{
     // Determines whether the current y is
     //  before ysp => (below ysp if ysp > y0) || (above ysp if ysp < y0)
     //  after ysp => (above ysp if ysp > y0) || (below ysp if ysp < y0)
@@ -410,7 +225,7 @@ PIDController.prototype =
         return u;
     },
 
-    // Called from the benchmark to tune its test. It uses Ziegler–Nichols method
+    // Called from the benchmark to tune its test. It uses Ziegler-Nichols method
     // to calculate the controller parameters. It then returns a PID tuning value.
     tune: function(t, h, y)
     {
@@ -426,66 +241,27 @@ PIDController.prototype =
         // Apply back-calculation and tracking to avoid integrator windup
         return this._saturate(v, e);
     }
-}
-
-function KalmanEstimator(initX)
-{
-    // Initialize state transition matrix.
-    this._matA = Matrix3.identity();
-    this._matA[Matrix3.pos(0, 2)] = 1;
-
-    // Initialize measurement matrix.
-    this._vecH = Vector3.zeros();
-    this._vecH[0] = 1;
-
-    this._matQ = Matrix3.identity();
-    this._R = 1000;
-
-    // Initial state conditions.
-    this._vecX_est = Vector3.zeros();
-    this._vecX_est[0] = initX;
-    this._matP_est = Matrix3.zeros();
-}
-
-KalmanEstimator.prototype =
-{
-    estimate: function(current)
-    {
-        // Project the state ahead
-        //  X_prd(k) = A * X_est(k-1)
-        var vecX_prd = Matrix3.multiplyVector3(this._matA, this._vecX_est);
-
-        // Project the error covariance ahead
-        //  P_prd(k) = A * P_est(k-1) * A' + Q
-        var matP_prd = Matrix3.add(Matrix3.multiplyMatrix3(Matrix3.multiplyMatrix3(this._matA, this._matP_est), Matrix3.transpose(this._matA)), this._matQ);
-
-        // Compute Kalman gain
-        //  B = H * P_prd(k)';
-        //  S = B * H' + R;
-        //  K(k) = (S \ B)';
-        var vecB = Vector3.multiplyMatrix3(this._vecH, Matrix3.transpose(matP_prd));
-        var S = Vector3.multiplyVector3(vecB, this._vecH) + this._R;
-        var vecGain = Vector3.scale(1/S, vecB);
-
-        // Update the estimate via z(k)
-        //  X_est(k) = x_prd + K(k) * (z(k) - H * X_prd(k));
-        this._vecX_est = Vector3.add(vecX_prd, Vector3.scale(current - Vector3.multiplyVector3(this._vecH, vecX_prd), vecGain));
-
-        // Update the error covariance
-        //  P_est(k) = P_prd(k) - K(k) * H * P_prd(k);
-        this._matP_est = Matrix3.subtract(matP_prd, Matrix3.scale(Vector3.multiplyVector3(vecGain, this._vecH), matP_prd));
-
-        // Compute the estimated measurement.
-        //  y = H * X_est(k);
-        return Vector3.multiplyVector3(this._vecH,  this._vecX_est);
-    }
-}
-
-function IdentityEstimator() {}
-IdentityEstimator.prototype =
-{
-    estimate: function(current)
-    {
-        return current;
+});
+
+Utilities.extendObject(PIDController, {
+    // This enum will be used to tell whether the system output (or the controller input)
+    // is moving towards the set-point or away from it.
+    yPositions: {
+        BEFORE_SETPOINT: 0,
+        AFTER_SETPOINT: 1
+    },
+
+    // The Ziegler-Nichols method for is used tuning the PID controller. The workflow of
+    // the tuning is split into four stages. The first two stages determine the values
+    // of the PID controller gains. During these two stages we return the proportional
+    // term only. The third stage is used to determine the min-max values of the
+    // saturation actuator. In the last stage back-calculation and tracking are applied
+    // to avoid integrator windup. During the last two stages, we return a PID control
+    // value.
+    stages: {
+        WARMING: 0,         // Increase the value of the Kp until the system output reaches ysp.
+        OVERSHOOT: 1,       // Measure the oscillation period and the overshoot value
+        UNDERSHOOT: 2,      // Return PID value and measure the undershoot value
+        SATURATE: 3         // Return PID value and apply back-calculation and tracking.
     }
-};
+});
index 52068ec3f26a3223101c295bb773b9e2b24dbb33..e60995f30fb3b7e18c87fb2cacef2329829bc9f6 100644 (file)
@@ -81,13 +81,10 @@ Sampler = Utilities.createClass(
         this.marks[comment] = data;
     },
 
-    process: function(options)
+    process: function()
     {
         var results = {};
 
-        if (options["adjustment"] == "adaptive")
-            results[Strings.json.targetFPS] = +options["frame-rate"];
-
         // Remove unused capacity
         this.samples = this.samples.map(function(array) {
             return array.slice(0, this.sampleCount);
index e8e7c3991e5aae958562859f29d3655507dcc894..ec01957c0abfd4c0a0a047e073c75d74cd76326e 100644 (file)
@@ -1,3 +1,105 @@
+2016-02-07  Jon Lee  <jonlee@apple.com>
+
+        Update how the benchmark is run
+        https://bugs.webkit.org/show_bug.cgi?id=153960
+
+        Provisionally reviewed by Said Abou-Hallawa.
+
+        Introduce the notion of a Controller. It is responsible for recording, updating,
+        and processing the statistics and complexity of the benchmark. This allows
+        plugging in different Controllers.
+
+        This strips most of the functionality from Animator and BenchmarkState, so fold
+        what's left into Benchmark. Now, Benchmarks only own a stage and a controller, but
+        are responsible for driving the animation loop.
+
+        Rewrite Animator._shouldRequestAnotherFrame into two different Controllers. One
+        maintains a fixed complexity, and the other adapts the complexity to meet a
+        fixed FPS.
+
+        Fix the Kalman estimator to be modeled on a scalar variable with no model.
+
+        * Animometer/tests/resources/main.js: Remove BenchmarkState and Animator, and
+        replace it with a Controller. Add a FixedController and refactor the previous controller
+        to an AdaptiveController.
+
+        (Controller): Controllers own the estimator and the sampler. When a new frame is
+        displayed, the animation loop calls update(). The estimator and sampler record
+        stats, then tune. Samplers can track multiple series of data. The basic controller
+        tracks timestamp, complexity, and estimated frame rate.
+                The Kalman estimation is based on the frame length rather than the frame
+        rate. Because FPS is inverse proportional to frame length, in the case where the measured
+        frame length is very small, the FPS ends up being a wildly large number (in the order of
+        600-1000 "FPS"), and it pulls the estimator up drastically enough that it takes a while
+        for it to settle back down. Using frame length reduces the impact of these spikes.
+                Converging the estimation takes enough time to avoid initializing it immediately
+        when the benchmark starts. Instead, the benchmark runs for a brief period of time (100ms)
+        before running it in earnest. Allow controllers an opportunity to set the complexity
+        before starting recording.
+                When the benchmark is complete, the controller has an opportunity to process
+        the samples. The default implementation calculates the raw FPS based on the time
+        difference of the samples, and calculates the complexity score. This is moved from
+        Benchmark.processSamples.
+
+        (Controller): Initialize timestamps. These are at first relative to the start of the
+        benchmark, but are offset by the absolute start time during start(). By default maintain
+        3 data series, but subclasses can override.
+        (start): Calls recordFirstSample() for subclasses to override if needed.
+        (recordFirstSample): For basic controller, start sampling at the beginning.
+        (update): Update the frame length estimator and sample.
+        (shouldStop): Checks that the time is before _endTimestamp.
+        (results): Returns the processed samples.
+        (processSamples): Iterate through the sample data and collate them. Include scores.
+
+        (FixedComplexityController): Controller that tunes the stage to the desired complexity
+        prior to starting, and keeps it at that complexity.
+
+        (AdaptiveController): Have the estimator estimate the interval frame rate instead of the
+        raw frame rate.
+                The previous version of this controller ignored the frame that came after the
+        adjustment. The raw FPS show that whatever noise the scene change adds is negligible
+        compared to the noise of the system overall. Stop ignoring that frame and include all
+        frames in the measurements.
+
+        (Benchmark): Remove dependency on animator, and instantiate a runner based on what is
+        selected. Most of the loop's functionality is in Controller, so remove here.
+        (Benchmark.run): Remove start() since it is only called from run(), and fold it in here.
+        (Benchmark._animateLoop): Fold in from Animator.animateLoop. Let the benchmark run for
+        a brief period before calling Controller.start().
+
+        * Animometer/tests/resources/math.js: Fix the Kalman estimator. The filter estimates
+        a scalar variable, and makes basic assumptions regarding the model. As a result
+        none of the linear algebra classes are needed, so remove Matrix, Vector3, and Matrix3.
+        (SimpleKalmanEstimator): Calculate the gain based on the provided process and
+        measurement errors.
+        (KalmanEstimator): Deleted.
+        (IdentityEstimator): Deleted.
+        (PIDController): Refactor to use the Utilities.createClass() helper.
+
+        The Kalman filter algorithm is explained here http://greg.czerniak.info/guides/kalman1/.
+        The state, represented by a scalar, is the estimated frame length. There is no user
+        transition of the state, and the state is the same as the measurement. With this model,
+        the estimation error converges, so calculate the gain ahead of time.
+
+        * Animometer/developer.html: Remove fixed-after-warmup since it is not useful.
+        Replace the option to toggle the estimator, and make it possible to customize the
+        estimator's error parameters. Show raw FPS by default, and remove interval FPS,
+        which will be shown instead of the filtered raw FPS.
+        * Animometer/resources/debug-runner/animometer.css: Put the header behind the graph.
+        Remove #intervalFPS rules; move the color to #filteredFPS.
+        * Animometer/resources/debug-runner/graph.js:
+        (updateGraphData): Update the hr style to force the layout to be calculated
+        correctly. Change the tick format to be in terms of seconds, since the timestamps
+        are in milliseconds. Remove interval data.
+        * Animometer/resources/runner/animometer.js:
+        (window.benchmarkController.startBenchmark): Set Kalman parameters.
+        * Animometer/resources/runner/benchmark-runner.js:
+        (_runBenchmarkAndRecordResults): When a benchmark completes, expect it to return
+        the final data, rather than passing a sampler from the controller. This avoids
+        needing to expose the sampler variable in the benchmark.
+        * Animometer/tests/resources/sampler.js:
+        (process): Move the setting of the target frame rate to AdaptiveController.
+
 2016-02-06  Jon Lee  <jonlee@apple.com>
 
         Code clean up: Move Rotater function closer to Stage static methods.