Update animation benchmark and tests
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sat, 27 Feb 2016 04:32:09 +0000 (04:32 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sat, 27 Feb 2016 04:32:09 +0000 (04:32 +0000)
https://bugs.webkit.org/show_bug.cgi?id=154673

Reviewed by Dean Jackson.

Update the ramp controller.

The controller refines the complexity interval to test across.

* Animometer/resources/statistics.js: Add functions that estimate cumulative distribution function.
(Regression): For the flat regression, force the first segment to be at 60 fps.
(valueAt): Add convenience function to return interpolated value based on the regression used.
(_calculateRegression): Include the number of points included for both segments, and the piecewise
errors.
* Animometer/tests/resources/math.js: Make the Kalman estimator subclass Experiment, and allow it
to be reset.

* Animometer/tests/resources/main.js: Initialize the tier such that it starts at 10^0 = 1.
Increase the number of ramps. Maintain three FPS thresholds-- the frame rate of interest, a limit
on the lowest FPS we care to go for later interpolation, and a minimum FPS threshold we want to
aim for each ramp. Also keep three estimators: a running average of the change point, a minimum
boundary for each ramp, and an estimator for all the frames within an interval. The first two
are used to determine the parameters of the next ramp, and the latter allows us to refine the
parameters.
(update): During the tier phase, it is possible that the highest complexity possible for a test
won't stress the system enough to trigger stopping the tier phase and transitioning to the ramps.
If the complexity doesn't change when going to the next tier, we've maxed the test out, and move
on. When the tier phase completed, turn off Controller.frameLengthEstimator, which estimates the
FPS at each tier.
(tune): At each interval, look at the confidence distribution of being on the 60 FPS side or the
slow side. If the slowest FPS we achieve at the ramp's maximum complexity is not at least
_fpsRampSlowThreshold, then increase the maximum complexity. If we ever achieve 60 FPS, increase
the ramp's minimum complexity to that level. If, at an even lower complexity, a glitch causes the
FPS to drop, we reset the minimum complexity.

Have the bootstrap calculation occur between tests. Clean up harness.

* Animometer/resources/debug-runner/animometer.js: Run bootstrap after a test has
completed to avoid doing all of it at the end before showing the results. Clean up
parameters being passed around.
* Animometer/resources/debug-runner/tests.js:
(text):
* Animometer/resources/runner/animometer.js:
(this._processData.calculateScore): Save the results to the same object holding the data.
(this._processData._processData): In the case where a file is dragged, calculate the score
serially. Grab the results object and move it to the results variable and remove it from
the data object. This avoids serializing the results into the JSON.
(this._processData.findRegression): Include the samples used for bootstrapping. Reduce the
resample size to shorten the wait.
* Animometer/resources/runner/benchmark-runner.js:
* Animometer/resources/statistics.js:
(bootstrap): Update how bootstrapData is sorted. In some regression results the mix of
floats and integers causes an alphabetical sort to occur.
* Animometer/resources/strings.js:

Add meta charset so that encodings between harness and test match.

* Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
* Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-css-images.html:
* Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-svg-images.html:
* Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
* Animometer/tests/master/canvas-stage.html:
* Animometer/tests/master/focus.html:
* Animometer/tests/master/image-data.html:
* Animometer/tests/master/multiply.html:
* Animometer/tests/master/particles.html:
* Animometer/tests/misc/canvas-electrons.html:
* Animometer/tests/misc/canvas-stars.html:
* Animometer/tests/misc/compositing-transforms.html:
* Animometer/tests/simple/simple-canvas-paths.html:
* Animometer/tests/simple/tiled-canvas-image.html:
* Animometer/tests/template/template-canvas.html:
* Animometer/tests/template/template-css.html:
* Animometer/tests/template/template-svg.html:
* Animometer/tests/text/layering-text.html:
* Animometer/tests/text/text-boxes.html:

Update test harness reporting.

* Animometer/developer.html: Add missing meta charset.
* Animometer/index.html: Remove unnecessary utf-8 declaration.
* Animometer/resources/debug-runner/animometer.css: Add convenience classes for
formatting the results table.
* Animometer/resources/debug-runner/animometer.js: Adjust which stats are shown.
* Animometer/resources/debug-runner/tests.js: Display bootstrapping statistics.
* Animometer/resources/strings.js: Move strings not used by the release harness.

Switch to a pseudo-random number generator.

* Animometer/resources/statistics.js: Add a Pseudo class, with a simple
pseudo-random number generator.
(_calculateRegression): Reset the generator before running bootstrap.
(bootstrap): Deleted.

Replace Math.random with Pseudo.random.
* Animometer/tests/master/resources/canvas-tests.js:
* Animometer/tests/master/resources/focus.js:
* Animometer/tests/master/resources/particles.js:
* Animometer/tests/resources/main.js:

Use bootstrapping to get confidence interval in the breakpoint.

For the ramp controller, calculate the piecewise regression, and then use
bootstrapping in order to find the 95% confidence interval. Use the raw data.

* Animometer/developer.html: Default to the complexity graph. Add a legend
checkbox to toggle visibility of the bootstrap score and histogram.
* Animometer/resources/debug-runner/animometer.css: Make some more space to show
the old raw and average scores in the legend. Add new styles for the data.
* Animometer/resources/debug-runner/graph.js:
(_addRegressionLine): Allow passing an array for the variance bar tied to the
regression line. Now |stdev| is |range|.
(createComplexityGraph): Add bootstrap median, and overlay a histogram of
the bootstrap samples. Switch raw samples from circles to X's.
(onComplexityGraphOptionsChanged): Allow toggling of the bootstrap data.
(onGraphTypeChanged): Move the regressions for the raw and average samples to the
legend. In the subtitle use the bootstrap median, and include the 95% confidence
interval.
* Animometer/resources/runner/animometer.js:
(this._processData.findRegression): Factor out the code that determines which
samples to include when calculating the piecewise regression. For series that have
many samples, or a wider range of recorded complexities, throw away the 2.5%
lowest and highest samples before calculating the regression. Keep all samples
if the number of samples to regress is small or the range of complexities is
narrow.
(this._processData._calculateScore): Factor out regression calculation to
findRegression(). Bootstrap the change point of the regression. The score is the
median.
* Animometer/resources/statistics.js:
(_calculateRegression): Correct an issue in the calculation of the regression, where
the denominator can be 0.
(bootstrap): Template for bootstrapping. Create a bootstrap sample array, Create
re-samples by random selection with replacement. Return the 95% confidence samples,
the bootstrap median, mean, and the data itself.
* Animometer/resources/strings.js: Add bootstrap.
* Animometer/tests/resources/main.js:
(processSamples): Don't prematurely cut the sample data.

Fix graph drawing.

* Animometer/resources/debug-runner/animometer.js: Add spacing in the JSON output.
Multiple tests output a lot of JSON and can hang when selecting JSON with no whitespace.
* Animometer/resources/debug-runner/animometer.css:
(#complexity-graph .series.raw circle): Update the color.
* Animometer/resources/debug-runner/graph.js: Use the FPS axis instead of the
complexity axis, which can vary in domain. For determining the complexity domain,
only use samples after samplingTimeOffset.

Allow dropping results JSON.

* Animometer/developer.html: Add a button.
* Animometer/resources/debug-runner/animometer.css:
* Animometer/resources/debug-runner/animometer.js: Read the data and go straight
to the dashboard. With JSON output, write out only the options and the raw data.

Teach the harness to evaluate the samples and determine the test score.

This will allow us to update how the score is calculated separately from the samples recorded.
This also prepares the harness to be able to accept JSON of prior runs.

* Animometer/resources/strings.js: Clean up and remove unneeded strings and reduce some of the
hierarchy.
* Animometer/resources/debug-runner/tests.js: Update to use the new strings.

* Animometer/tests/resources/main.js: Allow all controllers to show a complexity-FPS graph.
(_processComplexitySamples): Factor out some of the sample processing done in the ramp
controller for the benefit of the other controllers. |complexitySamples| contains a list of
samples. Sort the samples by complexity. Optionally remove the top x% of samples.
Group them, and calculate distribution of samples within the same complexity, and add those as
new entries into |complexityAverageSamples|.
(Controller.processSamples): Move the code responsible for determining the complexity and FPS
scores out to ResultsDashboard. The structure of the data returned by the controller is:

{
    controller: [time-regression, time-regression, ...], // optional, data specific to controller
    marks: [...],
    samples: {                    // all of the sample data
        controller: [...],
        complexity: [...],        // processed from controller samples
        complexityAverage: [...], // processed from complexity samples
    }
}

(AdaptiveController.processSamples): Adding the target frame length is no longer necessary; we
now pass the test options to the graph.
(Regression): Move to statistics.js.
* Animometer/resources/statistics.js: Move Regression to here. Add a check if the sampling range
only contains one sample, since we cannot calculate a regression from one sample point.

Teach the test harness to evaluate the data.
* Animometer/resources/runner/animometer.js:
(ResultsDashboard): Store the options used to run the test and the computed results/score separately
from the data. The results are stored as:

{
    score: /* geomean of iteration score */,
    iterationsResults: [
        {
            score: /* geomean of tests */,
            testsResults: {
                suiteName: {
                    testName: {
                        controller: {
                            average:
                            concern:
                            stdev:
                            percent:
                        },
                        frameLength: { ... },
                        complexity: {
                            complexity:
                            stdev:
                            segment1:
                            segment2:
                        },
                        complexityAverage: { ... }
                    },
                    testName: { ... },
                },
                ... next suite ...
            }
        },
        { ...next iteration... }
    ]
}

* Animometer/resources/debug-runner/animometer.js: Pass options around instead of relying
on what was selected in the form. This will later allow for dropping previous results, and
using those runs' options when calculating scores.
(ResultsTable._addGraphButton): Simplify button action by using attached test data.
* Animometer/resources/debug-runner/graph.js: Refactor to use the data.

Consolidate JS files, and move statistics out to a separate JS.

Preparation for having the Controller only handle recording and storage of the samples,
and leave the evaluation of the test score out to the harness. Move Experiment to
a new statistics.js, where Regression will also eventually go. Get rid of algorithm.js
and move it to utilities.js since the Heap is used only for Experiments.

* Animometer/tests/resources/algorithm.js: Removed. Heap is in utilities.js.
* Animometer/tests/resources/sampler.js: Removed. Experiment is in statistics.js,
Sampler in main.js.
* Animometer/tests/resources/main.js: Move Sampler here.
* Animometer/resources/statistics.js: Added. Move Statistics and Experiment here.
* Animometer/resources/extensions.js: Move Heap here. Attach static method to create
a max or min heap to Heap, instead of a new Algorithm object.

Update JS files.
* Animometer/developer.html:
* Animometer/index.html:
* Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
* Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-css-images.html:
* Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-svg-images.html:
* Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
* Animometer/tests/master/canvas-stage.html:
* Animometer/tests/master/focus.html:
* Animometer/tests/master/image-data.html:
* Animometer/tests/master/multiply.html:
* Animometer/tests/master/particles.html:
* Animometer/tests/misc/canvas-electrons.html:
* Animometer/tests/misc/canvas-stars.html:
* Animometer/tests/misc/compositing-transforms.html:
* Animometer/tests/simple/simple-canvas-paths.html:
* Animometer/tests/simple/tiled-canvas-image.html:
* Animometer/tests/template/template-canvas.html:
* Animometer/tests/template/template-css.html:
* Animometer/tests/template/template-svg.html:
* Animometer/tests/text/layering-text.html:
* Animometer/tests/text/text-boxes.html:

Fix the cursor in the graph analysis when the min
complexity is not 0.

* Animometer/resources/debug-runner/graph.js:
(_addRegression):
(createComplexityGraph):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@197229 268f45cc-cd09-0410-ab3c-d52691b4dbfc

40 files changed:
PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/index.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/animometer.js
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/debug-runner/tests.js
PerformanceTests/Animometer/resources/extensions.js
PerformanceTests/Animometer/resources/runner/animometer.js
PerformanceTests/Animometer/resources/runner/benchmark-runner.js
PerformanceTests/Animometer/resources/statistics.js [new file with mode: 0644]
PerformanceTests/Animometer/resources/strings.js
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-canvas-images.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-css-images.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-css-shapes.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-svg-images.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-svg-shapes.html
PerformanceTests/Animometer/tests/master/canvas-stage.html
PerformanceTests/Animometer/tests/master/focus.html
PerformanceTests/Animometer/tests/master/image-data.html
PerformanceTests/Animometer/tests/master/multiply.html
PerformanceTests/Animometer/tests/master/particles.html
PerformanceTests/Animometer/tests/master/resources/canvas-tests.js
PerformanceTests/Animometer/tests/master/resources/focus.js
PerformanceTests/Animometer/tests/master/resources/particles.js
PerformanceTests/Animometer/tests/misc/canvas-electrons.html
PerformanceTests/Animometer/tests/misc/canvas-stars.html
PerformanceTests/Animometer/tests/misc/compositing-transforms.html
PerformanceTests/Animometer/tests/resources/algorithm.js [deleted file]
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/Animometer/tests/resources/math.js
PerformanceTests/Animometer/tests/resources/sampler.js [deleted file]
PerformanceTests/Animometer/tests/simple/simple-canvas-paths.html
PerformanceTests/Animometer/tests/simple/tiled-canvas-image.html
PerformanceTests/Animometer/tests/template/template-canvas.html
PerformanceTests/Animometer/tests/template/template-css.html
PerformanceTests/Animometer/tests/template/template-svg.html
PerformanceTests/Animometer/tests/text/layering-text.html
PerformanceTests/Animometer/tests/text/text-boxes.html
PerformanceTests/ChangeLog

index e2ad5f3..c065468 100644 (file)
@@ -2,11 +2,13 @@
 <html>
 <head>
     <title>Animometer - developer</title>
+    <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, user-scalable=no">
     <link rel="stylesheet" href="resources/runner/animometer.css">
     <link rel="stylesheet" href="resources/debug-runner/animometer.css">
     <script src="resources/strings.js"></script>
     <script src="resources/extensions.js"></script>
+    <script src="resources/statistics.js"></script>
 
     <script src="resources/runner/tests.js"></script>
     <script src="resources/debug-runner/tests.js" charset="utf-8"></script>
@@ -26,6 +28,7 @@
                 <div id="suites">
                     <h2>Suites:</h2>
                     <ul class="tree"></ul>
+                    <div><span id="drop-target">Drop results here</span></div>
                 </div>
                 <div id="options">
                     <h2>Options:</h2>
             <nav>
                 <form name="graph-type">
                     <ul>
-                        <li><label><input type="radio" name="graph-type" value="time" checked> Time graph</label></li>
-                        <li><label><input type="radio" name="graph-type" value="complexity"> Complexity graph</label></li>
+                        <li><label><input type="radio" name="graph-type" value="time"> Time graph</label></li>
+                        <li><label><input type="radio" name="graph-type" value="complexity" checked> Complexity graph</label></li>
                     </ul>
                 </form>
                 <form name="time-graph-options">
                 <form name="complexity-graph-options">
                     <ul class="series">
                         <li><label><input type="checkbox" name="series-raw" checked> Series raw</label></li>
-                        <li><label><input type="checkbox" name="series-average" checked> Series average</label></li>
+                        <li><label><input type="checkbox" name="series-average"> Series average</label></li>
                     </ul>
                     <ul>
-                        <li><label><input type="checkbox" name="regression-time-score" checked> Ramp regression score</label>
-                        <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label>
-                        <li><label><input type="checkbox" name="complexity-regression-aggregate-average" checked> Regression, series average</label>
+                        <li><label><input type="checkbox" name="regression-time-score"> Controller score</label></li>
+                        <li><label><input type="checkbox" name="bootstrap-score" checked> Bootstrap score and histogram</label></li>
+                        <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label><span id="complexity-regression-aggregate-raw"></span></li>
+                        <li><label><input type="checkbox" name="complexity-regression-aggregate-average"> Regression, series average</label><span id="complexity-regression-aggregate-average"></span></li>
                     </ul>
                 </form>
             </nav>
index 13fd05d..9f7370a 100644 (file)
@@ -1,14 +1,16 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <title>Animometer</title>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, user-scalable=no">
     <link rel="stylesheet" href="resources/runner/animometer.css">
     <script src="resources/strings.js"></script>
     <script src="resources/extensions.js"></script>
+    <script src="resources/statistics.js"></script>
 
     <script src="resources/runner/tests.js"></script>
-    <script src="resources/runner/animometer.js" charset="utf-8"></script>
+    <script src="resources/runner/animometer.js"></script>
 
     <script src="resources/runner/benchmark-runner.js"></script>
 </head>
index d847ec8..e6e7bb6 100644 (file)
@@ -171,6 +171,23 @@ label.tree-label {
     padding: 0 0 0 1em;
 }
 
+#suites > div {
+    margin-top: 3em;
+}
+
+#drop-target {
+    font-size: 1em;
+    border-radius: 10px;
+    padding: .5em 2em;
+    border: 2px solid rgb(235, 235, 235);
+    color: rgb(235, 235, 235);
+}
+
+#drop-target:hover {
+    background-color: rgba(255, 255, 255, .1);
+    cursor: pointer;
+}
+
 #options ul {
     margin: 0;
     padding: 0;
@@ -306,6 +323,26 @@ label.tree-label {
     padding-left: .25em;
 }
 
+#results-data .left {
+    text-align: left;
+}
+
+#results-data .right {
+    text-align: right;
+}
+
+#results-data .pad-left {
+    padding-left: 1em;
+}
+
+#results-data .pad-right {
+    padding-right: .25em;
+}
+
+#results-data .small {
+    font-size: .8em;
+}
+
 #results-tables td.noisy-results {
     color: rgb(255, 104, 104);
 }
@@ -357,7 +394,7 @@ label.tree-label {
     top: 1.5em;
     right: 0;
     font-size: .7em;
-    width: 23em;
+    width: 28em;
 }
 
 #test-graph nav ul {
@@ -520,11 +557,41 @@ label.tree-label {
     fill: rgba(253, 253, 253, .05);
 }
 
-#complexity-graph .series.average circle {
+#complexity-graph .raw.series line {
+    stroke: hsl(30, 96%, 56%);
+    stroke-width: 1px;
+}
+
+#complexity-graph .raw.regression line {
+    stroke: rgba(30, 96%, 86%, .6);
+}
+
+#complexity-graph .raw.regression polygon {
+    stroke: rgba(30, 96%, 86%, .05);
+}
+
+#complexity-graph .average.series circle {
     fill: hsl(170, 96%, 56%);
 }
 
-#complexity-graph .series.average line {
+#complexity-graph .average.series line {
     stroke: hsla(170, 96%, 56%, .2);
     stroke-width: 2px;
 }
+
+#complexity-graph .bootstrap .bar {
+    fill: hsla(260, 56%, 66%, .4);
+}
+
+#complexity-graph .bootstrap .median line {
+    stroke: hsla(300, 56%, 66%, .8);
+    stroke-width: 2px;
+}
+
+#complexity-graph .bootstrap .median circle {
+    fill: hsla(300, 56%, 66%, .8);
+}
+
+#complexity-graph .bootstrap .median polygon {
+    fill: hsla(300, 56%, 66%, .05);
+}
index e7635e2..4033379 100644 (file)
@@ -25,42 +25,17 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
         ResultsTable.call(this, element, headers);
     }, {
 
-    _addGraphButton: function(td, testName, testResults)
+    _addGraphButton: function(td, testName, testResult, testData)
     {
-        var data = testResults[Strings.json.samples];
-        if (!data)
-            return;
-
         var button = Utilities.createElement("button", { class: "small-button" }, td);
+        button.textContent = Strings.text.graph + "...";
+        button.testName = testName;
+        button.testResult = testResult;
+        button.testData = testData;
 
-        button.addEventListener("click", function() {
-            var graphData = {
-                axes: [Strings.text.complexity, Strings.text.frameRate],
-                samples: data,
-                complexityAverageSamples: testResults[Strings.json.complexityAverageSamples],
-                averages: {},
-                marks: testResults[Strings.json.marks]
-            };
-            [Strings.json.experiments.complexity, Strings.json.experiments.frameRate].forEach(function(experiment) {
-                if (experiment in testResults)
-                    graphData.averages[experiment] = testResults[experiment];
-            });
-
-            [
-                Strings.json.score,
-                Strings.json.regressions.timeRegressions,
-                Strings.json.regressions.complexityRegression,
-                Strings.json.regressions.complexityAverageRegression,
-                Strings.json.targetFrameLength
-            ].forEach(function(key) {
-                if (testResults[key])
-                    graphData[key] = testResults[key];
-            });
-
-            benchmarkController.showTestGraph(testName, graphData);
+        button.addEventListener("click", function(e) {
+            benchmarkController.showTestGraph(e.target.testName, e.target.testResult, e.target.testData);
         });
-
-        button.textContent = Strings.text.graph + "...";
     },
 
     _isNoisyMeasurement: function(jsonExperiment, data, measurement, options)
@@ -71,19 +46,19 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
         if (measurement == Strings.json.measurements.percent)
             return data[Strings.json.measurements.percent] >= percentThreshold;
 
-        if (jsonExperiment == Strings.json.experiments.frameRate && measurement == Strings.json.measurements.average)
+        if (jsonExperiment == Strings.json.frameLength && measurement == Strings.json.measurements.average)
             return Math.abs(data[Strings.json.measurements.average] - options["frame-rate"]) >= averageThreshold;
 
         return false;
     },
 
-    _addTest: function(testName, testResults, options)
+    _addTest: function(testName, testResult, options, testData)
     {
         var row = Utilities.createElement("tr", {}, this.element);
 
         var isNoisy = false;
-        [Strings.json.experiments.complexity, Strings.json.experiments.frameRate].forEach(function (experiment) {
-            var data = testResults[experiment];
+        [Strings.json.complexity, Strings.json.frameLength].forEach(function (experiment) {
+            var data = testResult[experiment];
             for (var measurement in data) {
                 if (this._isNoisyMeasurement(experiment, data, measurement, options))
                     isNoisy = true;
@@ -94,7 +69,7 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
             var className = "";
             if (header.className) {
                 if (typeof header.className == "function")
-                    className = header.className(testResults, options);
+                    className = header.className(testResult, options);
                 else
                     className = header.className;
             }
@@ -109,16 +84,16 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
 
             var td = Utilities.createElement("td", { class: className }, row);
             if (header.title == Strings.text.graph) {
-                this._addGraphButton(td, testName, testResults);
+                this._addGraphButton(td, testName, testResult, testData);
             } else if (!("text" in header)) {
-                td.textContent = testResults[header.title];
+                td.textContent = testResult[header.title];
             } else if (typeof header.text == "string") {
-                var data = testResults[header.text];
+                var data = testResult[header.text];
                 if (typeof data == "number")
                     data = data.toFixed(2);
                 td.textContent = data;
             } else {
-                td.textContent = header.text(testResults, testName);
+                td.textContent = header.text(testResult, testName);
             }
         }, this);
     }
@@ -134,15 +109,16 @@ Utilities.extendObject(window.benchmarkRunnerClient, {
         this.options = options;
     },
 
-    willStartFirstIteration: function ()
+    willStartFirstIteration: function()
     {
-        this.results = new ResultsDashboard();
+        this.results = new ResultsDashboard(this.options);
         this.progressBar = new ProgressBar(document.getElementById("progress-completed"), this.testsCount);
     },
 
-    didRunTest: function()
+    didRunTest: function(testData)
     {
         this.progressBar.incrementRange();
+        this.results.calculateScore(testData);
     }
 });
 
@@ -152,10 +128,10 @@ Utilities.extendObject(window.sectionsManager, {
         document.querySelector("#" + sectionIdentifier + " h1").textContent = title;
     },
 
-    populateTable: function(tableIdentifier, headers, data)
+    populateTable: function(tableIdentifier, headers, dashboard)
     {
         var table = new DeveloperResultsTable(document.getElementById(tableIdentifier), headers);
-        table.showIterations(data, benchmarkRunnerClient.options);
+        table.showIterations(dashboard);
     }
 });
 
@@ -431,14 +407,13 @@ window.suitesManager =
         return suites;
     },
 
-    updateLocalStorageFromJSON: function(iterationResults)
+    updateLocalStorageFromJSON: function(results)
     {
-        for (var suiteName in iterationResults[Strings.json.results.suites]) {
-            var suiteResults = iterationResults[Strings.json.results.suites][suiteName];
-
-            for (var testName in suiteResults[Strings.json.results.tests]) {
-                var testResults = suiteResults[Strings.json.results.tests][testName];
-                var data = testResults[Strings.json.experiments.complexity];
+        for (var suiteName in results[Strings.json.results.tests]) {
+            var suiteResults = results[Strings.json.results.tests][suiteName];
+            for (var testName in suiteResults) {
+                var testResults = suiteResults[testName];
+                var data = testResults[Strings.json.controller];
                 var complexity = Math.round(data[Strings.json.measurements.average]);
 
                 var value = { checked: true, complexity: complexity };
@@ -462,6 +437,36 @@ Utilities.extendObject(window.benchmarkController, {
         suitesManager.updateUIFromLocalStorage();
         suitesManager.updateDisplay();
         suitesManager.updateEditsElementsState();
+
+        var dropTarget = document.getElementById("drop-target");
+        function stopEvent(e) {
+            e.stopPropagation();
+            e.preventDefault();
+        }
+        dropTarget.addEventListener("dragenter", stopEvent, false);
+        dropTarget.addEventListener("dragover", stopEvent, false);
+        dropTarget.addEventListener("dragleave", stopEvent, false);
+        dropTarget.addEventListener("drop", function (e) {
+            e.stopPropagation();
+            e.preventDefault();
+
+            if (!e.dataTransfer.files.length)
+                return;
+
+            var file = e.dataTransfer.files[0];
+
+            var reader = new FileReader();
+            reader.filename = file.name;
+            reader.onload = function(e) {
+                var run = JSON.parse(e.target.result);
+                benchmarkRunnerClient.results = new ResultsDashboard(run.options, run.data);
+                benchmarkController.showResults();
+            };
+
+            reader.readAsText(file);
+            document.title = "File: " + reader.filename;
+        }, false);
+
     },
 
     onBenchmarkOptionsChanged: function(event)
@@ -479,12 +484,6 @@ Utilities.extendObject(window.benchmarkController, {
     {
         var options = optionsManager.updateLocalStorageFromUI();
         var suites = suitesManager.updateLocalStorageFromUI();
-        if (options["adjustment"] == "ramp") {
-            Headers.details[2].disabled = true;
-        } else {
-            Headers.details[3].disabled = true;
-            Headers.details[4].disabled = true;
-        }
         this._startBenchmark(suites, options, "running-test");
     },
 
@@ -495,32 +494,39 @@ Utilities.extendObject(window.benchmarkController, {
             this.addedKeyEvent = true;
         }
 
-        sectionsManager.setSectionScore("results", benchmarkRunnerClient.results.score.toFixed(2));
-        var data = benchmarkRunnerClient.results.data[Strings.json.results.iterations];
-        sectionsManager.populateTable("results-header", Headers.testName, data);
-        sectionsManager.populateTable("results-score", Headers.score, data);
-        sectionsManager.populateTable("results-data", Headers.details, data);
+        var dashboard = benchmarkRunnerClient.results;
+        if (dashboard.options["adjustment"] == "ramp") {
+            Headers.details[3].disabled = true;
+        } else {
+            Headers.details[1].disabled = true;
+            Headers.details[4].disabled = true;
+        }
+
+        sectionsManager.setSectionScore("results", dashboard.score.toFixed(2));
+        sectionsManager.populateTable("results-header", Headers.testName, dashboard);
+        sectionsManager.populateTable("results-score", Headers.score, dashboard);
+        sectionsManager.populateTable("results-data", Headers.details, dashboard);
         sectionsManager.showSection("results", true);
 
-        suitesManager.updateLocalStorageFromJSON(data[0]);
+        suitesManager.updateLocalStorageFromJSON(dashboard.results[0]);
     },
 
     showJSONResults: function()
     {
-        document.querySelector("#results-json textarea").textContent = JSON.stringify(benchmarkRunnerClient.results.data, function(key, value) {
-            if (typeof value == "number")
-                return value.toFixed(2);
-            return value;
-        });
+        var output = {
+            options: benchmarkRunnerClient.results.options,
+            data: benchmarkRunnerClient.results.data
+        };
+        var textarea = document.querySelector("#results-json textarea").textContent = JSON.stringify(output, null, 1);
         document.querySelector("#results-json button").remove();
         document.querySelector("#results-json div").classList.remove("hidden");
     },
 
-    showTestGraph: function(testName, graphData)
+    showTestGraph: function(testName, testResult, testData)
     {
         sectionsManager.setSectionHeader("test-graph", testName);
         sectionsManager.showSection("test-graph", true);
-        this.updateGraphData(graphData);
+        this.updateGraphData(testResult, testData, benchmarkRunnerClient.results.options);
     }
 });
 
index 7049463..0aabcc9 100644 (file)
@@ -1,44 +1,56 @@
 Utilities.extendObject(window.benchmarkController, {
     layoutCounter: 0,
 
-    updateGraphData: function(graphData)
+    updateGraphData: function(testResult, testData, options)
     {
         var element = document.getElementById("test-graph-data");
         element.innerHTML = "";
-        element.graphData = graphData;
+        element.testResult = testResult;
+        element.testData = testData;
+        element.options = options;
         document.querySelector("hr").style.width = this.layoutCounter++ + "px";
 
         var margins = new Insets(30, 30, 30, 40);
         var size = Point.elementClientSize(element).subtract(margins.size);
 
-        this.createTimeGraph(graphData, margins, size);
+        this.createTimeGraph(testResult, testData[Strings.json.samples][Strings.json.controller], testData[Strings.json.marks], testData[Strings.json.controller], options, margins, size);
         this.onTimeGraphOptionsChanged();
 
-        var hasComplexityRegression = !!graphData.complexityRegression;
-        this._showOrHideNodes(hasComplexityRegression, "form[name=graph-type]");
-        if (hasComplexityRegression) {
+        var hasComplexitySamples = !!testData[Strings.json.samples][Strings.json.complexity];
+        this._showOrHideNodes(hasComplexitySamples, "form[name=graph-type]");
+        if (hasComplexitySamples) {
             document.forms["graph-type"].elements["type"] = "complexity";
-            this.createComplexityGraph(graphData, margins, size);
+            this.createComplexityGraph(testResult, testData[Strings.json.controller], testData[Strings.json.samples], options, margins, size);
             this.onComplexityGraphOptionsChanged();
         }
 
         this.onGraphTypeChanged();
     },
 
-    _addRegressionLine: function(parent, xScale, yScale, points, stdev, isAlongYAxis)
+    _addRegressionLine: function(parent, xScale, yScale, points, range, isAlongYAxis)
     {
         var polygon = [];
         var line = []
-        var xStdev = isAlongYAxis ? stdev : 0;
-        var yStdev = isAlongYAxis ? 0 : stdev;
+        var xRange = isAlongYAxis ? range : 0;
+        var yRange = isAlongYAxis ? 0 : range;
         for (var i = 0; i < points.length; ++i) {
             var point = points[i];
-            polygon.push(xScale(point[0] + xStdev), yScale(point[1] + yStdev));
+            var x;
+            if (xRange instanceof Array)
+                x = xRange[0];
+            else
+                x = point[0] + xRange;
+            polygon.push(xScale(x), yScale(point[1] + yRange));
             line.push(xScale(point[0]), yScale(point[1]));
         }
         for (var i = points.length - 1; i >= 0; --i) {
             var point = points[i];
-            polygon.push(xScale(point[0] - xStdev), yScale(point[1] - yStdev));
+            var x;
+            if (xRange instanceof Array)
+                x = xRange[1];
+            else
+                x = point[0] - xRange;
+            polygon.push(xScale(x), yScale(point[1] - yRange));
         }
         parent.append("polygon")
             .attr("points", polygon.join(","));
@@ -52,14 +64,14 @@ Utilities.extendObject(window.benchmarkController, {
     _addRegression: function(data, svg, xScale, yScale)
     {
         svg.append("circle")
-            .attr("cx", xScale(data.segment2[1][0]))
-            .attr("cy", yScale(data.segment2[1][1]))
+            .attr("cx", xScale(data.segment1[1][0]))
+            .attr("cy", yScale(data.segment1[1][1]))
             .attr("r", 5);
         this._addRegressionLine(svg, xScale, yScale, data.segment1, data.stdev);
         this._addRegressionLine(svg, xScale, yScale, data.segment2, data.stdev);
     },
 
-    createComplexityGraph: function(graphData, margins, size)
+    createComplexityGraph: function(result, timeRegressions, data, options, margins, size)
     {
         var svg = d3.select("#test-graph-data").append("svg")
             .attr("id", "complexity-graph")
@@ -69,34 +81,36 @@ Utilities.extendObject(window.benchmarkController, {
             .append("g")
                 .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
 
+        var timeSamples = data[Strings.json.controller];
+
         var xMin = 100000, xMax = 0;
-        if (graphData.timeRegressions) {
-            graphData.timeRegressions.forEach(function(regression) {
+        if (timeRegressions) {
+            timeRegressions.forEach(function(regression) {
                 for (var i = regression.startIndex; i <= regression.endIndex; ++i) {
-                    xMin = Math.min(xMin, graphData.samples[i].complexity);
-                    xMax = Math.max(xMax, graphData.samples[i].complexity);
+                    xMin = Math.min(xMin, timeSamples[i].complexity);
+                    xMax = Math.max(xMax, timeSamples[i].complexity);
                 }
             });
         } else {
-            xMin = d3.min(graphData.samples, function(s) { return s.complexity; });
-            xMax = d3.max(graphData.samples, function(s) { return s.complexity; });
+            xMin = d3.min(timeSamples, function(s) { return s.complexity; });
+            xMax = d3.max(timeSamples, function(s) { return s.complexity; });
         }
 
         var xScale = d3.scale.linear()
             .range([0, size.width])
             .domain([xMin, xMax]);
         var yScale = d3.scale.linear()
-                .range([size.height, 0])
-                .domain([1000/20, 1000/60]);
+            .range([size.height, 0])
+            .domain([1000/20, 1000/60]);
 
         var xAxis = d3.svg.axis()
-                .scale(xScale)
-                .orient("bottom");
+            .scale(xScale)
+            .orient("bottom");
         var yAxis = d3.svg.axis()
-                .scale(yScale)
-                .tickValues([1000/20, 1000/25, 1000/30, 1000/35, 1000/40, 1000/45, 1000/50, 1000/55, 1000/60])
-                .tickFormat(function(d) { return (1000 / d).toFixed(0); })
-                .orient("left");
+            .scale(yScale)
+            .tickValues([1000/20, 1000/25, 1000/30, 1000/35, 1000/40, 1000/45, 1000/50, 1000/55, 1000/60])
+            .tickFormat(function(d) { return (1000 / d).toFixed(0); })
+            .orient("left");
 
         // x-axis
         svg.append("g")
@@ -109,38 +123,64 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("class", "y axis")
             .call(yAxis);
 
-        // time-based regression
+        // time result
         var mean = svg.append("g")
             .attr("class", "mean complexity");
-        var complexity = graphData.averages[Strings.json.experiments.complexity];
-        this._addRegressionLine(mean, xScale, yScale, [[complexity.average, yScale.domain()[0]], [complexity.average, yScale.domain()[1]]], complexity.stdev, true);
+        var timeResult = result[Strings.json.controller];
+        var yMin = yScale.domain()[0], yMax = yScale.domain()[1];
+        this._addRegressionLine(mean, xScale, yScale, [[timeResult.average, yMin], [timeResult.average, yMax]], timeResult.stdev, true);
 
         // regression
-        this._addRegression(graphData.complexityRegression, svg.append("g").attr("class", "regression raw"), xScale, yScale);
-        this._addRegression(graphData.complexityAverageRegression, svg.append("g").attr("class", "regression average"), xScale, yScale);
-
-        var svgGroup = svg.append("g")
-            .attr("class", "series raw");
-        var seriesCounter = 0;
-        graphData.timeRegressions.forEach(function(regression, i) {
-            seriesCounter++;
-            var group = svgGroup.append("g")
-                .attr("class", "series-" + seriesCounter)
-                .attr("fill", "hsl(" + (i / graphData.timeRegressions.length * 360).toFixed(0) + ", 96%, 56%)");
-            group.selectAll("circle")
-                .data(graphData.samples)
-                .enter()
-                .append("circle")
-                .filter(function(d, i) { return i >= regression.startIndex && i <= regression.endIndex; })
-                .attr("cx", function(d) { return xScale(d.complexity); })
-                .attr("cy", function(d) { return yScale(d.frameLength); })
-                .attr("r", 2);
-        });
+        this._addRegression(result[Strings.json.complexity], svg.append("g").attr("class", "regression raw"), xScale, yScale);
+        this._addRegression(result[Strings.json.complexityAverage], svg.append("g").attr("class", "regression average"), xScale, yScale);
+
+        var bootstrapResult = result[Strings.json.complexity][Strings.json.bootstrap];
+        if (bootstrapResult) {
+            var histogram = d3.layout.histogram()
+                .bins(xScale.ticks(100))(bootstrapResult.data);
+            var yBootstrapScale = d3.scale.linear()
+                .range([size.height/2, 0])
+                .domain([0, d3.max(histogram, function(d) { return d.y; })]);
+            group = svg.append("g").attr("class", "bootstrap");
+            var bar = group.selectAll(".bar")
+                .data(histogram)
+                .enter().append("g")
+                    .attr("class", "bar")
+                    .attr("transform", function(d) { return "translate(" + xScale(d.x) + "," + yBootstrapScale(d.y) + ")"; });
+            bar.append("rect")
+                .attr("x", 1)
+                .attr("y", size.height/2)
+                .attr("width", xScale(histogram[1].x) - xScale(histogram[0].x) - 1)
+                .attr("height", function(d) { return size.height/2 - yBootstrapScale(d.y); });
+            group = group.append("g").attr("class", "median");
+            this._addRegressionLine(group, xScale, yScale, [[bootstrapResult.median, yMin], [bootstrapResult.median, yMax]], [bootstrapResult.confidenceLow, bootstrapResult.confidenceHigh], true);
+            group.append("circle")
+                .attr("cx", xScale(bootstrapResult.median))
+                .attr("cy", yScale(1000/60))
+                .attr("r", 5);
+        }
+
+        // series
+        group = svg.append("g")
+            .attr("class", "series raw")
+            .selectAll("line")
+                .data(data[Strings.json.complexity])
+                .enter();
+        group.append("line")
+            .attr("x1", function(d) { return xScale(d.complexity) - 3; })
+            .attr("x2", function(d) { return xScale(d.complexity) + 3; })
+            .attr("y1", function(d) { return yScale(d.frameLength) - 3; })
+            .attr("y2", function(d) { return yScale(d.frameLength) + 3; });
+        group.append("line")
+            .attr("x1", function(d) { return xScale(d.complexity) - 3; })
+            .attr("x2", function(d) { return xScale(d.complexity) + 3; })
+            .attr("y1", function(d) { return yScale(d.frameLength) + 3; })
+            .attr("y2", function(d) { return yScale(d.frameLength) - 3; });
 
         group = svg.append("g")
             .attr("class", "series average")
             .selectAll("circle")
-                .data(graphData.complexityAverageSamples)
+                .data(data[Strings.json.complexityAverage])
                 .enter();
         group.append("circle")
             .attr("cx", function(d) { return xScale(d.complexity); })
@@ -162,7 +202,7 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("y2", yScale(yAxis.scale().domain()[1]));
         cursorGroup.append("line")
             .attr("class", "y")
-            .attr("x1", xScale(0) - 10)
+            .attr("x1", xScale(xAxis.scale().domain()[0]) - 10)
             .attr("x2", xScale(xAxis.scale().domain()[1]))
             .attr("y1", 0)
             .attr("y2", 0)
@@ -174,7 +214,7 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("text-anchor", "middle");
         cursorGroup.append("text")
             .attr("class", "label y")
-            .attr("x", xScale(0) - 15)
+            .attr("x", xScale(xAxis.scale().domain()[0]) - 15)
             .attr("y", 0)
             .attr("baseline-shift", "-30%")
             .attr("text-anchor", "end");
@@ -208,7 +248,7 @@ Utilities.extendObject(window.benchmarkController, {
         });
     },
 
-    createTimeGraph: function(graphData, margins, size)
+    createTimeGraph: function(result, samples, marks, regressions, options, margins, size)
     {
         var svg = d3.select("#test-graph-data").append("svg")
             .attr("id", "time-graph")
@@ -217,21 +257,17 @@ Utilities.extendObject(window.benchmarkController, {
             .append("g")
                 .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
 
-        var axes = graphData.axes;
-        var targetFrameLength = graphData.targetFrameLength;
-
         // Axis scales
         var x = d3.scale.linear()
                 .range([0, size.width])
                 .domain([
-                    Math.min(d3.min(graphData.samples, function(s) { return s.time; }), 0),
-                    d3.max(graphData.samples, function(s) { return s.time; })]);
-        var complexityMax = d3.max(graphData.samples, function(s) { return s.complexity; });
-        if (graphData.timeRegressions) {
-            complexityMax = Math.max.apply(Math, graphData.timeRegressions.map(function(regression) {
-                return regression.maxComplexity || 0;
-            }));
-        }
+                    Math.min(d3.min(samples, function(s) { return s.time; }), 0),
+                    d3.max(samples, function(s) { return s.time; })]);
+        var complexityMax = d3.max(samples, function(s) {
+            if (s.time > 0)
+                return s.complexity;
+            return 0;
+        });
 
         var yLeft = d3.scale.linear()
                 .range([size.height, 0])
@@ -280,7 +316,7 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("fill", "#7ADD49")
                 .attr("dy", ".71em")
                 .style("text-anchor", "end")
-                .text(axes[0]);
+                .text(Strings.text.complexity);
 
         // yRight-axis
         svg.append("g")
@@ -295,13 +331,13 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("fill", "#FA4925")
                 .attr("dy", ".71em")
                 .style("text-anchor", "start")
-                .text(axes[1]);
+                .text(Strings.text.frameRate);
 
         // marks
-        var yMin = yLeft(0);
-        var yMax = yLeft(yAxisLeft.scale().domain()[1]);
-        for (var markName in graphData.marks) {
-            var mark = graphData.marks[markName];
+        var yMin = yRight(yAxisRight.scale().domain()[0]);
+        var yMax = yRight(yAxisRight.scale().domain()[1]);
+        for (var markName in marks) {
+            var mark = marks[markName];
             var xLocation = x(mark.time);
 
             var markerGroup = svg.append("g")
@@ -318,21 +354,22 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("y2", yMax);
         }
 
-        if (Strings.json.experiments.complexity in graphData.averages) {
-            var complexity = graphData.averages[Strings.json.experiments.complexity];
+        if (Strings.json.controller in result) {
+            var complexity = result[Strings.json.controller];
             var regression = svg.append("g")
                 .attr("class", "complexity mean");
-            this._addRegressionLine(regression, x, yLeft, [[graphData.samples[0].time, complexity.average], [graphData.samples[graphData.samples.length - 1].time, complexity.average]], complexity.stdev);
+            this._addRegressionLine(regression, x, yLeft, [[samples[0].time, complexity.average], [samples[samples.length - 1].time, complexity.average]], complexity.stdev);
         }
-        if (Strings.json.experiments.frameRate in graphData.averages) {
-            var frameRate = graphData.averages[Strings.json.experiments.frameRate];
+        if (Strings.json.frameLength in result) {
+            var frameLength = result[Strings.json.frameLength];
             var regression = svg.append("g")
                 .attr("class", "fps mean");
-            this._addRegressionLine(regression, x, yRight, [[graphData.samples[0].time, 1000/frameRate.average], [graphData.samples[graphData.samples.length - 1].time, 1000/frameRate.average]], frameRate.stdev);
+            this._addRegressionLine(regression, x, yRight, [[samples[0].time, 1000/frameLength.average], [samples[samples.length - 1].time, 1000/frameLength.average]], frameLength.stdev);
         }
 
         // right-target
-        if (targetFrameLength) {
+        if (options["adjustment"] == "adaptive") {
+            var targetFrameLength = 1000 / options["frame-rate"];
             svg.append("line")
                 .attr("x1", x(0))
                 .attr("x2", size.width)
@@ -350,8 +387,8 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("y2", yMin);
 
         // Data
-        var allData = graphData.samples;
-        var filteredData = graphData.samples.filter(function (sample) {
+        var allData = samples;
+        var filteredData = samples.filter(function (sample) {
             return "smoothedFrameLength" in sample;
         });
 
@@ -384,19 +421,23 @@ Utilities.extendObject(window.benchmarkController, {
         // regressions
         var regressionGroup = svg.append("g")
             .attr("id", "regressions");
-        if (graphData.timeRegressions) {
+        if (regressions) {
             var complexities = [];
-            graphData.timeRegressions.forEach(function (regression) {
-                regressionGroup.append("line")
-                    .attr("x1", x(regression.segment1[0][0]))
-                    .attr("x2", x(regression.segment1[1][0]))
-                    .attr("y1", yRight(regression.segment1[0][1]))
-                    .attr("y2", yRight(regression.segment1[1][1]));
-                regressionGroup.append("line")
-                    .attr("x1", x(regression.segment2[0][0]))
-                    .attr("x2", x(regression.segment2[1][0]))
-                    .attr("y1", yRight(regression.segment2[0][1]))
-                    .attr("y2", yRight(regression.segment2[1][1]));
+            regressions.forEach(function (regression) {
+                if (!isNaN(regression.segment1[0][1]) && !isNaN(regression.segment1[1][1])) {
+                    regressionGroup.append("line")
+                        .attr("x1", x(regression.segment1[0][0]))
+                        .attr("x2", x(regression.segment1[1][0]))
+                        .attr("y1", yRight(regression.segment1[0][1]))
+                        .attr("y2", yRight(regression.segment1[1][1]));
+                }
+                if (!isNaN(regression.segment2[0][1]) && !isNaN(regression.segment2[1][1])) {
+                    regressionGroup.append("line")
+                        .attr("x1", x(regression.segment2[0][0]))
+                        .attr("x2", x(regression.segment2[1][0]))
+                        .attr("y1", yRight(regression.segment2[0][1]))
+                        .attr("y2", yRight(regression.segment2[1][1]));
+                }
                 // inflection point
                 regressionGroup.append("circle")
                     .attr("cx", x(regression.segment1[1][0]))
@@ -503,6 +544,7 @@ Utilities.extendObject(window.benchmarkController, {
         benchmarkController._showOrHideNodes(form["series-raw"].checked, "#complexity-graph .series.raw");
         benchmarkController._showOrHideNodes(form["series-average"].checked, "#complexity-graph .series.average");
         benchmarkController._showOrHideNodes(form["regression-time-score"].checked, "#complexity-graph .mean.complexity");
+        benchmarkController._showOrHideNodes(form["bootstrap-score"].checked, "#complexity-graph .bootstrap");
         benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-raw"].checked, "#complexity-graph .regression.raw");
         benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-average"].checked, "#complexity-graph .regression.average");
     },
@@ -514,11 +556,12 @@ Utilities.extendObject(window.benchmarkController, {
         benchmarkController._showOrHideNodes(form["complexity"].checked, "#complexity");
         benchmarkController._showOrHideNodes(form["rawFPS"].checked, "#rawFPS");
         benchmarkController._showOrHideNodes(form["filteredFPS"].checked, "#filteredFPS");
+        benchmarkController._showOrHideNodes(form["regressions"].checked, "#regressions");
     },
 
     onGraphTypeChanged: function() {
         var form = document.forms["graph-type"].elements;
-        var graphData = document.getElementById("test-graph-data").graphData;
+        var testResult = document.getElementById("test-graph-data").testResult;
         var isTimeSelected = form["graph-type"].value == "time";
 
         benchmarkController._showOrHideNodes(isTimeSelected, "#time-graph");
@@ -528,9 +571,9 @@ Utilities.extendObject(window.benchmarkController, {
 
         var score, mean;
         if (isTimeSelected) {
-            score = graphData.score.toFixed(2);
+            score = testResult[Strings.json.score].toFixed(2);
 
-            var regression = graphData.averages.complexity;
+            var regression = testResult[Strings.json.controller];
             mean = [
                 "mean: ",
                 regression.average.toFixed(2),
@@ -546,18 +589,20 @@ Utilities.extendObject(window.benchmarkController, {
             }
             mean = mean.join("");
         } else {
-            score = [
-                "raw: ",
-                graphData.complexityRegression.complexity.toFixed(2),
-                ", average: ",
-                graphData.complexityAverageRegression.complexity.toFixed(2)].join("");
+            var complexityRegression = testResult[Strings.json.complexity];
+            var complexityAverageRegression = testResult[Strings.json.complexityAverage];
+
+            document.getElementById("complexity-regression-aggregate-raw").textContent = complexityRegression.complexity.toFixed(2) + ", ±" + complexityRegression.stdev.toFixed(2) + "ms";
+            document.getElementById("complexity-regression-aggregate-average").textContent = complexityAverageRegression.complexity.toFixed(2) + ", ±" + complexityAverageRegression.stdev.toFixed(2) + "ms";
 
+            var bootstrap = complexityRegression[Strings.json.bootstrap];
+            score = bootstrap.median.toFixed(2);
             mean = [
-                "raw: ±",
-                graphData.complexityRegression.stdev.toFixed(2),
-                "ms, average: ±",
-                graphData.complexityAverageRegression.stdev.toFixed(2),
-                "ms"].join("");
+                "95% CI: ",
+                bootstrap.confidenceLow.toFixed(2),
+                "",
+                bootstrap.confidenceHigh.toFixed(2)
+            ].join("");
         }
 
         sectionsManager.setSectionScore("test-graph", score, mean);
index 26e3d38..6f1c815 100644 (file)
@@ -1,15 +1,57 @@
+Utilities.extendObject(Strings.text, {
+    samples: "Samples",
+    complexity: "Time Complexity",
+    frameRate: "FPS",
+    confidenceInterval: "95% Confidence Interval",
+    mergedRawComplexity: "Raw Complexity",
+    graph: "Graph"
+});
+
+
 Utilities.extendObject(Headers, {
     details: [
         {
             title: Strings.text.graph
         },
         {
+            title: Strings.text.confidenceInterval,
+            children:
+            [
+                {
+                    text: function(data) {
+                        return data[Strings.json.complexity][Strings.json.bootstrap].confidenceLow.toFixed(2);
+                    },
+                    className: "right pad-left pad-right"
+                },
+                {
+                    text: function(data) {
+                        return " - " + data[Strings.json.complexity][Strings.json.bootstrap].confidenceHigh.toFixed(2);
+                    },
+                    className: "left"
+                },
+                {
+                    text: function(data) {
+                        var bootstrap = data[Strings.json.complexity][Strings.json.bootstrap];
+                        return (100 * (bootstrap.confidenceLow / bootstrap.median - 1)).toFixed(2) + "%";
+                    },
+                    className: "left pad-left small"
+                },
+                {
+                    text: function(data) {
+                        var bootstrap = data[Strings.json.complexity][Strings.json.bootstrap];
+                        return "+" + (100 * (bootstrap.confidenceHigh / bootstrap.median - 1)).toFixed(2) + "%";
+                    },
+                    className: "left pad-left small"
+                }
+            ]
+        },
+        {
             title: Strings.text.complexity,
             children:
             [
                 {
                     text: function(data) {
-                        return data[Strings.json.experiments.complexity][Strings.json.measurements.average].toFixed(2);
+                        return data[Strings.json.controller][Strings.json.measurements.average].toFixed(2);
                     },
                     className: "average"
                 },
@@ -17,14 +59,14 @@ Utilities.extendObject(Headers, {
                     text: function(data) {
                         return [
                             "± ",
-                            data[Strings.json.experiments.complexity][Strings.json.measurements.percent].toFixed(2),
+                            data[Strings.json.controller][Strings.json.measurements.percent].toFixed(2),
                             "%"
                         ].join("");
                     },
                     className: function(data) {
                         var className = "stdev";
 
-                        if (data[Strings.json.experiments.complexity][Strings.json.measurements.percent] >= 10)
+                        if (data[Strings.json.controller][Strings.json.measurements.percent] >= 10)
                             className += " noisy-results";
                         return className;
                     }
@@ -37,18 +79,18 @@ Utilities.extendObject(Headers, {
             [
                 {
                     text: function(data) {
-                        return data[Strings.json.experiments.frameRate][Strings.json.measurements.average].toFixed(2);
+                        return data[Strings.json.frameLength][Strings.json.measurements.average].toFixed(2);
                     },
                     className: function(data, options) {
                         var className = "average";
-                        if (Math.abs(data[Strings.json.experiments.frameRate][Strings.json.measurements.average] - options["frame-rate"]) >= 2)
+                        if (Math.abs(data[Strings.json.frameLength][Strings.json.measurements.average] - options["frame-rate"]) >= 2)
                             className += " noisy-results";
                         return className;
                     }
                 },
                 {
                     text: function(data) {
-                        var frameRateData = data[Strings.json.experiments.frameRate];
+                        var frameRateData = data[Strings.json.frameLength];
                         return [
                             "± ",
                             frameRateData[Strings.json.measurements.percent].toFixed(2),
@@ -58,7 +100,7 @@ Utilities.extendObject(Headers, {
                     className: function(data) {
                         var className = "stdev";
 
-                        if (data[Strings.json.experiments.frameRate][Strings.json.measurements.percent] >= 10)
+                        if (data[Strings.json.frameLength][Strings.json.measurements.percent] >= 10)
                             className += " noisy-results";
                         return className;
                     }
@@ -71,7 +113,7 @@ Utilities.extendObject(Headers, {
             [
                 {
                     text: function(data) {
-                        return data[Strings.json.regressions.complexityRegression][Strings.json.regressions.complexity].toFixed(2);
+                        return data[Strings.json.complexity][Strings.json.complexity].toFixed(2);
                     },
                     className: "average"
                 },
@@ -79,36 +121,14 @@ Utilities.extendObject(Headers, {
                     text: function(data) {
                         return [
                             "± ",
-                            data[Strings.json.regressions.complexityRegression][Strings.json.measurements.stdev].toFixed(2),
+                            data[Strings.json.complexity][Strings.json.measurements.stdev].toFixed(2),
                             "ms"
                         ].join("");
                     },
                     className: "stdev"
                 }
             ]
-        },
-        {
-            title: Strings.text.mergedAverageComplexity,
-            children:
-            [
-                {
-                    text: function(data) {
-                        return data[Strings.json.regressions.complexityAverageRegression][Strings.json.regressions.complexity].toFixed(2);
-                    },
-                    className: "average"
-                },
-                {
-                    text: function(data) {
-                        return [
-                            "± ",
-                            data[Strings.json.regressions.complexityAverageRegression][Strings.json.measurements.stdev].toFixed(2),
-                            "ms"
-                        ].join("");
-                    },
-                    className: "stdev"
-                }
-            ]
-        },
+        }
     ]
 })
 
index 11c6bff..6f653fe 100644 (file)
@@ -352,29 +352,136 @@ SimplePromise = Utilities.createClass(
     }
 });
 
-Statistics =
-{
-    sampleMean: function(numberOfSamples, sum)
+var Heap = Utilities.createClass(
+    function(maxSize, compare)
+    {
+        this._maxSize = maxSize;
+        this._compare = compare;
+        this._size = 0;
+        this._values = new Array(this._maxSize);
+    }, {
+
+    // This is a binary heap represented in an array. The root element is stored
+    // in the first element in the array. The root is followed by its two children.
+    // Then its four grandchildren and so on. So every level in the binary heap is
+    // doubled in the following level. Here is an example of the node indices and
+    // how they are related to their parents and children.
+    // ===========================================================================
+    //              0       1       2       3       4       5       6
+    // PARENT       -1      0       0       1       1       2       2
+    // LEFT         1       3       5       7       9       11      13
+    // RIGHT        2       4       6       8       10      12      14
+    // ===========================================================================
+    _parentIndex: function(i)
+    {
+        return i > 0 ? Math.floor((i - 1) / 2) : -1;
+    },
+
+    _leftIndex: function(i)
+    {
+        var leftIndex = i * 2 + 1;
+        return leftIndex < this._size ? leftIndex : -1;
+    },
+
+    _rightIndex: function(i)
+    {
+        var rightIndex = i * 2 + 2;
+        return rightIndex < this._size ? rightIndex : -1;
+    },
+
+    // Return the child index that may violate the heap property at index i.
+    _childIndex: function(i)
     {
-        if (numberOfSamples < 1)
-            return 0;
-        return sum / numberOfSamples;
+        var left = this._leftIndex(i);
+        var right = this._rightIndex(i);
+
+        if (left != -1 && right != -1)
+            return this._compare(this._values[left], this._values[right]) > 0 ? left : right;
+
+        return left != -1 ? left : right;
+    },
+
+    init: function()
+    {
+        this._size = 0;
+    },
+
+    top: function()
+    {
+        return this._size ? this._values[0] : NaN;
     },
 
-    // With sum and sum of squares, we can compute the sample standard deviation in O(1).
-    // See https://rniwa.com/2012-11-10/sample-standard-deviation-in-terms-of-sum-and-square-sum-of-samples/
-    unbiasedSampleStandardDeviation: function(numberOfSamples, sum, squareSum)
+    push: function(value)
     {
-        if (numberOfSamples < 2)
-            return 0;
-        return Math.sqrt((squareSum - sum * sum / numberOfSamples) / (numberOfSamples - 1));
+        if (this._size == this._maxSize) {
+            // If size is bounded and the new value can be a parent of the top()
+            // if the size were unbounded, just ignore the new value.
+            if (this._compare(value, this.top()) > 0)
+                return;
+            this.pop();
+        }
+        this._values[this._size++] = value;
+        this._bubble(this._size - 1);
     },
 
-    geometricMean: function(values)
+    pop: function()
     {
-        if (!values.length)
-            return 0;
-        var roots = values.map(function(value) { return  Math.pow(value, 1 / values.length); })
-        return roots.reduce(function(a, b) { return a * b; });
+        if (!this._size)
+            return NaN;
+
+        this._values[0] = this._values[--this._size];
+        this._sink(0);
+    },
+
+    _bubble: function(i)
+    {
+        // Fix the heap property at index i given that parent is the only node that
+        // may violate the heap property.
+        for (var pi = this._parentIndex(i); pi != -1; i = pi, pi = this._parentIndex(pi)) {
+            if (this._compare(this._values[pi], this._values[i]) > 0)
+                break;
+
+            this._values.swap(pi, i);
+        }
+    },
+
+    _sink: function(i)
+    {
+        // Fix the heap property at index i given that each of the left and the right
+        // sub-trees satisfies the heap property.
+        for (var ci = this._childIndex(i); ci != -1; i = ci, ci = this._childIndex(ci)) {
+            if (this._compare(this._values[i], this._values[ci]) > 0)
+                break;
+
+            this._values.swap(ci, i);
+        }
+    },
+
+    str: function()
+    {
+        var out = "Heap[" + this._size + "] = [";
+        for (var i = 0; i < this._size; ++i) {
+            out += this._values[i];
+            if (i < this._size - 1)
+                out += ", ";
+        }
+        return out + "]";
+    },
+
+    values: function(size) {
+        // Return the last "size" heap elements values.
+        var values = this._values.slice(0, this._size);
+        return values.sort(this._compare).slice(0, Math.min(size, this._size));
     }
-};
+});
+
+Utilities.extendObject(Heap, {
+    createMinHeap: function(maxSize)
+    {
+        return new Heap(maxSize, function(a, b) { return b - a; });
+    },
+
+    createMaxHeap: function(maxSize) {
+        return new Heap(maxSize, function(a, b) { return a - b; });
+    }
+});
index 06e43b4..3ac2dcc 100644 (file)
@@ -1,8 +1,13 @@
 ResultsDashboard = Utilities.createClass(
-    function()
+    function(options, testData)
     {
         this._iterationsSamplers = [];
-        this._processedData = undefined;
+        this._options = options;
+        this._results = null;
+        if (testData) {
+            this._iterationsSamplers = testData;
+            this._processData();
+        }
     }, {
 
     push: function(suitesSamplers)
@@ -12,53 +17,179 @@ ResultsDashboard = Utilities.createClass(
 
     _processData: function()
     {
-        var iterationsResults = [];
+        this._results = {};
+        this._results[Strings.json.results.iterations] = [];
+
         var iterationsScores = [];
+        this._iterationsSamplers.forEach(function(iteration, index) {
+            var testsScores = [];
+
+            var result = {};
+            this._results[Strings.json.results.iterations][index] = result;
+
+            var suitesResult = {};
+            result[Strings.json.results.tests] = suitesResult;
 
-        this._iterationsSamplers.forEach(function(iterationSamplers, index) {
-            var suitesResults = {};
-            var suitesScores = [];
+            for (var suiteName in iteration) {
+                var suiteData = iteration[suiteName];
 
-            for (var suiteName in iterationSamplers) {
-                var suite = suiteFromName(suiteName);
-                var suiteSamplerData = iterationSamplers[suiteName];
+                var suiteResult = {};
+                suitesResult[suiteName] = suiteResult;
 
-                var testsResults = {};
-                var testsScores = [];
+                for (var testName in suiteData) {
+                    if (!suiteData[testName][Strings.json.result])
+                        this.calculateScore(suiteData[testName]);
 
-                for (var testName in suiteSamplerData) {
-                    testsResults[testName] = suiteSamplerData[testName];
-                    testsScores.push(testsResults[testName][Strings.json.score]);
+                    suiteResult[testName] = suiteData[testName][Strings.json.result];
+                    delete suiteData[testName][Strings.json.result];
+
+                    testsScores.push(suiteResult[testName][Strings.json.score]);
                 }
+            }
+
+            result[Strings.json.score] = Statistics.geometricMean(testsScores);
+            iterationsScores.push(result[Strings.json.score]);
+        }, this);
 
-                suitesResults[suiteName] =  {};
-                suitesResults[suiteName][Strings.json.score] = Statistics.geometricMean(testsScores);
-                suitesResults[suiteName][Strings.json.results.tests] = testsResults;
-                suitesScores.push(suitesResults[suiteName][Strings.json.score]);
+        this._results[Strings.json.score] = Statistics.sampleMean(iterationsScores.length, iterationsScores.reduce(function(a, b) { return a + b; }));
+    },
+
+    calculateScore: function(data)
+    {
+        var result = {};
+        data[Strings.json.result] = result;
+        var samples = data[Strings.json.samples];
+
+        function findRegression(series) {
+            var minIndex = Math.round(.025 * series.length);
+            var maxIndex = Math.round(.975 * (series.length - 1));
+            var minComplexity = series[minIndex].complexity;
+            var maxComplexity = series[maxIndex].complexity;
+            if (Math.abs(maxComplexity - minComplexity) < 20 && maxIndex - minIndex < 20) {
+                minIndex = 0;
+                maxIndex = series.length - 1;
+                minComplexity = series[minIndex].complexity;
+                maxComplexity = series[maxIndex].complexity;
             }
 
-            iterationsResults[index] = {};
-            iterationsResults[index][Strings.json.score] = Statistics.geometricMean(suitesScores);
-            iterationsResults[index][Strings.json.results.suites] = suitesResults;
-            iterationsScores.push(iterationsResults[index][Strings.json.score]);
+            return {
+                minComplexity: minComplexity,
+                maxComplexity: maxComplexity,
+                samples: series.slice(minIndex, maxIndex + 1),
+                regression: new Regression(
+                    series,
+                    function (datum, i) { return datum[i].complexity; },
+                    function (datum, i) { return datum[i].frameLength; },
+                    minIndex, maxIndex)
+            };
+        }
+
+        var complexitySamples;
+        [Strings.json.complexity, Strings.json.complexityAverage].forEach(function(seriesName) {
+            if (!(seriesName in samples))
+                return;
+
+            var regression = {};
+            result[seriesName] = regression;
+            var regressionResult = findRegression(samples[seriesName]);
+            if (seriesName == Strings.json.complexity)
+                complexitySamples = regressionResult.samples;
+            var calculation = regressionResult.regression;
+            regression[Strings.json.regressions.segment1] = [
+                [regressionResult.minComplexity, calculation.s1 + calculation.t1 * regressionResult.minComplexity],
+                [calculation.complexity, calculation.s1 + calculation.t1 * calculation.complexity]
+            ];
+            regression[Strings.json.regressions.segment2] = [
+                [calculation.complexity, calculation.s2 + calculation.t2 * calculation.complexity],
+                [regressionResult.maxComplexity, calculation.s2 + calculation.t2 * regressionResult.maxComplexity]
+            ];
+            regression[Strings.json.complexity] = calculation.complexity;
+            regression[Strings.json.measurements.stdev] = Math.sqrt(calculation.error / samples[seriesName].length);
         });
 
-        this._processedData = {};
-        this._processedData[Strings.json.score] = Statistics.sampleMean(iterationsScores.length, iterationsScores.reduce(function(a, b) { return a * b; }));
-        this._processedData[Strings.json.results.iterations] = iterationsResults;
+        if (this._options["adjustment"] == "ramp") {
+            var timeComplexity = new Experiment;
+            data[Strings.json.controller].forEach(function(regression) {
+                timeComplexity.sample(regression[Strings.json.complexity]);
+            });
+
+            var experimentResult = {};
+            result[Strings.json.controller] = experimentResult;
+            experimentResult[Strings.json.score] = timeComplexity.mean();
+            experimentResult[Strings.json.measurements.average] = timeComplexity.mean();
+            experimentResult[Strings.json.measurements.stdev] = timeComplexity.standardDeviation();
+            experimentResult[Strings.json.measurements.percent] = timeComplexity.percentage();
+
+            result[Strings.json.complexity][Strings.json.bootstrap] = Regression.bootstrap(complexitySamples, 2500, function(resample) {
+                    resample.sort(function(a, b) {
+                        return a.complexity - b.complexity;
+                    });
+
+                    var regressionResult = findRegression(resample);
+                    return regressionResult.regression.complexity;
+                }, .95);
+            result[Strings.json.score] = result[Strings.json.complexity][Strings.json.bootstrap].median;
+
+        } else {
+            var marks = data[Strings.json.marks];
+            var samplingStartIndex = 0, samplingEndIndex = -1;
+            if (Strings.json.samplingStartTimeOffset in marks)
+                samplingStartIndex = marks[Strings.json.samplingStartTimeOffset].index;
+            if (Strings.json.samplingEndTimeOffset in marks)
+                samplingEndIndex = marks[Strings.json.samplingEndTimeOffset].index;
+
+            var averageComplexity = new Experiment;
+            var averageFrameLength = new Experiment;
+            samples[Strings.json.controller].forEach(function (sample, i) {
+                if (i >= samplingStartIndex && (samplingEndIndex == -1 || i < samplingEndIndex)) {
+                    averageComplexity.sample(sample.complexity);
+                    if (sample.smoothedFrameLength && sample.smoothedFrameLength != -1)
+                        averageFrameLength.sample(sample.smoothedFrameLength);
+                }
+            });
+
+            var experimentResult = {};
+            result[Strings.json.controller] = experimentResult;
+            experimentResult[Strings.json.measurements.average] = averageComplexity.mean();
+            experimentResult[Strings.json.measurements.concern] = averageComplexity.concern(Experiment.defaults.CONCERN);
+            experimentResult[Strings.json.measurements.stdev] = averageComplexity.standardDeviation();
+            experimentResult[Strings.json.measurements.percent] = averageComplexity.percentage();
+
+            experimentResult = {};
+            result[Strings.json.frameLength] = experimentResult;
+            experimentResult[Strings.json.measurements.average] = 1000 / averageFrameLength.mean();
+            experimentResult[Strings.json.measurements.concern] = averageFrameLength.concern(Experiment.defaults.CONCERN);
+            experimentResult[Strings.json.measurements.stdev] = averageFrameLength.standardDeviation();
+            experimentResult[Strings.json.measurements.percent] = averageFrameLength.percentage();
+
+            result[Strings.json.score] = averageComplexity.score(Experiment.defaults.CONCERN);
+        }
     },
 
     get data()
     {
-        if (this._processedData)
-            return this._processedData;
+        return this._iterationsSamplers;
+    },
+
+    get results()
+    {
+        if (this._results)
+            return this._results[Strings.json.results.iterations];
         this._processData();
-        return this._processedData;
+        return this._results[Strings.json.results.iterations];
+    },
+
+    get options()
+    {
+        return this._options;
     },
 
     get score()
     {
-        return this.data[Strings.json.score];
+        if (this._results)
+            return this._results[Strings.json.score];
+        this._processData();
+        return this._results[Strings.json.score];
     }
 });
 
@@ -133,29 +264,26 @@ ResultsTable = Utilities.createClass(
         }, this);
     },
 
-    _addSuite: function(suiteName, suiteResults, options)
+    _addIteration: function(iterationResult, iterationData, options)
     {
-        for (var testName in suiteResults[Strings.json.results.tests]) {
-            var testResults = suiteResults[Strings.json.results.tests][testName];
-            this._addTest(testName, testResults, options);
-        }
-    },
-
-    _addIteration: function(iterationResult, options)
-    {
-        for (var suiteName in iterationResult[Strings.json.results.suites]) {
+        var testsResults = iterationResult[Strings.json.results.tests];
+        for (var suiteName in testsResults) {
             this._addEmptyRow();
-            this._addSuite(suiteName, iterationResult[Strings.json.results.suites][suiteName], options);
+            var suiteResult = testsResults[suiteName];
+            var suiteData = iterationData[suiteName];
+            for (var testName in suiteResult)
+                this._addTest(testName, suiteResult[testName], options, suiteData[testName]);
         }
     },
 
-    showIterations: function(iterationsResults, options)
+    showIterations: function(dashboard)
     {
         this.clear();
         this._addHeader();
 
-        iterationsResults.forEach(function(iterationResult) {
-            this._addIteration(iterationResult, options);
+        var iterationsResults = dashboard.results;
+        iterationsResults.forEach(function(iterationResult, index) {
+            this._addIteration(iterationResult, dashboard.data[index], dashboard.options);
         }, this);
     }
 });
@@ -172,7 +300,7 @@ window.benchmarkRunnerClient = {
 
     willStartFirstIteration: function()
     {
-        this.results = new ResultsDashboard();
+        this.results = new ResultsDashboard(this.options);
     },
 
     didRunSuites: function(suitesSamplers)
@@ -180,6 +308,11 @@ window.benchmarkRunnerClient = {
         this.results.push(suitesSamplers);
     },
 
+    didRunTest: function(testData)
+    {
+        this.results.calculateScore(testData);
+    },
+
     didFinishLastIteration: function()
     {
         benchmarkController.showResults();
@@ -210,10 +343,10 @@ window.sectionsManager =
             document.querySelector("#" + sectionIdentifier + " .mean").innerHTML = mean;
     },
 
-    populateTable: function(tableIdentifier, headers, data)
+    populateTable: function(tableIdentifier, headers, dashboard)
     {
         var table = new ResultsTable(document.getElementById(tableIdentifier), headers);
-        table.showIterations(data, benchmarkRunnerClient.options);
+        table.showIterations(dashboard);
     }
 };
 
@@ -249,10 +382,11 @@ window.benchmarkController = {
             this.addedKeyEvent = true;
         }
 
-        sectionsManager.setSectionScore("results", benchmarkRunnerClient.results.score.toFixed(2));
-        var data = benchmarkRunnerClient.results.data[Strings.json.results.iterations];
-        sectionsManager.populateTable("results-header", Headers.testName, data);
-        sectionsManager.populateTable("results-score", Headers.score, data);
+        var dashboard = benchmarkRunnerClient.results;
+
+        sectionsManager.setSectionScore("results", dashboard.score.toFixed(2));
+        sectionsManager.populateTable("results-header", Headers.testName, dashboard);
+        sectionsManager.populateTable("results-score", Headers.score, dashboard);
         sectionsManager.showSection("results", true);
     },
 
index a2ac7b2..1aa6303 100644 (file)
@@ -95,13 +95,13 @@ BenchmarkRunner = Utilities.createClass(
 
         var benchmark = new contentWindow.benchmarkClass(options);
         document.body.style.backgroundColor = benchmark.backgroundColor();
-        benchmark.run().then(function(results) {
+        benchmark.run().then(function(testData) {
             var suiteResults = self._suitesResults[suite.name] || {};
-            suiteResults[test.name] = results;
+            suiteResults[test.name] = testData;
             self._suitesResults[suite.name] = suiteResults;
 
             if (self._client && self._client.didRunTest)
-                self._client.didRunTest(suite, test);
+                self._client.didRunTest(testData);
 
             state.next();
             if (state.currentSuite() != suite)
diff --git a/PerformanceTests/Animometer/resources/statistics.js b/PerformanceTests/Animometer/resources/statistics.js
new file mode 100644 (file)
index 0000000..ee7c24c
--- /dev/null
@@ -0,0 +1,377 @@
+Pseudo =
+{
+    initialRandomSeed: 49734321,
+    randomSeed: 49734321,
+
+    resetRandomSeed: function()
+    {
+        Pseudo.randomSeed = Pseudo.initialRandomSeed;
+    },
+
+    random: function()
+    {
+        var randomSeed = Pseudo.randomSeed;
+        randomSeed = ((randomSeed + 0x7ed55d16) + (randomSeed << 12))  & 0xffffffff;
+        randomSeed = ((randomSeed ^ 0xc761c23c) ^ (randomSeed >>> 19)) & 0xffffffff;
+        randomSeed = ((randomSeed + 0x165667b1) + (randomSeed << 5))   & 0xffffffff;
+        randomSeed = ((randomSeed + 0xd3a2646c) ^ (randomSeed << 9))   & 0xffffffff;
+        randomSeed = ((randomSeed + 0xfd7046c5) + (randomSeed << 3))   & 0xffffffff;
+        randomSeed = ((randomSeed ^ 0xb55a4f09) ^ (randomSeed >>> 16)) & 0xffffffff;
+        Pseudo.randomSeed = randomSeed;
+        return (randomSeed & 0xfffffff) / 0x10000000;
+    }
+};
+
+Statistics =
+{
+    sampleMean: function(numberOfSamples, sum)
+    {
+        if (numberOfSamples < 1)
+            return 0;
+        return sum / numberOfSamples;
+    },
+
+    // With sum and sum of squares, we can compute the sample standard deviation in O(1).
+    // See https://rniwa.com/2012-11-10/sample-standard-deviation-in-terms-of-sum-and-square-sum-of-samples/
+    unbiasedSampleStandardDeviation: function(numberOfSamples, sum, squareSum)
+    {
+        if (numberOfSamples < 2)
+            return 0;
+        return Math.sqrt((squareSum - sum * sum / numberOfSamples) / (numberOfSamples - 1));
+    },
+
+    geometricMean: function(values)
+    {
+        if (!values.length)
+            return 0;
+        var roots = values.map(function(value) { return Math.pow(value, 1 / values.length); })
+        return roots.reduce(function(a, b) { return a * b; });
+    },
+
+    // Cumulative distribution function
+    cdf: function(value, mean, standardDeviation)
+    {
+        return 0.5 * (1 + Statistics.erf((value - mean) / (Math.sqrt(2 * standardDeviation * standardDeviation))));
+    },
+
+    // Approximation of Gauss error function, Abramowitz and Stegun 7.1.26
+    erf: function(value)
+    {
+          var sign = (value >= 0) ? 1 : -1;
+          value = Math.abs(value);
+
+          var a1 = 0.254829592;
+          var a2 = -0.284496736;
+          var a3 = 1.421413741;
+          var a4 = -1.453152027;
+          var a5 = 1.061405429;
+          var p = 0.3275911;
+
+          var t = 1.0 / (1.0 + p * value);
+          var y = 1.0 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-value * value);
+          return sign * y;
+    }
+};
+
+Experiment = Utilities.createClass(
+    function(includeConcern)
+    {
+        if (includeConcern)
+            this._maxHeap = Heap.createMaxHeap(Experiment.defaults.CONCERN_SIZE);
+        this.reset();
+    }, {
+
+    reset: function()
+    {
+        this._sum = 0;
+        this._squareSum = 0;
+        this._numberOfSamples = 0;
+        if (this._maxHeap)
+            this._maxHeap.init();
+    },
+
+    get sampleCount()
+    {
+        return this._numberOfSamples;
+    },
+
+    sample: function(value)
+    {
+        this._sum += value;
+        this._squareSum += value * value;
+        if (this._maxHeap)
+            this._maxHeap.push(value);
+        ++this._numberOfSamples;
+    },
+
+    mean: function()
+    {
+        return Statistics.sampleMean(this._numberOfSamples, this._sum);
+    },
+
+    standardDeviation: function()
+    {
+        return Statistics.unbiasedSampleStandardDeviation(this._numberOfSamples, this._sum, this._squareSum);
+    },
+
+    cdf: function(value)
+    {
+        return Statistics.cdf(value, this.mean(), this.standardDeviation());
+    },
+
+    percentage: function()
+    {
+        var mean = this.mean();
+        return mean ? this.standardDeviation() * 100 / mean : 0;
+    },
+
+    concern: function(percentage)
+    {
+        if (!this._maxHeap)
+            return this.mean();
+
+        var size = Math.ceil(this._numberOfSamples * percentage / 100);
+        var values = this._maxHeap.values(size);
+        return values.length ? values.reduce(function(a, b) { return a + b; }) / values.length : 0;
+    },
+
+    score: function(percentage)
+    {
+        return Statistics.geometricMean([this.mean(), Math.max(this.concern(percentage), 1)]);
+    }
+});
+
+Experiment.defaults =
+{
+    CONCERN: 5,
+    CONCERN_SIZE: 100,
+};
+
+Regression = Utilities.createClass(
+    function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+    {
+        var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+            shouldClip: true,
+            s1: 1000/60,
+            t1: 0
+        });
+        var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+            shouldClip: true,
+            s1: 1000/60,
+            t1: 0,
+            t2: 0
+        });
+        var desired;
+        if (slope.error < flat.error)
+            desired = slope;
+        else
+            desired = flat;
+
+        this.startIndex = Math.min(startIndex, endIndex);
+        this.endIndex = Math.max(startIndex, endIndex);
+
+        this.complexity = desired.complexity;
+        this.s1 = desired.s1;
+        this.t1 = desired.t1;
+        this.s2 = desired.s2;
+        this.t2 = desired.t2;
+        this.stdev1 = desired.stdev1;
+        this.stdev2 = desired.stdev2;
+        this.n1 = desired.n1;
+        this.n2 = desired.n2;
+        this.error = desired.error;
+    }, {
+
+    valueAt: function(complexity)
+    {
+        if (this.n1 == 1 || complexity > this.complexity)
+            return this.s2 + this.t2 * complexity;
+        return this.s1 + this.t1 * complexity;
+    },
+
+    // A generic two-segment piecewise regression calculator. Based on Kundu/Ubhaya
+    //
+    // Minimize sum of (y - y')^2
+    // where                        y = s1 + t1*x
+    //                              y = s2 + t2*x
+    //                y' = s1 + t1*x' = s2 + t2*x'   if x_0 <= x' <= x_n
+    //
+    // Allows for fixing s1, t1, s2, t2
+    //
+    // x is assumed to be complexity, y is frame length. Can be used for pure complexity-FPS
+    // analysis or for ramp controllers since complexity monotonically decreases with time.
+    _calculateRegression: function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+    {
+        if (startIndex == endIndex) {
+            // Only one sample point; we can't calculate any regression.
+            var x = getComplexity(samples, startIndex);
+            return {
+                complexity: x,
+                s1: x,
+                t1: 0,
+                s2: x,
+                t2: 0,
+                error1: 0,
+                error2: 0
+            };
+        }
+
+        var iterationDirection = endIndex > startIndex ? 1 : -1;
+        var lowComplexity = getComplexity(samples, startIndex);
+        var highComplexity = getComplexity(samples, endIndex);
+        var a1 = 0, b1 = 0, c1 = 0, d1 = 0, h1 = 0, k1 = 0;
+        var a2 = 0, b2 = 0, c2 = 0, d2 = 0, h2 = 0, k2 = 0;
+
+        // Iterate from low to high complexity
+        for (var i = startIndex; iterationDirection * (endIndex - i) > -1; i += iterationDirection) {
+            var x = getComplexity(samples, i);
+            var y = getFrameLength(samples, i);
+            a2 += 1;
+            b2 += x;
+            c2 += x * x;
+            d2 += y;
+            h2 += y * x;
+            k2 += y * y;
+        }
+
+        var s1_best, t1_best, s2_best, t2_best, n1_best, n2_best, error1_best, error2_best, x_best, x_prime;
+
+        function setBest(s1, t1, error1, s2, t2, error2, splitIndex, x_prime, x)
+        {
+            s1_best = s1;
+            t1_best = t1;
+            error1_best = error1;
+            s2_best = s2;
+            t2_best = t2;
+            error2_best = error2;
+            n1_best = iterationDirection * (splitIndex - startIndex) + 1;
+            n2_best = iterationDirection * (endIndex - splitIndex);
+            if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
+                x_best = x_prime;
+            else {
+                // Discontinuous piecewise regression
+                x_best = x;
+            }
+        }
+
+        // Iterate from startIndex to endIndex - 1, inclusive
+        for (var i = startIndex; iterationDirection * (endIndex - i) > 0; i += iterationDirection) {
+            var x = getComplexity(samples, i);
+            var y = getFrameLength(samples, i);
+            var xx = x * x;
+            var yx = y * x;
+            var yy = y * y;
+            // a1, b1, etc. is sum from startIndex to i, inclusive
+            a1 += 1;
+            b1 += x;
+            c1 += xx;
+            d1 += y;
+            h1 += yx;
+            k1 += yy;
+            // a2, b2, etc. is sum from i+1 to endIndex, inclusive
+            a2 -= 1;
+            b2 -= x;
+            c2 -= xx;
+            d2 -= y;
+            h2 -= yx;
+            k2 -= yy;
+
+            var A = c1*d1 - b1*h1;
+            var B = a1*h1 - b1*d1;
+            var C = a1*c1 - b1*b1;
+            var D = c2*d2 - b2*h2;
+            var E = a2*h2 - b2*d2;
+            var F = a2*c2 - b2*b2;
+            var s1 = options.s1 !== undefined ? options.s1 : (A / C);
+            var t1 = options.t1 !== undefined ? options.t1 : (B / C);
+            var s2 = options.s2 !== undefined ? options.s2 : (D / F);
+            var t2 = options.t2 !== undefined ? options.t2 : (E / F);
+            // Assumes that the two segments meet
+            var x_prime = (s1 - s2) / (t2 - t1);
+
+            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
+            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
+
+            if (i == startIndex) {
+                setBest(s1, t1, error1, s2, t2, error2, i, x_prime, x);
+                continue;
+            }
+
+            if (C == 0 || F == 0)
+                continue;
+
+            // Projected point is not between this and the next sample
+            if (x_prime > getComplexity(samples, i + iterationDirection) || x_prime < x) {
+                // Calculate lambda, which divides the weight of this sample between the two lines
+
+                // These values remove the influence of this sample
+                var I = c1 - 2*b1*x + a1*xx;
+                var H = C - I;
+                var G = A + B*x - C*y;
+
+                var J = D + E*x - F*y;
+                var K = c2 - 2*b2*x + a2*xx;
+
+                var lambda = (G*F + G*K - H*J) / (I*J + G*K);
+                if (lambda > 0 && lambda < 1) {
+                    var lambda1 = 1 - lambda;
+                    s1 = options.s1 !== undefined ? options.s1 : ((A - lambda1*(-h1*x + d1*xx + c1*y - b1*yx)) / (C - lambda1*I));
+                    t1 = options.t1 !== undefined ? options.t1 : ((B - lambda1*(h1 - d1*x - b1*y + a1*yx)) / (C - lambda1*I));
+                    s2 = options.s2 !== undefined ? options.s2 : ((D + lambda1*(-h2*x + d2*xx + c2*y - b2*yx)) / (F + lambda1*K));
+                    t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
+                    x_prime = (s1 - s2) / (t2 - t1);
+
+                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
+                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
+                } else if (t1 != t2)
+                    continue;
+            }
+
+            if (error1 + error2 < error1_best + error2_best)
+                setBest(s1, t1, error1, s2, t2, error2, i, x_prime, x);
+        }
+
+        return {
+            complexity: x_best,
+            s1: s1_best,
+            t1: t1_best,
+            stdev1: Math.sqrt(error1_best / n1_best),
+            s2: s2_best,
+            t2: t2_best,
+            stdev2: Math.sqrt(error2_best / n2_best),
+            error: error1_best + error2_best,
+            n1: n1_best,
+            n2: n2_best
+        };
+    }
+});
+
+Utilities.extendObject(Regression, {
+    bootstrap: function(samples, iterationCount, processResample, confidencePercentage)
+    {
+        var sampleLength = samples.length;
+        var resample = new Array(sampleLength);
+
+        var bootstrapEstimator = new Experiment;
+        var bootstrapData = new Array(iterationCount);
+
+        Pseudo.resetRandomSeed();
+        for (var i = 0; i < iterationCount; ++i) {
+            for (var j = 0; j < sampleLength; ++j)
+                resample[j] = samples[Math.floor(Pseudo.random() * sampleLength)];
+
+            var resampleResult = processResample(resample);
+            bootstrapEstimator.sample(resampleResult);
+            bootstrapData[i] = resampleResult;
+        }
+
+        bootstrapData.sort(function(a, b) { return a - b; });
+        return {
+            confidenceLow: bootstrapData[Math.round((iterationCount - 1) * (1 - confidencePercentage) / 2)],
+            confidenceHigh: bootstrapData[Math.round((iterationCount - 1) * (1 + confidencePercentage) / 2)],
+            median: bootstrapData[Math.round(iterationCount / 2)],
+            mean: bootstrapEstimator.mean(),
+            data: bootstrapData
+        };
+    }
+});
index cc03bf0..19229aa 100644 (file)
@@ -1,53 +1,38 @@
 var Strings = {
     text: {
         testName: "Test Name",
-        score: "Score",
-        samples: "Samples",
-
-        complexity: "Complexity",
-        frameRate: "FPS",
-        mergedRawComplexity: "Merged raw",
-        mergedAverageComplexity: "Merged average",
-        graph: "Graph"
+        score: "Score"
     },
     json: {
-        score: "score",
-        samples: "samples",
-        complexityAverageSamples: "complexityAverageSamples",
         marks: "marks",
-
-        targetFrameLength: "targetFrameLength",
         samplingStartTimeOffset: "Start sampling",
         samplingEndTimeOffset: "End sampling",
 
-        experiments: {
-            complexity: "complexity",
-            frameRate: "frameRate"
+        samples: "samples",
+        controller: "controller",
+        complexity: "complexity",
+        complexityAverage: "complexityAverage",
+        frameLength: "frameLength",
+
+        result: "result",
+        score: "score",
+        bootstrap: "bootstrap",
+        measurements: {
+            average: "average",
+            concern: "concern",
+            stdev: "stdev",
+            percent: "percent"
         },
 
         regressions: {
-            timeRegressions: "timeRegressions",
-            complexity: "complexity",
-            maxComplexity: "maxComplexity",
             startIndex: "startIndex",
             endIndex: "endIndex",
-
-            complexityRegression: "complexityRegression",
-            complexityAverageRegression: "complexityAverageRegression",
             segment1: "segment1",
             segment2: "segment2"
         },
 
-        measurements: {
-            average: "average",
-            concern: "concern",
-            stdev: "stdev",
-            percent: "percent"
-        },
-
         results: {
             iterations: "iterationsResults",
-            suites: "suitesResults",
             tests: "testsResults"
         }
     }
index 84bcf18..864c19a 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         img.hidden {
@@ -15,8 +16,7 @@
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index ac47fc6..c759d45 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index bfaca49..058de48 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         img {
@@ -12,8 +13,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index 0f0f579..0c1248f 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         .circle {
@@ -23,8 +24,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index bb29050..2b6a0b9 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <svg id="stage"></svg>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index ae5193e..a821231 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <svg id="stage"></svg>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index 36fc2f7..123c4bc 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/canvas-stage.js"></script>
index 5263473..2a5c600 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
     <style type="text/css">
 
@@ -33,8 +34,7 @@
     </div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/focus.js"></script>
index bc92e65..4c5fa8d 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style type="text/css">
 
@@ -18,8 +19,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/image-data.js"></script>
index be1758d..7e7f07b 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
     <style type="text/css">
 
@@ -51,8 +52,7 @@
     </div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/multiply.js"></script>
index 607d714..8de065e 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
     <style>
         #stage div {
@@ -14,8 +15,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/particles.js"></script>
index a9e957b..b1d6db7 100644 (file)
@@ -7,15 +7,15 @@ CanvasLineSegment = Utilities.createClass(
     {
         var circle = Stage.randomInt(0, 2);
         this._color = ["#e01040", "#10c030", "#e05010"][circle];
-        this._lineWidth = Math.pow(Math.random(), 12) * 20 + 3;
-        this._omega = Math.random() * 3 + 0.2;
+        this._lineWidth = Math.pow(Pseudo.random(), 12) * 20 + 3;
+        this._omega = Pseudo.random() * 3 + 0.2;
         var theta = Stage.randomAngle();
         this._cosTheta = Math.cos(theta);
         this._sinTheta = Math.sin(theta);
         this._startX = stage.circleRadius * this._cosTheta + (0.5 + circle) / 3 * stage.size.x;
         this._startY = stage.circleRadius * this._sinTheta + stage.size.y / 2;
-        this._length = Math.pow(Math.random(), 8) * 40 + 20;
-        this._segmentDirection = Math.random() > 0.5 ? -1 : 1;
+        this._length = Math.pow(Pseudo.random(), 8) * 40 + 20;
+        this._segmentDirection = Pseudo.random() > 0.5 ? -1 : 1;
     }, {
 
     draw: function(context)
@@ -44,15 +44,15 @@ CanvasArc = Utilities.createClass(
 
         this._point = new Point(distanceX * (randX + (randY % 2) / 2), distanceY * (randY + .5));
 
-        this._radius = 20 + Math.pow(Math.random(), 5) * (Math.min(distanceX, distanceY) / 1.8);
+        this._radius = 20 + Math.pow(Pseudo.random(), 5) * (Math.min(distanceX, distanceY) / 1.8);
         this._startAngle = Stage.randomAngle();
         this._endAngle = Stage.randomAngle();
-        this._omega = (Math.random() - 0.5) * 0.3;
+        this._omega = (Pseudo.random() - 0.5) * 0.3;
         this._counterclockwise = Stage.randomBool();
         var colors = ["#101010", "#808080", "#c0c0c0"];
         colors.push(["#e01040", "#10c030", "#e05010"][(randX + Math.ceil(randY / 2)) % 3]);
-        this._color = colors[Math.floor(Math.random() * colors.length)];
-        this._lineWidth = 1 + Math.pow(Math.random(), 5) * 30;
+        this._color = colors[Math.floor(Pseudo.random() * colors.length)];
+        this._lineWidth = 1 + Math.pow(Pseudo.random(), 5) * 30;
         this._doStroke = Stage.randomInt(0, 3) != 0;
     }, {
 
@@ -87,7 +87,7 @@ CanvasLinePoint = Utilities.createClass(
         var Y_LOOPS = 20;
 
         var offsets = [[-2, -1], [2, 1], [-1, 0], [1, 0], [-1, 2], [1, -2]];
-        var offset = offsets[Math.floor(Math.random() * offsets.length)];
+        var offset = offsets[Math.floor(Pseudo.random() * offsets.length)];
 
         this.coordinate = new Point(X_LOOPS/2, Y_LOOPS/2);
         if (stage.objects.length) {
@@ -111,10 +111,10 @@ CanvasLinePoint = Utilities.createClass(
         var randX = (xOff + this.coordinate.x) * stage.size.x / X_LOOPS;
         var randY = this.coordinate.y * stage.size.y / Y_LOOPS;
         var colors = ["#101010", "#808080", "#c0c0c0", "#101010", "#808080", "#c0c0c0", "#e01040"];
-        this.color = colors[Math.floor(Math.random() * colors.length)];
+        this.color = colors[Math.floor(Pseudo.random() * colors.length)];
 
-        this.width = Math.pow(Math.random(), 5) * 20 + 1;
-        this.isSplit = Math.random() > 0.9;
+        this.width = Math.pow(Pseudo.random(), 5) * 20 + 1;
+        this.isSplit = Pseudo.random() > 0.9;
         this.point = new Point(randX, randY);
     }
 );
@@ -189,7 +189,7 @@ CanvasLinePathStage = Utilities.createSubclass(SimpleCanvasStage,
 
                 context.lineTo(object.point.x, object.point.y);
 
-                if (Math.random() > 0.999)
+                if (Pseudo.random() > 0.999)
                     object.isSplit = !object.isSplit;
             }
         }
index 482e7bc..fc0ca59 100644 (file)
@@ -18,7 +18,7 @@ var FocusElement = Utilities.createClass(
         var left = Stage.random(0, stage.size.width - 2 * radius - sizeVariance);
 
         // size and blurring are a function of depth
-        this._depth = Utilities.lerp(1 - Math.pow(Math.random(), 2), minObjectDepth, maxObjectDepth);
+        this._depth = Utilities.lerp(1 - Math.pow(Pseudo.random(), 2), minObjectDepth, maxObjectDepth);
         var distance = Utilities.lerp(this._depth, 1, sizeVariance);
         var size = 2 * radius + sizeVariance - distance;
 
@@ -32,8 +32,8 @@ var FocusElement = Utilities.createClass(
         Utilities.setElementPrefixedProperty(this.element, "filter", "blur(" + stage.getBlurValue(this._depth) + "px) opacity(" + stage.getOpacityValue(this._depth) + "%)");
 
         var depthMultiplier = Utilities.lerp(1 - this._depth, 0.8, 1);
-        this._sinMultiplier = Math.random() * Stage.randomSign() * depthMultiplier;
-        this._cosMultiplier = Math.random() * Stage.randomSign() * depthMultiplier;
+        this._sinMultiplier = Pseudo.random() * Stage.randomSign() * depthMultiplier;
+        this._cosMultiplier = Pseudo.random() * Stage.randomSign() * depthMultiplier;
     }, {
 
     animate: function(stage, sinTime, cosTime)
index d51bf51..d251a60 100644 (file)
@@ -10,7 +10,7 @@ Particle.prototype =
 {
     reset: function()
     {
-        var randSize = Math.pow(Math.random(), 4) * 25 + 15;
+        var randSize = Math.pow(Pseudo.random(), 4) * 25 + 15;
         this.size = new Point(randSize, randSize);
         this.maxPosition = this.stage.size.subtract(this.size);
         this.position = new Point(this.stage.size.x / 2, this.stage.size.y / 4);
index 0d1e9d5..bf933bd 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/canvas-electrons.js"></script>
index ca20c81..da5b0b2 100644 (file)
@@ -1,11 +1,11 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/canvas-stars.js"></script>
index 9ccb045..6d5b0bf 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         img {
@@ -14,8 +15,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="../bouncing-particles/resources/bouncing-particles.js"></script>
diff --git a/PerformanceTests/Animometer/tests/resources/algorithm.js b/PerformanceTests/Animometer/tests/resources/algorithm.js
deleted file mode 100644 (file)
index 41188b1..0000000
+++ /dev/null
@@ -1,134 +0,0 @@
-function Heap(maxSize, compare)
-{
-    this._maxSize = maxSize;
-    this._compare = compare;
-    this._size = 0;
-    this._values = new Array(this._maxSize);
-}
-
-Heap.prototype =
-{
-    // This is a binary heap represented in an array. The root element is stored
-    // in the first element in the array. The root is followed by its two children.
-    // Then its four grandchildren and so on. So every level in the binary heap is
-    // doubled in the following level. Here is an example of the node indices and
-    // how they are related to their parents and children.
-    // ===========================================================================
-    //              0       1       2       3       4       5       6
-    // PARENT       -1      0       0       1       1       2       2
-    // LEFT         1       3       5       7       9       11      13
-    // RIGHT        2       4       6       8       10      12      14
-    // ===========================================================================
-    _parentIndex: function(i)
-    {
-        return i > 0 ? Math.floor((i - 1) / 2) : -1;
-    },
-
-    _leftIndex: function(i)
-    {
-        var leftIndex = i * 2 + 1;
-        return leftIndex < this._size ? leftIndex : -1;
-    },
-
-    _rightIndex: function(i)
-    {
-        var rightIndex = i * 2 + 2;
-        return rightIndex < this._size ? rightIndex : -1;
-    },
-
-    // Return the child index that may violate the heap property at index i.
-    _childIndex: function(i)
-    {
-        var left = this._leftIndex(i);
-        var right = this._rightIndex(i);
-
-        if (left != -1 && right != -1)
-            return this._compare(this._values[left], this._values[right]) > 0 ? left : right;
-
-        return left != -1 ? left : right;
-    },
-
-    init: function()
-    {
-        this._size = 0;
-    },
-
-    top: function()
-    {
-        return this._size ? this._values[0] : NaN;
-    },
-
-    push: function(value)
-    {
-        if (this._size == this._maxSize) {
-            // If size is bounded and the new value can be a parent of the top()
-            // if the size were unbounded, just ignore the new value.
-            if (this._compare(value, this.top()) > 0)
-                return;
-            this.pop();
-        }
-        this._values[this._size++] = value;
-        this._bubble(this._size - 1);
-    },
-
-    pop: function()
-    {
-        if (!this._size)
-            return NaN;
-
-        this._values[0] = this._values[--this._size];
-        this._sink(0);
-    },
-
-    _bubble: function(i)
-    {
-        // Fix the heap property at index i given that parent is the only node that
-        // may violate the heap property.
-        for (var pi = this._parentIndex(i); pi != -1; i = pi, pi = this._parentIndex(pi)) {
-            if (this._compare(this._values[pi], this._values[i]) > 0)
-                break;
-
-            this._values.swap(pi, i);
-        }
-    },
-
-    _sink: function(i)
-    {
-        // Fix the heap property at index i given that each of the left and the right
-        // sub-trees satisfies the heap property.
-        for (var ci = this._childIndex(i); ci != -1; i = ci, ci = this._childIndex(ci)) {
-            if (this._compare(this._values[i], this._values[ci]) > 0)
-                break;
-
-            this._values.swap(ci, i);
-        }
-    },
-
-    str: function()
-    {
-        var out = "Heap[" + this._size + "] = [";
-        for (var i = 0; i < this._size; ++i) {
-            out += this._values[i];
-            if (i < this._size - 1)
-                out += ", ";
-        }
-        return out + "]";
-    },
-
-    values: function(size) {
-        // Return the last "size" heap elements values.
-        var values = this._values.slice(0, this._size);
-        return values.sort(this._compare).slice(0, Math.min(size, this._size));
-    }
-}
-
-var Algorithm = {
-    createMinHeap: function(maxSize)
-    {
-        return new Heap(maxSize, function(a, b) { return b - a; });
-    },
-
-    createMaxHeap: function(maxSize) {
-        return new Heap(maxSize, function(a, b) { return a - b; });
-    }
-}
index dafd6db..529e1ce 100644 (file)
@@ -1,3 +1,40 @@
+Sampler = Utilities.createClass(
+    function(seriesCount, expectedSampleCount, processor)
+    {
+        this._processor = processor;
+
+        this.samples = [];
+        for (var i = 0; i < seriesCount; ++i) {
+            var array = new Array(expectedSampleCount);
+            array.fill(0);
+            this.samples[i] = array;
+        }
+        this.sampleCount = 0;
+    }, {
+
+    record: function() {
+        // Assume that arguments.length == this.samples.length
+        for (var i = 0; i < arguments.length; i++) {
+            this.samples[i][this.sampleCount] = arguments[i];
+        }
+        ++this.sampleCount;
+    },
+
+    processSamples: function()
+    {
+        var results = {};
+
+        // Remove unused capacity
+        this.samples = this.samples.map(function(array) {
+            return array.slice(0, this.sampleCount);
+        }, this);
+
+        this._processor.processSamples(results);
+
+        return results;
+    }
+});
+
 Controller = Utilities.createClass(
     function(benchmark, options)
     {
@@ -27,6 +64,7 @@ Controller = Utilities.createClass(
     {
         this._startTimestamp = startTimestamp;
         this._endTimestamp += startTimestamp;
+        this._previousTimestamp = startTimestamp;
         this._measureAndResetInterval(startTimestamp);
         this.recordFirstSample(startTimestamp, stage);
     },
@@ -66,11 +104,14 @@ Controller = Utilities.createClass(
 
     update: function(timestamp, stage)
     {
-        var frameLengthEstimate = -1;
+        var lastFrameLength = timestamp - this._previousTimestamp;
+        this._previousTimestamp = timestamp;
+
+        var frameLengthEstimate = -1, intervalAverageFrameLength = -1;
         var didFinishInterval = false;
         if (!this._intervalLength) {
             if (this._isFrameLengthEstimatorEnabled) {
-                this._frameLengthEstimator.sample(timestamp - this._sampler.samples[0][this._sampler.sampleCount - 1]);
+                this._frameLengthEstimator.sample(lastFrameLength);
                 frameLengthEstimate = this._frameLengthEstimator.estimate;
             }
         } else if (timestamp >= this._intervalEndTimestamp) {
@@ -85,14 +126,14 @@ Controller = Utilities.createClass(
         }
 
         this._sampler.record(timestamp, stage.complexity(), frameLengthEstimate);
-        this.tune(timestamp, stage, didFinishInterval);
+        this.tune(timestamp, stage, lastFrameLength, didFinishInterval, intervalAverageFrameLength);
     },
 
     didFinishInterval: function(timestamp, stage, intervalAverageFrameLength)
     {
     },
 
-    tune: function(timestamp, stage, didFinishInterval)
+    tune: function(timestamp, stage, lastFrameLength, didFinishInterval, intervalAverageFrameLength)
     {
     },
 
@@ -106,6 +147,38 @@ Controller = Utilities.createClass(
         return this._sampler.processSamples();
     },
 
+    _processComplexitySamples: function(complexitySamples, complexityAverageSamples)
+    {
+        complexitySamples.sort(function(a, b) {
+            return a.complexity - b.complexity;
+        });
+
+        // Samples averaged based on complexity
+        var currentComplexity = -1;
+        var experimentAtComplexity;
+        function addSample() {
+            var mean = experimentAtComplexity.mean();
+            var stdev = experimentAtComplexity.standardDeviation();
+            complexityAverageSamples.push({
+                complexity: currentComplexity,
+                frameLength: mean,
+                stdev: stdev
+            });
+        }
+        complexitySamples.forEach(function(sample) {
+            if (sample.complexity != currentComplexity) {
+                if (currentComplexity > -1)
+                    addSample();
+
+                currentComplexity = sample.complexity;
+                experimentAtComplexity = new Experiment;
+            }
+            experimentAtComplexity.sample(sample.frameLength);
+        });
+        // Finish off the last one
+        addSample();
+    },
+
     processSamples: function(results)
     {
         var complexityExperiment = new Experiment;
@@ -113,56 +186,36 @@ Controller = Utilities.createClass(
 
         var samples = this._sampler.samples;
 
-        var samplingStartIndex = 0, samplingEndIndex = -1;
-        if (Strings.json.samplingStartTimeOffset in this._marks)
-            samplingStartIndex = this._marks[Strings.json.samplingStartTimeOffset].index;
-        if (Strings.json.samplingEndTimeOffset in this._marks)
-            samplingEndIndex = this._marks[Strings.json.samplingEndTimeOffset].index;
-
         for (var markName in this._marks)
             this._marks[markName].time -= this._startTimestamp;
         results[Strings.json.marks] = this._marks;
 
-        results[Strings.json.samples] = samples[0].map(function(timestamp, i) {
-            var result = {
+        results[Strings.json.samples] = {};
+
+        var complexitySamples = [], complexityAverageSamples = [];
+        results[Strings.json.samples][Strings.json.complexity] = complexitySamples;
+        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
+
+        results[Strings.json.samples][Strings.json.controller] = samples[0].map(function(timestamp, i) {
+            var sample = {
                 // Represent time in milliseconds
                 time: timestamp - this._startTimestamp,
                 complexity: samples[1][i]
             };
 
             if (i == 0)
-                result.frameLength = 1000/60;
+                sample.frameLength = 1000/60;
             else
-                result.frameLength = timestamp - samples[0][i - 1];
+                sample.frameLength = timestamp - samples[0][i - 1];
 
             if (samples[2][i] != -1)
-                result.smoothedFrameLength = samples[2][i];
-
-            // Don't start adding data to the experiments until we reach the sampling timestamp
-            if (i >= samplingStartIndex && (samplingEndIndex == -1 || i < samplingEndIndex)) {
-                complexityExperiment.sample(result.complexity);
-                if (result.smoothedFrameLength && result.smoothedFrameLength != -1)
-                    smoothedFrameLengthExperiment.sample(result.smoothedFrameLength);
-            }
+                sample.smoothedFrameLength = samples[2][i];
 
-            return result;
+            complexitySamples.push(sample);
+            return sample;
         }, this);
 
-        results[Strings.json.score] = complexityExperiment.score(Experiment.defaults.CONCERN);
-
-        var complexityResults = {};
-        results[Strings.json.experiments.complexity] = complexityResults;
-        complexityResults[Strings.json.measurements.average] = complexityExperiment.mean();
-        complexityResults[Strings.json.measurements.concern] = complexityExperiment.concern(Experiment.defaults.CONCERN);
-        complexityResults[Strings.json.measurements.stdev] = complexityExperiment.standardDeviation();
-        complexityResults[Strings.json.measurements.percent] = complexityExperiment.percentage();
-
-        var smoothedFrameLengthResults = {};
-        results[Strings.json.experiments.frameRate] = smoothedFrameLengthResults;
-        smoothedFrameLengthResults[Strings.json.measurements.average] = 1000 / smoothedFrameLengthExperiment.mean();
-        smoothedFrameLengthResults[Strings.json.measurements.concern] = smoothedFrameLengthExperiment.concern(Experiment.defaults.CONCERN);
-        smoothedFrameLengthResults[Strings.json.measurements.stdev] = smoothedFrameLengthExperiment.standardDeviation();
-        smoothedFrameLengthResults[Strings.json.measurements.percent] = smoothedFrameLengthExperiment.percentage();
+        this._processComplexitySamples(complexitySamples, complexityAverageSamples);
     }
 });
 
@@ -250,174 +303,6 @@ AdaptiveController = Utilities.createSubclass(Controller,
         // Start the next interval.
         this._intervalFrameCount = 0;
         this._intervalTimestamp = timestamp;
-    },
-
-    processSamples: function(results)
-    {
-        Controller.prototype.processSamples.call(this, results);
-        results[Strings.json.targetFrameLength] = 1000 / this._targetFrameRate;
-    }
-});
-
-Regression = Utilities.createClass(
-    function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
-    {
-        var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
-            shouldClip: true,
-            s1: 1000/60,
-            t1: 0
-        });
-        var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
-            shouldClip: true,
-            t1: 0,
-            t2: 0
-        });
-        var desired;
-        if (slope.error < flat.error)
-            desired = slope;
-        else
-            desired = flat;
-
-        this.startIndex = Math.min(startIndex, endIndex);
-        this.endIndex = Math.max(startIndex, endIndex);
-
-        this.complexity = desired.complexity;
-        this.s1 = desired.s1;
-        this.t1 = desired.t1;
-        this.s2 = desired.s2;
-        this.t2 = desired.t2;
-        this.error = desired.error;
-    }, {
-
-    // A generic two-segment piecewise regression calculator. Based on Kundu/Ubhaya
-    //
-    // Minimize sum of (y - y')^2
-    // where                        y = s1 + t1*x
-    //                              y = s2 + t2*x
-    //                y' = s1 + t1*x' = s2 + t2*x'   if x_0 <= x' <= x_n
-    //
-    // Allows for fixing s1, t1, s2, t2
-    //
-    // x is assumed to be complexity, y is frame length. Can be used for pure complexity-FPS
-    // analysis or for ramp controllers since complexity monotonically decreases with time.
-    _calculateRegression: function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
-    {
-        var iterationDirection = endIndex > startIndex ? 1 : -1;
-        var lowComplexity = getComplexity(samples, startIndex);
-        var highComplexity = getComplexity(samples, endIndex);
-        var a1 = 0, b1 = 0, c1 = 0, d1 = 0, h1 = 0, k1 = 0;
-        var a2 = 0, b2 = 0, c2 = 0, d2 = 0, h2 = 0, k2 = 0;
-
-        // Iterate from low to high complexity
-        for (var i = startIndex; iterationDirection * (endIndex - i) > -1; i += iterationDirection) {
-            var x = getComplexity(samples, i);
-            var y = getFrameLength(samples, i);
-            a2 += 1;
-            b2 += x;
-            c2 += x * x;
-            d2 += y;
-            h2 += y * x;
-            k2 += y * y;
-        }
-
-        var s1_best, t1_best, s2_best, t2_best, x_best, error_best, x_prime;
-
-        function setBest(s1, t1, s2, t2, error, x_prime, x)
-        {
-            s1_best = s1;
-            t1_best = t1;
-            s2_best = s2;
-            t2_best = t2;
-            error_best = error;
-            if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
-                x_best = x_prime;
-            else {
-                // Discontinuous piecewise regression
-                x_best = x;
-            }
-        }
-
-        // Iterate from startIndex to endIndex - 1, inclusive
-        for (var i = startIndex; iterationDirection * (endIndex - i) > 0; i += iterationDirection) {
-            var x = getComplexity(samples, i);
-            var y = getFrameLength(samples, i);
-            var xx = x * x;
-            var yx = y * x;
-            var yy = y * y;
-            // a1, b1, etc. is sum from startIndex to i, inclusive
-            a1 += 1;
-            b1 += x;
-            c1 += xx;
-            d1 += y;
-            h1 += yx;
-            k1 += yy;
-            // a2, b2, etc. is sum from i+1 to endIndex, inclusive
-            a2 -= 1;
-            b2 -= x;
-            c2 -= xx;
-            d2 -= y;
-            h2 -= yx;
-            k2 -= yy;
-
-            var A = c1*d1 - b1*h1;
-            var B = a1*h1 - b1*d1;
-            var C = a1*c1 - b1*b1;
-            var D = c2*d2 - b2*h2;
-            var E = a2*h2 - b2*d2;
-            var F = a2*c2 - b2*b2;
-            var s1 = options.s1 !== undefined ? options.s1 : (A / C);
-            var t1 = options.t1 !== undefined ? options.t1 : (B / C);
-            var s2 = options.s2 !== undefined ? options.s2 : (D / F);
-            var t2 = options.t2 !== undefined ? options.t2 : (E / F);
-            // Assumes that the two segments meet
-            var x_prime = (s1 - s2) / (t2 - t1);
-
-            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
-            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
-
-            if (i == startIndex) {
-                setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
-                continue;
-            }
-
-            // Projected point is not between this and the next sample
-            if (x_prime > getComplexity(samples, i + iterationDirection) || x_prime < x) {
-                // Calculate lambda, which divides the weight of this sample between the two lines
-
-                // These values remove the influence of this sample
-                var I = c1 - 2*b1*x + a1*xx;
-                var H = C - I;
-                var G = A + B*x - C*y;
-
-                var J = D + E*x - F*y;
-                var K = c2 - 2*b2*x + a2*xx;
-
-                var lambda = (G*F + G*K - H*J) / (I*J + G*K);
-                if (lambda > 0 && lambda < 1) {
-                    var lambda1 = 1 - lambda;
-                    s1 = options.s1 !== undefined ? options.s1 : ((A - lambda1*(-h1*x + d1*xx + c1*y - b1*yx)) / (C - lambda1*I));
-                    t1 = options.t1 !== undefined ? options.t1 : ((B - lambda1*(h1 - d1*x - b1*y + a1*yx)) / (C - lambda1*I));
-                    s2 = options.s2 !== undefined ? options.s2 : ((D + lambda1*(-h2*x + d2*xx + c2*y - b2*yx)) / (F + lambda1*K));
-                    t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
-                    x_prime = (s1 - s2) / (t2 - t1);
-
-                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
-                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
-                }
-            }
-
-            if (error1 + error2 < error_best)
-                setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
-        }
-
-        return {
-            complexity: x_best,
-            s1: s1_best,
-            t1: t1_best,
-            s2: s2_best,
-            t2: t2_best,
-            error: error_best
-        };
     }
 });
 
@@ -430,7 +315,7 @@ RampController = Utilities.createSubclass(Controller,
 
         // Initially start with a tier test to find the bounds
         // The number of objects in a tier test is 10^|_tier|
-        this._tier = 0;
+        this._tier = -.5;
         // The timestamp is first set after the first interval completes
         this._tierStartTimestamp = 0;
         // If the engine can handle the tier's complexity at 60 FPS, test for a short
@@ -439,11 +324,11 @@ RampController = Utilities.createSubclass(Controller,
         // If the engine is under stress, let the test run a little longer to let
         // the measurement settle
         this._tierSlowTestLength = 750;
+        this._minimumComplexity = 0;
         this._maximumComplexity = 0;
-        this._minimumTier = 0;
 
         // After the tier range is determined, figure out the number of ramp iterations
-        var minimumRampLength = 3000;
+        var minimumRampLength = 2500;
         var totalRampIterations = Math.max(1, Math.floor(this._endTimestamp / minimumRampLength));
         // Give a little extra room to run since the ramps won't be exactly this length
         this._rampLength = Math.floor((this._endTimestamp - totalRampIterations * this._intervalLength) / totalRampIterations);
@@ -455,10 +340,14 @@ RampController = Utilities.createSubclass(Controller,
         this._fps60Threshold = 1000/58;
         // We are looking for the complexity that will get us at least as slow this threshold
         this._fpsLowestThreshold = 1000/30;
+        // Try to make each ramp get this slow so that we can cross the break point
+        this._fpsRampSlowThreshold = 1000/45;
 
         this._finishedTierSampling = false;
-        this._startedRamps = false;
-        this._complexityPrime = new Experiment;
+        this._changePointEstimator = new Experiment;
+        this._minimumComplexityEstimator = new Experiment;
+        // Estimates all frames within an interval
+        this._intervalFrameLengthEstimator = new Experiment;
     }, {
 
     start: function(startTimestamp, stage)
@@ -479,38 +368,60 @@ RampController = Utilities.createSubclass(Controller,
                 var isAnimatingAt60FPS = currentFrameLength < this._fps60Threshold;
                 var hasFinishedSlowTierTest = timestamp > this._tierStartTimestamp + this._tierSlowTestLength;
 
+                if (!isAnimatingAt60FPS && !hasFinishedSlowTierTest)
+                    return;
+
                 // We're measuring at 60 fps, so quickly move on to the next tier, or
                 // we've slower than 60 fps, but we've let this tier run long enough to
                 // get an estimate
-                if (currentFrameLength < this._fps60Threshold || timestamp > this._tierStartTimestamp + this._tierSlowTestLength) {
-                    this._lastComplexity = currentComplexity;
-                    this._lastFrameLength = currentFrameLength;
+                this._lastTierComplexity = currentComplexity;
+                this._lastTierFrameLength = currentFrameLength;
 
+                this._tier += .5;
+                var nextTierComplexity = Math.round(Math.pow(10, this._tier));
+                stage.tune(nextTierComplexity - currentComplexity);
+
+                // Some tests may be unable to go beyond a certain capacity. If so, don't keep moving up tiers
+                if (stage.complexity() - currentComplexity > 0 || nextTierComplexity == 1) {
                     this._tierStartTimestamp = timestamp;
-                    this._tier += .5;
-                    var nextTierComplexity = Math.round(Math.pow(10, this._tier));
                     this.mark("Complexity: " + nextTierComplexity, timestamp);
-
-                    stage.tune(nextTierComplexity - currentComplexity);
+                    return;
                 }
-                return;
             } else if (timestamp < this._tierStartTimestamp + this._tierSlowTestLength)
                 return;
 
             this._finishedTierSampling = true;
+            this.isFrameLengthEstimatorEnabled = false;
+
             // Extend the test length so that the full test length is made of the ramps
             this._endTimestamp += timestamp;
             this.mark(Strings.json.samplingStartTimeOffset, timestamp);
 
+            this._minimumComplexity = 0;
+            this._possibleMinimumComplexity = this._minimumComplexity;
+            this._minimumComplexityEstimator.sample(this._minimumComplexity);
+
             // Sometimes this last tier will drop the frame length well below the threshold
             // Avoid going down that far since it means fewer measurements are taken in the 60 fps area
             // Interpolate a maximum complexity that gets us around the lowest threshold
-            this._maximumComplexity = Math.floor(this._lastComplexity + (this._fpsLowestThreshold - this._lastFrameLength) / (currentFrameLength - this._lastFrameLength) * (currentComplexity - this._lastComplexity));
+            if (this._lastTierComplexity != currentComplexity)
+                this._maximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this._fpsLowestThreshold, this._lastTierFrameLength, currentFrameLength), this._lastTierComplexity, currentComplexity));
+            else {
+                // If the browser is capable of handling the most complex version of the test, use that
+                this._maximumComplexity = currentComplexity;
+            }
+            this._possibleMaximumComplexity = this._maximumComplexity;
+
+            // If we get ourselves onto a ramp where the maximum complexity does not yield slow enough FPS,
+            // We'll use this as a boundary to find a higher maximum complexity for the next ramp
+            this._lastTierComplexity = currentComplexity;
+            this._lastTierFrameLength = currentFrameLength;
+
+            // First ramp
             stage.tune(this._maximumComplexity - currentComplexity);
-            this._rampStartTimestamp = timestamp;
             this._rampDidWarmup = false;
-            this.isFrameLengthEstimatorEnabled = false;
-            this._intervalCount = 0;
+            // Start timestamp represents start of ramp iteration and warm up
+            this._rampStartTimestamp = timestamp;
             return;
         }
 
@@ -527,15 +438,34 @@ RampController = Utilities.createSubclass(Controller,
         this._rampStartIndex = this._sampler.sampleCount;
     },
 
-    tune: function(timestamp, stage, didFinishInterval)
+    tune: function(timestamp, stage, lastFrameLength, didFinishInterval, intervalAverageFrameLength)
     {
-        if (!didFinishInterval || !this._rampDidWarmup)
+        if (!this._rampDidWarmup)
+            return;
+
+        this._intervalFrameLengthEstimator.sample(lastFrameLength);
+        if (!didFinishInterval)
             return;
 
+        var currentComplexity = stage.complexity();
+        var intervalFrameLengthMean = this._intervalFrameLengthEstimator.mean();
+        var intervalFrameLengthStandardDeviation = this._intervalFrameLengthEstimator.standardDeviation();
+
+        if (intervalFrameLengthMean < this._fps60Threshold && this._intervalFrameLengthEstimator.cdf(this._fps60Threshold) > .95) {
+            this._possibleMinimumComplexity = Math.max(this._possibleMinimumComplexity, currentComplexity);
+        } else if (intervalFrameLengthStandardDeviation > 2) {
+            // In the case where we might have found a previous interval where 60fps was reached. We hit a significant blip,
+            // so we should resample this area in the next ramp.
+            this._possibleMinimumComplexity = 0;
+        }
+        if (intervalFrameLengthMean - intervalFrameLengthStandardDeviation > this._fpsRampSlowThreshold)
+            this._possibleMaximumComplexity = Math.min(this._possibleMaximumComplexity, currentComplexity);
+        this._intervalFrameLengthEstimator.reset();
+
         var progress = (timestamp - this._rampStartTimestamp) / this._currentRampLength;
 
         if (progress < 1) {
-            stage.tune(Math.round((1 - progress) * this._maximumComplexity) - stage.complexity());
+            stage.tune(Math.floor(Utilities.lerp(progress, this._maximumComplexity, this._minimumComplexity)) - currentComplexity);
             return;
         }
 
@@ -543,14 +473,28 @@ RampController = Utilities.createSubclass(Controller,
             this._sampler.sampleCount - 1, this._rampStartIndex);
         this._rampRegressions.push(regression);
 
-        this._complexityPrime.sample(regression.complexity);
-        this._maximumComplexity = Math.max(5, Math.round(this._complexityPrime.mean() * 2));
+        var interpolatedFrameLength = regression.valueAt(this._maximumComplexity);
+        if (interpolatedFrameLength < this._fpsRampSlowThreshold)
+            this._possibleMaximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this._fpsRampSlowThreshold, interpolatedFrameLength, this._lastTierFrameLength), this._maximumComplexity, this._lastTierComplexity));
+
+        interpolatedFrameLength = regression.valueAt(this._minimumComplexity);
+        this._minimumComplexityEstimator.sample(this._possibleMinimumComplexity);
+
+        this._changePointEstimator.sample(regression.complexity);
+
+        this._minimumComplexity = Math.round(this._minimumComplexityEstimator.mean());
+        this._maximumComplexity = Math.round(this._minimumComplexity +
+            Math.max(5,
+                this._possibleMaximumComplexity - this._minimumComplexity,
+                (this._changePointEstimator.mean() - this._minimumComplexity) * 2));
 
         // Next ramp
+        stage.tune(this._maximumComplexity - stage.complexity());
         this._rampDidWarmup = false;
         // Start timestamp represents start of ramp iteration and warm up
         this._rampStartTimestamp = timestamp;
-        stage.tune(this._maximumComplexity - stage.complexity());
+        this._possibleMinimumComplexity = 0;
+        this._possibleMaximumComplexity = this._maximumComplexity;
     },
 
     _getComplexity: function(samples, i) {
@@ -567,26 +511,29 @@ RampController = Utilities.createSubclass(Controller,
 
         // Have samplingTimeOffset represent time 0
         var startTimestamp = this._marks[Strings.json.samplingStartTimeOffset].time;
-        results[Strings.json.samples].forEach(function(sample) {
-            sample.time -= startTimestamp;
-        });
+
         for (var markName in results[Strings.json.marks]) {
             results[Strings.json.marks][markName].time -= startTimestamp;
         }
 
-        var samples = results[Strings.json.samples];
-        results[Strings.json.regressions.timeRegressions] = [];
-        var complexityRegressionSamples = [];
-        var timeComplexityScore = new Experiment;
+        var timeSamples = results[Strings.json.samples][Strings.json.controller];
+        timeSamples.forEach(function(timeSample) {
+            timeSample.time -= startTimestamp;
+        });
+
+        // Aggregate all of the ramps into one big complexity-frameLength dataset
+        var complexitySamples = [], complexityAverageSamples = [];
+        results[Strings.json.samples][Strings.json.complexity] = complexitySamples;
+        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
+
+        results[Strings.json.controller] = [];
         this._rampRegressions.forEach(function(ramp) {
             var startIndex = ramp.startIndex, endIndex = ramp.endIndex;
-            var startTime = samples[startIndex].time, endTime = samples[endIndex].time;
-            var startComplexity = samples[startIndex].complexity, endComplexity = samples[endIndex].complexity;
-
-            timeComplexityScore.sample(ramp.complexity);
+            var startTime = timeSamples[startIndex].time, endTime = timeSamples[endIndex].time;
+            var startComplexity = timeSamples[startIndex].complexity, endComplexity = timeSamples[endIndex].complexity;
 
             var regression = {};
-            results[Strings.json.regressions.timeRegressions].push(regression);
+            results[Strings.json.controller].push(regression);
 
             var percentage = (ramp.complexity - startComplexity) / (endComplexity - startComplexity);
             var inflectionTime = startTime + percentage * (endTime - startTime);
@@ -599,78 +546,14 @@ RampController = Utilities.createSubclass(Controller,
                 [inflectionTime, ramp.s1 + ramp.t1 * ramp.complexity],
                 [endTime, ramp.s1 + ramp.t1 * endComplexity]
             ];
-            regression[Strings.json.regressions.complexity] = ramp.complexity;
-            regression[Strings.json.regressions.maxComplexity] = Math.max(startComplexity, endComplexity);
+            regression[Strings.json.complexity] = ramp.complexity;
             regression[Strings.json.regressions.startIndex] = startIndex;
             regression[Strings.json.regressions.endIndex] = endIndex;
 
             for (var j = startIndex; j <= endIndex; ++j)
-                complexityRegressionSamples.push(samples[j]);
-        });
-
-        // Aggregate all of the ramps into one big dataset and calculate a regression from this
-        complexityRegressionSamples.sort(function(a, b) {
-            return a.complexity - b.complexity;
-        });
-
-        // Samples averaged based on complexity
-        results[Strings.json.complexityAverageSamples] = [];
-        var currentComplexity = -1;
-        var experimentAtComplexity;
-        function addSample() {
-            results[Strings.json.complexityAverageSamples].push({
-                complexity: currentComplexity,
-                frameLength: experimentAtComplexity.mean(),
-                stdev: experimentAtComplexity.standardDeviation(),
-            });
-        }
-        complexityRegressionSamples.forEach(function(sample) {
-            if (sample.complexity != currentComplexity) {
-                if (currentComplexity > -1)
-                    addSample();
-
-                currentComplexity = sample.complexity;
-                experimentAtComplexity = new Experiment;
-            }
-            experimentAtComplexity.sample(sample.frameLength);
+                complexitySamples.push(timeSamples[j]);
         });
-        // Finish off the last one
-        addSample();
-
-        function calculateRegression(samples, key) {
-            var complexityRegression = new Regression(
-                samples,
-                function (samples, i) { return samples[i].complexity; },
-                function (samples, i) { return samples[i].frameLength; },
-                0, samples.length - 1
-            );
-            var minComplexity = samples[0].complexity;
-            var maxComplexity = samples[samples.length - 1].complexity;
-            var regression = {};
-            results[key] = regression;
-            regression[Strings.json.regressions.segment1] = [
-                [minComplexity, complexityRegression.s1 + complexityRegression.t1 * minComplexity],
-                [complexityRegression.complexity, complexityRegression.s1 + complexityRegression.t1 * complexityRegression.complexity]
-            ];
-            regression[Strings.json.regressions.segment2] = [
-                [complexityRegression.complexity, complexityRegression.s2 + complexityRegression.t2 * complexityRegression.complexity],
-                [maxComplexity, complexityRegression.s2 + complexityRegression.t2 * maxComplexity]
-            ];
-            regression[Strings.json.regressions.complexity] = complexityRegression.complexity;
-            regression[Strings.json.measurements.stdev] = Math.sqrt(complexityRegression.error / samples.length);
-        }
-
-        calculateRegression(complexityRegressionSamples, Strings.json.regressions.complexityRegression);
-        calculateRegression(results[Strings.json.complexityAverageSamples], Strings.json.regressions.complexityAverageRegression);
-
-        // Frame rate experiment result is unneeded
-        delete results[Strings.json.experiments.frameRate];
-
-        results[Strings.json.score] = timeComplexityScore.mean();
-        results[Strings.json.experiments.complexity] = {};
-        results[Strings.json.experiments.complexity][Strings.json.measurements.average] = timeComplexityScore.mean();
-        results[Strings.json.experiments.complexity][Strings.json.measurements.stdev] = timeComplexityScore.standardDeviation();
-        results[Strings.json.experiments.complexity][Strings.json.measurements.percent] = timeComplexityScore.percentage();
+        this._processComplexitySamples(complexitySamples, complexityAverageSamples);
     }
 });
 
@@ -722,17 +605,17 @@ Stage = Utilities.createClass(
 Utilities.extendObject(Stage, {
     random: function(min, max)
     {
-        return (Math.random() * (max - min)) + min;
+        return (Pseudo.random() * (max - min)) + min;
     },
 
     randomBool: function()
     {
-        return !!Math.round(Math.random());
+        return !!Math.round(Pseudo.random());
     },
 
     randomSign: function()
     {
-        return Math.random() >= .5 ? 1 : -1;
+        return Pseudo.random() >= .5 ? 1 : -1;
     },
 
     randomInt: function(min, max)
index bcbe902..9c2706e 100644 (file)
@@ -1,7 +1,6 @@
-SimpleKalmanEstimator = Utilities.createClass(
+SimpleKalmanEstimator = Utilities.createSubclass(Experiment,
     function(processError, measurementError) {
-        this._initialized = false;
-
+        Experiment.call(this, false);
         var error = .5 * (Math.sqrt(processError * processError + 4 * processError * measurementError) - processError);
         this._gain = error / (error + measurementError);
     }, {
@@ -10,16 +9,18 @@ SimpleKalmanEstimator = Utilities.createClass(
     {
         if (!this._initialized) {
             this._initialized = true;
-            this._estimatedMeasurement = newMeasurement;
+            this.estimate = newMeasurement;
             return;
         }
 
-        this._estimatedMeasurement = this._estimatedMeasurement + this._gain * (newMeasurement - this._estimatedMeasurement);
+        this.estimate = this.estimate + this._gain * (newMeasurement - this.estimate);
     },
 
-    get estimate()
+    reset: function()
     {
-        return this._estimatedMeasurement;
+        Experiment.prototype.reset.call(this);
+        this._initialized = false;
+        this.estimate = 0;
     }
 });
 
diff --git a/PerformanceTests/Animometer/tests/resources/sampler.js b/PerformanceTests/Animometer/tests/resources/sampler.js
deleted file mode 100644 (file)
index 56335b1..0000000
+++ /dev/null
@@ -1,88 +0,0 @@
-Experiment = Utilities.createClass(
-    function()
-    {
-        this._sum = 0;
-        this._squareSum = 0;
-        this._numberOfSamples = 0;
-        this._maxHeap = Algorithm.createMaxHeap(Experiment.defaults.CONCERN_SIZE);
-    }, {
-
-    sample: function(value)
-    {
-        this._sum += value;
-        this._squareSum += value * value;
-        this._maxHeap.push(value);
-        ++this._numberOfSamples;
-    },
-
-    mean: function()
-    {
-        return Statistics.sampleMean(this._numberOfSamples, this._sum);
-    },
-
-    standardDeviation: function()
-    {
-        return Statistics.unbiasedSampleStandardDeviation(this._numberOfSamples, this._sum, this._squareSum);
-    },
-
-    percentage: function()
-    {
-        var mean = this.mean();
-        return mean ? this.standardDeviation() * 100 / mean : 0;
-    },
-
-    concern: function(percentage)
-    {
-        var size = Math.ceil(this._numberOfSamples * percentage / 100);
-        var values = this._maxHeap.values(size);
-        return values.length ? values.reduce(function(a, b) { return a + b; }) / values.length : 0;
-    },
-
-    score: function(percentage)
-    {
-        return Statistics.geometricMean([this.mean(), Math.max(this.concern(percentage), 1)]);
-    }
-});
-
-Experiment.defaults =
-{
-    CONCERN: 5,
-    CONCERN_SIZE: 100,
-}
-
-Sampler = Utilities.createClass(
-    function(seriesCount, expectedSampleCount, processor)
-    {
-        this._processor = processor;
-
-        this.samples = [];
-        for (var i = 0; i < seriesCount; ++i) {
-            var array = new Array(expectedSampleCount);
-            array.fill(0);
-            this.samples[i] = array;
-        }
-        this.sampleCount = 0;
-    }, {
-
-    record: function() {
-        // Assume that arguments.length == this.samples.length
-        for (var i = 0; i < arguments.length; i++) {
-            this.samples[i][this.sampleCount] = arguments[i];
-        }
-        ++this.sampleCount;
-    },
-
-    processSamples: function()
-    {
-        var results = {};
-
-        // Remove unused capacity
-        this.samples = this.samples.map(function(array) {
-            return array.slice(0, this.sampleCount);
-        }, this);
-
-        this._processor.processSamples(results);
-
-        return results;
-    }
-});
index f267820..5bb69bc 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="../master/resources/canvas-stage.js"></script>
index 5c5b4c8..c7c0fef 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/tiled-canvas-image.js"></script>
index 11ddfaa..dcb7f52 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/template-canvas.js"></script>
index b2ba479..70eb214 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/template-css.js"></script>
index 425ec36..7a14fc4 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <svg id="stage"></svg>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/template-svg.js"></script>
index 4180bba..d7abc53 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         .text-layer {
@@ -24,8 +25,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/layering-text.js"></script>
index 828ca71..6c489c3 100644 (file)
@@ -1,7 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
-    <meta charset="UTF-8">
+    <meta charset="utf-8">
     <style>
         .text-layer {
             position: absolute;
@@ -52,8 +52,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="../bouncing-particles/resources/bouncing-particles.js"></script>
index 01b7e0e..6029ac4 100644 (file)
@@ -1,3 +1,286 @@
+2016-02-24  Jon Lee  <jonlee@apple.com>
+
+        Update animation benchmark and tests
+        https://bugs.webkit.org/show_bug.cgi?id=154673
+
+        Reviewed by Dean Jackson.
+
+        Update the ramp controller.
+
+        The controller refines the complexity interval to test across.
+
+        * Animometer/resources/statistics.js: Add functions that estimate cumulative distribution function.
+        (Regression): For the flat regression, force the first segment to be at 60 fps.
+        (valueAt): Add convenience function to return interpolated value based on the regression used.
+        (_calculateRegression): Include the number of points included for both segments, and the piecewise
+        errors.
+        * Animometer/tests/resources/math.js: Make the Kalman estimator subclass Experiment, and allow it
+        to be reset.
+
+        * Animometer/tests/resources/main.js: Initialize the tier such that it starts at 10^0 = 1.
+        Increase the number of ramps. Maintain three FPS thresholds-- the frame rate of interest, a limit
+        on the lowest FPS we care to go for later interpolation, and a minimum FPS threshold we want to
+        aim for each ramp. Also keep three estimators: a running average of the change point, a minimum
+        boundary for each ramp, and an estimator for all the frames within an interval. The first two
+        are used to determine the parameters of the next ramp, and the latter allows us to refine the
+        parameters.
+        (update): During the tier phase, it is possible that the highest complexity possible for a test
+        won't stress the system enough to trigger stopping the tier phase and transitioning to the ramps.
+        If the complexity doesn't change when going to the next tier, we've maxed the test out, and move
+        on. When the tier phase completed, turn off Controller.frameLengthEstimator, which estimates the
+        FPS at each tier.
+        (tune): At each interval, look at the confidence distribution of being on the 60 FPS side or the
+        slow side. If the slowest FPS we achieve at the ramp's maximum complexity is not at least
+        _fpsRampSlowThreshold, then increase the maximum complexity. If we ever achieve 60 FPS, increase
+        the ramp's minimum complexity to that level. If, at an even lower complexity, a glitch causes the
+        FPS to drop, we reset the minimum complexity.
+
+        Have the bootstrap calculation occur between tests. Clean up harness.
+
+        * Animometer/resources/debug-runner/animometer.js: Run bootstrap after a test has
+        completed to avoid doing all of it at the end before showing the results. Clean up
+        parameters being passed around.
+        * Animometer/resources/debug-runner/tests.js:
+        (text):
+        * Animometer/resources/runner/animometer.js:
+        (this._processData.calculateScore): Save the results to the same object holding the data.
+        (this._processData._processData): In the case where a file is dragged, calculate the score
+        serially. Grab the results object and move it to the results variable and remove it from
+        the data object. This avoids serializing the results into the JSON.
+        (this._processData.findRegression): Include the samples used for bootstrapping. Reduce the
+        resample size to shorten the wait.
+        * Animometer/resources/runner/benchmark-runner.js:
+        * Animometer/resources/statistics.js:
+        (bootstrap): Update how bootstrapData is sorted. In some regression results the mix of
+        floats and integers causes an alphabetical sort to occur.
+        * Animometer/resources/strings.js:
+
+        Add meta charset so that encodings between harness and test match.
+
+        * Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
+        * Animometer/tests/master/canvas-stage.html:
+        * Animometer/tests/master/focus.html:
+        * Animometer/tests/master/image-data.html:
+        * Animometer/tests/master/multiply.html:
+        * Animometer/tests/master/particles.html:
+        * Animometer/tests/misc/canvas-electrons.html:
+        * Animometer/tests/misc/canvas-stars.html:
+        * Animometer/tests/misc/compositing-transforms.html:
+        * Animometer/tests/simple/simple-canvas-paths.html:
+        * Animometer/tests/simple/tiled-canvas-image.html:
+        * Animometer/tests/template/template-canvas.html:
+        * Animometer/tests/template/template-css.html:
+        * Animometer/tests/template/template-svg.html:
+        * Animometer/tests/text/layering-text.html:
+        * Animometer/tests/text/text-boxes.html:
+
+        Update test harness reporting.
+
+        * Animometer/developer.html: Add missing meta charset.
+        * Animometer/index.html: Remove unnecessary utf-8 declaration.
+        * Animometer/resources/debug-runner/animometer.css: Add convenience classes for
+        formatting the results table.
+        * Animometer/resources/debug-runner/animometer.js: Adjust which stats are shown.
+        * Animometer/resources/debug-runner/tests.js: Display bootstrapping statistics.
+        * Animometer/resources/strings.js: Move strings not used by the release harness.
+
+        Switch to a pseudo-random number generator.
+
+        * Animometer/resources/statistics.js: Add a Pseudo class, with a simple
+        pseudo-random number generator.
+        (_calculateRegression): Reset the generator before running bootstrap.
+        (bootstrap): Deleted.
+
+        Replace Math.random with Pseudo.random.
+        * Animometer/tests/master/resources/canvas-tests.js:
+        * Animometer/tests/master/resources/focus.js:
+        * Animometer/tests/master/resources/particles.js:
+        * Animometer/tests/resources/main.js:
+
+        Use bootstrapping to get confidence interval in the breakpoint.
+
+        For the ramp controller, calculate the piecewise regression, and then use
+        bootstrapping in order to find the 95% confidence interval. Use the raw data.
+
+        * Animometer/developer.html: Default to the complexity graph. Add a legend
+        checkbox to toggle visibility of the bootstrap score and histogram.
+        * Animometer/resources/debug-runner/animometer.css: Make some more space to show
+        the old raw and average scores in the legend. Add new styles for the data.
+        * Animometer/resources/debug-runner/graph.js:
+        (_addRegressionLine): Allow passing an array for the variance bar tied to the
+        regression line. Now |stdev| is |range|.
+        (createComplexityGraph): Add bootstrap median, and overlay a histogram of
+        the bootstrap samples. Switch raw samples from circles to X's.
+        (onComplexityGraphOptionsChanged): Allow toggling of the bootstrap data.
+        (onGraphTypeChanged): Move the regressions for the raw and average samples to the
+        legend. In the subtitle use the bootstrap median, and include the 95% confidence
+        interval.
+        * Animometer/resources/runner/animometer.js:
+        (this._processData.findRegression): Factor out the code that determines which
+        samples to include when calculating the piecewise regression. For series that have
+        many samples, or a wider range of recorded complexities, throw away the 2.5%
+        lowest and highest samples before calculating the regression. Keep all samples
+        if the number of samples to regress is small or the range of complexities is
+        narrow.
+        (this._processData._calculateScore): Factor out regression calculation to
+        findRegression(). Bootstrap the change point of the regression. The score is the
+        median.
+        * Animometer/resources/statistics.js:
+        (_calculateRegression): Correct an issue in the calculation of the regression, where
+        the denominator can be 0.
+        (bootstrap): Template for bootstrapping. Create a bootstrap sample array, Create
+        re-samples by random selection with replacement. Return the 95% confidence samples,
+        the bootstrap median, mean, and the data itself.
+        * Animometer/resources/strings.js: Add bootstrap.
+        * Animometer/tests/resources/main.js:
+        (processSamples): Don't prematurely cut the sample data.
+
+        Fix graph drawing.
+
+        * Animometer/resources/debug-runner/animometer.js: Add spacing in the JSON output.
+        Multiple tests output a lot of JSON and can hang when selecting JSON with no whitespace.
+        * Animometer/resources/debug-runner/animometer.css:
+        (#complexity-graph .series.raw circle): Update the color.
+        * Animometer/resources/debug-runner/graph.js: Use the FPS axis instead of the
+        complexity axis, which can vary in domain. For determining the complexity domain,
+        only use samples after samplingTimeOffset.
+
+        Allow dropping results JSON.
+
+        * Animometer/developer.html: Add a button.
+        * Animometer/resources/debug-runner/animometer.css:
+        * Animometer/resources/debug-runner/animometer.js: Read the data and go straight
+        to the dashboard. With JSON output, write out only the options and the raw data.
+
+        Teach the harness to evaluate the samples and determine the test score.
+
+        This will allow us to update how the score is calculated separately from the samples recorded.
+        This also prepares the harness to be able to accept JSON of prior runs.
+
+        * Animometer/resources/strings.js: Clean up and remove unneeded strings and reduce some of the
+        hierarchy.
+        * Animometer/resources/debug-runner/tests.js: Update to use the new strings.
+
+        * Animometer/tests/resources/main.js: Allow all controllers to show a complexity-FPS graph.
+        (_processComplexitySamples): Factor out some of the sample processing done in the ramp
+        controller for the benefit of the other controllers. |complexitySamples| contains a list of
+        samples. Sort the samples by complexity. Optionally remove the top x% of samples.
+        Group them, and calculate distribution of samples within the same complexity, and add those as
+        new entries into |complexityAverageSamples|.
+        (Controller.processSamples): Move the code responsible for determining the complexity and FPS
+        scores out to ResultsDashboard. The structure of the data returned by the controller is:
+
+        {
+            controller: [time-regression, time-regression, ...], // optional, data specific to controller
+            marks: [...],
+            samples: {                    // all of the sample data
+                controller: [...],
+                complexity: [...],        // processed from controller samples
+                complexityAverage: [...], // processed from complexity samples
+            }
+        }
+
+        (AdaptiveController.processSamples): Adding the target frame length is no longer necessary; we
+        now pass the test options to the graph.
+        (Regression): Move to statistics.js.
+        * Animometer/resources/statistics.js: Move Regression to here. Add a check if the sampling range
+        only contains one sample, since we cannot calculate a regression from one sample point.
+
+        Teach the test harness to evaluate the data.
+        * Animometer/resources/runner/animometer.js:
+        (ResultsDashboard): Store the options used to run the test and the computed results/score separately
+        from the data. The results are stored as:
+
+        {
+            score: /* geomean of iteration score */,
+            iterationsResults: [
+                {
+                    score: /* geomean of tests */,
+                    testsResults: {
+                        suiteName: {
+                            testName: {
+                                controller: {
+                                    average:
+                                    concern:
+                                    stdev:
+                                    percent:
+                                },
+                                frameLength: { ... },
+                                complexity: {
+                                    complexity:
+                                    stdev:
+                                    segment1:
+                                    segment2:
+                                },
+                                complexityAverage: { ... }
+                            },
+                            testName: { ... },
+                        },
+                        ... next suite ...
+                    }
+                },
+                { ...next iteration... }
+            ]
+        }
+
+        * Animometer/resources/debug-runner/animometer.js: Pass options around instead of relying
+        on what was selected in the form. This will later allow for dropping previous results, and
+        using those runs' options when calculating scores.
+        (ResultsTable._addGraphButton): Simplify button action by using attached test data.
+        * Animometer/resources/debug-runner/graph.js: Refactor to use the data.
+
+        Consolidate JS files, and move statistics out to a separate JS.
+
+        Preparation for having the Controller only handle recording and storage of the samples,
+        and leave the evaluation of the test score out to the harness. Move Experiment to
+        a new statistics.js, where Regression will also eventually go. Get rid of algorithm.js
+        and move it to utilities.js since the Heap is used only for Experiments.
+
+        * Animometer/tests/resources/algorithm.js: Removed. Heap is in utilities.js.
+        * Animometer/tests/resources/sampler.js: Removed. Experiment is in statistics.js,
+        Sampler in main.js.
+        * Animometer/tests/resources/main.js: Move Sampler here.
+        * Animometer/resources/statistics.js: Added. Move Statistics and Experiment here.
+        * Animometer/resources/extensions.js: Move Heap here. Attach static method to create
+        a max or min heap to Heap, instead of a new Algorithm object.
+
+        Update JS files.
+        * Animometer/developer.html:
+        * Animometer/index.html:
+        * Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
+        * Animometer/tests/master/canvas-stage.html:
+        * Animometer/tests/master/focus.html:
+        * Animometer/tests/master/image-data.html:
+        * Animometer/tests/master/multiply.html:
+        * Animometer/tests/master/particles.html:
+        * Animometer/tests/misc/canvas-electrons.html:
+        * Animometer/tests/misc/canvas-stars.html:
+        * Animometer/tests/misc/compositing-transforms.html:
+        * Animometer/tests/simple/simple-canvas-paths.html:
+        * Animometer/tests/simple/tiled-canvas-image.html:
+        * Animometer/tests/template/template-canvas.html:
+        * Animometer/tests/template/template-css.html:
+        * Animometer/tests/template/template-svg.html:
+        * Animometer/tests/text/layering-text.html:
+        * Animometer/tests/text/text-boxes.html:
+
+        Fix the cursor in the graph analysis when the min
+        complexity is not 0.
+
+        * Animometer/resources/debug-runner/graph.js:
+        (_addRegression):
+        (createComplexityGraph):
+
 2016-02-23  Geoffrey Garen  <ggaren@apple.com>
 
         Fix some issues in MallocBench
         Added new capabilities to MallocBench.  These include:
             Added a recording of http://nim-lang.org/docs/lib.html.
             Added thread id to the recording and the ability to playback switching threads in MallocBench
-            Added aligned allocations to recordings and the ability to playback 
+            Added aligned allocations to recordings and the ability to playback
             Added --use-thread-id option to honor recorded thread ids
             Added --detailed-report to output remaining allocations by size after playback
             Added --no-warmup to not run the warm up iteration