Update animation benchmark and tests
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sat, 27 Feb 2016 04:32:09 +0000 (04:32 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sat, 27 Feb 2016 04:32:09 +0000 (04:32 +0000)
https://bugs.webkit.org/show_bug.cgi?id=154673

Reviewed by Dean Jackson.

Update the ramp controller.

The controller refines the complexity interval to test across.

* Animometer/resources/statistics.js: Add functions that estimate cumulative distribution function.
(Regression): For the flat regression, force the first segment to be at 60 fps.
(valueAt): Add convenience function to return interpolated value based on the regression used.
(_calculateRegression): Include the number of points included for both segments, and the piecewise
errors.
* Animometer/tests/resources/math.js: Make the Kalman estimator subclass Experiment, and allow it
to be reset.

* Animometer/tests/resources/main.js: Initialize the tier such that it starts at 10^0 = 1.
Increase the number of ramps. Maintain three FPS thresholds-- the frame rate of interest, a limit
on the lowest FPS we care to go for later interpolation, and a minimum FPS threshold we want to
aim for each ramp. Also keep three estimators: a running average of the change point, a minimum
boundary for each ramp, and an estimator for all the frames within an interval. The first two
are used to determine the parameters of the next ramp, and the latter allows us to refine the
parameters.
(update): During the tier phase, it is possible that the highest complexity possible for a test
won't stress the system enough to trigger stopping the tier phase and transitioning to the ramps.
If the complexity doesn't change when going to the next tier, we've maxed the test out, and move
on. When the tier phase completed, turn off Controller.frameLengthEstimator, which estimates the
FPS at each tier.
(tune): At each interval, look at the confidence distribution of being on the 60 FPS side or the
slow side. If the slowest FPS we achieve at the ramp's maximum complexity is not at least
_fpsRampSlowThreshold, then increase the maximum complexity. If we ever achieve 60 FPS, increase
the ramp's minimum complexity to that level. If, at an even lower complexity, a glitch causes the
FPS to drop, we reset the minimum complexity.

Have the bootstrap calculation occur between tests. Clean up harness.

* Animometer/resources/debug-runner/animometer.js: Run bootstrap after a test has
completed to avoid doing all of it at the end before showing the results. Clean up
parameters being passed around.
* Animometer/resources/debug-runner/tests.js:
(text):
* Animometer/resources/runner/animometer.js:
(this._processData.calculateScore): Save the results to the same object holding the data.
(this._processData._processData): In the case where a file is dragged, calculate the score
serially. Grab the results object and move it to the results variable and remove it from
the data object. This avoids serializing the results into the JSON.
(this._processData.findRegression): Include the samples used for bootstrapping. Reduce the
resample size to shorten the wait.
* Animometer/resources/runner/benchmark-runner.js:
* Animometer/resources/statistics.js:
(bootstrap): Update how bootstrapData is sorted. In some regression results the mix of
floats and integers causes an alphabetical sort to occur.
* Animometer/resources/strings.js:

Add meta charset so that encodings between harness and test match.

* Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
* Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-css-images.html:
* Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-svg-images.html:
* Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
* Animometer/tests/master/canvas-stage.html:
* Animometer/tests/master/focus.html:
* Animometer/tests/master/image-data.html:
* Animometer/tests/master/multiply.html:
* Animometer/tests/master/particles.html:
* Animometer/tests/misc/canvas-electrons.html:
* Animometer/tests/misc/canvas-stars.html:
* Animometer/tests/misc/compositing-transforms.html:
* Animometer/tests/simple/simple-canvas-paths.html:
* Animometer/tests/simple/tiled-canvas-image.html:
* Animometer/tests/template/template-canvas.html:
* Animometer/tests/template/template-css.html:
* Animometer/tests/template/template-svg.html:
* Animometer/tests/text/layering-text.html:
* Animometer/tests/text/text-boxes.html:

Update test harness reporting.

* Animometer/developer.html: Add missing meta charset.
* Animometer/index.html: Remove unnecessary utf-8 declaration.
* Animometer/resources/debug-runner/animometer.css: Add convenience classes for
formatting the results table.
* Animometer/resources/debug-runner/animometer.js: Adjust which stats are shown.
* Animometer/resources/debug-runner/tests.js: Display bootstrapping statistics.
* Animometer/resources/strings.js: Move strings not used by the release harness.

Switch to a pseudo-random number generator.

* Animometer/resources/statistics.js: Add a Pseudo class, with a simple
pseudo-random number generator.
(_calculateRegression): Reset the generator before running bootstrap.
(bootstrap): Deleted.

Replace Math.random with Pseudo.random.
* Animometer/tests/master/resources/canvas-tests.js:
* Animometer/tests/master/resources/focus.js:
* Animometer/tests/master/resources/particles.js:
* Animometer/tests/resources/main.js:

Use bootstrapping to get confidence interval in the breakpoint.

For the ramp controller, calculate the piecewise regression, and then use
bootstrapping in order to find the 95% confidence interval. Use the raw data.

* Animometer/developer.html: Default to the complexity graph. Add a legend
checkbox to toggle visibility of the bootstrap score and histogram.
* Animometer/resources/debug-runner/animometer.css: Make some more space to show
the old raw and average scores in the legend. Add new styles for the data.
* Animometer/resources/debug-runner/graph.js:
(_addRegressionLine): Allow passing an array for the variance bar tied to the
regression line. Now |stdev| is |range|.
(createComplexityGraph): Add bootstrap median, and overlay a histogram of
the bootstrap samples. Switch raw samples from circles to X's.
(onComplexityGraphOptionsChanged): Allow toggling of the bootstrap data.
(onGraphTypeChanged): Move the regressions for the raw and average samples to the
legend. In the subtitle use the bootstrap median, and include the 95% confidence
interval.
* Animometer/resources/runner/animometer.js:
(this._processData.findRegression): Factor out the code that determines which
samples to include when calculating the piecewise regression. For series that have
many samples, or a wider range of recorded complexities, throw away the 2.5%
lowest and highest samples before calculating the regression. Keep all samples
if the number of samples to regress is small or the range of complexities is
narrow.
(this._processData._calculateScore): Factor out regression calculation to
findRegression(). Bootstrap the change point of the regression. The score is the
median.
* Animometer/resources/statistics.js:
(_calculateRegression): Correct an issue in the calculation of the regression, where
the denominator can be 0.
(bootstrap): Template for bootstrapping. Create a bootstrap sample array, Create
re-samples by random selection with replacement. Return the 95% confidence samples,
the bootstrap median, mean, and the data itself.
* Animometer/resources/strings.js: Add bootstrap.
* Animometer/tests/resources/main.js:
(processSamples): Don't prematurely cut the sample data.

Fix graph drawing.

* Animometer/resources/debug-runner/animometer.js: Add spacing in the JSON output.
Multiple tests output a lot of JSON and can hang when selecting JSON with no whitespace.
* Animometer/resources/debug-runner/animometer.css:
(#complexity-graph .series.raw circle): Update the color.
* Animometer/resources/debug-runner/graph.js: Use the FPS axis instead of the
complexity axis, which can vary in domain. For determining the complexity domain,
only use samples after samplingTimeOffset.

Allow dropping results JSON.

* Animometer/developer.html: Add a button.
* Animometer/resources/debug-runner/animometer.css:
* Animometer/resources/debug-runner/animometer.js: Read the data and go straight
to the dashboard. With JSON output, write out only the options and the raw data.

Teach the harness to evaluate the samples and determine the test score.

This will allow us to update how the score is calculated separately from the samples recorded.
This also prepares the harness to be able to accept JSON of prior runs.

* Animometer/resources/strings.js: Clean up and remove unneeded strings and reduce some of the
hierarchy.
* Animometer/resources/debug-runner/tests.js: Update to use the new strings.

* Animometer/tests/resources/main.js: Allow all controllers to show a complexity-FPS graph.
(_processComplexitySamples): Factor out some of the sample processing done in the ramp
controller for the benefit of the other controllers. |complexitySamples| contains a list of
samples. Sort the samples by complexity. Optionally remove the top x% of samples.
Group them, and calculate distribution of samples within the same complexity, and add those as
new entries into |complexityAverageSamples|.
(Controller.processSamples): Move the code responsible for determining the complexity and FPS
scores out to ResultsDashboard. The structure of the data returned by the controller is:

{
    controller: [time-regression, time-regression, ...], // optional, data specific to controller
    marks: [...],
    samples: {                    // all of the sample data
        controller: [...],
        complexity: [...],        // processed from controller samples
        complexityAverage: [...], // processed from complexity samples
    }
}

(AdaptiveController.processSamples): Adding the target frame length is no longer necessary; we
now pass the test options to the graph.
(Regression): Move to statistics.js.
* Animometer/resources/statistics.js: Move Regression to here. Add a check if the sampling range
only contains one sample, since we cannot calculate a regression from one sample point.

Teach the test harness to evaluate the data.
* Animometer/resources/runner/animometer.js:
(ResultsDashboard): Store the options used to run the test and the computed results/score separately
from the data. The results are stored as:

{
    score: /* geomean of iteration score */,
    iterationsResults: [
        {
            score: /* geomean of tests */,
            testsResults: {
                suiteName: {
                    testName: {
                        controller: {
                            average:
                            concern:
                            stdev:
                            percent:
                        },
                        frameLength: { ... },
                        complexity: {
                            complexity:
                            stdev:
                            segment1:
                            segment2:
                        },
                        complexityAverage: { ... }
                    },
                    testName: { ... },
                },
                ... next suite ...
            }
        },
        { ...next iteration... }
    ]
}

* Animometer/resources/debug-runner/animometer.js: Pass options around instead of relying
on what was selected in the form. This will later allow for dropping previous results, and
using those runs' options when calculating scores.
(ResultsTable._addGraphButton): Simplify button action by using attached test data.
* Animometer/resources/debug-runner/graph.js: Refactor to use the data.

Consolidate JS files, and move statistics out to a separate JS.

Preparation for having the Controller only handle recording and storage of the samples,
and leave the evaluation of the test score out to the harness. Move Experiment to
a new statistics.js, where Regression will also eventually go. Get rid of algorithm.js
and move it to utilities.js since the Heap is used only for Experiments.

* Animometer/tests/resources/algorithm.js: Removed. Heap is in utilities.js.
* Animometer/tests/resources/sampler.js: Removed. Experiment is in statistics.js,
Sampler in main.js.
* Animometer/tests/resources/main.js: Move Sampler here.
* Animometer/resources/statistics.js: Added. Move Statistics and Experiment here.
* Animometer/resources/extensions.js: Move Heap here. Attach static method to create
a max or min heap to Heap, instead of a new Algorithm object.

Update JS files.
* Animometer/developer.html:
* Animometer/index.html:
* Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
* Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-css-images.html:
* Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
* Animometer/tests/bouncing-particles/bouncing-svg-images.html:
* Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
* Animometer/tests/master/canvas-stage.html:
* Animometer/tests/master/focus.html:
* Animometer/tests/master/image-data.html:
* Animometer/tests/master/multiply.html:
* Animometer/tests/master/particles.html:
* Animometer/tests/misc/canvas-electrons.html:
* Animometer/tests/misc/canvas-stars.html:
* Animometer/tests/misc/compositing-transforms.html:
* Animometer/tests/simple/simple-canvas-paths.html:
* Animometer/tests/simple/tiled-canvas-image.html:
* Animometer/tests/template/template-canvas.html:
* Animometer/tests/template/template-css.html:
* Animometer/tests/template/template-svg.html:
* Animometer/tests/text/layering-text.html:
* Animometer/tests/text/text-boxes.html:

Fix the cursor in the graph analysis when the min
complexity is not 0.

* Animometer/resources/debug-runner/graph.js:
(_addRegression):
(createComplexityGraph):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@197229 268f45cc-cd09-0410-ab3c-d52691b4dbfc

40 files changed:
PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/index.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/animometer.js
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/debug-runner/tests.js
PerformanceTests/Animometer/resources/extensions.js
PerformanceTests/Animometer/resources/runner/animometer.js
PerformanceTests/Animometer/resources/runner/benchmark-runner.js
PerformanceTests/Animometer/resources/statistics.js [new file with mode: 0644]
PerformanceTests/Animometer/resources/strings.js
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-canvas-images.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-css-images.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-css-shapes.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-svg-images.html
PerformanceTests/Animometer/tests/bouncing-particles/bouncing-svg-shapes.html
PerformanceTests/Animometer/tests/master/canvas-stage.html
PerformanceTests/Animometer/tests/master/focus.html
PerformanceTests/Animometer/tests/master/image-data.html
PerformanceTests/Animometer/tests/master/multiply.html
PerformanceTests/Animometer/tests/master/particles.html
PerformanceTests/Animometer/tests/master/resources/canvas-tests.js
PerformanceTests/Animometer/tests/master/resources/focus.js
PerformanceTests/Animometer/tests/master/resources/particles.js
PerformanceTests/Animometer/tests/misc/canvas-electrons.html
PerformanceTests/Animometer/tests/misc/canvas-stars.html
PerformanceTests/Animometer/tests/misc/compositing-transforms.html
PerformanceTests/Animometer/tests/resources/algorithm.js [deleted file]
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/Animometer/tests/resources/math.js
PerformanceTests/Animometer/tests/resources/sampler.js [deleted file]
PerformanceTests/Animometer/tests/simple/simple-canvas-paths.html
PerformanceTests/Animometer/tests/simple/tiled-canvas-image.html
PerformanceTests/Animometer/tests/template/template-canvas.html
PerformanceTests/Animometer/tests/template/template-css.html
PerformanceTests/Animometer/tests/template/template-svg.html
PerformanceTests/Animometer/tests/text/layering-text.html
PerformanceTests/Animometer/tests/text/text-boxes.html
PerformanceTests/ChangeLog

index e2ad5f36306bcdf5cdf64d27d70a030a2029ea1a..c0654685f827f55cb87ade872c8de0cf3f52d044 100644 (file)
@@ -2,11 +2,13 @@
 <html>
 <head>
     <title>Animometer - developer</title>
+    <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, user-scalable=no">
     <link rel="stylesheet" href="resources/runner/animometer.css">
     <link rel="stylesheet" href="resources/debug-runner/animometer.css">
     <script src="resources/strings.js"></script>
     <script src="resources/extensions.js"></script>
+    <script src="resources/statistics.js"></script>
 
     <script src="resources/runner/tests.js"></script>
     <script src="resources/debug-runner/tests.js" charset="utf-8"></script>
@@ -26,6 +28,7 @@
                 <div id="suites">
                     <h2>Suites:</h2>
                     <ul class="tree"></ul>
+                    <div><span id="drop-target">Drop results here</span></div>
                 </div>
                 <div id="options">
                     <h2>Options:</h2>
             <nav>
                 <form name="graph-type">
                     <ul>
-                        <li><label><input type="radio" name="graph-type" value="time" checked> Time graph</label></li>
-                        <li><label><input type="radio" name="graph-type" value="complexity"> Complexity graph</label></li>
+                        <li><label><input type="radio" name="graph-type" value="time"> Time graph</label></li>
+                        <li><label><input type="radio" name="graph-type" value="complexity" checked> Complexity graph</label></li>
                     </ul>
                 </form>
                 <form name="time-graph-options">
                 <form name="complexity-graph-options">
                     <ul class="series">
                         <li><label><input type="checkbox" name="series-raw" checked> Series raw</label></li>
-                        <li><label><input type="checkbox" name="series-average" checked> Series average</label></li>
+                        <li><label><input type="checkbox" name="series-average"> Series average</label></li>
                     </ul>
                     <ul>
-                        <li><label><input type="checkbox" name="regression-time-score" checked> Ramp regression score</label>
-                        <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label>
-                        <li><label><input type="checkbox" name="complexity-regression-aggregate-average" checked> Regression, series average</label>
+                        <li><label><input type="checkbox" name="regression-time-score"> Controller score</label></li>
+                        <li><label><input type="checkbox" name="bootstrap-score" checked> Bootstrap score and histogram</label></li>
+                        <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label><span id="complexity-regression-aggregate-raw"></span></li>
+                        <li><label><input type="checkbox" name="complexity-regression-aggregate-average"> Regression, series average</label><span id="complexity-regression-aggregate-average"></span></li>
                     </ul>
                 </form>
             </nav>
index 13fd05dc6dc3d286792ce747f7d990ca274b275e..9f7370ac72ca18683a84cbb24e816fed7246fcaf 100644 (file)
@@ -1,14 +1,16 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <title>Animometer</title>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, user-scalable=no">
     <link rel="stylesheet" href="resources/runner/animometer.css">
     <script src="resources/strings.js"></script>
     <script src="resources/extensions.js"></script>
+    <script src="resources/statistics.js"></script>
 
     <script src="resources/runner/tests.js"></script>
-    <script src="resources/runner/animometer.js" charset="utf-8"></script>
+    <script src="resources/runner/animometer.js"></script>
 
     <script src="resources/runner/benchmark-runner.js"></script>
 </head>
index d847ec8cb6e08f1ef8706e1c00c60ec6c3039fbb..e6e7bb61e8d5cd24011428dab5cb411d554d9a26 100644 (file)
@@ -171,6 +171,23 @@ label.tree-label {
     padding: 0 0 0 1em;
 }
 
+#suites > div {
+    margin-top: 3em;
+}
+
+#drop-target {
+    font-size: 1em;
+    border-radius: 10px;
+    padding: .5em 2em;
+    border: 2px solid rgb(235, 235, 235);
+    color: rgb(235, 235, 235);
+}
+
+#drop-target:hover {
+    background-color: rgba(255, 255, 255, .1);
+    cursor: pointer;
+}
+
 #options ul {
     margin: 0;
     padding: 0;
@@ -306,6 +323,26 @@ label.tree-label {
     padding-left: .25em;
 }
 
+#results-data .left {
+    text-align: left;
+}
+
+#results-data .right {
+    text-align: right;
+}
+
+#results-data .pad-left {
+    padding-left: 1em;
+}
+
+#results-data .pad-right {
+    padding-right: .25em;
+}
+
+#results-data .small {
+    font-size: .8em;
+}
+
 #results-tables td.noisy-results {
     color: rgb(255, 104, 104);
 }
@@ -357,7 +394,7 @@ label.tree-label {
     top: 1.5em;
     right: 0;
     font-size: .7em;
-    width: 23em;
+    width: 28em;
 }
 
 #test-graph nav ul {
@@ -520,11 +557,41 @@ label.tree-label {
     fill: rgba(253, 253, 253, .05);
 }
 
-#complexity-graph .series.average circle {
+#complexity-graph .raw.series line {
+    stroke: hsl(30, 96%, 56%);
+    stroke-width: 1px;
+}
+
+#complexity-graph .raw.regression line {
+    stroke: rgba(30, 96%, 86%, .6);
+}
+
+#complexity-graph .raw.regression polygon {
+    stroke: rgba(30, 96%, 86%, .05);
+}
+
+#complexity-graph .average.series circle {
     fill: hsl(170, 96%, 56%);
 }
 
-#complexity-graph .series.average line {
+#complexity-graph .average.series line {
     stroke: hsla(170, 96%, 56%, .2);
     stroke-width: 2px;
 }
+
+#complexity-graph .bootstrap .bar {
+    fill: hsla(260, 56%, 66%, .4);
+}
+
+#complexity-graph .bootstrap .median line {
+    stroke: hsla(300, 56%, 66%, .8);
+    stroke-width: 2px;
+}
+
+#complexity-graph .bootstrap .median circle {
+    fill: hsla(300, 56%, 66%, .8);
+}
+
+#complexity-graph .bootstrap .median polygon {
+    fill: hsla(300, 56%, 66%, .05);
+}
index e7635e2bcd9b974dec74f6d7e704477cc3d6ab61..4033379f0197731deec942d7553989474bab0231 100644 (file)
@@ -25,42 +25,17 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
         ResultsTable.call(this, element, headers);
     }, {
 
-    _addGraphButton: function(td, testName, testResults)
+    _addGraphButton: function(td, testName, testResult, testData)
     {
-        var data = testResults[Strings.json.samples];
-        if (!data)
-            return;
-
         var button = Utilities.createElement("button", { class: "small-button" }, td);
+        button.textContent = Strings.text.graph + "...";
+        button.testName = testName;
+        button.testResult = testResult;
+        button.testData = testData;
 
-        button.addEventListener("click", function() {
-            var graphData = {
-                axes: [Strings.text.complexity, Strings.text.frameRate],
-                samples: data,
-                complexityAverageSamples: testResults[Strings.json.complexityAverageSamples],
-                averages: {},
-                marks: testResults[Strings.json.marks]
-            };
-            [Strings.json.experiments.complexity, Strings.json.experiments.frameRate].forEach(function(experiment) {
-                if (experiment in testResults)
-                    graphData.averages[experiment] = testResults[experiment];
-            });
-
-            [
-                Strings.json.score,
-                Strings.json.regressions.timeRegressions,
-                Strings.json.regressions.complexityRegression,
-                Strings.json.regressions.complexityAverageRegression,
-                Strings.json.targetFrameLength
-            ].forEach(function(key) {
-                if (testResults[key])
-                    graphData[key] = testResults[key];
-            });
-
-            benchmarkController.showTestGraph(testName, graphData);
+        button.addEventListener("click", function(e) {
+            benchmarkController.showTestGraph(e.target.testName, e.target.testResult, e.target.testData);
         });
-
-        button.textContent = Strings.text.graph + "...";
     },
 
     _isNoisyMeasurement: function(jsonExperiment, data, measurement, options)
@@ -71,19 +46,19 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
         if (measurement == Strings.json.measurements.percent)
             return data[Strings.json.measurements.percent] >= percentThreshold;
 
-        if (jsonExperiment == Strings.json.experiments.frameRate && measurement == Strings.json.measurements.average)
+        if (jsonExperiment == Strings.json.frameLength && measurement == Strings.json.measurements.average)
             return Math.abs(data[Strings.json.measurements.average] - options["frame-rate"]) >= averageThreshold;
 
         return false;
     },
 
-    _addTest: function(testName, testResults, options)
+    _addTest: function(testName, testResult, options, testData)
     {
         var row = Utilities.createElement("tr", {}, this.element);
 
         var isNoisy = false;
-        [Strings.json.experiments.complexity, Strings.json.experiments.frameRate].forEach(function (experiment) {
-            var data = testResults[experiment];
+        [Strings.json.complexity, Strings.json.frameLength].forEach(function (experiment) {
+            var data = testResult[experiment];
             for (var measurement in data) {
                 if (this._isNoisyMeasurement(experiment, data, measurement, options))
                     isNoisy = true;
@@ -94,7 +69,7 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
             var className = "";
             if (header.className) {
                 if (typeof header.className == "function")
-                    className = header.className(testResults, options);
+                    className = header.className(testResult, options);
                 else
                     className = header.className;
             }
@@ -109,16 +84,16 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
 
             var td = Utilities.createElement("td", { class: className }, row);
             if (header.title == Strings.text.graph) {
-                this._addGraphButton(td, testName, testResults);
+                this._addGraphButton(td, testName, testResult, testData);
             } else if (!("text" in header)) {
-                td.textContent = testResults[header.title];
+                td.textContent = testResult[header.title];
             } else if (typeof header.text == "string") {
-                var data = testResults[header.text];
+                var data = testResult[header.text];
                 if (typeof data == "number")
                     data = data.toFixed(2);
                 td.textContent = data;
             } else {
-                td.textContent = header.text(testResults, testName);
+                td.textContent = header.text(testResult, testName);
             }
         }, this);
     }
@@ -134,15 +109,16 @@ Utilities.extendObject(window.benchmarkRunnerClient, {
         this.options = options;
     },
 
-    willStartFirstIteration: function ()
+    willStartFirstIteration: function()
     {
-        this.results = new ResultsDashboard();
+        this.results = new ResultsDashboard(this.options);
         this.progressBar = new ProgressBar(document.getElementById("progress-completed"), this.testsCount);
     },
 
-    didRunTest: function()
+    didRunTest: function(testData)
     {
         this.progressBar.incrementRange();
+        this.results.calculateScore(testData);
     }
 });
 
@@ -152,10 +128,10 @@ Utilities.extendObject(window.sectionsManager, {
         document.querySelector("#" + sectionIdentifier + " h1").textContent = title;
     },
 
-    populateTable: function(tableIdentifier, headers, data)
+    populateTable: function(tableIdentifier, headers, dashboard)
     {
         var table = new DeveloperResultsTable(document.getElementById(tableIdentifier), headers);
-        table.showIterations(data, benchmarkRunnerClient.options);
+        table.showIterations(dashboard);
     }
 });
 
@@ -431,14 +407,13 @@ window.suitesManager =
         return suites;
     },
 
-    updateLocalStorageFromJSON: function(iterationResults)
+    updateLocalStorageFromJSON: function(results)
     {
-        for (var suiteName in iterationResults[Strings.json.results.suites]) {
-            var suiteResults = iterationResults[Strings.json.results.suites][suiteName];
-
-            for (var testName in suiteResults[Strings.json.results.tests]) {
-                var testResults = suiteResults[Strings.json.results.tests][testName];
-                var data = testResults[Strings.json.experiments.complexity];
+        for (var suiteName in results[Strings.json.results.tests]) {
+            var suiteResults = results[Strings.json.results.tests][suiteName];
+            for (var testName in suiteResults) {
+                var testResults = suiteResults[testName];
+                var data = testResults[Strings.json.controller];
                 var complexity = Math.round(data[Strings.json.measurements.average]);
 
                 var value = { checked: true, complexity: complexity };
@@ -462,6 +437,36 @@ Utilities.extendObject(window.benchmarkController, {
         suitesManager.updateUIFromLocalStorage();
         suitesManager.updateDisplay();
         suitesManager.updateEditsElementsState();
+
+        var dropTarget = document.getElementById("drop-target");
+        function stopEvent(e) {
+            e.stopPropagation();
+            e.preventDefault();
+        }
+        dropTarget.addEventListener("dragenter", stopEvent, false);
+        dropTarget.addEventListener("dragover", stopEvent, false);
+        dropTarget.addEventListener("dragleave", stopEvent, false);
+        dropTarget.addEventListener("drop", function (e) {
+            e.stopPropagation();
+            e.preventDefault();
+
+            if (!e.dataTransfer.files.length)
+                return;
+
+            var file = e.dataTransfer.files[0];
+
+            var reader = new FileReader();
+            reader.filename = file.name;
+            reader.onload = function(e) {
+                var run = JSON.parse(e.target.result);
+                benchmarkRunnerClient.results = new ResultsDashboard(run.options, run.data);
+                benchmarkController.showResults();
+            };
+
+            reader.readAsText(file);
+            document.title = "File: " + reader.filename;
+        }, false);
+
     },
 
     onBenchmarkOptionsChanged: function(event)
@@ -479,12 +484,6 @@ Utilities.extendObject(window.benchmarkController, {
     {
         var options = optionsManager.updateLocalStorageFromUI();
         var suites = suitesManager.updateLocalStorageFromUI();
-        if (options["adjustment"] == "ramp") {
-            Headers.details[2].disabled = true;
-        } else {
-            Headers.details[3].disabled = true;
-            Headers.details[4].disabled = true;
-        }
         this._startBenchmark(suites, options, "running-test");
     },
 
@@ -495,32 +494,39 @@ Utilities.extendObject(window.benchmarkController, {
             this.addedKeyEvent = true;
         }
 
-        sectionsManager.setSectionScore("results", benchmarkRunnerClient.results.score.toFixed(2));
-        var data = benchmarkRunnerClient.results.data[Strings.json.results.iterations];
-        sectionsManager.populateTable("results-header", Headers.testName, data);
-        sectionsManager.populateTable("results-score", Headers.score, data);
-        sectionsManager.populateTable("results-data", Headers.details, data);
+        var dashboard = benchmarkRunnerClient.results;
+        if (dashboard.options["adjustment"] == "ramp") {
+            Headers.details[3].disabled = true;
+        } else {
+            Headers.details[1].disabled = true;
+            Headers.details[4].disabled = true;
+        }
+
+        sectionsManager.setSectionScore("results", dashboard.score.toFixed(2));
+        sectionsManager.populateTable("results-header", Headers.testName, dashboard);
+        sectionsManager.populateTable("results-score", Headers.score, dashboard);
+        sectionsManager.populateTable("results-data", Headers.details, dashboard);
         sectionsManager.showSection("results", true);
 
-        suitesManager.updateLocalStorageFromJSON(data[0]);
+        suitesManager.updateLocalStorageFromJSON(dashboard.results[0]);
     },
 
     showJSONResults: function()
     {
-        document.querySelector("#results-json textarea").textContent = JSON.stringify(benchmarkRunnerClient.results.data, function(key, value) {
-            if (typeof value == "number")
-                return value.toFixed(2);
-            return value;
-        });
+        var output = {
+            options: benchmarkRunnerClient.results.options,
+            data: benchmarkRunnerClient.results.data
+        };
+        var textarea = document.querySelector("#results-json textarea").textContent = JSON.stringify(output, null, 1);
         document.querySelector("#results-json button").remove();
         document.querySelector("#results-json div").classList.remove("hidden");
     },
 
-    showTestGraph: function(testName, graphData)
+    showTestGraph: function(testName, testResult, testData)
     {
         sectionsManager.setSectionHeader("test-graph", testName);
         sectionsManager.showSection("test-graph", true);
-        this.updateGraphData(graphData);
+        this.updateGraphData(testResult, testData, benchmarkRunnerClient.results.options);
     }
 });
 
index 7049463dd2b432458cb498015ab9dafd7a72ca1e..0aabcc98900600b5096ec9656c53bcd876037e82 100644 (file)
@@ -1,44 +1,56 @@
 Utilities.extendObject(window.benchmarkController, {
     layoutCounter: 0,
 
-    updateGraphData: function(graphData)
+    updateGraphData: function(testResult, testData, options)
     {
         var element = document.getElementById("test-graph-data");
         element.innerHTML = "";
-        element.graphData = graphData;
+        element.testResult = testResult;
+        element.testData = testData;
+        element.options = options;
         document.querySelector("hr").style.width = this.layoutCounter++ + "px";
 
         var margins = new Insets(30, 30, 30, 40);
         var size = Point.elementClientSize(element).subtract(margins.size);
 
-        this.createTimeGraph(graphData, margins, size);
+        this.createTimeGraph(testResult, testData[Strings.json.samples][Strings.json.controller], testData[Strings.json.marks], testData[Strings.json.controller], options, margins, size);
         this.onTimeGraphOptionsChanged();
 
-        var hasComplexityRegression = !!graphData.complexityRegression;
-        this._showOrHideNodes(hasComplexityRegression, "form[name=graph-type]");
-        if (hasComplexityRegression) {
+        var hasComplexitySamples = !!testData[Strings.json.samples][Strings.json.complexity];
+        this._showOrHideNodes(hasComplexitySamples, "form[name=graph-type]");
+        if (hasComplexitySamples) {
             document.forms["graph-type"].elements["type"] = "complexity";
-            this.createComplexityGraph(graphData, margins, size);
+            this.createComplexityGraph(testResult, testData[Strings.json.controller], testData[Strings.json.samples], options, margins, size);
             this.onComplexityGraphOptionsChanged();
         }
 
         this.onGraphTypeChanged();
     },
 
-    _addRegressionLine: function(parent, xScale, yScale, points, stdev, isAlongYAxis)
+    _addRegressionLine: function(parent, xScale, yScale, points, range, isAlongYAxis)
     {
         var polygon = [];
         var line = []
-        var xStdev = isAlongYAxis ? stdev : 0;
-        var yStdev = isAlongYAxis ? 0 : stdev;
+        var xRange = isAlongYAxis ? range : 0;
+        var yRange = isAlongYAxis ? 0 : range;
         for (var i = 0; i < points.length; ++i) {
             var point = points[i];
-            polygon.push(xScale(point[0] + xStdev), yScale(point[1] + yStdev));
+            var x;
+            if (xRange instanceof Array)
+                x = xRange[0];
+            else
+                x = point[0] + xRange;
+            polygon.push(xScale(x), yScale(point[1] + yRange));
             line.push(xScale(point[0]), yScale(point[1]));
         }
         for (var i = points.length - 1; i >= 0; --i) {
             var point = points[i];
-            polygon.push(xScale(point[0] - xStdev), yScale(point[1] - yStdev));
+            var x;
+            if (xRange instanceof Array)
+                x = xRange[1];
+            else
+                x = point[0] - xRange;
+            polygon.push(xScale(x), yScale(point[1] - yRange));
         }
         parent.append("polygon")
             .attr("points", polygon.join(","));
@@ -52,14 +64,14 @@ Utilities.extendObject(window.benchmarkController, {
     _addRegression: function(data, svg, xScale, yScale)
     {
         svg.append("circle")
-            .attr("cx", xScale(data.segment2[1][0]))
-            .attr("cy", yScale(data.segment2[1][1]))
+            .attr("cx", xScale(data.segment1[1][0]))
+            .attr("cy", yScale(data.segment1[1][1]))
             .attr("r", 5);
         this._addRegressionLine(svg, xScale, yScale, data.segment1, data.stdev);
         this._addRegressionLine(svg, xScale, yScale, data.segment2, data.stdev);
     },
 
-    createComplexityGraph: function(graphData, margins, size)
+    createComplexityGraph: function(result, timeRegressions, data, options, margins, size)
     {
         var svg = d3.select("#test-graph-data").append("svg")
             .attr("id", "complexity-graph")
@@ -69,34 +81,36 @@ Utilities.extendObject(window.benchmarkController, {
             .append("g")
                 .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
 
+        var timeSamples = data[Strings.json.controller];
+
         var xMin = 100000, xMax = 0;
-        if (graphData.timeRegressions) {
-            graphData.timeRegressions.forEach(function(regression) {
+        if (timeRegressions) {
+            timeRegressions.forEach(function(regression) {
                 for (var i = regression.startIndex; i <= regression.endIndex; ++i) {
-                    xMin = Math.min(xMin, graphData.samples[i].complexity);
-                    xMax = Math.max(xMax, graphData.samples[i].complexity);
+                    xMin = Math.min(xMin, timeSamples[i].complexity);
+                    xMax = Math.max(xMax, timeSamples[i].complexity);
                 }
             });
         } else {
-            xMin = d3.min(graphData.samples, function(s) { return s.complexity; });
-            xMax = d3.max(graphData.samples, function(s) { return s.complexity; });
+            xMin = d3.min(timeSamples, function(s) { return s.complexity; });
+            xMax = d3.max(timeSamples, function(s) { return s.complexity; });
         }
 
         var xScale = d3.scale.linear()
             .range([0, size.width])
             .domain([xMin, xMax]);
         var yScale = d3.scale.linear()
-                .range([size.height, 0])
-                .domain([1000/20, 1000/60]);
+            .range([size.height, 0])
+            .domain([1000/20, 1000/60]);
 
         var xAxis = d3.svg.axis()
-                .scale(xScale)
-                .orient("bottom");
+            .scale(xScale)
+            .orient("bottom");
         var yAxis = d3.svg.axis()
-                .scale(yScale)
-                .tickValues([1000/20, 1000/25, 1000/30, 1000/35, 1000/40, 1000/45, 1000/50, 1000/55, 1000/60])
-                .tickFormat(function(d) { return (1000 / d).toFixed(0); })
-                .orient("left");
+            .scale(yScale)
+            .tickValues([1000/20, 1000/25, 1000/30, 1000/35, 1000/40, 1000/45, 1000/50, 1000/55, 1000/60])
+            .tickFormat(function(d) { return (1000 / d).toFixed(0); })
+            .orient("left");
 
         // x-axis
         svg.append("g")
@@ -109,38 +123,64 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("class", "y axis")
             .call(yAxis);
 
-        // time-based regression
+        // time result
         var mean = svg.append("g")
             .attr("class", "mean complexity");
-        var complexity = graphData.averages[Strings.json.experiments.complexity];
-        this._addRegressionLine(mean, xScale, yScale, [[complexity.average, yScale.domain()[0]], [complexity.average, yScale.domain()[1]]], complexity.stdev, true);
+        var timeResult = result[Strings.json.controller];
+        var yMin = yScale.domain()[0], yMax = yScale.domain()[1];
+        this._addRegressionLine(mean, xScale, yScale, [[timeResult.average, yMin], [timeResult.average, yMax]], timeResult.stdev, true);
 
         // regression
-        this._addRegression(graphData.complexityRegression, svg.append("g").attr("class", "regression raw"), xScale, yScale);
-        this._addRegression(graphData.complexityAverageRegression, svg.append("g").attr("class", "regression average"), xScale, yScale);
-
-        var svgGroup = svg.append("g")
-            .attr("class", "series raw");
-        var seriesCounter = 0;
-        graphData.timeRegressions.forEach(function(regression, i) {
-            seriesCounter++;
-            var group = svgGroup.append("g")
-                .attr("class", "series-" + seriesCounter)
-                .attr("fill", "hsl(" + (i / graphData.timeRegressions.length * 360).toFixed(0) + ", 96%, 56%)");
-            group.selectAll("circle")
-                .data(graphData.samples)
-                .enter()
-                .append("circle")
-                .filter(function(d, i) { return i >= regression.startIndex && i <= regression.endIndex; })
-                .attr("cx", function(d) { return xScale(d.complexity); })
-                .attr("cy", function(d) { return yScale(d.frameLength); })
-                .attr("r", 2);
-        });
+        this._addRegression(result[Strings.json.complexity], svg.append("g").attr("class", "regression raw"), xScale, yScale);
+        this._addRegression(result[Strings.json.complexityAverage], svg.append("g").attr("class", "regression average"), xScale, yScale);
+
+        var bootstrapResult = result[Strings.json.complexity][Strings.json.bootstrap];
+        if (bootstrapResult) {
+            var histogram = d3.layout.histogram()
+                .bins(xScale.ticks(100))(bootstrapResult.data);
+            var yBootstrapScale = d3.scale.linear()
+                .range([size.height/2, 0])
+                .domain([0, d3.max(histogram, function(d) { return d.y; })]);
+            group = svg.append("g").attr("class", "bootstrap");
+            var bar = group.selectAll(".bar")
+                .data(histogram)
+                .enter().append("g")
+                    .attr("class", "bar")
+                    .attr("transform", function(d) { return "translate(" + xScale(d.x) + "," + yBootstrapScale(d.y) + ")"; });
+            bar.append("rect")
+                .attr("x", 1)
+                .attr("y", size.height/2)
+                .attr("width", xScale(histogram[1].x) - xScale(histogram[0].x) - 1)
+                .attr("height", function(d) { return size.height/2 - yBootstrapScale(d.y); });
+            group = group.append("g").attr("class", "median");
+            this._addRegressionLine(group, xScale, yScale, [[bootstrapResult.median, yMin], [bootstrapResult.median, yMax]], [bootstrapResult.confidenceLow, bootstrapResult.confidenceHigh], true);
+            group.append("circle")
+                .attr("cx", xScale(bootstrapResult.median))
+                .attr("cy", yScale(1000/60))
+                .attr("r", 5);
+        }
+
+        // series
+        group = svg.append("g")
+            .attr("class", "series raw")
+            .selectAll("line")
+                .data(data[Strings.json.complexity])
+                .enter();
+        group.append("line")
+            .attr("x1", function(d) { return xScale(d.complexity) - 3; })
+            .attr("x2", function(d) { return xScale(d.complexity) + 3; })
+            .attr("y1", function(d) { return yScale(d.frameLength) - 3; })
+            .attr("y2", function(d) { return yScale(d.frameLength) + 3; });
+        group.append("line")
+            .attr("x1", function(d) { return xScale(d.complexity) - 3; })
+            .attr("x2", function(d) { return xScale(d.complexity) + 3; })
+            .attr("y1", function(d) { return yScale(d.frameLength) + 3; })
+            .attr("y2", function(d) { return yScale(d.frameLength) - 3; });
 
         group = svg.append("g")
             .attr("class", "series average")
             .selectAll("circle")
-                .data(graphData.complexityAverageSamples)
+                .data(data[Strings.json.complexityAverage])
                 .enter();
         group.append("circle")
             .attr("cx", function(d) { return xScale(d.complexity); })
@@ -162,7 +202,7 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("y2", yScale(yAxis.scale().domain()[1]));
         cursorGroup.append("line")
             .attr("class", "y")
-            .attr("x1", xScale(0) - 10)
+            .attr("x1", xScale(xAxis.scale().domain()[0]) - 10)
             .attr("x2", xScale(xAxis.scale().domain()[1]))
             .attr("y1", 0)
             .attr("y2", 0)
@@ -174,7 +214,7 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("text-anchor", "middle");
         cursorGroup.append("text")
             .attr("class", "label y")
-            .attr("x", xScale(0) - 15)
+            .attr("x", xScale(xAxis.scale().domain()[0]) - 15)
             .attr("y", 0)
             .attr("baseline-shift", "-30%")
             .attr("text-anchor", "end");
@@ -208,7 +248,7 @@ Utilities.extendObject(window.benchmarkController, {
         });
     },
 
-    createTimeGraph: function(graphData, margins, size)
+    createTimeGraph: function(result, samples, marks, regressions, options, margins, size)
     {
         var svg = d3.select("#test-graph-data").append("svg")
             .attr("id", "time-graph")
@@ -217,21 +257,17 @@ Utilities.extendObject(window.benchmarkController, {
             .append("g")
                 .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
 
-        var axes = graphData.axes;
-        var targetFrameLength = graphData.targetFrameLength;
-
         // Axis scales
         var x = d3.scale.linear()
                 .range([0, size.width])
                 .domain([
-                    Math.min(d3.min(graphData.samples, function(s) { return s.time; }), 0),
-                    d3.max(graphData.samples, function(s) { return s.time; })]);
-        var complexityMax = d3.max(graphData.samples, function(s) { return s.complexity; });
-        if (graphData.timeRegressions) {
-            complexityMax = Math.max.apply(Math, graphData.timeRegressions.map(function(regression) {
-                return regression.maxComplexity || 0;
-            }));
-        }
+                    Math.min(d3.min(samples, function(s) { return s.time; }), 0),
+                    d3.max(samples, function(s) { return s.time; })]);
+        var complexityMax = d3.max(samples, function(s) {
+            if (s.time > 0)
+                return s.complexity;
+            return 0;
+        });
 
         var yLeft = d3.scale.linear()
                 .range([size.height, 0])
@@ -280,7 +316,7 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("fill", "#7ADD49")
                 .attr("dy", ".71em")
                 .style("text-anchor", "end")
-                .text(axes[0]);
+                .text(Strings.text.complexity);
 
         // yRight-axis
         svg.append("g")
@@ -295,13 +331,13 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("fill", "#FA4925")
                 .attr("dy", ".71em")
                 .style("text-anchor", "start")
-                .text(axes[1]);
+                .text(Strings.text.frameRate);
 
         // marks
-        var yMin = yLeft(0);
-        var yMax = yLeft(yAxisLeft.scale().domain()[1]);
-        for (var markName in graphData.marks) {
-            var mark = graphData.marks[markName];
+        var yMin = yRight(yAxisRight.scale().domain()[0]);
+        var yMax = yRight(yAxisRight.scale().domain()[1]);
+        for (var markName in marks) {
+            var mark = marks[markName];
             var xLocation = x(mark.time);
 
             var markerGroup = svg.append("g")
@@ -318,21 +354,22 @@ Utilities.extendObject(window.benchmarkController, {
                 .attr("y2", yMax);
         }
 
-        if (Strings.json.experiments.complexity in graphData.averages) {
-            var complexity = graphData.averages[Strings.json.experiments.complexity];
+        if (Strings.json.controller in result) {
+            var complexity = result[Strings.json.controller];
             var regression = svg.append("g")
                 .attr("class", "complexity mean");
-            this._addRegressionLine(regression, x, yLeft, [[graphData.samples[0].time, complexity.average], [graphData.samples[graphData.samples.length - 1].time, complexity.average]], complexity.stdev);
+            this._addRegressionLine(regression, x, yLeft, [[samples[0].time, complexity.average], [samples[samples.length - 1].time, complexity.average]], complexity.stdev);
         }
-        if (Strings.json.experiments.frameRate in graphData.averages) {
-            var frameRate = graphData.averages[Strings.json.experiments.frameRate];
+        if (Strings.json.frameLength in result) {
+            var frameLength = result[Strings.json.frameLength];
             var regression = svg.append("g")
                 .attr("class", "fps mean");
-            this._addRegressionLine(regression, x, yRight, [[graphData.samples[0].time, 1000/frameRate.average], [graphData.samples[graphData.samples.length - 1].time, 1000/frameRate.average]], frameRate.stdev);
+            this._addRegressionLine(regression, x, yRight, [[samples[0].time, 1000/frameLength.average], [samples[samples.length - 1].time, 1000/frameLength.average]], frameLength.stdev);
         }
 
         // right-target
-        if (targetFrameLength) {
+        if (options["adjustment"] == "adaptive") {
+            var targetFrameLength = 1000 / options["frame-rate"];
             svg.append("line")
                 .attr("x1", x(0))
                 .attr("x2", size.width)
@@ -350,8 +387,8 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("y2", yMin);
 
         // Data
-        var allData = graphData.samples;
-        var filteredData = graphData.samples.filter(function (sample) {
+        var allData = samples;
+        var filteredData = samples.filter(function (sample) {
             return "smoothedFrameLength" in sample;
         });
 
@@ -384,19 +421,23 @@ Utilities.extendObject(window.benchmarkController, {
         // regressions
         var regressionGroup = svg.append("g")
             .attr("id", "regressions");
-        if (graphData.timeRegressions) {
+        if (regressions) {
             var complexities = [];
-            graphData.timeRegressions.forEach(function (regression) {
-                regressionGroup.append("line")
-                    .attr("x1", x(regression.segment1[0][0]))
-                    .attr("x2", x(regression.segment1[1][0]))
-                    .attr("y1", yRight(regression.segment1[0][1]))
-                    .attr("y2", yRight(regression.segment1[1][1]));
-                regressionGroup.append("line")
-                    .attr("x1", x(regression.segment2[0][0]))
-                    .attr("x2", x(regression.segment2[1][0]))
-                    .attr("y1", yRight(regression.segment2[0][1]))
-                    .attr("y2", yRight(regression.segment2[1][1]));
+            regressions.forEach(function (regression) {
+                if (!isNaN(regression.segment1[0][1]) && !isNaN(regression.segment1[1][1])) {
+                    regressionGroup.append("line")
+                        .attr("x1", x(regression.segment1[0][0]))
+                        .attr("x2", x(regression.segment1[1][0]))
+                        .attr("y1", yRight(regression.segment1[0][1]))
+                        .attr("y2", yRight(regression.segment1[1][1]));
+                }
+                if (!isNaN(regression.segment2[0][1]) && !isNaN(regression.segment2[1][1])) {
+                    regressionGroup.append("line")
+                        .attr("x1", x(regression.segment2[0][0]))
+                        .attr("x2", x(regression.segment2[1][0]))
+                        .attr("y1", yRight(regression.segment2[0][1]))
+                        .attr("y2", yRight(regression.segment2[1][1]));
+                }
                 // inflection point
                 regressionGroup.append("circle")
                     .attr("cx", x(regression.segment1[1][0]))
@@ -503,6 +544,7 @@ Utilities.extendObject(window.benchmarkController, {
         benchmarkController._showOrHideNodes(form["series-raw"].checked, "#complexity-graph .series.raw");
         benchmarkController._showOrHideNodes(form["series-average"].checked, "#complexity-graph .series.average");
         benchmarkController._showOrHideNodes(form["regression-time-score"].checked, "#complexity-graph .mean.complexity");
+        benchmarkController._showOrHideNodes(form["bootstrap-score"].checked, "#complexity-graph .bootstrap");
         benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-raw"].checked, "#complexity-graph .regression.raw");
         benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-average"].checked, "#complexity-graph .regression.average");
     },
@@ -514,11 +556,12 @@ Utilities.extendObject(window.benchmarkController, {
         benchmarkController._showOrHideNodes(form["complexity"].checked, "#complexity");
         benchmarkController._showOrHideNodes(form["rawFPS"].checked, "#rawFPS");
         benchmarkController._showOrHideNodes(form["filteredFPS"].checked, "#filteredFPS");
+        benchmarkController._showOrHideNodes(form["regressions"].checked, "#regressions");
     },
 
     onGraphTypeChanged: function() {
         var form = document.forms["graph-type"].elements;
-        var graphData = document.getElementById("test-graph-data").graphData;
+        var testResult = document.getElementById("test-graph-data").testResult;
         var isTimeSelected = form["graph-type"].value == "time";
 
         benchmarkController._showOrHideNodes(isTimeSelected, "#time-graph");
@@ -528,9 +571,9 @@ Utilities.extendObject(window.benchmarkController, {
 
         var score, mean;
         if (isTimeSelected) {
-            score = graphData.score.toFixed(2);
+            score = testResult[Strings.json.score].toFixed(2);
 
-            var regression = graphData.averages.complexity;
+            var regression = testResult[Strings.json.controller];
             mean = [
                 "mean: ",
                 regression.average.toFixed(2),
@@ -546,18 +589,20 @@ Utilities.extendObject(window.benchmarkController, {
             }
             mean = mean.join("");
         } else {
-            score = [
-                "raw: ",
-                graphData.complexityRegression.complexity.toFixed(2),
-                ", average: ",
-                graphData.complexityAverageRegression.complexity.toFixed(2)].join("");
+            var complexityRegression = testResult[Strings.json.complexity];
+            var complexityAverageRegression = testResult[Strings.json.complexityAverage];
+
+            document.getElementById("complexity-regression-aggregate-raw").textContent = complexityRegression.complexity.toFixed(2) + ", ±" + complexityRegression.stdev.toFixed(2) + "ms";
+            document.getElementById("complexity-regression-aggregate-average").textContent = complexityAverageRegression.complexity.toFixed(2) + ", ±" + complexityAverageRegression.stdev.toFixed(2) + "ms";
 
+            var bootstrap = complexityRegression[Strings.json.bootstrap];
+            score = bootstrap.median.toFixed(2);
             mean = [
-                "raw: ±",
-                graphData.complexityRegression.stdev.toFixed(2),
-                "ms, average: ±",
-                graphData.complexityAverageRegression.stdev.toFixed(2),
-                "ms"].join("");
+                "95% CI: ",
+                bootstrap.confidenceLow.toFixed(2),
+                "",
+                bootstrap.confidenceHigh.toFixed(2)
+            ].join("");
         }
 
         sectionsManager.setSectionScore("test-graph", score, mean);
index 26e3d38438ecf7d436302a070fb179ec9721bb27..6f1c815b2a2fb75964f5fe114a478a5260b992fa 100644 (file)
@@ -1,15 +1,57 @@
+Utilities.extendObject(Strings.text, {
+    samples: "Samples",
+    complexity: "Time Complexity",
+    frameRate: "FPS",
+    confidenceInterval: "95% Confidence Interval",
+    mergedRawComplexity: "Raw Complexity",
+    graph: "Graph"
+});
+
+
 Utilities.extendObject(Headers, {
     details: [
         {
             title: Strings.text.graph
         },
+        {
+            title: Strings.text.confidenceInterval,
+            children:
+            [
+                {
+                    text: function(data) {
+                        return data[Strings.json.complexity][Strings.json.bootstrap].confidenceLow.toFixed(2);
+                    },
+                    className: "right pad-left pad-right"
+                },
+                {
+                    text: function(data) {
+                        return " - " + data[Strings.json.complexity][Strings.json.bootstrap].confidenceHigh.toFixed(2);
+                    },
+                    className: "left"
+                },
+                {
+                    text: function(data) {
+                        var bootstrap = data[Strings.json.complexity][Strings.json.bootstrap];
+                        return (100 * (bootstrap.confidenceLow / bootstrap.median - 1)).toFixed(2) + "%";
+                    },
+                    className: "left pad-left small"
+                },
+                {
+                    text: function(data) {
+                        var bootstrap = data[Strings.json.complexity][Strings.json.bootstrap];
+                        return "+" + (100 * (bootstrap.confidenceHigh / bootstrap.median - 1)).toFixed(2) + "%";
+                    },
+                    className: "left pad-left small"
+                }
+            ]
+        },
         {
             title: Strings.text.complexity,
             children:
             [
                 {
                     text: function(data) {
-                        return data[Strings.json.experiments.complexity][Strings.json.measurements.average].toFixed(2);
+                        return data[Strings.json.controller][Strings.json.measurements.average].toFixed(2);
                     },
                     className: "average"
                 },
@@ -17,14 +59,14 @@ Utilities.extendObject(Headers, {
                     text: function(data) {
                         return [
                             "± ",
-                            data[Strings.json.experiments.complexity][Strings.json.measurements.percent].toFixed(2),
+                            data[Strings.json.controller][Strings.json.measurements.percent].toFixed(2),
                             "%"
                         ].join("");
                     },
                     className: function(data) {
                         var className = "stdev";
 
-                        if (data[Strings.json.experiments.complexity][Strings.json.measurements.percent] >= 10)
+                        if (data[Strings.json.controller][Strings.json.measurements.percent] >= 10)
                             className += " noisy-results";
                         return className;
                     }
@@ -37,18 +79,18 @@ Utilities.extendObject(Headers, {
             [
                 {
                     text: function(data) {
-                        return data[Strings.json.experiments.frameRate][Strings.json.measurements.average].toFixed(2);
+                        return data[Strings.json.frameLength][Strings.json.measurements.average].toFixed(2);
                     },
                     className: function(data, options) {
                         var className = "average";
-                        if (Math.abs(data[Strings.json.experiments.frameRate][Strings.json.measurements.average] - options["frame-rate"]) >= 2)
+                        if (Math.abs(data[Strings.json.frameLength][Strings.json.measurements.average] - options["frame-rate"]) >= 2)
                             className += " noisy-results";
                         return className;
                     }
                 },
                 {
                     text: function(data) {
-                        var frameRateData = data[Strings.json.experiments.frameRate];
+                        var frameRateData = data[Strings.json.frameLength];
                         return [
                             "± ",
                             frameRateData[Strings.json.measurements.percent].toFixed(2),
@@ -58,7 +100,7 @@ Utilities.extendObject(Headers, {
                     className: function(data) {
                         var className = "stdev";
 
-                        if (data[Strings.json.experiments.frameRate][Strings.json.measurements.percent] >= 10)
+                        if (data[Strings.json.frameLength][Strings.json.measurements.percent] >= 10)
                             className += " noisy-results";
                         return className;
                     }
@@ -71,7 +113,7 @@ Utilities.extendObject(Headers, {
             [
                 {
                     text: function(data) {
-                        return data[Strings.json.regressions.complexityRegression][Strings.json.regressions.complexity].toFixed(2);
+                        return data[Strings.json.complexity][Strings.json.complexity].toFixed(2);
                     },
                     className: "average"
                 },
@@ -79,36 +121,14 @@ Utilities.extendObject(Headers, {
                     text: function(data) {
                         return [
                             "± ",
-                            data[Strings.json.regressions.complexityRegression][Strings.json.measurements.stdev].toFixed(2),
+                            data[Strings.json.complexity][Strings.json.measurements.stdev].toFixed(2),
                             "ms"
                         ].join("");
                     },
                     className: "stdev"
                 }
             ]
-        },
-        {
-            title: Strings.text.mergedAverageComplexity,
-            children:
-            [
-                {
-                    text: function(data) {
-                        return data[Strings.json.regressions.complexityAverageRegression][Strings.json.regressions.complexity].toFixed(2);
-                    },
-                    className: "average"
-                },
-                {
-                    text: function(data) {
-                        return [
-                            "± ",
-                            data[Strings.json.regressions.complexityAverageRegression][Strings.json.measurements.stdev].toFixed(2),
-                            "ms"
-                        ].join("");
-                    },
-                    className: "stdev"
-                }
-            ]
-        },
+        }
     ]
 })
 
index 11c6bff2c014cfd25ae6567394b3c01b630c234a..6f653fef25ab714f15f66c6be23bf6fce9198c92 100644 (file)
@@ -352,29 +352,136 @@ SimplePromise = Utilities.createClass(
     }
 });
 
-Statistics =
-{
-    sampleMean: function(numberOfSamples, sum)
+var Heap = Utilities.createClass(
+    function(maxSize, compare)
+    {
+        this._maxSize = maxSize;
+        this._compare = compare;
+        this._size = 0;
+        this._values = new Array(this._maxSize);
+    }, {
+
+    // This is a binary heap represented in an array. The root element is stored
+    // in the first element in the array. The root is followed by its two children.
+    // Then its four grandchildren and so on. So every level in the binary heap is
+    // doubled in the following level. Here is an example of the node indices and
+    // how they are related to their parents and children.
+    // ===========================================================================
+    //              0       1       2       3       4       5       6
+    // PARENT       -1      0       0       1       1       2       2
+    // LEFT         1       3       5       7       9       11      13
+    // RIGHT        2       4       6       8       10      12      14
+    // ===========================================================================
+    _parentIndex: function(i)
+    {
+        return i > 0 ? Math.floor((i - 1) / 2) : -1;
+    },
+
+    _leftIndex: function(i)
+    {
+        var leftIndex = i * 2 + 1;
+        return leftIndex < this._size ? leftIndex : -1;
+    },
+
+    _rightIndex: function(i)
+    {
+        var rightIndex = i * 2 + 2;
+        return rightIndex < this._size ? rightIndex : -1;
+    },
+
+    // Return the child index that may violate the heap property at index i.
+    _childIndex: function(i)
     {
-        if (numberOfSamples < 1)
-            return 0;
-        return sum / numberOfSamples;
+        var left = this._leftIndex(i);
+        var right = this._rightIndex(i);
+
+        if (left != -1 && right != -1)
+            return this._compare(this._values[left], this._values[right]) > 0 ? left : right;
+
+        return left != -1 ? left : right;
+    },
+
+    init: function()
+    {
+        this._size = 0;
+    },
+
+    top: function()
+    {
+        return this._size ? this._values[0] : NaN;
     },
 
-    // With sum and sum of squares, we can compute the sample standard deviation in O(1).
-    // See https://rniwa.com/2012-11-10/sample-standard-deviation-in-terms-of-sum-and-square-sum-of-samples/
-    unbiasedSampleStandardDeviation: function(numberOfSamples, sum, squareSum)
+    push: function(value)
     {
-        if (numberOfSamples < 2)
-            return 0;
-        return Math.sqrt((squareSum - sum * sum / numberOfSamples) / (numberOfSamples - 1));
+        if (this._size == this._maxSize) {
+            // If size is bounded and the new value can be a parent of the top()
+            // if the size were unbounded, just ignore the new value.
+            if (this._compare(value, this.top()) > 0)
+                return;
+            this.pop();
+        }
+        this._values[this._size++] = value;
+        this._bubble(this._size - 1);
     },
 
-    geometricMean: function(values)
+    pop: function()
     {
-        if (!values.length)
-            return 0;
-        var roots = values.map(function(value) { return  Math.pow(value, 1 / values.length); })
-        return roots.reduce(function(a, b) { return a * b; });
+        if (!this._size)
+            return NaN;
+
+        this._values[0] = this._values[--this._size];
+        this._sink(0);
+    },
+
+    _bubble: function(i)
+    {
+        // Fix the heap property at index i given that parent is the only node that
+        // may violate the heap property.
+        for (var pi = this._parentIndex(i); pi != -1; i = pi, pi = this._parentIndex(pi)) {
+            if (this._compare(this._values[pi], this._values[i]) > 0)
+                break;
+
+            this._values.swap(pi, i);
+        }
+    },
+
+    _sink: function(i)
+    {
+        // Fix the heap property at index i given that each of the left and the right
+        // sub-trees satisfies the heap property.
+        for (var ci = this._childIndex(i); ci != -1; i = ci, ci = this._childIndex(ci)) {
+            if (this._compare(this._values[i], this._values[ci]) > 0)
+                break;
+
+            this._values.swap(ci, i);
+        }
+    },
+
+    str: function()
+    {
+        var out = "Heap[" + this._size + "] = [";
+        for (var i = 0; i < this._size; ++i) {
+            out += this._values[i];
+            if (i < this._size - 1)
+                out += ", ";
+        }
+        return out + "]";
+    },
+
+    values: function(size) {
+        // Return the last "size" heap elements values.
+        var values = this._values.slice(0, this._size);
+        return values.sort(this._compare).slice(0, Math.min(size, this._size));
     }
-};
+});
+
+Utilities.extendObject(Heap, {
+    createMinHeap: function(maxSize)
+    {
+        return new Heap(maxSize, function(a, b) { return b - a; });
+    },
+
+    createMaxHeap: function(maxSize) {
+        return new Heap(maxSize, function(a, b) { return a - b; });
+    }
+});
index 06e43b4cfb6a76748925e8a4d990722a74464fa1..3ac2dccde87b47abb9ded319aa711656a0ba3ad6 100644 (file)
@@ -1,8 +1,13 @@
 ResultsDashboard = Utilities.createClass(
-    function()
+    function(options, testData)
     {
         this._iterationsSamplers = [];
-        this._processedData = undefined;
+        this._options = options;
+        this._results = null;
+        if (testData) {
+            this._iterationsSamplers = testData;
+            this._processData();
+        }
     }, {
 
     push: function(suitesSamplers)
@@ -12,53 +17,179 @@ ResultsDashboard = Utilities.createClass(
 
     _processData: function()
     {
-        var iterationsResults = [];
+        this._results = {};
+        this._results[Strings.json.results.iterations] = [];
+
         var iterationsScores = [];
+        this._iterationsSamplers.forEach(function(iteration, index) {
+            var testsScores = [];
+
+            var result = {};
+            this._results[Strings.json.results.iterations][index] = result;
+
+            var suitesResult = {};
+            result[Strings.json.results.tests] = suitesResult;
 
-        this._iterationsSamplers.forEach(function(iterationSamplers, index) {
-            var suitesResults = {};
-            var suitesScores = [];
+            for (var suiteName in iteration) {
+                var suiteData = iteration[suiteName];
 
-            for (var suiteName in iterationSamplers) {
-                var suite = suiteFromName(suiteName);
-                var suiteSamplerData = iterationSamplers[suiteName];
+                var suiteResult = {};
+                suitesResult[suiteName] = suiteResult;
 
-                var testsResults = {};
-                var testsScores = [];
+                for (var testName in suiteData) {
+                    if (!suiteData[testName][Strings.json.result])
+                        this.calculateScore(suiteData[testName]);
 
-                for (var testName in suiteSamplerData) {
-                    testsResults[testName] = suiteSamplerData[testName];
-                    testsScores.push(testsResults[testName][Strings.json.score]);
+                    suiteResult[testName] = suiteData[testName][Strings.json.result];
+                    delete suiteData[testName][Strings.json.result];
+
+                    testsScores.push(suiteResult[testName][Strings.json.score]);
                 }
+            }
+
+            result[Strings.json.score] = Statistics.geometricMean(testsScores);
+            iterationsScores.push(result[Strings.json.score]);
+        }, this);
 
-                suitesResults[suiteName] =  {};
-                suitesResults[suiteName][Strings.json.score] = Statistics.geometricMean(testsScores);
-                suitesResults[suiteName][Strings.json.results.tests] = testsResults;
-                suitesScores.push(suitesResults[suiteName][Strings.json.score]);
+        this._results[Strings.json.score] = Statistics.sampleMean(iterationsScores.length, iterationsScores.reduce(function(a, b) { return a + b; }));
+    },
+
+    calculateScore: function(data)
+    {
+        var result = {};
+        data[Strings.json.result] = result;
+        var samples = data[Strings.json.samples];
+
+        function findRegression(series) {
+            var minIndex = Math.round(.025 * series.length);
+            var maxIndex = Math.round(.975 * (series.length - 1));
+            var minComplexity = series[minIndex].complexity;
+            var maxComplexity = series[maxIndex].complexity;
+            if (Math.abs(maxComplexity - minComplexity) < 20 && maxIndex - minIndex < 20) {
+                minIndex = 0;
+                maxIndex = series.length - 1;
+                minComplexity = series[minIndex].complexity;
+                maxComplexity = series[maxIndex].complexity;
             }
 
-            iterationsResults[index] = {};
-            iterationsResults[index][Strings.json.score] = Statistics.geometricMean(suitesScores);
-            iterationsResults[index][Strings.json.results.suites] = suitesResults;
-            iterationsScores.push(iterationsResults[index][Strings.json.score]);
+            return {
+                minComplexity: minComplexity,
+                maxComplexity: maxComplexity,
+                samples: series.slice(minIndex, maxIndex + 1),
+                regression: new Regression(
+                    series,
+                    function (datum, i) { return datum[i].complexity; },
+                    function (datum, i) { return datum[i].frameLength; },
+                    minIndex, maxIndex)
+            };
+        }
+
+        var complexitySamples;
+        [Strings.json.complexity, Strings.json.complexityAverage].forEach(function(seriesName) {
+            if (!(seriesName in samples))
+                return;
+
+            var regression = {};
+            result[seriesName] = regression;
+            var regressionResult = findRegression(samples[seriesName]);
+            if (seriesName == Strings.json.complexity)
+                complexitySamples = regressionResult.samples;
+            var calculation = regressionResult.regression;
+            regression[Strings.json.regressions.segment1] = [
+                [regressionResult.minComplexity, calculation.s1 + calculation.t1 * regressionResult.minComplexity],
+                [calculation.complexity, calculation.s1 + calculation.t1 * calculation.complexity]
+            ];
+            regression[Strings.json.regressions.segment2] = [
+                [calculation.complexity, calculation.s2 + calculation.t2 * calculation.complexity],
+                [regressionResult.maxComplexity, calculation.s2 + calculation.t2 * regressionResult.maxComplexity]
+            ];
+            regression[Strings.json.complexity] = calculation.complexity;
+            regression[Strings.json.measurements.stdev] = Math.sqrt(calculation.error / samples[seriesName].length);
         });
 
-        this._processedData = {};
-        this._processedData[Strings.json.score] = Statistics.sampleMean(iterationsScores.length, iterationsScores.reduce(function(a, b) { return a * b; }));
-        this._processedData[Strings.json.results.iterations] = iterationsResults;
+        if (this._options["adjustment"] == "ramp") {
+            var timeComplexity = new Experiment;
+            data[Strings.json.controller].forEach(function(regression) {
+                timeComplexity.sample(regression[Strings.json.complexity]);
+            });
+
+            var experimentResult = {};
+            result[Strings.json.controller] = experimentResult;
+            experimentResult[Strings.json.score] = timeComplexity.mean();
+            experimentResult[Strings.json.measurements.average] = timeComplexity.mean();
+            experimentResult[Strings.json.measurements.stdev] = timeComplexity.standardDeviation();
+            experimentResult[Strings.json.measurements.percent] = timeComplexity.percentage();
+
+            result[Strings.json.complexity][Strings.json.bootstrap] = Regression.bootstrap(complexitySamples, 2500, function(resample) {
+                    resample.sort(function(a, b) {
+                        return a.complexity - b.complexity;
+                    });
+
+                    var regressionResult = findRegression(resample);
+                    return regressionResult.regression.complexity;
+                }, .95);
+            result[Strings.json.score] = result[Strings.json.complexity][Strings.json.bootstrap].median;
+
+        } else {
+            var marks = data[Strings.json.marks];
+            var samplingStartIndex = 0, samplingEndIndex = -1;
+            if (Strings.json.samplingStartTimeOffset in marks)
+                samplingStartIndex = marks[Strings.json.samplingStartTimeOffset].index;
+            if (Strings.json.samplingEndTimeOffset in marks)
+                samplingEndIndex = marks[Strings.json.samplingEndTimeOffset].index;
+
+            var averageComplexity = new Experiment;
+            var averageFrameLength = new Experiment;
+            samples[Strings.json.controller].forEach(function (sample, i) {
+                if (i >= samplingStartIndex && (samplingEndIndex == -1 || i < samplingEndIndex)) {
+                    averageComplexity.sample(sample.complexity);
+                    if (sample.smoothedFrameLength && sample.smoothedFrameLength != -1)
+                        averageFrameLength.sample(sample.smoothedFrameLength);
+                }
+            });
+
+            var experimentResult = {};
+            result[Strings.json.controller] = experimentResult;
+            experimentResult[Strings.json.measurements.average] = averageComplexity.mean();
+            experimentResult[Strings.json.measurements.concern] = averageComplexity.concern(Experiment.defaults.CONCERN);
+            experimentResult[Strings.json.measurements.stdev] = averageComplexity.standardDeviation();
+            experimentResult[Strings.json.measurements.percent] = averageComplexity.percentage();
+
+            experimentResult = {};
+            result[Strings.json.frameLength] = experimentResult;
+            experimentResult[Strings.json.measurements.average] = 1000 / averageFrameLength.mean();
+            experimentResult[Strings.json.measurements.concern] = averageFrameLength.concern(Experiment.defaults.CONCERN);
+            experimentResult[Strings.json.measurements.stdev] = averageFrameLength.standardDeviation();
+            experimentResult[Strings.json.measurements.percent] = averageFrameLength.percentage();
+
+            result[Strings.json.score] = averageComplexity.score(Experiment.defaults.CONCERN);
+        }
     },
 
     get data()
     {
-        if (this._processedData)
-            return this._processedData;
+        return this._iterationsSamplers;
+    },
+
+    get results()
+    {
+        if (this._results)
+            return this._results[Strings.json.results.iterations];
         this._processData();
-        return this._processedData;
+        return this._results[Strings.json.results.iterations];
+    },
+
+    get options()
+    {
+        return this._options;
     },
 
     get score()
     {
-        return this.data[Strings.json.score];
+        if (this._results)
+            return this._results[Strings.json.score];
+        this._processData();
+        return this._results[Strings.json.score];
     }
 });
 
@@ -133,29 +264,26 @@ ResultsTable = Utilities.createClass(
         }, this);
     },
 
-    _addSuite: function(suiteName, suiteResults, options)
+    _addIteration: function(iterationResult, iterationData, options)
     {
-        for (var testName in suiteResults[Strings.json.results.tests]) {
-            var testResults = suiteResults[Strings.json.results.tests][testName];
-            this._addTest(testName, testResults, options);
-        }
-    },
-
-    _addIteration: function(iterationResult, options)
-    {
-        for (var suiteName in iterationResult[Strings.json.results.suites]) {
+        var testsResults = iterationResult[Strings.json.results.tests];
+        for (var suiteName in testsResults) {
             this._addEmptyRow();
-            this._addSuite(suiteName, iterationResult[Strings.json.results.suites][suiteName], options);
+            var suiteResult = testsResults[suiteName];
+            var suiteData = iterationData[suiteName];
+            for (var testName in suiteResult)
+                this._addTest(testName, suiteResult[testName], options, suiteData[testName]);
         }
     },
 
-    showIterations: function(iterationsResults, options)
+    showIterations: function(dashboard)
     {
         this.clear();
         this._addHeader();
 
-        iterationsResults.forEach(function(iterationResult) {
-            this._addIteration(iterationResult, options);
+        var iterationsResults = dashboard.results;
+        iterationsResults.forEach(function(iterationResult, index) {
+            this._addIteration(iterationResult, dashboard.data[index], dashboard.options);
         }, this);
     }
 });
@@ -172,7 +300,7 @@ window.benchmarkRunnerClient = {
 
     willStartFirstIteration: function()
     {
-        this.results = new ResultsDashboard();
+        this.results = new ResultsDashboard(this.options);
     },
 
     didRunSuites: function(suitesSamplers)
@@ -180,6 +308,11 @@ window.benchmarkRunnerClient = {
         this.results.push(suitesSamplers);
     },
 
+    didRunTest: function(testData)
+    {
+        this.results.calculateScore(testData);
+    },
+
     didFinishLastIteration: function()
     {
         benchmarkController.showResults();
@@ -210,10 +343,10 @@ window.sectionsManager =
             document.querySelector("#" + sectionIdentifier + " .mean").innerHTML = mean;
     },
 
-    populateTable: function(tableIdentifier, headers, data)
+    populateTable: function(tableIdentifier, headers, dashboard)
     {
         var table = new ResultsTable(document.getElementById(tableIdentifier), headers);
-        table.showIterations(data, benchmarkRunnerClient.options);
+        table.showIterations(dashboard);
     }
 };
 
@@ -249,10 +382,11 @@ window.benchmarkController = {
             this.addedKeyEvent = true;
         }
 
-        sectionsManager.setSectionScore("results", benchmarkRunnerClient.results.score.toFixed(2));
-        var data = benchmarkRunnerClient.results.data[Strings.json.results.iterations];
-        sectionsManager.populateTable("results-header", Headers.testName, data);
-        sectionsManager.populateTable("results-score", Headers.score, data);
+        var dashboard = benchmarkRunnerClient.results;
+
+        sectionsManager.setSectionScore("results", dashboard.score.toFixed(2));
+        sectionsManager.populateTable("results-header", Headers.testName, dashboard);
+        sectionsManager.populateTable("results-score", Headers.score, dashboard);
         sectionsManager.showSection("results", true);
     },
 
index a2ac7b2b0bd6e725e69ff77cb48ae4fe5f6d47c8..1aa630356cff167ec370961d7256920a21d6c508 100644 (file)
@@ -95,13 +95,13 @@ BenchmarkRunner = Utilities.createClass(
 
         var benchmark = new contentWindow.benchmarkClass(options);
         document.body.style.backgroundColor = benchmark.backgroundColor();
-        benchmark.run().then(function(results) {
+        benchmark.run().then(function(testData) {
             var suiteResults = self._suitesResults[suite.name] || {};
-            suiteResults[test.name] = results;
+            suiteResults[test.name] = testData;
             self._suitesResults[suite.name] = suiteResults;
 
             if (self._client && self._client.didRunTest)
-                self._client.didRunTest(suite, test);
+                self._client.didRunTest(testData);
 
             state.next();
             if (state.currentSuite() != suite)
diff --git a/PerformanceTests/Animometer/resources/statistics.js b/PerformanceTests/Animometer/resources/statistics.js
new file mode 100644 (file)
index 0000000..ee7c24c
--- /dev/null
@@ -0,0 +1,377 @@
+Pseudo =
+{
+    initialRandomSeed: 49734321,
+    randomSeed: 49734321,
+
+    resetRandomSeed: function()
+    {
+        Pseudo.randomSeed = Pseudo.initialRandomSeed;
+    },
+
+    random: function()
+    {
+        var randomSeed = Pseudo.randomSeed;
+        randomSeed = ((randomSeed + 0x7ed55d16) + (randomSeed << 12))  & 0xffffffff;
+        randomSeed = ((randomSeed ^ 0xc761c23c) ^ (randomSeed >>> 19)) & 0xffffffff;
+        randomSeed = ((randomSeed + 0x165667b1) + (randomSeed << 5))   & 0xffffffff;
+        randomSeed = ((randomSeed + 0xd3a2646c) ^ (randomSeed << 9))   & 0xffffffff;
+        randomSeed = ((randomSeed + 0xfd7046c5) + (randomSeed << 3))   & 0xffffffff;
+        randomSeed = ((randomSeed ^ 0xb55a4f09) ^ (randomSeed >>> 16)) & 0xffffffff;
+        Pseudo.randomSeed = randomSeed;
+        return (randomSeed & 0xfffffff) / 0x10000000;
+    }
+};
+
+Statistics =
+{
+    sampleMean: function(numberOfSamples, sum)
+    {
+        if (numberOfSamples < 1)
+            return 0;
+        return sum / numberOfSamples;
+    },
+
+    // With sum and sum of squares, we can compute the sample standard deviation in O(1).
+    // See https://rniwa.com/2012-11-10/sample-standard-deviation-in-terms-of-sum-and-square-sum-of-samples/
+    unbiasedSampleStandardDeviation: function(numberOfSamples, sum, squareSum)
+    {
+        if (numberOfSamples < 2)
+            return 0;
+        return Math.sqrt((squareSum - sum * sum / numberOfSamples) / (numberOfSamples - 1));
+    },
+
+    geometricMean: function(values)
+    {
+        if (!values.length)
+            return 0;
+        var roots = values.map(function(value) { return Math.pow(value, 1 / values.length); })
+        return roots.reduce(function(a, b) { return a * b; });
+    },
+
+    // Cumulative distribution function
+    cdf: function(value, mean, standardDeviation)
+    {
+        return 0.5 * (1 + Statistics.erf((value - mean) / (Math.sqrt(2 * standardDeviation * standardDeviation))));
+    },
+
+    // Approximation of Gauss error function, Abramowitz and Stegun 7.1.26
+    erf: function(value)
+    {
+          var sign = (value >= 0) ? 1 : -1;
+          value = Math.abs(value);
+
+          var a1 = 0.254829592;
+          var a2 = -0.284496736;
+          var a3 = 1.421413741;
+          var a4 = -1.453152027;
+          var a5 = 1.061405429;
+          var p = 0.3275911;
+
+          var t = 1.0 / (1.0 + p * value);
+          var y = 1.0 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-value * value);
+          return sign * y;
+    }
+};
+
+Experiment = Utilities.createClass(
+    function(includeConcern)
+    {
+        if (includeConcern)
+            this._maxHeap = Heap.createMaxHeap(Experiment.defaults.CONCERN_SIZE);
+        this.reset();
+    }, {
+
+    reset: function()
+    {
+        this._sum = 0;
+        this._squareSum = 0;
+        this._numberOfSamples = 0;
+        if (this._maxHeap)
+            this._maxHeap.init();
+    },
+
+    get sampleCount()
+    {
+        return this._numberOfSamples;
+    },
+
+    sample: function(value)
+    {
+        this._sum += value;
+        this._squareSum += value * value;
+        if (this._maxHeap)
+            this._maxHeap.push(value);
+        ++this._numberOfSamples;
+    },
+
+    mean: function()
+    {
+        return Statistics.sampleMean(this._numberOfSamples, this._sum);
+    },
+
+    standardDeviation: function()
+    {
+        return Statistics.unbiasedSampleStandardDeviation(this._numberOfSamples, this._sum, this._squareSum);
+    },
+
+    cdf: function(value)
+    {
+        return Statistics.cdf(value, this.mean(), this.standardDeviation());
+    },
+
+    percentage: function()
+    {
+        var mean = this.mean();
+        return mean ? this.standardDeviation() * 100 / mean : 0;
+    },
+
+    concern: function(percentage)
+    {
+        if (!this._maxHeap)
+            return this.mean();
+
+        var size = Math.ceil(this._numberOfSamples * percentage / 100);
+        var values = this._maxHeap.values(size);
+        return values.length ? values.reduce(function(a, b) { return a + b; }) / values.length : 0;
+    },
+
+    score: function(percentage)
+    {
+        return Statistics.geometricMean([this.mean(), Math.max(this.concern(percentage), 1)]);
+    }
+});
+
+Experiment.defaults =
+{
+    CONCERN: 5,
+    CONCERN_SIZE: 100,
+};
+
+Regression = Utilities.createClass(
+    function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+    {
+        var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+            shouldClip: true,
+            s1: 1000/60,
+            t1: 0
+        });
+        var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+            shouldClip: true,
+            s1: 1000/60,
+            t1: 0,
+            t2: 0
+        });
+        var desired;
+        if (slope.error < flat.error)
+            desired = slope;
+        else
+            desired = flat;
+
+        this.startIndex = Math.min(startIndex, endIndex);
+        this.endIndex = Math.max(startIndex, endIndex);
+
+        this.complexity = desired.complexity;
+        this.s1 = desired.s1;
+        this.t1 = desired.t1;
+        this.s2 = desired.s2;
+        this.t2 = desired.t2;
+        this.stdev1 = desired.stdev1;
+        this.stdev2 = desired.stdev2;
+        this.n1 = desired.n1;
+        this.n2 = desired.n2;
+        this.error = desired.error;
+    }, {
+
+    valueAt: function(complexity)
+    {
+        if (this.n1 == 1 || complexity > this.complexity)
+            return this.s2 + this.t2 * complexity;
+        return this.s1 + this.t1 * complexity;
+    },
+
+    // A generic two-segment piecewise regression calculator. Based on Kundu/Ubhaya
+    //
+    // Minimize sum of (y - y')^2
+    // where                        y = s1 + t1*x
+    //                              y = s2 + t2*x
+    //                y' = s1 + t1*x' = s2 + t2*x'   if x_0 <= x' <= x_n
+    //
+    // Allows for fixing s1, t1, s2, t2
+    //
+    // x is assumed to be complexity, y is frame length. Can be used for pure complexity-FPS
+    // analysis or for ramp controllers since complexity monotonically decreases with time.
+    _calculateRegression: function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+    {
+        if (startIndex == endIndex) {
+            // Only one sample point; we can't calculate any regression.
+            var x = getComplexity(samples, startIndex);
+            return {
+                complexity: x,
+                s1: x,
+                t1: 0,
+                s2: x,
+                t2: 0,
+                error1: 0,
+                error2: 0
+            };
+        }
+
+        var iterationDirection = endIndex > startIndex ? 1 : -1;
+        var lowComplexity = getComplexity(samples, startIndex);
+        var highComplexity = getComplexity(samples, endIndex);
+        var a1 = 0, b1 = 0, c1 = 0, d1 = 0, h1 = 0, k1 = 0;
+        var a2 = 0, b2 = 0, c2 = 0, d2 = 0, h2 = 0, k2 = 0;
+
+        // Iterate from low to high complexity
+        for (var i = startIndex; iterationDirection * (endIndex - i) > -1; i += iterationDirection) {
+            var x = getComplexity(samples, i);
+            var y = getFrameLength(samples, i);
+            a2 += 1;
+            b2 += x;
+            c2 += x * x;
+            d2 += y;
+            h2 += y * x;
+            k2 += y * y;
+        }
+
+        var s1_best, t1_best, s2_best, t2_best, n1_best, n2_best, error1_best, error2_best, x_best, x_prime;
+
+        function setBest(s1, t1, error1, s2, t2, error2, splitIndex, x_prime, x)
+        {
+            s1_best = s1;
+            t1_best = t1;
+            error1_best = error1;
+            s2_best = s2;
+            t2_best = t2;
+            error2_best = error2;
+            n1_best = iterationDirection * (splitIndex - startIndex) + 1;
+            n2_best = iterationDirection * (endIndex - splitIndex);
+            if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
+                x_best = x_prime;
+            else {
+                // Discontinuous piecewise regression
+                x_best = x;
+            }
+        }
+
+        // Iterate from startIndex to endIndex - 1, inclusive
+        for (var i = startIndex; iterationDirection * (endIndex - i) > 0; i += iterationDirection) {
+            var x = getComplexity(samples, i);
+            var y = getFrameLength(samples, i);
+            var xx = x * x;
+            var yx = y * x;
+            var yy = y * y;
+            // a1, b1, etc. is sum from startIndex to i, inclusive
+            a1 += 1;
+            b1 += x;
+            c1 += xx;
+            d1 += y;
+            h1 += yx;
+            k1 += yy;
+            // a2, b2, etc. is sum from i+1 to endIndex, inclusive
+            a2 -= 1;
+            b2 -= x;
+            c2 -= xx;
+            d2 -= y;
+            h2 -= yx;
+            k2 -= yy;
+
+            var A = c1*d1 - b1*h1;
+            var B = a1*h1 - b1*d1;
+            var C = a1*c1 - b1*b1;
+            var D = c2*d2 - b2*h2;
+            var E = a2*h2 - b2*d2;
+            var F = a2*c2 - b2*b2;
+            var s1 = options.s1 !== undefined ? options.s1 : (A / C);
+            var t1 = options.t1 !== undefined ? options.t1 : (B / C);
+            var s2 = options.s2 !== undefined ? options.s2 : (D / F);
+            var t2 = options.t2 !== undefined ? options.t2 : (E / F);
+            // Assumes that the two segments meet
+            var x_prime = (s1 - s2) / (t2 - t1);
+
+            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
+            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
+
+            if (i == startIndex) {
+                setBest(s1, t1, error1, s2, t2, error2, i, x_prime, x);
+                continue;
+            }
+
+            if (C == 0 || F == 0)
+                continue;
+
+            // Projected point is not between this and the next sample
+            if (x_prime > getComplexity(samples, i + iterationDirection) || x_prime < x) {
+                // Calculate lambda, which divides the weight of this sample between the two lines
+
+                // These values remove the influence of this sample
+                var I = c1 - 2*b1*x + a1*xx;
+                var H = C - I;
+                var G = A + B*x - C*y;
+
+                var J = D + E*x - F*y;
+                var K = c2 - 2*b2*x + a2*xx;
+
+                var lambda = (G*F + G*K - H*J) / (I*J + G*K);
+                if (lambda > 0 && lambda < 1) {
+                    var lambda1 = 1 - lambda;
+                    s1 = options.s1 !== undefined ? options.s1 : ((A - lambda1*(-h1*x + d1*xx + c1*y - b1*yx)) / (C - lambda1*I));
+                    t1 = options.t1 !== undefined ? options.t1 : ((B - lambda1*(h1 - d1*x - b1*y + a1*yx)) / (C - lambda1*I));
+                    s2 = options.s2 !== undefined ? options.s2 : ((D + lambda1*(-h2*x + d2*xx + c2*y - b2*yx)) / (F + lambda1*K));
+                    t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
+                    x_prime = (s1 - s2) / (t2 - t1);
+
+                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
+                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
+                } else if (t1 != t2)
+                    continue;
+            }
+
+            if (error1 + error2 < error1_best + error2_best)
+                setBest(s1, t1, error1, s2, t2, error2, i, x_prime, x);
+        }
+
+        return {
+            complexity: x_best,
+            s1: s1_best,
+            t1: t1_best,
+            stdev1: Math.sqrt(error1_best / n1_best),
+            s2: s2_best,
+            t2: t2_best,
+            stdev2: Math.sqrt(error2_best / n2_best),
+            error: error1_best + error2_best,
+            n1: n1_best,
+            n2: n2_best
+        };
+    }
+});
+
+Utilities.extendObject(Regression, {
+    bootstrap: function(samples, iterationCount, processResample, confidencePercentage)
+    {
+        var sampleLength = samples.length;
+        var resample = new Array(sampleLength);
+
+        var bootstrapEstimator = new Experiment;
+        var bootstrapData = new Array(iterationCount);
+
+        Pseudo.resetRandomSeed();
+        for (var i = 0; i < iterationCount; ++i) {
+            for (var j = 0; j < sampleLength; ++j)
+                resample[j] = samples[Math.floor(Pseudo.random() * sampleLength)];
+
+            var resampleResult = processResample(resample);
+            bootstrapEstimator.sample(resampleResult);
+            bootstrapData[i] = resampleResult;
+        }
+
+        bootstrapData.sort(function(a, b) { return a - b; });
+        return {
+            confidenceLow: bootstrapData[Math.round((iterationCount - 1) * (1 - confidencePercentage) / 2)],
+            confidenceHigh: bootstrapData[Math.round((iterationCount - 1) * (1 + confidencePercentage) / 2)],
+            median: bootstrapData[Math.round(iterationCount / 2)],
+            mean: bootstrapEstimator.mean(),
+            data: bootstrapData
+        };
+    }
+});
index cc03bf07a0c1a8ffa61d9059190005660d295d1a..19229aa64b365e516c94a9443270e039185b6748 100644 (file)
@@ -1,53 +1,38 @@
 var Strings = {
     text: {
         testName: "Test Name",
-        score: "Score",
-        samples: "Samples",
-
-        complexity: "Complexity",
-        frameRate: "FPS",
-        mergedRawComplexity: "Merged raw",
-        mergedAverageComplexity: "Merged average",
-        graph: "Graph"
+        score: "Score"
     },
     json: {
-        score: "score",
-        samples: "samples",
-        complexityAverageSamples: "complexityAverageSamples",
         marks: "marks",
-
-        targetFrameLength: "targetFrameLength",
         samplingStartTimeOffset: "Start sampling",
         samplingEndTimeOffset: "End sampling",
 
-        experiments: {
-            complexity: "complexity",
-            frameRate: "frameRate"
+        samples: "samples",
+        controller: "controller",
+        complexity: "complexity",
+        complexityAverage: "complexityAverage",
+        frameLength: "frameLength",
+
+        result: "result",
+        score: "score",
+        bootstrap: "bootstrap",
+        measurements: {
+            average: "average",
+            concern: "concern",
+            stdev: "stdev",
+            percent: "percent"
         },
 
         regressions: {
-            timeRegressions: "timeRegressions",
-            complexity: "complexity",
-            maxComplexity: "maxComplexity",
             startIndex: "startIndex",
             endIndex: "endIndex",
-
-            complexityRegression: "complexityRegression",
-            complexityAverageRegression: "complexityAverageRegression",
             segment1: "segment1",
             segment2: "segment2"
         },
 
-        measurements: {
-            average: "average",
-            concern: "concern",
-            stdev: "stdev",
-            percent: "percent"
-        },
-
         results: {
             iterations: "iterationsResults",
-            suites: "suitesResults",
             tests: "testsResults"
         }
     }
index 84bcf18ef100b4f44e89b3acee7d5e30fad17b19..864c19adcd6200b2956585d2e5d223236870d324 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         img.hidden {
@@ -15,8 +16,7 @@
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index ac47fc6ab3542a7ecda049ebb90c2eca2dc91f6e..c759d45c088750802202c5ad5f3897ef21ff0a16 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index bfaca496e85f2313b357bcfa557bae2053a94afb..058de486de6431f7965dffd701266f2867c15c26 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         img {
@@ -12,8 +13,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index 0f0f579a30da149c4b1749ac53bd5ea443221319..0c1248f4903525062b61ad55ae380599f723f533 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         .circle {
@@ -23,8 +24,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index bb2905056f60961b877aa065e2e1d98b8b1770e7..2b6a0b9db50abdab8decff6f1e6df61e818dfdb7 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <svg id="stage"></svg>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index ae5193ef5c9371bb64c9ce30f087775c4764d665..a821231672fe67774f42b19e3471c90cdfa949d1 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <svg id="stage"></svg>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/bouncing-particles.js"></script>
index 36fc2f7182a14626cd7224bdd7f57983c6ed1748..123c4bc6737c50673f8d47c16b1ae4ca365a7169 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/canvas-stage.js"></script>
index 526347379cf04427fe8d5c68cf5e6477d57c8337..2a5c600e1d0cd031f29d3991b07c58864361487d 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
     <style type="text/css">
 
@@ -33,8 +34,7 @@
     </div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/focus.js"></script>
index bc92e65b81458ea2534b32ff3f7cbbdc02916f1e..4c5fa8dcf1c5483f5a4fe666fe75133cff92960a 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style type="text/css">
 
@@ -18,8 +19,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/image-data.js"></script>
index be1758db7e92e13aa7bf722bd5098ca9a3052c8b..7e7f07b31b9b0cecccaa7aa32b23e7eb0b6ee19b 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
     <style type="text/css">
 
@@ -51,8 +52,7 @@
     </div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/multiply.js"></script>
index 607d714832674d3b5a2afa5c79ab8252de7e7f2b..8de065e10766d2236ec21b873b51c29c4604c129 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="resources/stage.css">
     <style>
         #stage div {
@@ -14,8 +15,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/particles.js"></script>
index a9e957b6975cbccf8d3587b078c51c97edfe787e..b1d6db7ed836478430ade15271a51914879959a3 100644 (file)
@@ -7,15 +7,15 @@ CanvasLineSegment = Utilities.createClass(
     {
         var circle = Stage.randomInt(0, 2);
         this._color = ["#e01040", "#10c030", "#e05010"][circle];
-        this._lineWidth = Math.pow(Math.random(), 12) * 20 + 3;
-        this._omega = Math.random() * 3 + 0.2;
+        this._lineWidth = Math.pow(Pseudo.random(), 12) * 20 + 3;
+        this._omega = Pseudo.random() * 3 + 0.2;
         var theta = Stage.randomAngle();
         this._cosTheta = Math.cos(theta);
         this._sinTheta = Math.sin(theta);
         this._startX = stage.circleRadius * this._cosTheta + (0.5 + circle) / 3 * stage.size.x;
         this._startY = stage.circleRadius * this._sinTheta + stage.size.y / 2;
-        this._length = Math.pow(Math.random(), 8) * 40 + 20;
-        this._segmentDirection = Math.random() > 0.5 ? -1 : 1;
+        this._length = Math.pow(Pseudo.random(), 8) * 40 + 20;
+        this._segmentDirection = Pseudo.random() > 0.5 ? -1 : 1;
     }, {
 
     draw: function(context)
@@ -44,15 +44,15 @@ CanvasArc = Utilities.createClass(
 
         this._point = new Point(distanceX * (randX + (randY % 2) / 2), distanceY * (randY + .5));
 
-        this._radius = 20 + Math.pow(Math.random(), 5) * (Math.min(distanceX, distanceY) / 1.8);
+        this._radius = 20 + Math.pow(Pseudo.random(), 5) * (Math.min(distanceX, distanceY) / 1.8);
         this._startAngle = Stage.randomAngle();
         this._endAngle = Stage.randomAngle();
-        this._omega = (Math.random() - 0.5) * 0.3;
+        this._omega = (Pseudo.random() - 0.5) * 0.3;
         this._counterclockwise = Stage.randomBool();
         var colors = ["#101010", "#808080", "#c0c0c0"];
         colors.push(["#e01040", "#10c030", "#e05010"][(randX + Math.ceil(randY / 2)) % 3]);
-        this._color = colors[Math.floor(Math.random() * colors.length)];
-        this._lineWidth = 1 + Math.pow(Math.random(), 5) * 30;
+        this._color = colors[Math.floor(Pseudo.random() * colors.length)];
+        this._lineWidth = 1 + Math.pow(Pseudo.random(), 5) * 30;
         this._doStroke = Stage.randomInt(0, 3) != 0;
     }, {
 
@@ -87,7 +87,7 @@ CanvasLinePoint = Utilities.createClass(
         var Y_LOOPS = 20;
 
         var offsets = [[-2, -1], [2, 1], [-1, 0], [1, 0], [-1, 2], [1, -2]];
-        var offset = offsets[Math.floor(Math.random() * offsets.length)];
+        var offset = offsets[Math.floor(Pseudo.random() * offsets.length)];
 
         this.coordinate = new Point(X_LOOPS/2, Y_LOOPS/2);
         if (stage.objects.length) {
@@ -111,10 +111,10 @@ CanvasLinePoint = Utilities.createClass(
         var randX = (xOff + this.coordinate.x) * stage.size.x / X_LOOPS;
         var randY = this.coordinate.y * stage.size.y / Y_LOOPS;
         var colors = ["#101010", "#808080", "#c0c0c0", "#101010", "#808080", "#c0c0c0", "#e01040"];
-        this.color = colors[Math.floor(Math.random() * colors.length)];
+        this.color = colors[Math.floor(Pseudo.random() * colors.length)];
 
-        this.width = Math.pow(Math.random(), 5) * 20 + 1;
-        this.isSplit = Math.random() > 0.9;
+        this.width = Math.pow(Pseudo.random(), 5) * 20 + 1;
+        this.isSplit = Pseudo.random() > 0.9;
         this.point = new Point(randX, randY);
     }
 );
@@ -189,7 +189,7 @@ CanvasLinePathStage = Utilities.createSubclass(SimpleCanvasStage,
 
                 context.lineTo(object.point.x, object.point.y);
 
-                if (Math.random() > 0.999)
+                if (Pseudo.random() > 0.999)
                     object.isSplit = !object.isSplit;
             }
         }
index 482e7bcca403153d7382ddcc6046451f665fc287..fc0ca59c2249fb1dacfa15c8fef6f59f681563ce 100644 (file)
@@ -18,7 +18,7 @@ var FocusElement = Utilities.createClass(
         var left = Stage.random(0, stage.size.width - 2 * radius - sizeVariance);
 
         // size and blurring are a function of depth
-        this._depth = Utilities.lerp(1 - Math.pow(Math.random(), 2), minObjectDepth, maxObjectDepth);
+        this._depth = Utilities.lerp(1 - Math.pow(Pseudo.random(), 2), minObjectDepth, maxObjectDepth);
         var distance = Utilities.lerp(this._depth, 1, sizeVariance);
         var size = 2 * radius + sizeVariance - distance;
 
@@ -32,8 +32,8 @@ var FocusElement = Utilities.createClass(
         Utilities.setElementPrefixedProperty(this.element, "filter", "blur(" + stage.getBlurValue(this._depth) + "px) opacity(" + stage.getOpacityValue(this._depth) + "%)");
 
         var depthMultiplier = Utilities.lerp(1 - this._depth, 0.8, 1);
-        this._sinMultiplier = Math.random() * Stage.randomSign() * depthMultiplier;
-        this._cosMultiplier = Math.random() * Stage.randomSign() * depthMultiplier;
+        this._sinMultiplier = Pseudo.random() * Stage.randomSign() * depthMultiplier;
+        this._cosMultiplier = Pseudo.random() * Stage.randomSign() * depthMultiplier;
     }, {
 
     animate: function(stage, sinTime, cosTime)
index d51bf514038160e4ca7f2d74d816201769d8b0b5..d251a606b847cd2885e68971d668d318e6818fa1 100644 (file)
@@ -10,7 +10,7 @@ Particle.prototype =
 {
     reset: function()
     {
-        var randSize = Math.pow(Math.random(), 4) * 25 + 15;
+        var randSize = Math.pow(Pseudo.random(), 4) * 25 + 15;
         this.size = new Point(randSize, randSize);
         this.maxPosition = this.stage.size.subtract(this.size);
         this.position = new Point(this.stage.size.x / 2, this.stage.size.y / 4);
index 0d1e9d5a003ee4aef27d4b1ee16f7f3bd1fe2fe2..bf933bd36a466a73dfdfad8c8f209d9454016f26 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/canvas-electrons.js"></script>
index ca20c819b7ee54f4d9da637d8d4d4dd6ea241828..da5b0b2036ad2c3b1f1a959e7fd76d9c1b76c8ce 100644 (file)
@@ -1,11 +1,11 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/canvas-stars.js"></script>
index 9ccb0450572b23aff4dd029f11f70874660441b5..6d5b0bf5e632890774ae57feabdb922cb4a0ea1c 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         img {
@@ -14,8 +15,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="../bouncing-particles/resources/bouncing-particles.js"></script>
diff --git a/PerformanceTests/Animometer/tests/resources/algorithm.js b/PerformanceTests/Animometer/tests/resources/algorithm.js
deleted file mode 100644 (file)
index 41188b1..0000000
+++ /dev/null
@@ -1,134 +0,0 @@
-function Heap(maxSize, compare)
-{
-    this._maxSize = maxSize;
-    this._compare = compare;
-    this._size = 0;
-    this._values = new Array(this._maxSize);
-}
-
-Heap.prototype =
-{
-    // This is a binary heap represented in an array. The root element is stored
-    // in the first element in the array. The root is followed by its two children.
-    // Then its four grandchildren and so on. So every level in the binary heap is
-    // doubled in the following level. Here is an example of the node indices and
-    // how they are related to their parents and children.
-    // ===========================================================================
-    //              0       1       2       3       4       5       6
-    // PARENT       -1      0       0       1       1       2       2
-    // LEFT         1       3       5       7       9       11      13
-    // RIGHT        2       4       6       8       10      12      14
-    // ===========================================================================
-    _parentIndex: function(i)
-    {
-        return i > 0 ? Math.floor((i - 1) / 2) : -1;
-    },
-
-    _leftIndex: function(i)
-    {
-        var leftIndex = i * 2 + 1;
-        return leftIndex < this._size ? leftIndex : -1;
-    },
-
-    _rightIndex: function(i)
-    {
-        var rightIndex = i * 2 + 2;
-        return rightIndex < this._size ? rightIndex : -1;
-    },
-
-    // Return the child index that may violate the heap property at index i.
-    _childIndex: function(i)
-    {
-        var left = this._leftIndex(i);
-        var right = this._rightIndex(i);
-
-        if (left != -1 && right != -1)
-            return this._compare(this._values[left], this._values[right]) > 0 ? left : right;
-
-        return left != -1 ? left : right;
-    },
-
-    init: function()
-    {
-        this._size = 0;
-    },
-
-    top: function()
-    {
-        return this._size ? this._values[0] : NaN;
-    },
-
-    push: function(value)
-    {
-        if (this._size == this._maxSize) {
-            // If size is bounded and the new value can be a parent of the top()
-            // if the size were unbounded, just ignore the new value.
-            if (this._compare(value, this.top()) > 0)
-                return;
-            this.pop();
-        }
-        this._values[this._size++] = value;
-        this._bubble(this._size - 1);
-    },
-
-    pop: function()
-    {
-        if (!this._size)
-            return NaN;
-
-        this._values[0] = this._values[--this._size];
-        this._sink(0);
-    },
-
-    _bubble: function(i)
-    {
-        // Fix the heap property at index i given that parent is the only node that
-        // may violate the heap property.
-        for (var pi = this._parentIndex(i); pi != -1; i = pi, pi = this._parentIndex(pi)) {
-            if (this._compare(this._values[pi], this._values[i]) > 0)
-                break;
-
-            this._values.swap(pi, i);
-        }
-    },
-
-    _sink: function(i)
-    {
-        // Fix the heap property at index i given that each of the left and the right
-        // sub-trees satisfies the heap property.
-        for (var ci = this._childIndex(i); ci != -1; i = ci, ci = this._childIndex(ci)) {
-            if (this._compare(this._values[i], this._values[ci]) > 0)
-                break;
-
-            this._values.swap(ci, i);
-        }
-    },
-
-    str: function()
-    {
-        var out = "Heap[" + this._size + "] = [";
-        for (var i = 0; i < this._size; ++i) {
-            out += this._values[i];
-            if (i < this._size - 1)
-                out += ", ";
-        }
-        return out + "]";
-    },
-
-    values: function(size) {
-        // Return the last "size" heap elements values.
-        var values = this._values.slice(0, this._size);
-        return values.sort(this._compare).slice(0, Math.min(size, this._size));
-    }
-}
-
-var Algorithm = {
-    createMinHeap: function(maxSize)
-    {
-        return new Heap(maxSize, function(a, b) { return b - a; });
-    },
-
-    createMaxHeap: function(maxSize) {
-        return new Heap(maxSize, function(a, b) { return a - b; });
-    }
-}
index dafd6db21cd094147a15d733e2e2edc0fbbbf774..529e1ce9c810b9cc997297c2f186293834abd5b8 100644 (file)
@@ -1,3 +1,40 @@
+Sampler = Utilities.createClass(
+    function(seriesCount, expectedSampleCount, processor)
+    {
+        this._processor = processor;
+
+        this.samples = [];
+        for (var i = 0; i < seriesCount; ++i) {
+            var array = new Array(expectedSampleCount);
+            array.fill(0);
+            this.samples[i] = array;
+        }
+        this.sampleCount = 0;
+    }, {
+
+    record: function() {
+        // Assume that arguments.length == this.samples.length
+        for (var i = 0; i < arguments.length; i++) {
+            this.samples[i][this.sampleCount] = arguments[i];
+        }
+        ++this.sampleCount;
+    },
+
+    processSamples: function()
+    {
+        var results = {};
+
+        // Remove unused capacity
+        this.samples = this.samples.map(function(array) {
+            return array.slice(0, this.sampleCount);
+        }, this);
+
+        this._processor.processSamples(results);
+
+        return results;
+    }
+});
+
 Controller = Utilities.createClass(
     function(benchmark, options)
     {
@@ -27,6 +64,7 @@ Controller = Utilities.createClass(
     {
         this._startTimestamp = startTimestamp;
         this._endTimestamp += startTimestamp;
+        this._previousTimestamp = startTimestamp;
         this._measureAndResetInterval(startTimestamp);
         this.recordFirstSample(startTimestamp, stage);
     },
@@ -66,11 +104,14 @@ Controller = Utilities.createClass(
 
     update: function(timestamp, stage)
     {
-        var frameLengthEstimate = -1;
+        var lastFrameLength = timestamp - this._previousTimestamp;
+        this._previousTimestamp = timestamp;
+
+        var frameLengthEstimate = -1, intervalAverageFrameLength = -1;
         var didFinishInterval = false;
         if (!this._intervalLength) {
             if (this._isFrameLengthEstimatorEnabled) {
-                this._frameLengthEstimator.sample(timestamp - this._sampler.samples[0][this._sampler.sampleCount - 1]);
+                this._frameLengthEstimator.sample(lastFrameLength);
                 frameLengthEstimate = this._frameLengthEstimator.estimate;
             }
         } else if (timestamp >= this._intervalEndTimestamp) {
@@ -85,14 +126,14 @@ Controller = Utilities.createClass(
         }
 
         this._sampler.record(timestamp, stage.complexity(), frameLengthEstimate);
-        this.tune(timestamp, stage, didFinishInterval);
+        this.tune(timestamp, stage, lastFrameLength, didFinishInterval, intervalAverageFrameLength);
     },
 
     didFinishInterval: function(timestamp, stage, intervalAverageFrameLength)
     {
     },
 
-    tune: function(timestamp, stage, didFinishInterval)
+    tune: function(timestamp, stage, lastFrameLength, didFinishInterval, intervalAverageFrameLength)
     {
     },
 
@@ -106,6 +147,38 @@ Controller = Utilities.createClass(
         return this._sampler.processSamples();
     },
 
+    _processComplexitySamples: function(complexitySamples, complexityAverageSamples)
+    {
+        complexitySamples.sort(function(a, b) {
+            return a.complexity - b.complexity;
+        });
+
+        // Samples averaged based on complexity
+        var currentComplexity = -1;
+        var experimentAtComplexity;
+        function addSample() {
+            var mean = experimentAtComplexity.mean();
+            var stdev = experimentAtComplexity.standardDeviation();
+            complexityAverageSamples.push({
+                complexity: currentComplexity,
+                frameLength: mean,
+                stdev: stdev
+            });
+        }
+        complexitySamples.forEach(function(sample) {
+            if (sample.complexity != currentComplexity) {
+                if (currentComplexity > -1)
+                    addSample();
+
+                currentComplexity = sample.complexity;
+                experimentAtComplexity = new Experiment;
+            }
+            experimentAtComplexity.sample(sample.frameLength);
+        });
+        // Finish off the last one
+        addSample();
+    },
+
     processSamples: function(results)
     {
         var complexityExperiment = new Experiment;
@@ -113,56 +186,36 @@ Controller = Utilities.createClass(
 
         var samples = this._sampler.samples;
 
-        var samplingStartIndex = 0, samplingEndIndex = -1;
-        if (Strings.json.samplingStartTimeOffset in this._marks)
-            samplingStartIndex = this._marks[Strings.json.samplingStartTimeOffset].index;
-        if (Strings.json.samplingEndTimeOffset in this._marks)
-            samplingEndIndex = this._marks[Strings.json.samplingEndTimeOffset].index;
-
         for (var markName in this._marks)
             this._marks[markName].time -= this._startTimestamp;
         results[Strings.json.marks] = this._marks;
 
-        results[Strings.json.samples] = samples[0].map(function(timestamp, i) {
-            var result = {
+        results[Strings.json.samples] = {};
+
+        var complexitySamples = [], complexityAverageSamples = [];
+        results[Strings.json.samples][Strings.json.complexity] = complexitySamples;
+        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
+
+        results[Strings.json.samples][Strings.json.controller] = samples[0].map(function(timestamp, i) {
+            var sample = {
                 // Represent time in milliseconds
                 time: timestamp - this._startTimestamp,
                 complexity: samples[1][i]
             };
 
             if (i == 0)
-                result.frameLength = 1000/60;
+                sample.frameLength = 1000/60;
             else
-                result.frameLength = timestamp - samples[0][i - 1];
+                sample.frameLength = timestamp - samples[0][i - 1];
 
             if (samples[2][i] != -1)
-                result.smoothedFrameLength = samples[2][i];
-
-            // Don't start adding data to the experiments until we reach the sampling timestamp
-            if (i >= samplingStartIndex && (samplingEndIndex == -1 || i < samplingEndIndex)) {
-                complexityExperiment.sample(result.complexity);
-                if (result.smoothedFrameLength && result.smoothedFrameLength != -1)
-                    smoothedFrameLengthExperiment.sample(result.smoothedFrameLength);
-            }
+                sample.smoothedFrameLength = samples[2][i];
 
-            return result;
+            complexitySamples.push(sample);
+            return sample;
         }, this);
 
-        results[Strings.json.score] = complexityExperiment.score(Experiment.defaults.CONCERN);
-
-        var complexityResults = {};
-        results[Strings.json.experiments.complexity] = complexityResults;
-        complexityResults[Strings.json.measurements.average] = complexityExperiment.mean();
-        complexityResults[Strings.json.measurements.concern] = complexityExperiment.concern(Experiment.defaults.CONCERN);
-        complexityResults[Strings.json.measurements.stdev] = complexityExperiment.standardDeviation();
-        complexityResults[Strings.json.measurements.percent] = complexityExperiment.percentage();
-
-        var smoothedFrameLengthResults = {};
-        results[Strings.json.experiments.frameRate] = smoothedFrameLengthResults;
-        smoothedFrameLengthResults[Strings.json.measurements.average] = 1000 / smoothedFrameLengthExperiment.mean();
-        smoothedFrameLengthResults[Strings.json.measurements.concern] = smoothedFrameLengthExperiment.concern(Experiment.defaults.CONCERN);
-        smoothedFrameLengthResults[Strings.json.measurements.stdev] = smoothedFrameLengthExperiment.standardDeviation();
-        smoothedFrameLengthResults[Strings.json.measurements.percent] = smoothedFrameLengthExperiment.percentage();
+        this._processComplexitySamples(complexitySamples, complexityAverageSamples);
     }
 });
 
@@ -250,174 +303,6 @@ AdaptiveController = Utilities.createSubclass(Controller,
         // Start the next interval.
         this._intervalFrameCount = 0;
         this._intervalTimestamp = timestamp;
-    },
-
-    processSamples: function(results)
-    {
-        Controller.prototype.processSamples.call(this, results);
-        results[Strings.json.targetFrameLength] = 1000 / this._targetFrameRate;
-    }
-});
-
-Regression = Utilities.createClass(
-    function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
-    {
-        var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
-            shouldClip: true,
-            s1: 1000/60,
-            t1: 0
-        });
-        var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
-            shouldClip: true,
-            t1: 0,
-            t2: 0
-        });
-        var desired;
-        if (slope.error < flat.error)
-            desired = slope;
-        else
-            desired = flat;
-
-        this.startIndex = Math.min(startIndex, endIndex);
-        this.endIndex = Math.max(startIndex, endIndex);
-
-        this.complexity = desired.complexity;
-        this.s1 = desired.s1;
-        this.t1 = desired.t1;
-        this.s2 = desired.s2;
-        this.t2 = desired.t2;
-        this.error = desired.error;
-    }, {
-
-    // A generic two-segment piecewise regression calculator. Based on Kundu/Ubhaya
-    //
-    // Minimize sum of (y - y')^2
-    // where                        y = s1 + t1*x
-    //                              y = s2 + t2*x
-    //                y' = s1 + t1*x' = s2 + t2*x'   if x_0 <= x' <= x_n
-    //
-    // Allows for fixing s1, t1, s2, t2
-    //
-    // x is assumed to be complexity, y is frame length. Can be used for pure complexity-FPS
-    // analysis or for ramp controllers since complexity monotonically decreases with time.
-    _calculateRegression: function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
-    {
-        var iterationDirection = endIndex > startIndex ? 1 : -1;
-        var lowComplexity = getComplexity(samples, startIndex);
-        var highComplexity = getComplexity(samples, endIndex);
-        var a1 = 0, b1 = 0, c1 = 0, d1 = 0, h1 = 0, k1 = 0;
-        var a2 = 0, b2 = 0, c2 = 0, d2 = 0, h2 = 0, k2 = 0;
-
-        // Iterate from low to high complexity
-        for (var i = startIndex; iterationDirection * (endIndex - i) > -1; i += iterationDirection) {
-            var x = getComplexity(samples, i);
-            var y = getFrameLength(samples, i);
-            a2 += 1;
-            b2 += x;
-            c2 += x * x;
-            d2 += y;
-            h2 += y * x;
-            k2 += y * y;
-        }
-
-        var s1_best, t1_best, s2_best, t2_best, x_best, error_best, x_prime;
-
-        function setBest(s1, t1, s2, t2, error, x_prime, x)
-        {
-            s1_best = s1;
-            t1_best = t1;
-            s2_best = s2;
-            t2_best = t2;
-            error_best = error;
-            if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
-                x_best = x_prime;
-            else {
-                // Discontinuous piecewise regression
-                x_best = x;
-            }
-        }
-
-        // Iterate from startIndex to endIndex - 1, inclusive
-        for (var i = startIndex; iterationDirection * (endIndex - i) > 0; i += iterationDirection) {
-            var x = getComplexity(samples, i);
-            var y = getFrameLength(samples, i);
-            var xx = x * x;
-            var yx = y * x;
-            var yy = y * y;
-            // a1, b1, etc. is sum from startIndex to i, inclusive
-            a1 += 1;
-            b1 += x;
-            c1 += xx;
-            d1 += y;
-            h1 += yx;
-            k1 += yy;
-            // a2, b2, etc. is sum from i+1 to endIndex, inclusive
-            a2 -= 1;
-            b2 -= x;
-            c2 -= xx;
-            d2 -= y;
-            h2 -= yx;
-            k2 -= yy;
-
-            var A = c1*d1 - b1*h1;
-            var B = a1*h1 - b1*d1;
-            var C = a1*c1 - b1*b1;
-            var D = c2*d2 - b2*h2;
-            var E = a2*h2 - b2*d2;
-            var F = a2*c2 - b2*b2;
-            var s1 = options.s1 !== undefined ? options.s1 : (A / C);
-            var t1 = options.t1 !== undefined ? options.t1 : (B / C);
-            var s2 = options.s2 !== undefined ? options.s2 : (D / F);
-            var t2 = options.t2 !== undefined ? options.t2 : (E / F);
-            // Assumes that the two segments meet
-            var x_prime = (s1 - s2) / (t2 - t1);
-
-            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
-            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
-
-            if (i == startIndex) {
-                setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
-                continue;
-            }
-
-            // Projected point is not between this and the next sample
-            if (x_prime > getComplexity(samples, i + iterationDirection) || x_prime < x) {
-                // Calculate lambda, which divides the weight of this sample between the two lines
-
-                // These values remove the influence of this sample
-                var I = c1 - 2*b1*x + a1*xx;
-                var H = C - I;
-                var G = A + B*x - C*y;
-
-                var J = D + E*x - F*y;
-                var K = c2 - 2*b2*x + a2*xx;
-
-                var lambda = (G*F + G*K - H*J) / (I*J + G*K);
-                if (lambda > 0 && lambda < 1) {
-                    var lambda1 = 1 - lambda;
-                    s1 = options.s1 !== undefined ? options.s1 : ((A - lambda1*(-h1*x + d1*xx + c1*y - b1*yx)) / (C - lambda1*I));
-                    t1 = options.t1 !== undefined ? options.t1 : ((B - lambda1*(h1 - d1*x - b1*y + a1*yx)) / (C - lambda1*I));
-                    s2 = options.s2 !== undefined ? options.s2 : ((D + lambda1*(-h2*x + d2*xx + c2*y - b2*yx)) / (F + lambda1*K));
-                    t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
-                    x_prime = (s1 - s2) / (t2 - t1);
-
-                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
-                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
-                }
-            }
-
-            if (error1 + error2 < error_best)
-                setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
-        }
-
-        return {
-            complexity: x_best,
-            s1: s1_best,
-            t1: t1_best,
-            s2: s2_best,
-            t2: t2_best,
-            error: error_best
-        };
     }
 });
 
@@ -430,7 +315,7 @@ RampController = Utilities.createSubclass(Controller,
 
         // Initially start with a tier test to find the bounds
         // The number of objects in a tier test is 10^|_tier|
-        this._tier = 0;
+        this._tier = -.5;
         // The timestamp is first set after the first interval completes
         this._tierStartTimestamp = 0;
         // If the engine can handle the tier's complexity at 60 FPS, test for a short
@@ -439,11 +324,11 @@ RampController = Utilities.createSubclass(Controller,
         // If the engine is under stress, let the test run a little longer to let
         // the measurement settle
         this._tierSlowTestLength = 750;
+        this._minimumComplexity = 0;
         this._maximumComplexity = 0;
-        this._minimumTier = 0;
 
         // After the tier range is determined, figure out the number of ramp iterations
-        var minimumRampLength = 3000;
+        var minimumRampLength = 2500;
         var totalRampIterations = Math.max(1, Math.floor(this._endTimestamp / minimumRampLength));
         // Give a little extra room to run since the ramps won't be exactly this length
         this._rampLength = Math.floor((this._endTimestamp - totalRampIterations * this._intervalLength) / totalRampIterations);
@@ -455,10 +340,14 @@ RampController = Utilities.createSubclass(Controller,
         this._fps60Threshold = 1000/58;
         // We are looking for the complexity that will get us at least as slow this threshold
         this._fpsLowestThreshold = 1000/30;
+        // Try to make each ramp get this slow so that we can cross the break point
+        this._fpsRampSlowThreshold = 1000/45;
 
         this._finishedTierSampling = false;
-        this._startedRamps = false;
-        this._complexityPrime = new Experiment;
+        this._changePointEstimator = new Experiment;
+        this._minimumComplexityEstimator = new Experiment;
+        // Estimates all frames within an interval
+        this._intervalFrameLengthEstimator = new Experiment;
     }, {
 
     start: function(startTimestamp, stage)
@@ -479,38 +368,60 @@ RampController = Utilities.createSubclass(Controller,
                 var isAnimatingAt60FPS = currentFrameLength < this._fps60Threshold;
                 var hasFinishedSlowTierTest = timestamp > this._tierStartTimestamp + this._tierSlowTestLength;
 
+                if (!isAnimatingAt60FPS && !hasFinishedSlowTierTest)
+                    return;
+
                 // We're measuring at 60 fps, so quickly move on to the next tier, or
                 // we've slower than 60 fps, but we've let this tier run long enough to
                 // get an estimate
-                if (currentFrameLength < this._fps60Threshold || timestamp > this._tierStartTimestamp + this._tierSlowTestLength) {
-                    this._lastComplexity = currentComplexity;
-                    this._lastFrameLength = currentFrameLength;
+                this._lastTierComplexity = currentComplexity;
+                this._lastTierFrameLength = currentFrameLength;
 
+                this._tier += .5;
+                var nextTierComplexity = Math.round(Math.pow(10, this._tier));
+                stage.tune(nextTierComplexity - currentComplexity);
+
+                // Some tests may be unable to go beyond a certain capacity. If so, don't keep moving up tiers
+                if (stage.complexity() - currentComplexity > 0 || nextTierComplexity == 1) {
                     this._tierStartTimestamp = timestamp;
-                    this._tier += .5;
-                    var nextTierComplexity = Math.round(Math.pow(10, this._tier));
                     this.mark("Complexity: " + nextTierComplexity, timestamp);
-
-                    stage.tune(nextTierComplexity - currentComplexity);
+                    return;
                 }
-                return;
             } else if (timestamp < this._tierStartTimestamp + this._tierSlowTestLength)
                 return;
 
             this._finishedTierSampling = true;
+            this.isFrameLengthEstimatorEnabled = false;
+
             // Extend the test length so that the full test length is made of the ramps
             this._endTimestamp += timestamp;
             this.mark(Strings.json.samplingStartTimeOffset, timestamp);
 
+            this._minimumComplexity = 0;
+            this._possibleMinimumComplexity = this._minimumComplexity;
+            this._minimumComplexityEstimator.sample(this._minimumComplexity);
+
             // Sometimes this last tier will drop the frame length well below the threshold
             // Avoid going down that far since it means fewer measurements are taken in the 60 fps area
             // Interpolate a maximum complexity that gets us around the lowest threshold
-            this._maximumComplexity = Math.floor(this._lastComplexity + (this._fpsLowestThreshold - this._lastFrameLength) / (currentFrameLength - this._lastFrameLength) * (currentComplexity - this._lastComplexity));
+            if (this._lastTierComplexity != currentComplexity)
+                this._maximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this._fpsLowestThreshold, this._lastTierFrameLength, currentFrameLength), this._lastTierComplexity, currentComplexity));
+            else {
+                // If the browser is capable of handling the most complex version of the test, use that
+                this._maximumComplexity = currentComplexity;
+            }
+            this._possibleMaximumComplexity = this._maximumComplexity;
+
+            // If we get ourselves onto a ramp where the maximum complexity does not yield slow enough FPS,
+            // We'll use this as a boundary to find a higher maximum complexity for the next ramp
+            this._lastTierComplexity = currentComplexity;
+            this._lastTierFrameLength = currentFrameLength;
+
+            // First ramp
             stage.tune(this._maximumComplexity - currentComplexity);
-            this._rampStartTimestamp = timestamp;
             this._rampDidWarmup = false;
-            this.isFrameLengthEstimatorEnabled = false;
-            this._intervalCount = 0;
+            // Start timestamp represents start of ramp iteration and warm up
+            this._rampStartTimestamp = timestamp;
             return;
         }
 
@@ -527,15 +438,34 @@ RampController = Utilities.createSubclass(Controller,
         this._rampStartIndex = this._sampler.sampleCount;
     },
 
-    tune: function(timestamp, stage, didFinishInterval)
+    tune: function(timestamp, stage, lastFrameLength, didFinishInterval, intervalAverageFrameLength)
     {
-        if (!didFinishInterval || !this._rampDidWarmup)
+        if (!this._rampDidWarmup)
+            return;
+
+        this._intervalFrameLengthEstimator.sample(lastFrameLength);
+        if (!didFinishInterval)
             return;
 
+        var currentComplexity = stage.complexity();
+        var intervalFrameLengthMean = this._intervalFrameLengthEstimator.mean();
+        var intervalFrameLengthStandardDeviation = this._intervalFrameLengthEstimator.standardDeviation();
+
+        if (intervalFrameLengthMean < this._fps60Threshold && this._intervalFrameLengthEstimator.cdf(this._fps60Threshold) > .95) {
+            this._possibleMinimumComplexity = Math.max(this._possibleMinimumComplexity, currentComplexity);
+        } else if (intervalFrameLengthStandardDeviation > 2) {
+            // In the case where we might have found a previous interval where 60fps was reached. We hit a significant blip,
+            // so we should resample this area in the next ramp.
+            this._possibleMinimumComplexity = 0;
+        }
+        if (intervalFrameLengthMean - intervalFrameLengthStandardDeviation > this._fpsRampSlowThreshold)
+            this._possibleMaximumComplexity = Math.min(this._possibleMaximumComplexity, currentComplexity);
+        this._intervalFrameLengthEstimator.reset();
+
         var progress = (timestamp - this._rampStartTimestamp) / this._currentRampLength;
 
         if (progress < 1) {
-            stage.tune(Math.round((1 - progress) * this._maximumComplexity) - stage.complexity());
+            stage.tune(Math.floor(Utilities.lerp(progress, this._maximumComplexity, this._minimumComplexity)) - currentComplexity);
             return;
         }
 
@@ -543,14 +473,28 @@ RampController = Utilities.createSubclass(Controller,
             this._sampler.sampleCount - 1, this._rampStartIndex);
         this._rampRegressions.push(regression);
 
-        this._complexityPrime.sample(regression.complexity);
-        this._maximumComplexity = Math.max(5, Math.round(this._complexityPrime.mean() * 2));
+        var interpolatedFrameLength = regression.valueAt(this._maximumComplexity);
+        if (interpolatedFrameLength < this._fpsRampSlowThreshold)
+            this._possibleMaximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this._fpsRampSlowThreshold, interpolatedFrameLength, this._lastTierFrameLength), this._maximumComplexity, this._lastTierComplexity));
+
+        interpolatedFrameLength = regression.valueAt(this._minimumComplexity);
+        this._minimumComplexityEstimator.sample(this._possibleMinimumComplexity);
+
+        this._changePointEstimator.sample(regression.complexity);
+
+        this._minimumComplexity = Math.round(this._minimumComplexityEstimator.mean());
+        this._maximumComplexity = Math.round(this._minimumComplexity +
+            Math.max(5,
+                this._possibleMaximumComplexity - this._minimumComplexity,
+                (this._changePointEstimator.mean() - this._minimumComplexity) * 2));
 
         // Next ramp
+        stage.tune(this._maximumComplexity - stage.complexity());
         this._rampDidWarmup = false;
         // Start timestamp represents start of ramp iteration and warm up
         this._rampStartTimestamp = timestamp;
-        stage.tune(this._maximumComplexity - stage.complexity());
+        this._possibleMinimumComplexity = 0;
+        this._possibleMaximumComplexity = this._maximumComplexity;
     },
 
     _getComplexity: function(samples, i) {
@@ -567,26 +511,29 @@ RampController = Utilities.createSubclass(Controller,
 
         // Have samplingTimeOffset represent time 0
         var startTimestamp = this._marks[Strings.json.samplingStartTimeOffset].time;
-        results[Strings.json.samples].forEach(function(sample) {
-            sample.time -= startTimestamp;
-        });
+
         for (var markName in results[Strings.json.marks]) {
             results[Strings.json.marks][markName].time -= startTimestamp;
         }
 
-        var samples = results[Strings.json.samples];
-        results[Strings.json.regressions.timeRegressions] = [];
-        var complexityRegressionSamples = [];
-        var timeComplexityScore = new Experiment;
+        var timeSamples = results[Strings.json.samples][Strings.json.controller];
+        timeSamples.forEach(function(timeSample) {
+            timeSample.time -= startTimestamp;
+        });
+
+        // Aggregate all of the ramps into one big complexity-frameLength dataset
+        var complexitySamples = [], complexityAverageSamples = [];
+        results[Strings.json.samples][Strings.json.complexity] = complexitySamples;
+        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
+
+        results[Strings.json.controller] = [];
         this._rampRegressions.forEach(function(ramp) {
             var startIndex = ramp.startIndex, endIndex = ramp.endIndex;
-            var startTime = samples[startIndex].time, endTime = samples[endIndex].time;
-            var startComplexity = samples[startIndex].complexity, endComplexity = samples[endIndex].complexity;
-
-            timeComplexityScore.sample(ramp.complexity);
+            var startTime = timeSamples[startIndex].time, endTime = timeSamples[endIndex].time;
+            var startComplexity = timeSamples[startIndex].complexity, endComplexity = timeSamples[endIndex].complexity;
 
             var regression = {};
-            results[Strings.json.regressions.timeRegressions].push(regression);
+            results[Strings.json.controller].push(regression);
 
             var percentage = (ramp.complexity - startComplexity) / (endComplexity - startComplexity);
             var inflectionTime = startTime + percentage * (endTime - startTime);
@@ -599,78 +546,14 @@ RampController = Utilities.createSubclass(Controller,
                 [inflectionTime, ramp.s1 + ramp.t1 * ramp.complexity],
                 [endTime, ramp.s1 + ramp.t1 * endComplexity]
             ];
-            regression[Strings.json.regressions.complexity] = ramp.complexity;
-            regression[Strings.json.regressions.maxComplexity] = Math.max(startComplexity, endComplexity);
+            regression[Strings.json.complexity] = ramp.complexity;
             regression[Strings.json.regressions.startIndex] = startIndex;
             regression[Strings.json.regressions.endIndex] = endIndex;
 
             for (var j = startIndex; j <= endIndex; ++j)
-                complexityRegressionSamples.push(samples[j]);
-        });
-
-        // Aggregate all of the ramps into one big dataset and calculate a regression from this
-        complexityRegressionSamples.sort(function(a, b) {
-            return a.complexity - b.complexity;
-        });
-
-        // Samples averaged based on complexity
-        results[Strings.json.complexityAverageSamples] = [];
-        var currentComplexity = -1;
-        var experimentAtComplexity;
-        function addSample() {
-            results[Strings.json.complexityAverageSamples].push({
-                complexity: currentComplexity,
-                frameLength: experimentAtComplexity.mean(),
-                stdev: experimentAtComplexity.standardDeviation(),
-            });
-        }
-        complexityRegressionSamples.forEach(function(sample) {
-            if (sample.complexity != currentComplexity) {
-                if (currentComplexity > -1)
-                    addSample();
-
-                currentComplexity = sample.complexity;
-                experimentAtComplexity = new Experiment;
-            }
-            experimentAtComplexity.sample(sample.frameLength);
+                complexitySamples.push(timeSamples[j]);
         });
-        // Finish off the last one
-        addSample();
-
-        function calculateRegression(samples, key) {
-            var complexityRegression = new Regression(
-                samples,
-                function (samples, i) { return samples[i].complexity; },
-                function (samples, i) { return samples[i].frameLength; },
-                0, samples.length - 1
-            );
-            var minComplexity = samples[0].complexity;
-            var maxComplexity = samples[samples.length - 1].complexity;
-            var regression = {};
-            results[key] = regression;
-            regression[Strings.json.regressions.segment1] = [
-                [minComplexity, complexityRegression.s1 + complexityRegression.t1 * minComplexity],
-                [complexityRegression.complexity, complexityRegression.s1 + complexityRegression.t1 * complexityRegression.complexity]
-            ];
-            regression[Strings.json.regressions.segment2] = [
-                [complexityRegression.complexity, complexityRegression.s2 + complexityRegression.t2 * complexityRegression.complexity],
-                [maxComplexity, complexityRegression.s2 + complexityRegression.t2 * maxComplexity]
-            ];
-            regression[Strings.json.regressions.complexity] = complexityRegression.complexity;
-            regression[Strings.json.measurements.stdev] = Math.sqrt(complexityRegression.error / samples.length);
-        }
-
-        calculateRegression(complexityRegressionSamples, Strings.json.regressions.complexityRegression);
-        calculateRegression(results[Strings.json.complexityAverageSamples], Strings.json.regressions.complexityAverageRegression);
-
-        // Frame rate experiment result is unneeded
-        delete results[Strings.json.experiments.frameRate];
-
-        results[Strings.json.score] = timeComplexityScore.mean();
-        results[Strings.json.experiments.complexity] = {};
-        results[Strings.json.experiments.complexity][Strings.json.measurements.average] = timeComplexityScore.mean();
-        results[Strings.json.experiments.complexity][Strings.json.measurements.stdev] = timeComplexityScore.standardDeviation();
-        results[Strings.json.experiments.complexity][Strings.json.measurements.percent] = timeComplexityScore.percentage();
+        this._processComplexitySamples(complexitySamples, complexityAverageSamples);
     }
 });
 
@@ -722,17 +605,17 @@ Stage = Utilities.createClass(
 Utilities.extendObject(Stage, {
     random: function(min, max)
     {
-        return (Math.random() * (max - min)) + min;
+        return (Pseudo.random() * (max - min)) + min;
     },
 
     randomBool: function()
     {
-        return !!Math.round(Math.random());
+        return !!Math.round(Pseudo.random());
     },
 
     randomSign: function()
     {
-        return Math.random() >= .5 ? 1 : -1;
+        return Pseudo.random() >= .5 ? 1 : -1;
     },
 
     randomInt: function(min, max)
index bcbe902928f2014854f0e8fbde939750eea4328b..9c2706e2a3b985fcc1393b75737bad4846b43de0 100644 (file)
@@ -1,7 +1,6 @@
-SimpleKalmanEstimator = Utilities.createClass(
+SimpleKalmanEstimator = Utilities.createSubclass(Experiment,
     function(processError, measurementError) {
-        this._initialized = false;
-
+        Experiment.call(this, false);
         var error = .5 * (Math.sqrt(processError * processError + 4 * processError * measurementError) - processError);
         this._gain = error / (error + measurementError);
     }, {
@@ -10,16 +9,18 @@ SimpleKalmanEstimator = Utilities.createClass(
     {
         if (!this._initialized) {
             this._initialized = true;
-            this._estimatedMeasurement = newMeasurement;
+            this.estimate = newMeasurement;
             return;
         }
 
-        this._estimatedMeasurement = this._estimatedMeasurement + this._gain * (newMeasurement - this._estimatedMeasurement);
+        this.estimate = this.estimate + this._gain * (newMeasurement - this.estimate);
     },
 
-    get estimate()
+    reset: function()
     {
-        return this._estimatedMeasurement;
+        Experiment.prototype.reset.call(this);
+        this._initialized = false;
+        this.estimate = 0;
     }
 });
 
diff --git a/PerformanceTests/Animometer/tests/resources/sampler.js b/PerformanceTests/Animometer/tests/resources/sampler.js
deleted file mode 100644 (file)
index 56335b1..0000000
+++ /dev/null
@@ -1,88 +0,0 @@
-Experiment = Utilities.createClass(
-    function()
-    {
-        this._sum = 0;
-        this._squareSum = 0;
-        this._numberOfSamples = 0;
-        this._maxHeap = Algorithm.createMaxHeap(Experiment.defaults.CONCERN_SIZE);
-    }, {
-
-    sample: function(value)
-    {
-        this._sum += value;
-        this._squareSum += value * value;
-        this._maxHeap.push(value);
-        ++this._numberOfSamples;
-    },
-
-    mean: function()
-    {
-        return Statistics.sampleMean(this._numberOfSamples, this._sum);
-    },
-
-    standardDeviation: function()
-    {
-        return Statistics.unbiasedSampleStandardDeviation(this._numberOfSamples, this._sum, this._squareSum);
-    },
-
-    percentage: function()
-    {
-        var mean = this.mean();
-        return mean ? this.standardDeviation() * 100 / mean : 0;
-    },
-
-    concern: function(percentage)
-    {
-        var size = Math.ceil(this._numberOfSamples * percentage / 100);
-        var values = this._maxHeap.values(size);
-        return values.length ? values.reduce(function(a, b) { return a + b; }) / values.length : 0;
-    },
-
-    score: function(percentage)
-    {
-        return Statistics.geometricMean([this.mean(), Math.max(this.concern(percentage), 1)]);
-    }
-});
-
-Experiment.defaults =
-{
-    CONCERN: 5,
-    CONCERN_SIZE: 100,
-}
-
-Sampler = Utilities.createClass(
-    function(seriesCount, expectedSampleCount, processor)
-    {
-        this._processor = processor;
-
-        this.samples = [];
-        for (var i = 0; i < seriesCount; ++i) {
-            var array = new Array(expectedSampleCount);
-            array.fill(0);
-            this.samples[i] = array;
-        }
-        this.sampleCount = 0;
-    }, {
-
-    record: function() {
-        // Assume that arguments.length == this.samples.length
-        for (var i = 0; i < arguments.length; i++) {
-            this.samples[i][this.sampleCount] = arguments[i];
-        }
-        ++this.sampleCount;
-    },
-
-    processSamples: function()
-    {
-        var results = {};
-
-        // Remove unused capacity
-        this.samples = this.samples.map(function(array) {
-            return array.slice(0, this.sampleCount);
-        }, this);
-
-        this._processor.processSamples(results);
-
-        return results;
-    }
-});
index f267820bbf4b0874721b91690a70b3366d8b6e7b..5bb69bc54f702bd6018d6c95acfba1081fca5e93 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="../master/resources/canvas-stage.js"></script>
index 5c5b4c82da68ecb58122c5a0ded598fc0d82134a..c7c0fef77402d17ab992799c2b8015f0260d0915 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/tiled-canvas-image.js"></script>
index 11ddfaaea3dafffb92030ac785e94a08f43e8827..dcb7f52861bbfef0973b576ceb52462bda07a722 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <canvas id="stage"></canvas>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/template-canvas.js"></script>
index b2ba479e02d5cdb6dc911e7854d898712a94c8b2..70eb214e9505302b721b502be299406bdd1fc06e 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/template-css.js"></script>
index 425ec369d7ff9267139041603be96800f01b3c7e..7a14fc45d68d68a35ed5710103d038faff42361a 100644 (file)
@@ -1,14 +1,14 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
 </head>
 <body>
     <svg id="stage"></svg>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/template-svg.js"></script>
index 4180bbaa5e896e63d0f45b3f50007c85abec7696..d7abc537ebb9a6eb739232c78d935773148fbcb9 100644 (file)
@@ -1,6 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
+    <meta charset="utf-8">
     <link rel="stylesheet" type="text/css" href="../resources/stage.css">
     <style>
         .text-layer {
@@ -24,8 +25,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="resources/layering-text.js"></script>
index 828ca71ddaf1b2556271a45ada8687ee8388016e..6c489c3080e66723e1fd04671da718e8d802d82b 100644 (file)
@@ -1,7 +1,7 @@
 <!DOCTYPE html>
 <html>
 <head>
-    <meta charset="UTF-8">
+    <meta charset="utf-8">
     <style>
         .text-layer {
             position: absolute;
@@ -52,8 +52,7 @@
     <div id="stage"></div>
     <script src="../../resources/strings.js"></script>
     <script src="../../resources/extensions.js"></script>
-    <script src="../resources/algorithm.js"></script>
-    <script src="../resources/sampler.js"></script>
+    <script src="../../resources/statistics.js"></script>
     <script src="../resources/math.js"></script>
     <script src="../resources/main.js"></script>
     <script src="../bouncing-particles/resources/bouncing-particles.js"></script>
index 01b7e0eb20bbd800ff93d9fd16bcf59db135cf5d..6029ac41ddd980638bcb43789dd53453292cee62 100644 (file)
@@ -1,3 +1,286 @@
+2016-02-24  Jon Lee  <jonlee@apple.com>
+
+        Update animation benchmark and tests
+        https://bugs.webkit.org/show_bug.cgi?id=154673
+
+        Reviewed by Dean Jackson.
+
+        Update the ramp controller.
+
+        The controller refines the complexity interval to test across.
+
+        * Animometer/resources/statistics.js: Add functions that estimate cumulative distribution function.
+        (Regression): For the flat regression, force the first segment to be at 60 fps.
+        (valueAt): Add convenience function to return interpolated value based on the regression used.
+        (_calculateRegression): Include the number of points included for both segments, and the piecewise
+        errors.
+        * Animometer/tests/resources/math.js: Make the Kalman estimator subclass Experiment, and allow it
+        to be reset.
+
+        * Animometer/tests/resources/main.js: Initialize the tier such that it starts at 10^0 = 1.
+        Increase the number of ramps. Maintain three FPS thresholds-- the frame rate of interest, a limit
+        on the lowest FPS we care to go for later interpolation, and a minimum FPS threshold we want to
+        aim for each ramp. Also keep three estimators: a running average of the change point, a minimum
+        boundary for each ramp, and an estimator for all the frames within an interval. The first two
+        are used to determine the parameters of the next ramp, and the latter allows us to refine the
+        parameters.
+        (update): During the tier phase, it is possible that the highest complexity possible for a test
+        won't stress the system enough to trigger stopping the tier phase and transitioning to the ramps.
+        If the complexity doesn't change when going to the next tier, we've maxed the test out, and move
+        on. When the tier phase completed, turn off Controller.frameLengthEstimator, which estimates the
+        FPS at each tier.
+        (tune): At each interval, look at the confidence distribution of being on the 60 FPS side or the
+        slow side. If the slowest FPS we achieve at the ramp's maximum complexity is not at least
+        _fpsRampSlowThreshold, then increase the maximum complexity. If we ever achieve 60 FPS, increase
+        the ramp's minimum complexity to that level. If, at an even lower complexity, a glitch causes the
+        FPS to drop, we reset the minimum complexity.
+
+        Have the bootstrap calculation occur between tests. Clean up harness.
+
+        * Animometer/resources/debug-runner/animometer.js: Run bootstrap after a test has
+        completed to avoid doing all of it at the end before showing the results. Clean up
+        parameters being passed around.
+        * Animometer/resources/debug-runner/tests.js:
+        (text):
+        * Animometer/resources/runner/animometer.js:
+        (this._processData.calculateScore): Save the results to the same object holding the data.
+        (this._processData._processData): In the case where a file is dragged, calculate the score
+        serially. Grab the results object and move it to the results variable and remove it from
+        the data object. This avoids serializing the results into the JSON.
+        (this._processData.findRegression): Include the samples used for bootstrapping. Reduce the
+        resample size to shorten the wait.
+        * Animometer/resources/runner/benchmark-runner.js:
+        * Animometer/resources/statistics.js:
+        (bootstrap): Update how bootstrapData is sorted. In some regression results the mix of
+        floats and integers causes an alphabetical sort to occur.
+        * Animometer/resources/strings.js:
+
+        Add meta charset so that encodings between harness and test match.
+
+        * Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
+        * Animometer/tests/master/canvas-stage.html:
+        * Animometer/tests/master/focus.html:
+        * Animometer/tests/master/image-data.html:
+        * Animometer/tests/master/multiply.html:
+        * Animometer/tests/master/particles.html:
+        * Animometer/tests/misc/canvas-electrons.html:
+        * Animometer/tests/misc/canvas-stars.html:
+        * Animometer/tests/misc/compositing-transforms.html:
+        * Animometer/tests/simple/simple-canvas-paths.html:
+        * Animometer/tests/simple/tiled-canvas-image.html:
+        * Animometer/tests/template/template-canvas.html:
+        * Animometer/tests/template/template-css.html:
+        * Animometer/tests/template/template-svg.html:
+        * Animometer/tests/text/layering-text.html:
+        * Animometer/tests/text/text-boxes.html:
+
+        Update test harness reporting.
+
+        * Animometer/developer.html: Add missing meta charset.
+        * Animometer/index.html: Remove unnecessary utf-8 declaration.
+        * Animometer/resources/debug-runner/animometer.css: Add convenience classes for
+        formatting the results table.
+        * Animometer/resources/debug-runner/animometer.js: Adjust which stats are shown.
+        * Animometer/resources/debug-runner/tests.js: Display bootstrapping statistics.
+        * Animometer/resources/strings.js: Move strings not used by the release harness.
+
+        Switch to a pseudo-random number generator.
+
+        * Animometer/resources/statistics.js: Add a Pseudo class, with a simple
+        pseudo-random number generator.
+        (_calculateRegression): Reset the generator before running bootstrap.
+        (bootstrap): Deleted.
+
+        Replace Math.random with Pseudo.random.
+        * Animometer/tests/master/resources/canvas-tests.js:
+        * Animometer/tests/master/resources/focus.js:
+        * Animometer/tests/master/resources/particles.js:
+        * Animometer/tests/resources/main.js:
+
+        Use bootstrapping to get confidence interval in the breakpoint.
+
+        For the ramp controller, calculate the piecewise regression, and then use
+        bootstrapping in order to find the 95% confidence interval. Use the raw data.
+
+        * Animometer/developer.html: Default to the complexity graph. Add a legend
+        checkbox to toggle visibility of the bootstrap score and histogram.
+        * Animometer/resources/debug-runner/animometer.css: Make some more space to show
+        the old raw and average scores in the legend. Add new styles for the data.
+        * Animometer/resources/debug-runner/graph.js:
+        (_addRegressionLine): Allow passing an array for the variance bar tied to the
+        regression line. Now |stdev| is |range|.
+        (createComplexityGraph): Add bootstrap median, and overlay a histogram of
+        the bootstrap samples. Switch raw samples from circles to X's.
+        (onComplexityGraphOptionsChanged): Allow toggling of the bootstrap data.
+        (onGraphTypeChanged): Move the regressions for the raw and average samples to the
+        legend. In the subtitle use the bootstrap median, and include the 95% confidence
+        interval.
+        * Animometer/resources/runner/animometer.js:
+        (this._processData.findRegression): Factor out the code that determines which
+        samples to include when calculating the piecewise regression. For series that have
+        many samples, or a wider range of recorded complexities, throw away the 2.5%
+        lowest and highest samples before calculating the regression. Keep all samples
+        if the number of samples to regress is small or the range of complexities is
+        narrow.
+        (this._processData._calculateScore): Factor out regression calculation to
+        findRegression(). Bootstrap the change point of the regression. The score is the
+        median.
+        * Animometer/resources/statistics.js:
+        (_calculateRegression): Correct an issue in the calculation of the regression, where
+        the denominator can be 0.
+        (bootstrap): Template for bootstrapping. Create a bootstrap sample array, Create
+        re-samples by random selection with replacement. Return the 95% confidence samples,
+        the bootstrap median, mean, and the data itself.
+        * Animometer/resources/strings.js: Add bootstrap.
+        * Animometer/tests/resources/main.js:
+        (processSamples): Don't prematurely cut the sample data.
+
+        Fix graph drawing.
+
+        * Animometer/resources/debug-runner/animometer.js: Add spacing in the JSON output.
+        Multiple tests output a lot of JSON and can hang when selecting JSON with no whitespace.
+        * Animometer/resources/debug-runner/animometer.css:
+        (#complexity-graph .series.raw circle): Update the color.
+        * Animometer/resources/debug-runner/graph.js: Use the FPS axis instead of the
+        complexity axis, which can vary in domain. For determining the complexity domain,
+        only use samples after samplingTimeOffset.
+
+        Allow dropping results JSON.
+
+        * Animometer/developer.html: Add a button.
+        * Animometer/resources/debug-runner/animometer.css:
+        * Animometer/resources/debug-runner/animometer.js: Read the data and go straight
+        to the dashboard. With JSON output, write out only the options and the raw data.
+
+        Teach the harness to evaluate the samples and determine the test score.
+
+        This will allow us to update how the score is calculated separately from the samples recorded.
+        This also prepares the harness to be able to accept JSON of prior runs.
+
+        * Animometer/resources/strings.js: Clean up and remove unneeded strings and reduce some of the
+        hierarchy.
+        * Animometer/resources/debug-runner/tests.js: Update to use the new strings.
+
+        * Animometer/tests/resources/main.js: Allow all controllers to show a complexity-FPS graph.
+        (_processComplexitySamples): Factor out some of the sample processing done in the ramp
+        controller for the benefit of the other controllers. |complexitySamples| contains a list of
+        samples. Sort the samples by complexity. Optionally remove the top x% of samples.
+        Group them, and calculate distribution of samples within the same complexity, and add those as
+        new entries into |complexityAverageSamples|.
+        (Controller.processSamples): Move the code responsible for determining the complexity and FPS
+        scores out to ResultsDashboard. The structure of the data returned by the controller is:
+
+        {
+            controller: [time-regression, time-regression, ...], // optional, data specific to controller
+            marks: [...],
+            samples: {                    // all of the sample data
+                controller: [...],
+                complexity: [...],        // processed from controller samples
+                complexityAverage: [...], // processed from complexity samples
+            }
+        }
+
+        (AdaptiveController.processSamples): Adding the target frame length is no longer necessary; we
+        now pass the test options to the graph.
+        (Regression): Move to statistics.js.
+        * Animometer/resources/statistics.js: Move Regression to here. Add a check if the sampling range
+        only contains one sample, since we cannot calculate a regression from one sample point.
+
+        Teach the test harness to evaluate the data.
+        * Animometer/resources/runner/animometer.js:
+        (ResultsDashboard): Store the options used to run the test and the computed results/score separately
+        from the data. The results are stored as:
+
+        {
+            score: /* geomean of iteration score */,
+            iterationsResults: [
+                {
+                    score: /* geomean of tests */,
+                    testsResults: {
+                        suiteName: {
+                            testName: {
+                                controller: {
+                                    average:
+                                    concern:
+                                    stdev:
+                                    percent:
+                                },
+                                frameLength: { ... },
+                                complexity: {
+                                    complexity:
+                                    stdev:
+                                    segment1:
+                                    segment2:
+                                },
+                                complexityAverage: { ... }
+                            },
+                            testName: { ... },
+                        },
+                        ... next suite ...
+                    }
+                },
+                { ...next iteration... }
+            ]
+        }
+
+        * Animometer/resources/debug-runner/animometer.js: Pass options around instead of relying
+        on what was selected in the form. This will later allow for dropping previous results, and
+        using those runs' options when calculating scores.
+        (ResultsTable._addGraphButton): Simplify button action by using attached test data.
+        * Animometer/resources/debug-runner/graph.js: Refactor to use the data.
+
+        Consolidate JS files, and move statistics out to a separate JS.
+
+        Preparation for having the Controller only handle recording and storage of the samples,
+        and leave the evaluation of the test score out to the harness. Move Experiment to
+        a new statistics.js, where Regression will also eventually go. Get rid of algorithm.js
+        and move it to utilities.js since the Heap is used only for Experiments.
+
+        * Animometer/tests/resources/algorithm.js: Removed. Heap is in utilities.js.
+        * Animometer/tests/resources/sampler.js: Removed. Experiment is in statistics.js,
+        Sampler in main.js.
+        * Animometer/tests/resources/main.js: Move Sampler here.
+        * Animometer/resources/statistics.js: Added. Move Statistics and Experiment here.
+        * Animometer/resources/extensions.js: Move Heap here. Attach static method to create
+        a max or min heap to Heap, instead of a new Algorithm object.
+
+        Update JS files.
+        * Animometer/developer.html:
+        * Animometer/index.html:
+        * Animometer/tests/bouncing-particles/bouncing-canvas-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-canvas-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-css-shapes.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-images.html:
+        * Animometer/tests/bouncing-particles/bouncing-svg-shapes.html:
+        * Animometer/tests/master/canvas-stage.html:
+        * Animometer/tests/master/focus.html:
+        * Animometer/tests/master/image-data.html:
+        * Animometer/tests/master/multiply.html:
+        * Animometer/tests/master/particles.html:
+        * Animometer/tests/misc/canvas-electrons.html:
+        * Animometer/tests/misc/canvas-stars.html:
+        * Animometer/tests/misc/compositing-transforms.html:
+        * Animometer/tests/simple/simple-canvas-paths.html:
+        * Animometer/tests/simple/tiled-canvas-image.html:
+        * Animometer/tests/template/template-canvas.html:
+        * Animometer/tests/template/template-css.html:
+        * Animometer/tests/template/template-svg.html:
+        * Animometer/tests/text/layering-text.html:
+        * Animometer/tests/text/text-boxes.html:
+
+        Fix the cursor in the graph analysis when the min
+        complexity is not 0.
+
+        * Animometer/resources/debug-runner/graph.js:
+        (_addRegression):
+        (createComplexityGraph):
+
 2016-02-23  Geoffrey Garen  <ggaren@apple.com>
 
         Fix some issues in MallocBench
         Added new capabilities to MallocBench.  These include:
             Added a recording of http://nim-lang.org/docs/lib.html.
             Added thread id to the recording and the ability to playback switching threads in MallocBench
-            Added aligned allocations to recordings and the ability to playback 
+            Added aligned allocations to recordings and the ability to playback
             Added --use-thread-id option to honor recorded thread ids
             Added --detailed-report to output remaining allocations by size after playback
             Added --no-warmup to not run the warm up iteration