Improvements to Animometer benchmark
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 22 Jun 2016 03:11:15 +0000 (03:11 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 22 Jun 2016 03:11:15 +0000 (03:11 +0000)
https://bugs.webkit.org/show_bug.cgi?id=157738

Reviewed by Dean Jackson.
Provisionally reviewed by Said Abou-Hallawa.

Include confidence interval for the final score, and store the canvas
size in the serialization so that it is accurately shown in results.

* Animometer/developer.html: Add a "confidence" div.
* Animometer/index.html: Ditto. Convert "mean" to "confidence".
* Animometer/resources/debug-runner/animometer.js: Look at options, and
if the configuration is included, update the body class based on it
(similar to what we do when we first load the page). That way, if you
drag-and-drop previous results in, that configuration is reflected in
the dashboard. Show the full confidence interval.

* Animometer/resources/debug-runner/animometer.css:
* Animometer/resources/debug-runner/animometer.js: Style update.
* Animometer/resources/runner/animometer.css:

* Animometer/resources/runner/animometer.js:
(_processData): Propagate the confidence interval values out and calculate
the lower and upper bounds. For now, shortcut the aggregate calculation,
since we only go through one iteration.
(this._processData.calculateScore): Propagate the confidence interval out
to be next to the score. Depending on the controller these values are
calculated differently.
(this._processData._getResultsProperty): Convenience function.
(this._processData.get score): Refactor.
(this._processData.get scoreLowerBound): Get the aggregate lower bound.
(this._processData.get scoreUpperBound): Get the aggregate upper bound.
(window.sectionsManager.setSectionScore):
(window.benchmarkController._startBenchmark): When the benchmark starts, note
the canvas size and add it to options. That way it will be automatically be
serialized.
(window.benchmarkController.showResults): Include the maximum deviation
percentage.
* Animometer/resources/runner/lines.svg: Make the background line up with the
skew.
* Animometer/resources/runner/tests.js:
(Headers.details.text): Refactor.
* Animometer/resources/statistics.js:
(Statistics.largestDeviationPercentage): Convenience function to calculate
the deviation percentage on either end and return the largest deviation.
* Animometer/resources/strings.js:

Allow specifying a regression profile to use instead of taking the one
with the lowest error.

Address an issue in the focus test when the regression calculated ends
up overestimating the change point, causing a cascade of tougher
ramps. The reason behind this is that at max complexity of an initial
ramp, the frame length is very high, and it influences the second
segment of the piecewise regression strongly, causing it to be very
steep. As a result, the first segment, expected to be flat, ends up
covering a higher range of complexity. That makes the change point
much higher than it should be. To avoid this, we will add a sanity
check on the maximum value of the ramp. If the regression's projected
value at the maximum complexity of the current ramp is very slow (less
than 20 fps), 1) reduce the maximum complexity by 20%, and 2) do not
include the regression's change point in the change point estimator.
That estimator is used as the midpoint of the next ramp, and including
the change point from a poor regression can bake in the error. The
controller already knows how to adjust for ramps that are too easy for
the system.

* Animometer/resources/runner/animometer.js:
(this._processData.findRegression): Takes a preferred profile and gives that to
Regression.
(this._processData.calculateScore): With the ramp controller, take the profile
of the ramp that was used the most when calculating the ramp's regression. That
profile is what is used for the test's score.
* Animometer/resources/statistics.js:
(Regression.Utilities.createClass): Update to take an options object which can
specify a profile to calculate with. Otherwise it will continue to use both and
select the one with the lower error.
(_calculateRegression): Fix an issue where we claim 0 error if the regression
calculation fails due to divide-by-zero. Instead reject that regression calculation
by giving it Number.MAX_VALUE.
* Animometer/resources/strings.js: New strings for marking a regression as flat
or slope.
* Animometer/tests/resources/main.js:
(RampController): Rename the thresholds for clarity. Add a threshold that, if
exceeded, will lower the maximum complexity of the next ramp.
(tune): Relax the cdf check to consider whether the interval definitely falls in
the desired frame length threshold.
(processSamples): Include the profile in the ramp.

Update ramp controller test. Increase the length of the test to 30 seconds, and extend
the interval to 120 ms during sampling. Improve the estimation of the ramp parameters.

* Animometer/developer.html: Change default to 30 seconds, and don't show the progress bar
by default.
* Animometer/resources/runner/animometer.js: Change default to 30 seconds.
* Animometer/tests/resources/main.js: A number of improvements to the ramp controller, in
the order in which they appear in the patch:

- With a longer test length use longer ramps with longer intervals to get more data at each
complexity. Keep the 100 ms interval length during the ramp up phase since we don't need to
spend more time there to find the right order of magnitude, but increase it during the
ramps to 120 ms.
- The ramp linearly interpolates the complexity to render based on its timestamp, but it would
never sample the minimum complexity. Instead of lerping max to min complexity from time
0 to t where t is the ramp length, instead lerp from 0 to (t - intervalSampleLength) so that
we can have at least one interval sample at the min complexity for that ramp.
- Some regression calculations only come out with one line segment rather than the two
we expect. This could be due to a noisy ramp or the ramp's range is too narrow. If that's the
case, influence the minimum complexity of the next ramp towards the lowest bound of 1, so that
we ensure that at least part of the ramp is covering a complexity range that the system can
handle at full 60.
- Remove an assignment to interpolatedFrameLength since that is never subsequently used.

Update the format used to serialize the results for analysis.

Each data point used to have named properties for fields like complexity and frame rate.
In addition the serialized numbers had rounding errors that took up many characters.
Update the format by introducing a new data container called SampleData, which contains a
field map. The map maps a string to an array index. Each data point is an array, so, to
get a stat, use the field map to get the array index into the data point. This allows future
versions to track other data, and reduces the size of the output string by two-thirds.

* Animometer/resources/extensions.js:
(Utilities.toFixedNumber): Add convenience function that truncates the number to a fixed
precision, and converts it back to a number.
(SampleData): New class that contains sample data and a field map that maps properties to
an array index.
(get length): Number of data points.
(addField): Add a field to the field map.
(push): Add a data point.
(sort): Sort the data.
(slice): Return new SampleData object with sliced data.
(forEach): Iterate over the data with provided function.
(createDatum):
(getFieldInDatum): Returns the data point associated with the field name by looking it up
in the field map in the datum provided, which can be the datum object itself (an array) or
an index into the data member variable.
(setFieldInDatum): Sets the data point associated with the field name.
(at): Returns the data point at the provided index.
(toArray): Serializes the data where the field map serves as property names for each point.

* Animometer/resources/debug-runner/graph.js:
(updateGraphData): Remove unused _testData. Convert the data to the old array format for the
graph to use, since the old format was much easier to work with when displaying the graphs.
(onGraphTypeChanged): For some controllers, no alternative score or mean is provided.
* Animometer/resources/runner/animometer.css:
* Animometer/resources/runner/animometer.js: Refactor to use SampleData. Update JSON output
to only go to 3 digits of precision for purposes of reducing the data size.
* Animometer/resources/strings.js: Add new strings to put into the field maps.
* Animometer/tests/resources/main.js: Refactor to use SampleData.

* Animometer/developer.html:
* Animometer/index.html: Restructure results table for both pages. Add charset attribute to
tests.js include.
* Animometer/resources/debug-runner/animometer.css: Clear out styles from release runner.
* Animometer/resources/debug-runner/graph.js:
(onGraphTypeChanged): Update score and mean if bootstrap results are available from the
controller, since not all controllers do bootstrapping.
* Animometer/resources/debug-runner/tests.js: Update header text.
* Animometer/resources/runner/animometer.css: Include confidence interval in results.
* Animometer/resources/runner/animometer.js:
(ResultsTable._addHeader): Header contents can be HTML, so use innerHTML instead.
(ResultsTable._addBody): Add tbody element.
(ResultsTable._addTest): Allow a data cell to invoke a JS function to get its contents.
(window.benchmarkController.showResults): Add table that includes tests' confidence intervals.
* Animometer/resources/runner/tests.js:
(Headers.details.text): Add new details table that includes bootstrap confidence interval.
The interval can be asymmetric, but for simplicity, report the maximum deviation percentage
on either side of the bootstrap median.
* Animometer/resources/statistics.js:
(bootstrap): Include the confidence percentage in the return object.

Report canvas size in results.

* Animometer/developer.html: Add markup to indicate whether a small, medium, or large
canvas was used.
* Animometer/index.html: Ditto.
* Animometer/resources/debug-runner/animometer.js: Call determineCanvasSize().
* Animometer/resources/runner/animometer.css: Update styles to set the canvas based on the
body class size.
* Animometer/resources/runner/animometer.js:
(window.benchmarkController.initialize): Update styles to set the canvas based on the
body class size.
(window.benchmarkController.determineCanvasSize): Run various media queries and set the body
class based on the size of the device.

* Animometer/developer.html: Refactor to include the main CSS file, and redo
the layout so that it doesn't rely on flexbox.
* Animometer/resources/debug-runner/animometer.css:
* Animometer/resources/debug-runner/animometer.js:
(updateDisplay): Since various parts of the script alter the body class, we can't
replace the className directly. Instead, remove all display-based values and then add
the one that was selected.
* Animometer/resources/debug-runner/graph.js:
(updateGraphData): To set the size of the graph, use window.innerHeight.
* Animometer/resources/runner/animometer.js:
(window.sectionsManager.showSection): Since various parts of the script alter the body
class, we can't replace the className directly. Remove all of the section classes
individually and then add the one desired.
* Animometer/tests/resources/stage.css: Remove -apple-system as a font to use in the
stage.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@202314 268f45cc-cd09-0410-ab3c-d52691b4dbfc

16 files changed:
PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/index.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/animometer.js
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/debug-runner/tests.js
PerformanceTests/Animometer/resources/extensions.js
PerformanceTests/Animometer/resources/runner/animometer.css
PerformanceTests/Animometer/resources/runner/animometer.js
PerformanceTests/Animometer/resources/runner/lines.svg
PerformanceTests/Animometer/resources/runner/tests.js
PerformanceTests/Animometer/resources/statistics.js
PerformanceTests/Animometer/resources/strings.js
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/Animometer/tests/resources/stage.css
PerformanceTests/ChangeLog

index cf37ee0951e7239d17044adf68d24f233a3e22d7..9c14e0f87f87031518adda739b1f99c4a9d04d52 100644 (file)
@@ -1,15 +1,19 @@
 <!DOCTYPE html>
 <html>
 <head>
-    <title>Animometer - developer</title>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, user-scalable=no">
+
+    <title>MotionMark 1.0 - developer</title>
+
+    <link rel="stylesheet" href="resources/runner/animometer.css">
     <link rel="stylesheet" href="resources/debug-runner/animometer.css">
+
     <script src="resources/strings.js"></script>
     <script src="resources/extensions.js"></script>
     <script src="resources/statistics.js"></script>
 
-    <script src="resources/runner/tests.js"></script>
+    <script src="resources/runner/tests.js" charset="utf-8"></script>
     <script src="resources/debug-runner/tests.js" charset="utf-8"></script>
     <script src="resources/runner/animometer.js"></script>
     <script src="resources/debug-runner/animometer.js" charset="utf-8"></script>
     <script src="resources/debug-runner/d3.min.js"></script>
     <script src="resources/debug-runner/graph.js" charset="utf-8"></script>
 </head>
-<body>
+<body class="showing-intro">
     <main>
-        <hr>
         <section id="intro" class="selected">
-            <h1>Animometer</h1>
-            <div>
-                <div id="suites">
-                    <h2>Suites:</h2>
-                    <ul class="tree"></ul>
-                    <div><span id="drop-target">Drop results here</span></div>
+            <h1>MotionMark</h1>
+            <div class="body">
+                <div>
+                    <div id="suites">
+                        <h2>Suites:</h2>
+                        <ul class="tree"></ul>
+                        <div><span id="drop-target">Drop results here</span></div>
+                    </div>
+                    <div id="options">
+                        <h2>Options:</h2>
+                        <form name="benchmark-options">
+                            <ul>
+                                <li>
+                                    <label>Test length: <input type="number" id="test-interval" value="30"> seconds each</label>
+                                </li>
+                                <li>
+                                    <h3>Display:</h3>
+                                    <ul>
+                                        <li><label><input name="display" type="radio" value="minimal" checked> Minimal</label></li>
+                                        <li><label><input name="display" type="radio" value="progress-bar"> Progress bar</label></li>
+                                    </ul>
+                                </li>
+                                <li>
+                                    <h3>Adjusting the test complexity:</h3>
+                                    <ul>
+                                        <li><label><input name="controller" type="radio" value="fixed"> Keep at a fixed complexity</label></li>
+                                        <li><label><input name="controller" type="radio" value="step"> Keep at a fixed complexity, then make a big step</label></li>
+                                        <li><label><input name="controller" type="radio" value="adaptive"> Maintain target FPS</label></li>
+                                        <li><label><input name="controller" type="radio" value="ramp" checked> Ramp</label></li>
+                                        <li><label><input name="controller" type="radio" value="ramp30"> Ramp @ 30fps</label></li>
+                                    </ul>
+                                </li>
+                                <li>
+                                    <label>Target frame rate: <input type="number" id="frame-rate" value="50"> FPS</label>
+                                </li>
+                                <li>
+                                    <h3>Kalman filter estimated error:</h3>
+                                    <ul>
+                                        <li><label>Process error (Q): <input type="number" id="kalman-process-error" value="1"></label></li>
+                                        <li><label>Measurement error (R): <input type="number" id="kalman-measurement-error" value="4"></label></li>
+                                    </ul>
+                                </li>
+                                <li>
+                                    <h3>Time measurement method:</h3>
+                                    <ul>
+                                        <li><label><input name="time-measurement" type="radio" value="performance" checked> <code>performance.now()</code></label></li>
+                                        <li><label><input name="time-measurement" type="radio" value="date"> <code>Date.now()</code></label></li>
+                                    </ul>
+                                </li>
+                            </ul>
+                        </form>
+                    </div>
                 </div>
-                <div id="options">
-                    <h2>Options:</h2>
-                    <form name="benchmark-options">
-                    <ul>
-                    <li>
-                        <label>Test length: <input type="number" id="test-interval" value="20"> seconds each</label>
-                    </li>
-                    <li>
-                        <h3>Display:</h3>
-                        <ul>
-                            <li><label><input name="display" type="radio" value="minimal" checked> Minimal</label></li>
-                            <li><label><input name="display" type="radio" value="progress-bar" checked> Progress bar</label></li>
-                        </ul>
-                    </li>
-                    <li>
-                        <h3>Adjusting the test complexity:</h3>
-                        <ul>
-                            <li><label><input name="controller" type="radio" value="fixed"> Keep at a fixed complexity</label></li>
-                            <li><label><input name="controller" type="radio" value="step"> Keep at a fixed complexity, then make a big step</label></li>
-                            <li><label><input name="controller" type="radio" value="adaptive"> Maintain target FPS</label></li>
-                            <li><label><input name="controller" type="radio" value="ramp" checked> Ramp</label></li>
-                            <li><label><input name="controller" type="radio" value="ramp30"> Ramp @ 30fps</label></li>
-                        </ul>
-                    </li>
-                    <li>
-                        <label>Target frame rate: <input type="number" id="frame-rate" value="50"> FPS</label>
-                    </li>
-                    <li>
-                        <h3>Kalman filter estimated error:</h3>
-                        <ul>
-                            <li><label>Process error (Q): <input type="number" id="kalman-process-error" value="1"></label></li>
-                            <li><label>Measurement error (R): <input type="number" id="kalman-measurement-error" value="4"></label></li>
-                        </ul>
-                    </li>
-                    <li>
-                        <h3>Time measurement method:</h3>
-                        <ul>
-                            <li><label><input name="time-measurement" type="radio" value="performance" checked> <code>performance.now()</code></label></li>
-                            <li><label><input name="time-measurement" type="radio" value="date"> <code>Date.now()</code></label></li>
-                        </ul>
-                    </li>
-                    </ul>
-                    </form>
+                <p>For accurate results, please take the browser window full screen, or rotate the device to landscape orientation.</p>
+                <div class="start-benchmark">
+                    <p class="hidden">Please rotate the device to orientation before starting.</p>
+                    <button id="run-benchmark" onclick="benchmarkController.startBenchmark()">Run benchmark</button>
                 </div>
             </div>
-            <p>For accurate results, please take the browser window full screen, or rotate the device to landscape orientation.</p>
-            <div class="orientation-check">
-                <p class="hidden">Please rotate the device to orientation before starting.</p>
-                <button id="run-benchmark" onclick="benchmarkController.startBenchmark()">Run benchmark</button>
-            </div>
         </section>
+
         <section id="test-container">
             <div id="running-test" class="frame-container"></div>
             <div id="progress">
                 <div id="progress-completed"></div>
             </div>
         </section>
+
         <section id="results">
-            <h1>Animometer score</h1>
-            <p class="score" onclick="benchmarkController.showDebugInfo()"></p>
-            <div id="results-tables">
-                <div>
-                    <table id="results-score"></table>
-                    <table id="results-data"></table>
+            <div class="body">
+                <h1>MotionMark score</h1>
+                <div class="detail">
+                    <span class="small">on a small screen (phone)</span>
+                    <span class="medium">on a medium screen (laptop, tablet)</span>
+                    <span class="large">on a large screen (desktop)</span>
                 </div>
-                <table id="results-header"></table>
+                <p class="score" onclick="benchmarkController.showDebugInfo()"></p>
+                <p class="confidence"></p>
+                <div id="results-tables" class="table-container">
+                    <div>
+                        <table id="results-score"></table>
+                        <table id="results-data"></table>
+                    </div>
+                    <table id="results-header"></table>
+                </div>
+                <button onclick="benchmarkController.restartBenchmark()">Test Again</button>
+                <p>
+                    'j': Show JSON results<br/>
+                    's': Select various results for copy/paste (use repeatedly to cycle)
+                </p>
             </div>
-            <button onclick="benchmarkController.restartBenchmark()">Test Again</button>
-            <p>
-                'j': Show JSON results<br/>
-                's': Select various results for copy/paste (use repeatedly to cycle)
-            </p>
         </section>
         <section id="test-graph">
-            <header>
-                <button onclick="benchmarkController.showResults()">&lt; Results</button>
-                <h1>Graph:</h1>
-            </header>
-            <nav>
-                <form name="graph-type">
-                    <ul>
-                        <li><label><input type="radio" name="graph-type" value="time"> Time graph</label></li>
-                        <li><label><input type="radio" name="graph-type" value="complexity" checked> Complexity graph</label></li>
-                    </ul>
-                </form>
-                <form name="time-graph-options">
-                    <ul>
-                        <li><label><input type="checkbox" name="markers" checked> Markers</label>
-                            <span>time: <span class="time"></span></span></li>
-                        <li><label><input type="checkbox" name="averages" checked> Averages</label></li>
-                        <li><label><input type="checkbox" name="complexity" checked> Complexity</label>
-                            <span class="complexity"></span></li>
-                        <li><label><input type="checkbox" name="rawFPS" checked> Raw FPS</label>
-                            <span class="rawFPS"></span></li>
-                        <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
-                            <span class="filteredFPS"></span></li>
-                        <li><label><input type="checkbox" name="regressions" checked> Regressions</label>
-                    </ul>
-                </form>
-                <form name="complexity-graph-options">
-                    <ul class="series">
-                        <li><label><input type="checkbox" name="series-raw" checked> Series raw</label></li>
-                        <li><label><input type="checkbox" name="series-average"> Series average</label></li>
-                    </ul>
-                    <ul>
-                        <li><label><input type="checkbox" name="regression-time-score"> Controller score</label></li>
-                        <li><label><input type="checkbox" name="bootstrap-score" checked> Bootstrap score and histogram</label></li>
-                        <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label><span id="complexity-regression-aggregate-raw"></span></li>
-                        <li><label><input type="checkbox" name="complexity-regression-aggregate-average"> Regression, series average</label><span id="complexity-regression-aggregate-average"></span></li>
-                    </ul>
-                </form>
-            </nav>
-            <p class="score"></p>
-            <p class="mean"></p>
-            <div id="test-graph-data"></div>
+            <div class="body">
+                <header>
+                    <button onclick="benchmarkController.showResults()">&lt; Results</button>
+                    <h1>Graph:</h1>
+                    <p class="score"></p>
+                    <p class="confidence"></p>
+                </header>
+                <nav>
+                    <form name="graph-type">
+                        <ul>
+                            <li><label><input type="radio" name="graph-type" value="time"> Time graph</label></li>
+                            <li><label><input type="radio" name="graph-type" value="complexity" checked> Complexity graph</label></li>
+                        </ul>
+                    </form>
+                    <form name="time-graph-options">
+                        <ul>
+                            <li><label><input type="checkbox" name="markers" checked> Markers</label>
+                                <span>time: <span class="time"></span></span></li>
+                            <li><label><input type="checkbox" name="averages" checked> Averages</label></li>
+                            <li><label><input type="checkbox" name="complexity" checked> Complexity</label>
+                                <span class="complexity"></span></li>
+                            <li><label><input type="checkbox" name="rawFPS" checked> Raw FPS</label>
+                                <span class="rawFPS"></span></li>
+                            <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
+                                <span class="filteredFPS"></span></li>
+                            <li><label><input type="checkbox" name="regressions" checked> Regressions</label></li>
+                        </ul>
+                    </form>
+                    <form name="complexity-graph-options">
+                        <ul class="series">
+                            <li><label><input type="checkbox" name="series-raw" checked> Series raw</label></li>
+                            <li><label><input type="checkbox" name="series-average"> Series average</label></li>
+                        </ul>
+                        <ul>
+                            <li><label><input type="checkbox" name="regression-time-score"> Controller score</label></li>
+                            <li><label><input type="checkbox" name="bootstrap-score" checked> Bootstrap score and histogram</label></li>
+                            <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label><span id="complexity-regression-aggregate-raw"></span></li>
+                            <li><label><input type="checkbox" name="complexity-regression-aggregate-average"> Regression, series average</label><span id="complexity-regression-aggregate-average"></span></li>
+                        </ul>
+                    </form>
+                </nav>
+                <div id="test-graph-data"></div>
+            </div>
         </section>
-        <hr>
     </main>
 </body>
 </html>
index f66f63d6be5c136ccad70381553cdcad2b9056ec..38193a82e43842ded966accd6d9f59c19680e3e6 100644 (file)
@@ -4,7 +4,7 @@
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, user-scalable=no">
 
-    <title>Motion Mark 1.0</title>
+    <title>MotionMark 1.0</title>
 
     <link rel="stylesheet" href="resources/runner/animometer.css">
 
@@ -12,7 +12,7 @@
     <script src="resources/extensions.js" defer></script>
     <script src="resources/statistics.js" defer></script>
 
-    <script src="resources/runner/tests.js" defer></script>
+    <script src="resources/runner/tests.js" charset="utf-8" defer></script>
     <script src="resources/runner/animometer.js" defer></script>
 
     <script src="resources/runner/benchmark-runner.js" defer></script>
@@ -29,7 +29,7 @@
         <section id="intro" class="selected">
             <svg class="logo"><use xlink:href="resources/runner/logo.svg#root"></svg>
             <div class="body">
-                <p>Motion Mark is a graphics benchmark that measures a browser’s capability to animate complex scenes at a target frame rate.</p>
+                <p>MotionMark is a graphics benchmark that measures a browser’s capability to animate complex scenes at a target frame rate.</p>
                 <p>For accurate results, please take your browser window full screen, or rotate your device to landscape orientation.</p>
                 <p class="portrait-orientation-check"><b>Please rotate your device.</b></p>
                 <button class="landscape-orientation-check" onclick="benchmarkController.startBenchmark()">Run Benchmark</button>
         <section id="results">
             <svg class="logo"><use xlink:href="resources/runner/logo.svg#root"></svg>
             <div class="body">
-                <p class="score" onclick="benchmarkController.showDebugInfo()"></p>
-                <table id="results-header"></table><table id="results-score"></table><br>
+                <div class="score-container">
+                    <div class="score"></div>
+                    <div class="confidence"></div>
+                    <div class="detail">
+                        <span class="small">on a small screen (phone)</span>
+                        <span class="medium">on a medium screen (laptop, tablet)</span>
+                        <span class="large">on a large screen (desktop)</span>
+                    </div>
+                </div>
+                <div class="table-container">
+                    <div>
+                        <table id="results-score"></table>
+                        <table id="results-data"></table>
+                    </div>
+                    <table id="results-header"></table>
+                </div>
                 <button onclick="benchmarkController.startBenchmark()">Test Again</button>
             </div>
         </section>
index 1d3dd33adca21a6ad5690de3e72cd6918902e561..3f08d49eb25399f7109ae1119a3efd29f2f73db2 100644 (file)
@@ -1,93 +1,61 @@
-/* Outer harness */
-html,body {
-    height: 100%;
-    margin: 0px;
-    padding: 0px;
-}
-
 body {
-    background-color: rgb(241, 241, 241);
-    color: rgb(96, 96, 96);
-    font-family: -apple-system, "Helvetica Neue", Helvetica, Verdana, sans-serif;
-}
-
-main {
-    width: 100%;
-    height: 100%;
-
-    display: flex;
-    align-items: center;
-    justify-content: flex-start;
-    flex-flow: column;
+    font-size: initial;
 }
 
-hr {
-    flex: 1 0 20px;
-    width: 1px;
-    border: 0;
-    margin: 0;
-}
+body.showing-intro,
+body.showing-results,
+body.showing-test-graph {
+    background-color: rgb(96, 96, 96);
+    background-image: initial;
+    background-repeat: initial;
+    background-size: initial;
+    animation: initial;
+    will-change: initial;
 
-section {
-    box-sizing: border-box;
-    width: 100%;
-    display: none;
+    color: rgb(235, 235, 235);
 }
 
-section.selected {
-    display: initial;
+section .body {
+    margin-left: 0;
+    max-width: initial;
+    transform: none;
 }
 
 h1 {
     font-size: 3em;
+    margin: 1.5em 0 .5em;
     text-align: center;
-    font-weight: 600;
-    margin: 10vh 0;
-    flex: 0 1 1em;
-}
-
-.hidden {
-    display: none;
 }
 
 button {
-    flex: 0 0 auto;
+    transform: none !important;
+    min-width: initial;
+    transition: none;
+    animation: none;
+    will-change: initial;
+
     display: block;
     font-size: 1.5em;
-    border: 2px solid rgb(96, 96, 96);
-    color: rgb(96, 96, 96);
+    border: 2px solid rgb(235, 235, 235);
+    color: rgb(235, 235, 235);
     background: transparent;
     border-radius: 10px;
     padding: .5em 2em;
 }
 
 button:hover {
-    background-color: rgba(0, 0, 0, .1);
+    background-color: rgba(255, 255, 255, .1);
     cursor: pointer;
 }
 
 button:active {
     color: inherit;
-    background-color: rgba(0, 0, 0, .2);
+    background-color: rgba(255, 255, 255, .2);
 }
 
 button:disabled {
-    border-color: rgba(96, 96, 96, .5);
-    color: rgba(96, 96, 96, .5);
-}
-
-@media screen and (min-device-height: 1024px),
-    screen and (min-device-width: 1024px) and (orientation: landscape) {
-    section {
-        padding: 0 20px;
-    }
-
-    section.selected {
-        display: flex;
-        align-items: center;
-        justify-content: flex-start;
-        flex-flow: column;
-    }
+    border-color: rgba(235, 235, 235, .5);
+    color: rgba(235, 235, 235, .5);
 }
 
 @media screen and (max-device-width: 414px),
@@ -96,280 +64,14 @@ button:disabled {
         font-size: 2.5em;
     }
 
-    hr {
-        flex: 0 0 0;
-    }
-
     section {
         box-sizing: border-box;
         width: 100%;
         height: 100%;
-        align-self: stretch;
         padding: 0 5px;
     }
 }
 
-/* Intro section */
-#intro.selected {
-    flex: 1 0 auto;
-}
-
-#intro > p {
-    flex: 0 1 auto;
-    padding: .5em 0;
-    margin: 0;
-    font-size: 2em;
-}
-
-#intro .orientation-check {
-    padding: 10vh 0;
-    text-align: center;
-}
-
-#intro .orientation-check p {
-    font-size: 1em;
-    color: hsl(11, 72%, 50%);
-    margin-bottom: 1em;
-    -apple-trailing-word: -apple-partially-balanced;
-}
-
-#intro .orientation-check button {
-    margin: 0 auto;
-}
-
-@media screen and (min-device-height: 1024px),
-    screen and (min-device-width: 1024px) and (orientation: landscape) {
-    #intro p {
-        max-width: 800px;
-        margin: 0;
-    }
-}
-
-@media screen and (max-device-width: 414px),
-    screen and (max-device-height: 414px) and (orientation: landscape) {
-    #intro.selected {
-        display: flex;
-        align-items: center;
-        justify-content: flex-start;
-        flex-flow: column;
-    }
-
-    #intro p {
-        padding-left: 20px;
-        padding-right: 20px;
-        font-size: 1.5em;
-    }
-}
-
-/* Running test section */
-#test-container.selected {
-    display: flex;
-
-    align-items: center;
-    justify-content: center;
-}
-
-.frame-container > iframe {
-    width: 1200px;
-    height: 600px;
-    border: 0;
-    margin: 0 auto;
-}
-
-@media screen and (max-device-width: 414px),
-    screen and (max-device-height: 414px) and (orientation: landscape) {
-    #test-container {
-        padding: 0;
-    }
-
-    .frame-container {
-        width: 100%;
-        height: calc(100% + 1px);
-    }
-
-    .frame-container > iframe {
-        width: 100%;
-        height: 100%;
-    }
-}
-
-@media (min-device-height: 768px) and (max-device-height: 1024px) {
-    .frame-container > iframe {
-        width: 900px;
-        height: 600px;
-    }
-}
-
-@media screen and (max-device-width: 1024px) and (min-device-height: 1366px) {
-    .frame-container > iframe {
-        width: 1200px;
-        height: 800px;
-    }
-}
-
-@media screen and (min-width: 1800px) {
-    .frame-container > iframe {
-        width: 1600px;
-        height: 800px;
-    }
-}
-
-/* Results section */
-#results {
-    text-align: center;
-}
-
-#results.selected {
-    flex: 1 0 auto;
-}
-
-.score {
-    font-size: 5em;
-    font-weight: bold;
-    margin: 0;
-}
-
-#results-tables {
-    direction: rtl;
-
-    display: flex;
-
-    align-items: center;
-    justify-content: center;
-
-    margin: 3em 0;
-}
-
-#results table {
-    direction: ltr;
-    border-spacing: 0;
-}
-
-#results th {
-    padding: .5em 0;
-}
-
-#results tr td {
-    padding: .25em 0;
-}
-
-#results-header td, #results-header th {
-    text-align: left;
-}
-#results-header tr td {
-    padding-right: 1em;
-}
-#results-score td, #results-score th {
-    text-align: right;
-}
-#results footer {
-    padding-bottom: 10vh;
-}
-
-@media screen and (max-device-width: 414px),
-    screen and (max-device-height: 414px) and (orientation: landscape) {
-    #results.selected {
-        padding: 0 20px;
-        display: flex;
-        align-items: center;
-        justify-content: flex-start;
-        flex-flow: column;
-    }
-
-    .score {
-        font-size: 3em;
-    }
-}
-
-#overlay {
-    position: fixed;
-    width: 100%;
-    height: 100%;
-    top: 0;
-    left: 0;
-    color: rgb(255, 255, 255);
-    background: rgba(0, 0, 10, .8);
-}
-
-@supports (-webkit-backdrop-filter: blur(10px)) {
-    #overlay {
-        background: rgba(0, 0, 10, .4);
-        -webkit-backdrop-filter: blur(10px);
-    }
-}
-
-#overlay > div {
-    width: 500px;
-    height: 500px;
-    position: absolute;
-    margin-top: -250px;
-    margin-left: -250px;
-    top: 50%;
-    left: 50%;
-    display: flex;
-    justify-content: flex-start;
-    flex-flow: column;
-}
-
-#overlay > div div {
-    flex: 1 1 auto;
-    overflow: scroll;
-    border: 1px solid rgb(241, 241, 241);
-    padding: 2px;
-    box-sizing: border-box;
-}
-
-#overlay button {
-    margin: 1em 5em;
-    border-color: rgb(241, 241, 241);
-    color: rgb(241, 241, 241);
-}
-
-#overlay button:hover {
-    background-color: rgba(255, 255, 255, .1);
-}
-
-#overlay button:active {
-    background-color: rgba(255, 255, 255, .2);
-}
-
-/* Debug specific */
-body {
-    background-color: rgb(96, 96, 96);
-    color: rgb(235, 235, 235);
-}
-
-h1 {
-    margin: 5vh 0;
-}
-
-button {
-    border: 2px solid rgb(235, 235, 235);
-    color: rgb(235, 235, 235);
-}
-
-button:disabled {
-    border-color: rgba(235, 235, 235, .5);
-    color: rgba(235, 235, 235, .5);
-}
-
-button:hover {
-    background-color: rgba(255, 255, 255, .1);
-    cursor: pointer;
-}
-
-button:active {
-    color: inherit;
-    background-color: rgba(255, 255, 255, .2);
-}
-
-@media screen and (min-width: 1800px) {
-    section {
-        width: 1600px;
-        height: 800px;
-    }
-}
-
 /* -------------------------------------------------------------------------- */
 /*                               Tree                                         */
 /* -------------------------------------------------------------------------- */
@@ -460,20 +162,54 @@ label.tree-label {
 /* -------------------------------------------------------------------------- */
 
 #intro {
-    flex-direction: column;
-    justify-content: flex-start;
-    align-content: center;
+    padding: 0;
+    opacity: initial;
+    transition: none;
+}
+
+#intro .body > p {
+    padding: 1em 0;
+    margin: 0 auto;
+    text-align: center;
+}
+
+#intro .start-benchmark {
+    padding: 10vh 0;
+    text-align: center;
+}
+
+#intro .start-benchmark p {
+    color: hsl(11, 72%, 50%);
+    margin-bottom: 1em;
+    -apple-trailing-word: -apple-partially-balanced;
+}
+
+#intro .start-benchmark button {
+    margin: 0 auto;
+}
+
+
+@media screen and (max-device-width: 414px),
+    screen and (max-device-height: 414px) and (orientation: landscape) {
+    #intro.selected {
+        display: flex;
+        align-items: center;
+        justify-content: flex-start;
+        flex-flow: column;
+    }
 
-    min-height: 600px;
-    height: auto;
+    #intro p {
+        padding-left: 20px;
+        padding-right: 20px;
+        font-size: 1.5em;
+    }
 }
 
 #intro h2 {
-    margin-top: 0;
     font-size: 1.2em;
 }
 
-#intro > div:first-of-type {
+#intro .body > div:first-of-type {
     width: 100%;
     margin: 2em 0 0;
     flex-direction: row;
@@ -558,28 +294,18 @@ label.tree-label {
     font-size: 1em;
 }
 
-#intro .orientation-check {
+#intro .start-benchmark {
     padding: 0 0 10vh;
     margin-top: 0;
 }
 
-#intro .orientation-check p {
+#intro .start-benchmark p {
     color: hsl(11, 100%, 66%);
 }
 
-@media screen and (min-device-width: 1800px) {
-    #intro {
-        min-height: 800px;
-    }
-}
-
 @media screen and (max-device-width: 414px),
     screen and (max-device-height: 414px) and (orientation: landscape) {
-    #intro {
-        min-height: 100%;
-    }
-
-    #intro > div:first-of-type {
+    #intro .body > div:first-of-type {
         flex-direction: column;
     }
 
@@ -595,10 +321,6 @@ label.tree-label {
 /*                           Running Section                                  */
 /* -------------------------------------------------------------------------- */
 
-#test-container {
-    position: relative;
-}
-
 #running-test {
     display: flex;
     align-items: center;
@@ -632,6 +354,10 @@ label.tree-label {
 /*                           Results Section                                  */
 /* -------------------------------------------------------------------------- */
 
+#results {
+    text-align: center;
+}
+
 #results h1, #test-graph h1 {
     font-size: 2em;
 }
@@ -650,15 +376,117 @@ label.tree-label {
     color: inherit;
 }
 
-.score {
+#results .score,
+#test-graph .score {
     font-size: 3em;
+    font-weight: bold;
+    margin: 0;
 }
 
-.mean {
+#results .confidence,
+#test-graph .confidence {
     margin-top: 0;
     margin-bottom: 1em;
     font-size: 1.5em;
     font-weight: 400;
+    text-indent: inherit;
+    color: inherit;
+}
+
+#results-tables {
+    direction: rtl;
+
+    display: flex;
+
+    align-items: center;
+    justify-content: center;
+
+    margin: 3em 0;
+}
+
+#results .table-container > div {
+    margin-left: 0;
+}
+
+#results #results-score {
+    float: initial;
+}
+
+#results #results-header {
+    width: initial;
+    position: initial;
+}
+
+#results table {
+    direction: ltr;
+    min-width: initial;
+}
+
+#results table td.suites-separator {
+    padding: .5em 0;
+}
+
+#results table tr:nth-child(even) {
+    background-color: transparent;
+}
+
+#results th {
+    padding: .5em 0;
+}
+
+#results tr td {
+    padding: .25em 0;
+}
+
+#results-header td, #results-header th {
+    text-align: left;
+}
+#results-header tr td {
+    padding-right: 1em;
+}
+#results-score td, #results-score th {
+    text-align: right;
+}
+#results .body > button {
+    margin: 1.5em auto .5em;
+}
+#results footer {
+    padding-bottom: 10vh;
+}
+
+@media screen and (max-device-width: 414px),
+    screen and (max-device-height: 414px) and (orientation: landscape) {
+    #results.selected {
+        padding: 0 20px;
+    }
+}
+
+#overlay {
+    background: rgba(0, 0, 10, .8);
+}
+
+@supports (-webkit-backdrop-filter: blur(10px)) {
+    #overlay {
+        background: rgba(0, 0, 10, .4);
+    }
+}
+
+#overlay > div div {
+    border: 1px solid rgb(241, 241, 241);
+}
+
+#overlay button {
+    margin: 2em auto;
+    border-color: rgb(241, 241, 241);
+    color: rgb(241, 241, 241);
+}
+
+#overlay button:hover {
+    background-color: rgba(255, 255, 255, .1);
+}
+
+#overlay button:active {
+    background-color: rgba(255, 255, 255, .2);
 }
 
 #results-data .average {
@@ -711,22 +539,20 @@ label.tree-label {
 
 #test-graph header {
     position: relative;
-    width: 100%;
+    text-align:center;
 }
 
 #test-graph header button {
     position: absolute;
-    top: 1.5em;
+    top: 0;
     left: 0;
     border-width: 1px;
     font-size: 1em;
     padding: .5em 1em;
 }
 
-#test-graph-data {
-    flex: 1 1 auto;
-    align-self: stretch;
-    z-index: 1;
+#test-graph .score, #test-graph .confidence {
+    margin: 0;
 }
 
 #test-graph nav {
@@ -760,6 +586,7 @@ label.tree-label {
 /* -------------------------------------------------------------------------- */
 
 #test-graph-data {
+    z-index: 1;
     font: 10px sans-serif;
     color: rgb(235, 235, 235);
 }
@@ -898,7 +725,7 @@ label.tree-label {
 }
 
 #complexity-graph .raw.series line {
-    stroke: hsl(30, 96%, 56%);
+    stroke: hsla(30, 96%, 56%, .3);
     stroke-width: 1px;
 }
 
index 55d1a99108e14df58ba38e2dcb3ba292074f7117..5f0473bd68fc34a13bd9db794ac19f707a787cd2 100644 (file)
@@ -74,7 +74,7 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
                     className = header.className;
             }
 
-            if (header.title == Strings.text.testName) {
+            if (header.text == Strings.text.testName) {
                 if (isNoisy)
                     className += " noisy-results";
                 var td = Utilities.createElement("td", { class: className }, row);
@@ -92,9 +92,8 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
                 if (typeof data == "number")
                     data = data.toFixed(2);
                 td.textContent = data;
-            } else {
-                td.textContent = header.text(testResult, testName);
-            }
+            } else
+                td.textContent = header.text(testResult);
         }, this);
     }
 });
@@ -213,7 +212,10 @@ window.optionsManager =
 
     updateDisplay: function()
     {
-        document.body.className = "display-" + optionsManager.valueForOption("display");
+        document.body.classList.remove("display-minimal");
+        document.body.classList.remove("display-progress-bar");
+
+        document.body.classList.add("display-" + optionsManager.valueForOption("display"));
     }
 };
 
@@ -505,6 +507,7 @@ Utilities.extendObject(window.benchmarkController, {
         if (benchmarkController.startBenchmarkImmediatelyIfEncoded())
             return;
 
+        benchmarkController.determineCanvasSize();
         benchmarkController.addOrientationListenerIfNecessary();
         suitesManager.createElements();
         suitesManager.updateUIFromLocalStorage();
@@ -606,7 +609,15 @@ Utilities.extendObject(window.benchmarkController, {
             Headers.details[4].disabled = true;
         }
 
-        sectionsManager.setSectionScore("results", dashboard.score.toFixed(2));
+        if (dashboard.options[Strings.json.configuration]) {
+            document.body.classList.remove("small", "medium", "large");
+            document.body.classList.add(dashboard.options[Strings.json.configuration]);
+        }
+
+        var score = dashboard.score;
+        var confidence = ((dashboard.scoreLowerBound / score - 1) * 100).toFixed(2) +
+            "% / +" + ((dashboard.scoreUpperBound / score - 1) * 100).toFixed(2) + "%";
+        sectionsManager.setSectionScore("results", score.toFixed(2), confidence);
         sectionsManager.populateTable("results-header", Headers.testName, dashboard);
         sectionsManager.populateTable("results-score", Headers.score, dashboard);
         sectionsManager.populateTable("results-data", Headers.details, dashboard);
index 76a0a9804e1f572bb18bb6069c97ee98dcad31f7..48039362198be735929894edf98e489df82ba4c0 100644 (file)
@@ -1,28 +1,30 @@
 Utilities.extendObject(window.benchmarkController, {
-    layoutCounter: 0,
-
     updateGraphData: function(testResult, testData, options)
     {
         var element = document.getElementById("test-graph-data");
         element.innerHTML = "";
         element._testResult = testResult;
-        element._testData = testData;
         element._options = options;
-        document.querySelector("hr").style.width = this.layoutCounter++ + "px";
 
         var margins = new Insets(30, 30, 30, 40);
-        var size = Point.elementClientSize(element).subtract(margins.size);
-
-        this.createTimeGraph(testResult, testData[Strings.json.samples][Strings.json.controller], testData[Strings.json.marks], testData[Strings.json.controller], options, margins, size);
+        var size = Point.elementClientSize(element);
+        size.y = window.innerHeight - element.offsetTop;
+        size = size.subtract(margins.size);
+
+        // Convert from compact JSON output to propertied data
+        var samplesWithProperties = {};
+        [Strings.json.controller, Strings.json.complexity, Strings.json.complexityAverage].forEach(function(seriesName) {
+            var series = testData[Strings.json.samples][seriesName];
+            samplesWithProperties[seriesName] = series.toArray();
+        })
+
+        this.createTimeGraph(testResult, samplesWithProperties[Strings.json.controller], testData[Strings.json.marks], testData[Strings.json.controller], options, margins, size);
         this.onTimeGraphOptionsChanged();
 
-        var hasComplexitySamples = !!testData[Strings.json.samples][Strings.json.complexity];
-        this._showOrHideNodes(hasComplexitySamples, "form[name=graph-type]");
-        if (hasComplexitySamples) {
-            document.forms["graph-type"].elements["type"] = "complexity";
-            this.createComplexityGraph(testResult, testData[Strings.json.controller], testData[Strings.json.samples], options, margins, size);
-            this.onComplexityGraphOptionsChanged();
-        }
+        this._showOrHideNodes(true, "form[name=graph-type]");
+        document.forms["graph-type"].elements["type"] = "complexity";
+        this.createComplexityGraph(testResult, testData[Strings.json.controller], samplesWithProperties, options, margins, size);
+        this.onComplexityGraphOptionsChanged();
 
         this.onGraphTypeChanged();
     },
@@ -569,7 +571,7 @@ Utilities.extendObject(window.benchmarkController, {
         benchmarkController._showOrHideNodes(!isTimeSelected, "#complexity-graph");
         benchmarkController._showOrHideNodes(!isTimeSelected, "form[name=complexity-graph-options]");
 
-        var score, mean;
+        var score = "", mean = "";
         if (isTimeSelected) {
             score = testResult[Strings.json.score].toFixed(2);
 
@@ -596,13 +598,16 @@ Utilities.extendObject(window.benchmarkController, {
             document.getElementById("complexity-regression-aggregate-average").textContent = complexityAverageRegression.complexity.toFixed(2) + ", ±" + complexityAverageRegression.stdev.toFixed(2) + "ms";
 
             var bootstrap = complexityRegression[Strings.json.bootstrap];
-            score = bootstrap.median.toFixed(2);
-            mean = [
-                "95% CI: ",
-                bootstrap.confidenceLow.toFixed(2),
-                "–",
-                bootstrap.confidenceHigh.toFixed(2)
-            ].join("");
+            if (bootstrap) {
+                score = bootstrap.median.toFixed(2);
+                mean = [
+                    (bootstrap.confidencePercentage * 100).toFixed(0),
+                    "% CI: ",
+                    bootstrap.confidenceLow.toFixed(2),
+                    "–",
+                    bootstrap.confidenceHigh.toFixed(2)
+                ].join("");
+            }
         }
 
         sectionsManager.setSectionScore("test-graph", score, mean);
index e3233f56ac056f282a2860dd14fbb1cfd49ff57b..11bb242fc4451dbb370ddf8309e6cdf5867c5009 100644 (file)
@@ -2,7 +2,7 @@ Utilities.extendObject(Strings.text, {
     samples: "Samples",
     complexity: "Time Complexity",
     frameRate: "FPS",
-    confidenceInterval: "95% Confidence Interval",
+    confidenceInterval: "80% Confidence Interval",
     mergedRawComplexity: "Raw Complexity",
     graph: "Graph"
 });
index d81dd79b0e35e070a6ca09f0634c7609021f3c18..69fe374ee7ebb863db03fc85c02110c08cb87aad 100644 (file)
@@ -155,6 +155,13 @@ Utilities =
     lerp: function(value, min, max)
     {
         return min + (max - min) * value;
+    },
+
+    toFixedNumber: function(number, precision)
+    {
+        if (number.toFixed)
+            return Number(number.toFixed(precision));
+        return number;
     }
 };
 
@@ -581,3 +588,83 @@ Utilities.extendObject(Heap, {
         return new Heap(maxSize, function(a, b) { return a - b; });
     }
 });
+
+var SampleData = Utilities.createClass(
+    function(fieldMap, data)
+    {
+        this.fieldMap = fieldMap || {};
+        this.data = data || [];
+    }, {
+
+    get length()
+    {
+        return this.data.length;
+    },
+
+    addField: function(name, index)
+    {
+        this.fieldMap[name] = index;
+    },
+
+    push: function(datum)
+    {
+        this.data.push(datum);
+    },
+
+    sort: function(sortFunction)
+    {
+        this.data.sort(sortFunction);
+    },
+
+    slice: function(begin, end)
+    {
+        return new SampleData(this.fieldMap, this.data.slice(begin, end));
+    },
+
+    forEach: function(iterationFunction)
+    {
+        this.data.forEach(iterationFunction);
+    },
+
+    createDatum: function()
+    {
+        return [];
+    },
+
+    getFieldInDatum: function(datum, fieldName)
+    {
+        if (typeof datum === 'number')
+            datum = this.data[datum];
+        return datum[this.fieldMap[fieldName]];
+    },
+
+    setFieldInDatum: function(datum, fieldName, value)
+    {
+        if (typeof datum === 'number')
+            datum = this.data[datum];
+        return datum[this.fieldMap[fieldName]] = value;
+    },
+
+    at: function(index)
+    {
+        return this.data[index];
+    },
+
+    toArray: function()
+    {
+        var array = [];
+
+        this.data.forEach(function(datum) {
+            var newDatum = {};
+            array.push(newDatum);
+
+            for (var fieldName in this.fieldMap) {
+                var value = this.getFieldInDatum(datum, fieldName);
+                if (value !== null && value !== undefined)
+                    newDatum[fieldName] = value;
+            }
+        }, this);
+
+        return array;
+    }
+});
index c65bb9733fe638ddddde47beb763c35bfbb6193e..05b45f8c84634a37a87456b417381e8aaa6eb287 100644 (file)
@@ -292,20 +292,14 @@ body.images-loaded #intro {
 
 /* Running test section */
 
-#test-container {
+.frame-container {
     position: absolute;
 
-    width: 1200px;
-    height: 600px;
-
     top: 50%;
     left: 50%;
-
-    margin-left: -600px;
-    margin-top: -300px;
 }
 
-#test-container > iframe {
+.frame-container > iframe {
     width: 100%;
     height: 100%;
 
@@ -313,70 +307,72 @@ body.images-loaded #intro {
     margin: 0;
 }
 
-@media screen and (max-device-width: 414px), screen and (max-device-height: 414px) and (orientation: landscape) {
-    #test-container {
-        width: 100%;
-        height: 100%;
-
-        top: 0;
-        left: 0;
-
-        margin-top: 0;
-        margin-left: 0;
-    }
+body.small .frame-container {
+    width: 568px;
+    height: 320px;
+    margin-left: -284px;
+    margin-top: -160px;
 }
 
-@media screen and (min-device-height: 768px) and (max-device-height: 1024px) {
-    #test-container {
-        width: 900px;
-        height: 600px;
+body.medium .frame-container {
+    width: 900px;
+    height: 600px;
+    margin-left: -450px;
+    margin-top: -300px;
+}
 
-        margin-left: -450px;
-        margin-top: -300px;
-    }
+body.large .frame-container {
+    width: 1600px;
+    height: 800px;
+    margin-left: -800px;
+    margin-top: -400px;
 }
 
-@media screen and (max-device-width: 1024px) and (min-device-height: 1366px) {
-    #test-container {
-        width: 1200px;
-        height: 600px;
+/* Results section */
 
-        margin-left: -600px;
-        margin-top: -300px;
-    }
+#results {
+    padding: 2em;
 }
 
-@media screen and (min-width: 1800px) {
-    #test-container {
-        width: 1600px;
-        height: 800px;
+#results .body {
+    -webkit-user-select: text;
+}
 
-        margin-left: -800px;
-        margin-top: -400px;
-    }
+#results .score-container {
+    padding-bottom: 2em;
 }
 
-/* Results section */
+#results .table-container {
+    position: relative;
+}
 
-#results {
-    padding: 2em;
+#results .table-container > div {
+    margin-left: 40%;
 }
 
 #results .score {
     font-size: 5em;
     font-weight: bold;
     font-style: italic;
+    line-height: 1;
+    margin: 0;
+}
 
-    padding-left: 0.25em;
-
+#results .confidence {
+    font-size: 2em;
+    font-style: italic;
+    line-height: 1;
     margin: 0;
+    text-indent: 1.75em;
+    color: hsl(0, 0%, 40%);
+    padding-bottom: .3em;
 }
 
 #results table {
     border-spacing: 0;
     margin: 0;
     padding: 0;
-    min-width: 40%;
+    min-width: 25%;
 }
 
 #results table td,
@@ -392,9 +388,20 @@ body.images-loaded #intro {
     background-color: hsla(0, 0%, 0%, 0.05);
 }
 
-#results #results-header,
+#results #results-header {
+    top: 0;
+    left: 0;
+    width: 40%;
+    position: absolute;
+}
+
 #results #results-score {
-    display: inline-table;
+    float: left;
+}
+
+#results #results-data span {
+    font-size: .75em;
+    color: hsl(0, 0%, 40%);
 }
 
 #results #results-header td,
@@ -411,17 +418,26 @@ body.images-loaded #intro {
 }
 
 #results #results-score td {
-    -webkit-user-select: text;
     cursor: text;
 }
 
 @media screen and (min-width: 667px) {
-    #results .body .score {
+    #results .score,
+    #results .confidence {
         font-style: normal;
-        padding-left: 0.1em;
     }
 }
 
+.detail span {
+    display: none;
+}
+
+body.small .detail .small,
+body.medium .detail .medium,
+body.large .detail .large {
+    display: initial;
+}
+
 #overlay {
     position: fixed;
 
@@ -456,6 +472,7 @@ body.images-loaded #intro {
 #overlay > div div {
     overflow: scroll;
 
+    font-size: 12px;
     -webkit-user-select: text;
     cursor: text;
 
index 54f648ee308abf3993b195f005a9310b085b7f38..1c470c749cd654e5e28f7641461a594fcaca2660 100644 (file)
@@ -23,6 +23,8 @@ ResultsDashboard = Utilities.createClass(
         var iterationsScores = [];
         this._iterationsSamplers.forEach(function(iteration, index) {
             var testsScores = [];
+            var testsLowerBoundScores = [];
+            var testsUpperBoundScores = [];
 
             var result = {};
             this._results[Strings.json.results.iterations][index] = result;
@@ -44,14 +46,20 @@ ResultsDashboard = Utilities.createClass(
                     delete suiteData[testName][Strings.json.result];
 
                     testsScores.push(suiteResult[testName][Strings.json.score]);
+                    testsLowerBoundScores.push(suiteResult[testName][Strings.json.scoreLowerBound]);
+                    testsUpperBoundScores.push(suiteResult[testName][Strings.json.scoreUpperBound]);
                 }
             }
 
             result[Strings.json.score] = Statistics.geometricMean(testsScores);
+            result[Strings.json.scoreLowerBound] = Statistics.geometricMean(testsLowerBoundScores);
+            result[Strings.json.scoreUpperBound] = Statistics.geometricMean(testsUpperBoundScores);
             iterationsScores.push(result[Strings.json.score]);
         }, this);
 
         this._results[Strings.json.score] = Statistics.sampleMean(iterationsScores.length, iterationsScores.reduce(function(a, b) { return a + b; }));
+        this._results[Strings.json.scoreLowerBound] = this._results[Strings.json.results.iterations][0][Strings.json.scoreLowerBound];
+        this._results[Strings.json.scoreUpperBound] = this._results[Strings.json.results.iterations][0][Strings.json.scoreUpperBound];
     },
 
     calculateScore: function(data)
@@ -64,38 +72,71 @@ ResultsDashboard = Utilities.createClass(
         if (this._options["controller"] == "ramp30")
             desiredFrameLength = 1000/30;
 
-        function findRegression(series) {
+        function findRegression(series, profile) {
             var minIndex = Math.round(.025 * series.length);
             var maxIndex = Math.round(.975 * (series.length - 1));
-            var minComplexity = series[minIndex].complexity;
-            var maxComplexity = series[maxIndex].complexity;
+            var minComplexity = series.getFieldInDatum(minIndex, Strings.json.complexity);
+            var maxComplexity = series.getFieldInDatum(maxIndex, Strings.json.complexity);
+
             if (Math.abs(maxComplexity - minComplexity) < 20 && maxIndex - minIndex < 20) {
                 minIndex = 0;
                 maxIndex = series.length - 1;
-                minComplexity = series[minIndex].complexity;
-                maxComplexity = series[maxIndex].complexity;
+                minComplexity = series.getFieldInDatum(minIndex, Strings.json.complexity);
+                maxComplexity = series.getFieldInDatum(maxIndex, Strings.json.complexity);
             }
 
+            var complexityIndex = series.fieldMap[Strings.json.complexity];
+            var frameLengthIndex = series.fieldMap[Strings.json.frameLength];
+            var regressionOptions = { desiredFrameLength: desiredFrameLength };
+            if (profile)
+                regressionOptions.preferredProfile = profile;
             return {
                 minComplexity: minComplexity,
                 maxComplexity: maxComplexity,
                 samples: series.slice(minIndex, maxIndex + 1),
                 regression: new Regression(
-                    series,
-                    function (datum, i) { return datum[i].complexity; },
-                    function (datum, i) { return datum[i].frameLength; },
-                    minIndex, maxIndex, desiredFrameLength)
+                    series.data,
+                    function (data, i) { return data[i][complexityIndex]; },
+                    function (data, i) { return data[i][frameLengthIndex]; },
+                    minIndex, maxIndex, regressionOptions)
             };
         }
 
         var complexitySamples;
+        // Convert these samples into SampleData objects if needed
+        [Strings.json.complexity, Strings.json.complexityAverage, Strings.json.controller].forEach(function(seriesName) {
+            var series = samples[seriesName];
+            if (series && !(series instanceof SampleData))
+                samples[seriesName] = new SampleData(series.fieldMap, series.data);
+        });
+
+        var isRampController = ["ramp", "ramp30"].indexOf(this._options["controller"]) != -1;
+        var predominantProfile = "";
+        if (isRampController) {
+            var profiles = {};
+            data[Strings.json.controller].forEach(function(regression) {
+                if (regression[Strings.json.regressions.profile]) {
+                    var profile = regression[Strings.json.regressions.profile];
+                    profiles[profile] = (profiles[profile] || 0) + 1;
+                }
+            });
+
+            var maxProfileCount = 0;
+            for (var profile in profiles) {
+                if (profiles[profile] > maxProfileCount) {
+                    predominantProfile = profile;
+                    maxProfileCount = profiles[profile];
+                }
+            }
+        }
+
         [Strings.json.complexity, Strings.json.complexityAverage].forEach(function(seriesName) {
             if (!(seriesName in samples))
                 return;
 
             var regression = {};
             result[seriesName] = regression;
-            var regressionResult = findRegression(samples[seriesName]);
+            var regressionResult = findRegression(samples[seriesName], predominantProfile);
             if (seriesName == Strings.json.complexity)
                 complexitySamples = regressionResult.samples;
             var calculation = regressionResult.regression;
@@ -111,7 +152,7 @@ ResultsDashboard = Utilities.createClass(
             regression[Strings.json.measurements.stdev] = Math.sqrt(calculation.error / samples[seriesName].length);
         });
 
-        if (["ramp", "ramp30"].indexOf(this._options["controller"]) != -1) {
+        if (isRampController) {
             var timeComplexity = new Experiment;
             data[Strings.json.controller].forEach(function(regression) {
                 timeComplexity.sample(regression[Strings.json.complexity]);
@@ -124,16 +165,22 @@ ResultsDashboard = Utilities.createClass(
             experimentResult[Strings.json.measurements.stdev] = timeComplexity.standardDeviation();
             experimentResult[Strings.json.measurements.percent] = timeComplexity.percentage();
 
-            result[Strings.json.complexity][Strings.json.bootstrap] = Regression.bootstrap(complexitySamples, 2500, function(resample) {
-                    resample.sort(function(a, b) {
-                        return a.complexity - b.complexity;
-                    });
-
-                    var regressionResult = findRegression(resample);
-                    return regressionResult.regression.complexity;
-                }, .95);
-            result[Strings.json.score] = result[Strings.json.complexity][Strings.json.bootstrap].median;
-
+            const bootstrapIterations = 2500;
+            var bootstrapResult = Regression.bootstrap(complexitySamples.data, bootstrapIterations, function(resampleData) {
+                var complexityIndex = complexitySamples.fieldMap[Strings.json.complexity];
+                resampleData.sort(function(a, b) {
+                    return a[complexityIndex] - b[complexityIndex];
+                });
+
+                var resample = new SampleData(complexitySamples.fieldMap, resampleData);
+                var regressionResult = findRegression(resample, predominantProfile);
+                return regressionResult.regression.complexity;
+            }, .8);
+
+            result[Strings.json.complexity][Strings.json.bootstrap] = bootstrapResult;
+            result[Strings.json.score] = bootstrapResult.median;
+            result[Strings.json.scoreLowerBound] = bootstrapResult.confidenceLow;
+            result[Strings.json.scoreUpperBound] = bootstrapResult.confidenceHigh;
         } else {
             var marks = data[Strings.json.marks];
             var samplingStartIndex = 0, samplingEndIndex = -1;
@@ -144,11 +191,13 @@ ResultsDashboard = Utilities.createClass(
 
             var averageComplexity = new Experiment;
             var averageFrameLength = new Experiment;
-            samples[Strings.json.controller].forEach(function (sample, i) {
+            var controllerSamples = samples[Strings.json.controller];
+            controllerSamples.forEach(function (sample, i) {
                 if (i >= samplingStartIndex && (samplingEndIndex == -1 || i < samplingEndIndex)) {
-                    averageComplexity.sample(sample.complexity);
-                    if (sample.smoothedFrameLength && sample.smoothedFrameLength != -1)
-                        averageFrameLength.sample(sample.smoothedFrameLength);
+                    averageComplexity.sample(controllerSamples.getFieldInDatum(sample, Strings.json.complexity));
+                    var smoothedFrameLength = controllerSamples.getFieldInDatum(sample, Strings.json.smoothedFrameLength);
+                    if (smoothedFrameLength && smoothedFrameLength != -1)
+                        averageFrameLength.sample(smoothedFrameLength);
                 }
             });
 
@@ -167,6 +216,8 @@ ResultsDashboard = Utilities.createClass(
             experimentResult[Strings.json.measurements.percent] = averageFrameLength.percentage();
 
             result[Strings.json.score] = averageComplexity.score(Experiment.defaults.CONCERN);
+            result[Strings.json.scoreLowerBound] = result[Strings.json.score] - averageFrameLength.standardDeviation();
+            result[Strings.json.scoreUpperBound] = result[Strings.json.score] + averageFrameLength.standardDeviation();
         }
     },
 
@@ -188,12 +239,27 @@ ResultsDashboard = Utilities.createClass(
         return this._options;
     },
 
-    get score()
+    _getResultsProperty: function(property)
     {
         if (this._results)
-            return this._results[Strings.json.score];
+            return this._results[property];
         this._processData();
-        return this._results[Strings.json.score];
+        return this._results[property];
+    },
+
+    get score()
+    {
+        return this._getResultsProperty(Strings.json.score);
+    },
+
+    get scoreLowerBound()
+    {
+        return this._getResultsProperty(Strings.json.scoreLowerBound);
+    },
+
+    get scoreUpperBound()
+    {
+        return this._getResultsProperty(Strings.json.scoreUpperBound);
     }
 });
 
@@ -237,34 +303,40 @@ ResultsTable = Utilities.createClass(
 
             var th = Utilities.createElement("th", {}, row);
             if (header.title != Strings.text.graph)
-                th.textContent = header.title;
+                th.innerHTML = header.title;
             if (header.children)
                 th.colSpan = header.children.length;
         });
     },
 
+    _addBody: function()
+    {
+        this.tbody = Utilities.createElement("tbody", {}, this.element);
+    },
+
     _addEmptyRow: function()
     {
-        var row = Utilities.createElement("tr", {}, this.element);
+        var row = Utilities.createElement("tr", {}, this.tbody);
         this._flattenedHeaders.forEach(function (header) {
             return Utilities.createElement("td", { class: "suites-separator" }, row);
         });
     },
 
-    _addTest: function(testName, testResults, options)
+    _addTest: function(testName, testResult, options)
     {
-        var row = Utilities.createElement("tr", {}, this.element);
+        var row = Utilities.createElement("tr", {}, this.tbody);
 
         this._flattenedHeaders.forEach(function (header) {
             var td = Utilities.createElement("td", {}, row);
-            if (header.title == Strings.text.testName) {
+            if (header.text == Strings.text.testName) {
                 td.textContent = testName;
-            } else if (header.text) {
-                var data = testResults[header.text];
+            } else if (typeof header.text == "string") {
+                var data = testResult[header.text];
                 if (typeof data == "number")
                     data = data.toFixed(2);
-                td.textContent = data;
-            }
+                td.innerHTML = data;
+            } else
+                td.innerHTML = header.text(testResult);
         }, this);
     },
 
@@ -284,6 +356,7 @@ ResultsTable = Utilities.createClass(
     {
         this.clear();
         this._addHeader();
+        this._addBody();
 
         var iterationsResults = dashboard.results;
         iterationsResults.forEach(function(iterationResult, index) {
@@ -327,10 +400,10 @@ window.sectionsManager =
 {
     showSection: function(sectionIdentifier, pushState)
     {
-        document.body.classList.remove("showing-intro");
-        document.body.classList.remove("showing-results");
-        document.body.classList.remove("showing-test-container");
-
+        var sections = document.querySelectorAll("main > section");
+        for (var i = 0; i < sections.length; ++i) {
+            document.body.classList.remove("showing-" + sections[i].id);
+        }
         document.body.classList.add("showing-" + sectionIdentifier);
 
         var currentSectionElement = document.querySelector("section.selected");
@@ -346,11 +419,11 @@ window.sectionsManager =
             history.pushState({section: sectionIdentifier}, document.title);
     },
 
-    setSectionScore: function(sectionIdentifier, score, mean)
+    setSectionScore: function(sectionIdentifier, score, confidence)
     {
         document.querySelector("#" + sectionIdentifier + " .score").textContent = score;
-        if (mean)
-            document.querySelector("#" + sectionIdentifier + " .mean").textContent = mean;
+        if (confidence)
+            document.querySelector("#" + sectionIdentifier + " .confidence").textContent = confidence;
     },
 
     populateTable: function(tableIdentifier, headers, dashboard)
@@ -363,9 +436,32 @@ window.sectionsManager =
 window.benchmarkController = {
     initialize: function()
     {
+        benchmarkController.determineCanvasSize();
         benchmarkController.addOrientationListenerIfNecessary();
     },
 
+    determineCanvasSize: function() {
+        var match = window.matchMedia("(max-device-width: 760px)");
+        if (match.matches) {
+            document.body.classList.add("small");
+            return;
+        }
+
+        match = window.matchMedia("(max-device-width: 1600px)");
+        if (match.matches) {
+            document.body.classList.add("medium");
+            return;
+        }
+
+        match = window.matchMedia("(max-width: 1600px)");
+        if (match.matches) {
+            document.body.classList.add("medium");
+            return;
+        }
+
+        document.body.classList.add("large");
+    },
+
     addOrientationListenerIfNecessary: function() {
         if (!("orientation" in window))
             return;
@@ -379,9 +475,9 @@ window.benchmarkController = {
     {
         benchmarkController.isInLandscapeOrientation = match.matches;
         if (match.matches)
-            document.querySelector(".orientation-check p").classList.add("hidden");
+            document.querySelector(".start-benchmark p").classList.add("hidden");
         else
-            document.querySelector(".orientation-check p").classList.remove("hidden");
+            document.querySelector(".start-benchmark p").classList.remove("hidden");
         benchmarkController.updateStartButtonState();
     },
 
@@ -392,6 +488,10 @@ window.benchmarkController = {
 
     _startBenchmark: function(suites, options, frameContainerID)
     {
+        var configuration = document.body.className.match(/small|medium|large/);
+        if (configuration)
+            options[Strings.json.configuration] = configuration[0];
+
         benchmarkRunnerClient.initialize(suites, options);
         var frameContainer = document.getElementById(frameContainerID);
         var runner = new BenchmarkRunner(suites, frameContainer, benchmarkRunnerClient);
@@ -403,7 +503,7 @@ window.benchmarkController = {
     startBenchmark: function()
     {
         var options = {
-            "test-interval": 20,
+            "test-interval": 30,
             "display": "minimal",
             "controller": "ramp",
             "kalman-process-error": 1,
@@ -421,10 +521,12 @@ window.benchmarkController = {
         }
 
         var dashboard = benchmarkRunnerClient.results;
-
-        sectionsManager.setSectionScore("results", dashboard.score.toFixed(2));
+        var score = dashboard.score;
+        var confidence = "±" + (Statistics.largestDeviationPercentage(dashboard.scoreLowerBound, score, dashboard.scoreUpperBound) * 100).toFixed(2) + "%";
+        sectionsManager.setSectionScore("results", score.toFixed(2), confidence);
         sectionsManager.populateTable("results-header", Headers.testName, dashboard);
         sectionsManager.populateTable("results-score", Headers.score, dashboard);
+        sectionsManager.populateTable("results-data", Headers.details, dashboard);
         sectionsManager.showSection("results", true);
     },
 
@@ -472,7 +574,11 @@ window.benchmarkController = {
                 options: benchmarkRunnerClient.results.options,
                 data: benchmarkRunnerClient.results.data
             };
-            data.textContent = JSON.stringify(output, null, 1);
+            data.textContent = JSON.stringify(output, function(key, value) {
+                if (typeof value === 'number')
+                    return Utilities.toFixedNumber(value, 3);
+                return value;
+            }, 1);
         }, 0);
         data.onclick = function() {
             var selection = window.getSelection();
@@ -503,7 +609,7 @@ window.benchmarkController = {
             }
             case 1: {
                 range.setStart(document.querySelector("#results .score"), 0);
-                range.setEndAfter(document.querySelector("#results-score > tr:last-of-type"), 0);
+                range.setEndAfter(document.querySelector("#results-score"), 0);
                 break;
             }
             case 2: {
index 16c7dc71150ca30af352ffce8d5336cffca0e544..83458b8095a28963c319d550e1e139ce05dcb7c8 100644 (file)
@@ -2,25 +2,25 @@
 <!-- Copyright © 2016 Apple Inc. All rights reserved. -->
 <svg viewBox="0 0 1052 1251" version="1.1" xmlns="http://www.w3.org/2000/svg">
     <g fill-opacity="0.5" fill="white">
-        <polygon points="251.761719 0 253.761719 0 2.12833345 1250.36328 0.128333454 1250.36328"/>
-        <polygon points="291.761719 0 295.761719 0 44.1283335 1250.36328 40.1283335 1250.36328"/>
-        <polygon points="331.761719 0 337.761719 0 86.1283335 1250.36328 80.1283335 1250.36328"/>
-        <polygon points="371.761719 0 379.761719 0 128.128333 1250.36328 120.128333 1250.36328"/>
-        <polygon points="411.761719 0 421.761719 0 170.128333 1250.36328 160.128333 1250.36328"/>
-        <polygon points="451.761719 0 463.761719 0 212.128333 1250.36328 200.128333 1250.36328"/>
-        <polygon points="491.761719 0 505.761719 0 254.128333 1250.36328 240.128333 1250.36328"/>
-        <polygon points="531.761719 0 547.761719 0 296.128333 1250.36328 280.128333 1250.36328"/>
-        <polygon points="571.761719 0 589.761719 0 338.128333 1250.36328 320.128333 1250.36328"/>
-        <polygon points="611.761719 0 631.761719 0 380.128333 1250.36328 360.128333 1250.36328"/>
-        <polygon points="651.761719 0 673.761719 0 422.128333 1250.36328 400.128333 1250.36328"/>
-        <polygon points="691.761719 0 715.761719 0 464.128333 1250.36328 440.128333 1250.36328"/>
-        <polygon points="731.761719 0 757.761719 0 506.128333 1250.36328 480.128333 1250.36328"/>
-        <polygon points="771.761719 0 799.761719 0 548.128333 1250.36328 520.128333 1250.36328"/>
-        <polygon points="811.761719 0 841.761719 0 590.128333 1250.36328 560.128333 1250.36328"/>
-        <polygon points="851.761719 0 883.761719 0 632.128333 1250.36328 600.128333 1250.36328"/>
-        <polygon points="891.761719 0 925.761719 0 674.128333 1250.36328 640.128333 1250.36328"/>
-        <polygon points="931.761719 0 967.761719 0 716.128333 1250.36328 680.128333 1250.36328"/>
-        <polygon points="971.761719 0 1009.76172 0 758.128333 1250.36328 720.128333 1250.36328"/>
-        <polygon points="1011.76172 0 1051.76172 0 800.128333 1250.36328 760.128333 1250.36328"/>
+        <polygon points="221.761719 0 223.761719 0 2.12833345 1250.36328 0.128333454 1250.36328"/>
+        <polygon points="261.761719 0 265.761719 0 44.1283335 1250.36328 40.1283335 1250.36328"/>
+        <polygon points="301.761719 0 307.761719 0 86.1283335 1250.36328 80.1283335 1250.36328"/>
+        <polygon points="341.761719 0 349.761719 0 128.128333 1250.36328 120.128333 1250.36328"/>
+        <polygon points="381.761719 0 391.761719 0 170.128333 1250.36328 160.128333 1250.36328"/>
+        <polygon points="421.761719 0 433.761719 0 212.128333 1250.36328 200.128333 1250.36328"/>
+        <polygon points="461.761719 0 475.761719 0 254.128333 1250.36328 240.128333 1250.36328"/>
+        <polygon points="501.761719 0 517.761719 0 296.128333 1250.36328 280.128333 1250.36328"/>
+        <polygon points="541.761719 0 559.761719 0 338.128333 1250.36328 320.128333 1250.36328"/>
+        <polygon points="581.761719 0 601.761719 0 380.128333 1250.36328 360.128333 1250.36328"/>
+        <polygon points="621.761719 0 643.761719 0 422.128333 1250.36328 400.128333 1250.36328"/>
+        <polygon points="661.761719 0 685.761719 0 464.128333 1250.36328 440.128333 1250.36328"/>
+        <polygon points="701.761719 0 727.761719 0 506.128333 1250.36328 480.128333 1250.36328"/>
+        <polygon points="741.761719 0 769.761719 0 548.128333 1250.36328 520.128333 1250.36328"/>
+        <polygon points="781.761719 0 811.761719 0 590.128333 1250.36328 560.128333 1250.36328"/>
+        <polygon points="821.761719 0 853.761719 0 632.128333 1250.36328 600.128333 1250.36328"/>
+        <polygon points="861.761719 0 895.761719 0 674.128333 1250.36328 640.128333 1250.36328"/>
+        <polygon points="901.761719 0 937.761719 0 716.128333 1250.36328 680.128333 1250.36328"/>
+        <polygon points="941.761719 0 979.76172 0 758.128333 1250.36328 720.128333 1250.36328"/>
+        <polygon points="981.76172 0 1021.76172 0 800.128333 1250.36328 760.128333 1250.36328"/>
     </g>
 </svg>
index e349c63d101aebb029bbba393d9b948d1a41e2c1..aa938c5189e038a67529748082ece257041401a1 100644 (file)
@@ -1,6 +1,25 @@
 var Headers = {
-    testName: [{ title: Strings.text.testName }],
-    score: [{ title: Strings.text.score, text: Strings.json.score }]
+    testName: [
+        {
+            title: "<span onclick='benchmarkController.showDebugInfo()'>" + Strings.text.testName + "</span>",
+            text: Strings.text.testName
+        }
+    ],
+    score: [
+        {
+            title: Strings.text.score,
+            text: Strings.json.score
+        }
+    ],
+    details: [
+        {
+            title: "&nbsp;",
+            text: function(data) {
+                var bootstrap = data[Strings.json.complexity][Strings.json.bootstrap];
+                return "<span>±" + (Statistics.largestDeviationPercentage(bootstrap.confidenceLow, bootstrap.median, bootstrap.confidenceHigh) * 100).toFixed(2) + "%</span>";
+            }
+        }
+    ]
 };
 
 var Suite = function(name, tests) {
index 6586e8630e1be31076c56040ee1ea4561c39a29d..9322a797b677a1db956f2d856913fc7421718549 100644 (file)
@@ -70,6 +70,11 @@ Statistics =
           var t = 1.0 / (1.0 + p * value);
           var y = 1.0 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-value * value);
           return sign * y;
+    },
+
+    largestDeviationPercentage: function(low, mean, high)
+    {
+        return Math.max(Math.abs(low / mean - 1), (high / mean - 1));
     }
 };
 
@@ -148,39 +153,49 @@ Experiment.defaults =
 };
 
 Regression = Utilities.createClass(
-    function(samples, getComplexity, getFrameLength, startIndex, endIndex, desiredFrameLength)
+    function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
     {
-        desiredFrameLength = desiredFrameLength || 1000/60;
-        var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
-            shouldClip: true,
-            s1: desiredFrameLength,
-            t1: 0
-        });
-        var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
-            shouldClip: true,
-            s1: desiredFrameLength,
-            t1: 0,
-            t2: 0
-        });
-        var desired;
-        if (slope.error < flat.error)
-            desired = slope;
-        else
-            desired = flat;
+        var desiredFrameLength = options.desiredFrameLength || 1000/60;
+        var bestProfile;
+
+        if (!options.preferredProfile || options.preferredProfile == Strings.json.profiles.slope) {
+            var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+                shouldClip: true,
+                s1: desiredFrameLength,
+                t1: 0
+            });
+            if (!bestProfile || slope.error < bestProfile.error) {
+                bestProfile = slope;
+                this.profile = Strings.json.profiles.slope;
+            }
+        }
+        if (!options.preferredProfile || options.preferredProfile == Strings.json.profiles.flat) {
+            var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+                shouldClip: true,
+                s1: desiredFrameLength,
+                t1: 0,
+                t2: 0
+            });
+
+            if (!bestProfile || flat.error < bestProfile.error) {
+                bestProfile = flat;
+                this.profile = Strings.json.profiles.flat;
+            }
+        }
 
         this.startIndex = Math.min(startIndex, endIndex);
         this.endIndex = Math.max(startIndex, endIndex);
 
-        this.complexity = desired.complexity;
-        this.s1 = desired.s1;
-        this.t1 = desired.t1;
-        this.s2 = desired.s2;
-        this.t2 = desired.t2;
-        this.stdev1 = desired.stdev1;
-        this.stdev2 = desired.stdev2;
-        this.n1 = desired.n1;
-        this.n2 = desired.n2;
-        this.error = desired.error;
+        this.complexity = bestProfile.complexity;
+        this.s1 = bestProfile.s1;
+        this.t1 = bestProfile.t1;
+        this.s2 = bestProfile.s2;
+        this.t2 = bestProfile.t2;
+        this.stdev1 = bestProfile.stdev1;
+        this.stdev2 = bestProfile.stdev2;
+        this.n1 = bestProfile.n1;
+        this.n2 = bestProfile.n2;
+        this.error = bestProfile.error;
     }, {
 
     valueAt: function(complexity)
@@ -217,6 +232,7 @@ Regression = Utilities.createClass(
             };
         }
 
+        // x is expected to increase in complexity
         var iterationDirection = endIndex > startIndex ? 1 : -1;
         var lowComplexity = getComplexity(samples, startIndex);
         var highComplexity = getComplexity(samples, endIndex);
@@ -245,7 +261,9 @@ Regression = Utilities.createClass(
             s2_best = s2;
             t2_best = t2;
             error2_best = error2;
+            // Number of samples included in the first segment, inclusive of splitIndex
             n1_best = iterationDirection * (splitIndex - startIndex) + 1;
+            // Number of samples included in the second segment
             n2_best = iterationDirection * (endIndex - splitIndex);
             if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
                 x_best = x_prime;
@@ -290,8 +308,8 @@ Regression = Utilities.createClass(
             // Assumes that the two segments meet
             var x_prime = (s1 - s2) / (t2 - t1);
 
-            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
-            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
+            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || Number.MAX_VALUE;
+            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || Number.MAX_VALUE;
 
             if (i == startIndex) {
                 setBest(s1, t1, error1, s2, t2, error2, i, x_prime, x);
@@ -322,8 +340,8 @@ Regression = Utilities.createClass(
                     t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
                     x_prime = (s1 - s2) / (t2 - t1);
 
-                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
-                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
+                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || Number.MAX_VALUE;
+                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || Number.MAX_VALUE;
                 } else if (t1 != t2)
                     continue;
             }
@@ -372,7 +390,8 @@ Utilities.extendObject(Regression, {
             confidenceHigh: bootstrapData[Math.round((iterationCount - 1) * (1 + confidencePercentage) / 2)],
             median: bootstrapData[Math.round(iterationCount / 2)],
             mean: bootstrapEstimator.mean(),
-            data: bootstrapData
+            data: bootstrapData,
+            confidencePercentage: confidencePercentage
         };
     }
 });
index 19229aa64b365e516c94a9443270e039185b6748..b58f67e9912d6cbcad75db81ddc4535e48d1b678 100644 (file)
@@ -9,13 +9,19 @@ var Strings = {
         samplingEndTimeOffset: "End sampling",
 
         samples: "samples",
+        dataFieldMap: "dataFieldMap",
         controller: "controller",
+        time: "time",
         complexity: "complexity",
         complexityAverage: "complexityAverage",
         frameLength: "frameLength",
+        smoothedFrameLength: "smoothedFrameLength",
 
         result: "result",
+        configuration: "configuration",
         score: "score",
+        scoreLowerBound: "scoreLowerBound",
+        scoreUpperBound: "scoreUpperBound",
         bootstrap: "bootstrap",
         measurements: {
             average: "average",
@@ -28,7 +34,13 @@ var Strings = {
             startIndex: "startIndex",
             endIndex: "endIndex",
             segment1: "segment1",
-            segment2: "segment2"
+            segment2: "segment2",
+            profile: "profile"
+        },
+
+        profiles: {
+            slope: "slope",
+            flat: "flat"
         },
 
         results: {
index e14ed59187a50855fff60d8f3aa37ff771f4cd20..29bb158a4aea7d2bf9876ec2275c700580d04992 100644 (file)
@@ -149,8 +149,12 @@ Controller = Utilities.createClass(
 
     _processComplexitySamples: function(complexitySamples, complexityAverageSamples)
     {
+        complexityAverageSamples.addField(Strings.json.complexity, 0);
+        complexityAverageSamples.addField(Strings.json.frameLength, 1);
+        complexityAverageSamples.addField(Strings.json.measurements.stdev, 2);
+
         complexitySamples.sort(function(a, b) {
-            return a.complexity - b.complexity;
+            return complexitySamples.getFieldInDatum(a, Strings.json.complexity) - complexitySamples.getFieldInDatum(b, Strings.json.complexity);
         });
 
         // Samples averaged based on complexity
@@ -159,21 +163,23 @@ Controller = Utilities.createClass(
         function addSample() {
             var mean = experimentAtComplexity.mean();
             var stdev = experimentAtComplexity.standardDeviation();
-            complexityAverageSamples.push({
-                complexity: currentComplexity,
-                frameLength: mean,
-                stdev: stdev
-            });
+
+            var averageSample = complexityAverageSamples.createDatum();
+            complexityAverageSamples.push(averageSample);
+            complexityAverageSamples.setFieldInDatum(averageSample, Strings.json.complexity, currentComplexity);
+            complexityAverageSamples.setFieldInDatum(averageSample, Strings.json.frameLength, mean);
+            complexityAverageSamples.setFieldInDatum(averageSample, Strings.json.measurements.stdev, stdev);
         }
         complexitySamples.forEach(function(sample) {
-            if (sample.complexity != currentComplexity) {
+            var sampleComplexity = complexitySamples.getFieldInDatum(sample, Strings.json.complexity);
+            if (sampleComplexity != currentComplexity) {
                 if (currentComplexity > -1)
                     addSample();
 
-                currentComplexity = sample.complexity;
+                currentComplexity = sampleComplexity;
                 experimentAtComplexity = new Experiment;
             }
-            experimentAtComplexity.sample(sample.frameLength);
+            experimentAtComplexity.sample(complexitySamples.getFieldInDatum(sample, Strings.json.frameLength));
         });
         // Finish off the last one
         addSample();
@@ -192,29 +198,37 @@ Controller = Utilities.createClass(
 
         results[Strings.json.samples] = {};
 
-        var complexitySamples = [], complexityAverageSamples = [];
+        var controllerSamples = new SampleData;
+        results[Strings.json.samples][Strings.json.controller] = controllerSamples;
+
+        controllerSamples.addField(Strings.json.time, 0);
+        controllerSamples.addField(Strings.json.complexity, 1);
+        controllerSamples.addField(Strings.json.frameLength, 2);
+        controllerSamples.addField(Strings.json.smoothedFrameLength, 3);
+
+        var complexitySamples = new SampleData(controllerSamples.fieldMap);
         results[Strings.json.samples][Strings.json.complexity] = complexitySamples;
-        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
 
-        results[Strings.json.samples][Strings.json.controller] = samples[0].map(function(timestamp, i) {
-            var sample = {
-                // Represent time in milliseconds
-                time: timestamp - this._startTimestamp,
-                complexity: samples[1][i]
-            };
+        samples[0].forEach(function(timestamp, i) {
+            var sample = controllerSamples.createDatum();
+            controllerSamples.push(sample);
+            complexitySamples.push(sample);
+
+            // Represent time in milliseconds
+            controllerSamples.setFieldInDatum(sample, Strings.json.time, timestamp - this._startTimestamp);
+            controllerSamples.setFieldInDatum(sample, Strings.json.complexity, samples[1][i]);
 
             if (i == 0)
-                sample.frameLength = 1000/60;
+                controllerSamples.setFieldInDatum(sample, Strings.json.frameLength, 1000/60);
             else
-                sample.frameLength = timestamp - samples[0][i - 1];
+                controllerSamples.setFieldInDatum(sample, Strings.json.frameLength, timestamp - samples[0][i - 1]);
 
             if (samples[2][i] != -1)
-                sample.smoothedFrameLength = samples[2][i];
-
-            complexitySamples.push(sample);
-            return sample;
+                controllerSamples.setFieldInDatum(sample, Strings.json.smoothedFrameLength, samples[2][i]);
         }, this);
 
+        var complexityAverageSamples = new SampleData;
+        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
         this._processComplexitySamples(complexitySamples, complexityAverageSamples);
     }
 });
@@ -331,7 +345,7 @@ RampController = Utilities.createSubclass(Controller,
         this._maximumComplexity = 1;
 
         // After the tier range is determined, figure out the number of ramp iterations
-        var minimumRampLength = 2500;
+        var minimumRampLength = 3000;
         var totalRampIterations = Math.max(1, Math.floor(this._endTimestamp / minimumRampLength));
         // Give a little extra room to run since the ramps won't be exactly this length
         this._rampLength = Math.floor((this._endTimestamp - totalRampIterations * this.intervalSamplingLength) / totalRampIterations);
@@ -357,15 +371,18 @@ RampController = Utilities.createSubclass(Controller,
     frameLengthDesired: 1000/60,
     // Add some tolerance; frame lengths shorter than this are considered to be @ the desired frame length
     frameLengthDesiredThreshold: 1000/58,
-    // Represents the lower bound threshold in order to find the right complexity range to sample
-    frameLengthSlowestThreshold: 1000/30,
+    // During tier sampling get at least this slow to find the right complexity range
+    frameLengthTierThreshold: 1000/30,
     // Try to make each ramp get this slow so that we can cross the break point
     frameLengthRampLowerThreshold: 1000/45,
+    // Do not let the regression calculation at the maximum complexity of a ramp get slower than this threshold
+    frameLengthRampUpperThreshold: 1000/20,
 
     start: function(startTimestamp, stage)
     {
         Controller.prototype.start.call(this, startTimestamp, stage);
         this._rampStartTimestamp = 0;
+        this.intervalSamplingLength = 100;
     },
 
     didFinishInterval: function(timestamp, stage, intervalAverageFrameLength)
@@ -376,7 +393,7 @@ RampController = Utilities.createSubclass(Controller,
 
             var currentComplexity = stage.complexity();
             var currentFrameLength = this._frameLengthEstimator.estimate;
-            if (currentFrameLength < this.frameLengthSlowestThreshold) {
+            if (currentFrameLength < this.frameLengthTierThreshold) {
                 var isAnimatingAt60FPS = currentFrameLength < this.frameLengthDesiredThreshold;
                 var hasFinishedSlowTierTest = timestamp > this._tierStartTimestamp + this.tierSlowTestLength;
 
@@ -404,6 +421,7 @@ RampController = Utilities.createSubclass(Controller,
 
             this._finishedTierSampling = true;
             this.isFrameLengthEstimatorEnabled = false;
+            this.intervalSamplingLength = 120;
 
             // Extend the test length so that the full test length is made of the ramps
             this._endTimestamp += timestamp;
@@ -418,7 +436,7 @@ RampController = Utilities.createSubclass(Controller,
             // Interpolate a maximum complexity that gets us around the lowest threshold.
             // Avoid doing this calculation if we never get out of the first tier (where this._lastTierComplexity is undefined).
             if (this._lastTierComplexity && this._lastTierComplexity != currentComplexity)
-                this._maximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this.frameLengthSlowestThreshold, this._lastTierFrameLength, currentFrameLength), this._lastTierComplexity, currentComplexity));
+                this._maximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this.frameLengthTierThreshold, this._lastTierFrameLength, currentFrameLength), this._lastTierComplexity, currentComplexity));
             else {
                 // If the browser is capable of handling the most complex version of the test, use that
                 this._maximumComplexity = currentComplexity;
@@ -464,7 +482,7 @@ RampController = Utilities.createSubclass(Controller,
         var intervalFrameLengthMean = this._intervalFrameLengthEstimator.mean();
         var intervalFrameLengthStandardDeviation = this._intervalFrameLengthEstimator.standardDeviation();
 
-        if (intervalFrameLengthMean < this.frameLengthDesiredThreshold && this._intervalFrameLengthEstimator.cdf(this.frameLengthDesiredThreshold) > .95) {
+        if (intervalFrameLengthMean < this.frameLengthDesiredThreshold && this._intervalFrameLengthEstimator.cdf(this.frameLengthDesiredThreshold) > .9) {
             this._possibleMinimumComplexity = Math.max(this._possibleMinimumComplexity, currentComplexity);
         } else if (intervalFrameLengthStandardDeviation > 2) {
             // In the case where we might have found a previous interval where 60fps was reached. We hit a significant blip,
@@ -478,28 +496,39 @@ RampController = Utilities.createSubclass(Controller,
         var progress = (timestamp - this._rampStartTimestamp) / this._currentRampLength;
 
         if (progress < 1) {
-            stage.tune(Math.floor(Utilities.lerp(progress, this._maximumComplexity, this._minimumComplexity)) - currentComplexity);
+            // Reframe progress percentage so that the last interval of the ramp can sample at minimum complexity
+            progress = (timestamp - this._rampStartTimestamp) / (this._currentRampLength - this.intervalSamplingLength);
+            stage.tune(Math.max(this._minimumComplexity, Math.floor(Utilities.lerp(progress, this._maximumComplexity, this._minimumComplexity))) - currentComplexity);
             return;
         }
 
         var regression = new Regression(this._sampler.samples, this._getComplexity, this._getFrameLength,
-            this._sampler.sampleCount - 1, this._rampStartIndex, this.frameLengthDesired);
+            this._sampler.sampleCount - 1, this._rampStartIndex, { desiredFrameLength: this.frameLengthDesired });
         this._rampRegressions.push(regression);
 
-        var interpolatedFrameLength = regression.valueAt(this._maximumComplexity);
-        if (interpolatedFrameLength < this.frameLengthRampLowerThreshold)
-            this._possibleMaximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this.frameLengthRampLowerThreshold, interpolatedFrameLength, this._lastTierFrameLength), this._maximumComplexity, this._lastTierComplexity));
+        var frameLengthAtMaxComplexity = regression.valueAt(this._maximumComplexity);
+        if (frameLengthAtMaxComplexity < this.frameLengthRampLowerThreshold)
+            this._possibleMaximumComplexity = Math.floor(Utilities.lerp(Utilities.progressValue(this.frameLengthRampLowerThreshold, frameLengthAtMaxComplexity, this._lastTierFrameLength), this._maximumComplexity, this._lastTierComplexity));
+        // If the regression doesn't fit the first segment at all, keep the minimum bound at 1
+        if ((timestamp - this._sampler.samples[0][this._sampler.sampleCount - regression.n1]) / this._currentRampLength < .25)
+            this._possibleMinimumComplexity = 1;
 
-        interpolatedFrameLength = regression.valueAt(this._minimumComplexity);
         this._minimumComplexityEstimator.sample(this._possibleMinimumComplexity);
-
-        this._changePointEstimator.sample(regression.complexity);
-
         this._minimumComplexity = Math.round(this._minimumComplexityEstimator.mean());
-        this._maximumComplexity = Math.round(this._minimumComplexity +
-            Math.max(5,
-                this._possibleMaximumComplexity - this._minimumComplexity,
-                (this._changePointEstimator.mean() - this._minimumComplexity) * 2));
+
+        if (frameLengthAtMaxComplexity < this.frameLengthRampUpperThreshold) {
+            this._changePointEstimator.sample(regression.complexity);
+            // Ideally we'll target the change point in the middle of the ramp. If the range of the ramp is too small, there isn't enough
+            // range along the complexity (x) axis for a good regression calculation to be made, so force at least a range of 5
+            // particles. Make it possible to increase the maximum complexity in case unexpected noise caps the regression too low.
+            this._maximumComplexity = Math.round(this._minimumComplexity +
+                Math.max(5,
+                    this._possibleMaximumComplexity - this._minimumComplexity,
+                    (this._changePointEstimator.mean() - this._minimumComplexity) * 2));
+        } else {
+            // The slowest samples weighed the regression too heavily
+            this._maximumComplexity = Math.max(Math.round(.8 * this._maximumComplexity), this._minimumComplexity + 5);
+        }
 
         // Next ramp
         stage.tune(this._maximumComplexity - stage.complexity());
@@ -529,21 +558,22 @@ RampController = Utilities.createSubclass(Controller,
             results[Strings.json.marks][markName].time -= startTimestamp;
         }
 
-        var timeSamples = results[Strings.json.samples][Strings.json.controller];
-        timeSamples.forEach(function(timeSample) {
-            timeSample.time -= startTimestamp;
+        var controllerSamples = results[Strings.json.samples][Strings.json.controller];
+        controllerSamples.forEach(function(timeSample) {
+            controllerSamples.setFieldInDatum(timeSample, Strings.json.time, controllerSamples.getFieldInDatum(timeSample, Strings.json.time) - startTimestamp);
         });
 
         // Aggregate all of the ramps into one big complexity-frameLength dataset
-        var complexitySamples = [], complexityAverageSamples = [];
+        var complexitySamples = new SampleData(controllerSamples.fieldMap);
         results[Strings.json.samples][Strings.json.complexity] = complexitySamples;
-        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
 
         results[Strings.json.controller] = [];
         this._rampRegressions.forEach(function(ramp) {
             var startIndex = ramp.startIndex, endIndex = ramp.endIndex;
-            var startTime = timeSamples[startIndex].time, endTime = timeSamples[endIndex].time;
-            var startComplexity = timeSamples[startIndex].complexity, endComplexity = timeSamples[endIndex].complexity;
+            var startTime = controllerSamples.getFieldInDatum(startIndex, Strings.json.time);
+            var endTime = controllerSamples.getFieldInDatum(endIndex, Strings.json.time);
+            var startComplexity = controllerSamples.getFieldInDatum(startIndex, Strings.json.complexity);
+            var endComplexity = controllerSamples.getFieldInDatum(endIndex, Strings.json.complexity);
 
             var regression = {};
             results[Strings.json.controller].push(regression);
@@ -562,10 +592,14 @@ RampController = Utilities.createSubclass(Controller,
             regression[Strings.json.complexity] = ramp.complexity;
             regression[Strings.json.regressions.startIndex] = startIndex;
             regression[Strings.json.regressions.endIndex] = endIndex;
+            regression[Strings.json.regressions.profile] = ramp.profile;
 
             for (var j = startIndex; j <= endIndex; ++j)
-                complexitySamples.push(timeSamples[j]);
+                complexitySamples.push(controllerSamples.at(j));
         });
+
+        var complexityAverageSamples = new SampleData;
+        results[Strings.json.samples][Strings.json.complexityAverage] = complexityAverageSamples;
         this._processComplexitySamples(complexitySamples, complexityAverageSamples);
     }
 });
@@ -578,8 +612,9 @@ Ramp30Controller = Utilities.createSubclass(RampController,
 
     frameLengthDesired: 1000/30,
     frameLengthDesiredThreshold: 1000/29,
-    frameLengthSlowestThreshold: 1000/20,
-    frameLengthRampLowerThreshold: 1000/20
+    frameLengthTierThreshold: 1000/20,
+    frameLengthRampLowerThreshold: 1000/20,
+    frameLengthRampUpperThreshold: 1000/12
 });
 
 Stage = Utilities.createClass(
index edf714156118d34748e7c046a7ca79fa519f3995..0b6ffdc4a21c4f83183efabaed0b36a60293da73 100644 (file)
@@ -7,7 +7,7 @@ body {
     margin: 0;
     padding: 0;
     background-color: rgb(241, 241, 241);
-    font-family: -apple-system, "Helvetica Neue", Helvetica, Verdana, sans-serif;
+    font-family: "Helvetica Neue", Helvetica, Verdana, sans-serif;
 }
 
 #stage {
index 8c20d531183f18bf535cca3203e0871bead82843..922609501412e21405ea1af13589aa10e20b7881 100644 (file)
@@ -1,3 +1,207 @@
+2016-06-21  Jon Lee  <jonlee@apple.com>
+
+        Improvements to Animometer benchmark
+        https://bugs.webkit.org/show_bug.cgi?id=157738
+
+        Reviewed by Dean Jackson.
+        Provisionally reviewed by Said Abou-Hallawa.
+
+        Include confidence interval for the final score, and store the canvas
+        size in the serialization so that it is accurately shown in results.
+
+        * Animometer/developer.html: Add a "confidence" div.
+        * Animometer/index.html: Ditto. Convert "mean" to "confidence".
+        * Animometer/resources/debug-runner/animometer.js: Look at options, and
+        if the configuration is included, update the body class based on it
+        (similar to what we do when we first load the page). That way, if you
+        drag-and-drop previous results in, that configuration is reflected in
+        the dashboard. Show the full confidence interval.
+
+        * Animometer/resources/debug-runner/animometer.css:
+        * Animometer/resources/debug-runner/animometer.js: Style update.
+        * Animometer/resources/runner/animometer.css:
+
+        * Animometer/resources/runner/animometer.js:
+        (_processData): Propagate the confidence interval values out and calculate
+        the lower and upper bounds. For now, shortcut the aggregate calculation,
+        since we only go through one iteration.
+        (this._processData.calculateScore): Propagate the confidence interval out
+        to be next to the score. Depending on the controller these values are
+        calculated differently.
+        (this._processData._getResultsProperty): Convenience function.
+        (this._processData.get score): Refactor.
+        (this._processData.get scoreLowerBound): Get the aggregate lower bound.
+        (this._processData.get scoreUpperBound): Get the aggregate upper bound.
+        (window.sectionsManager.setSectionScore):
+        (window.benchmarkController._startBenchmark): When the benchmark starts, note
+        the canvas size and add it to options. That way it will be automatically be
+        serialized.
+        (window.benchmarkController.showResults): Include the maximum deviation
+        percentage.
+        * Animometer/resources/runner/lines.svg: Make the background line up with the
+        skew.
+        * Animometer/resources/runner/tests.js:
+        (Headers.details.text): Refactor.
+        * Animometer/resources/statistics.js:
+        (Statistics.largestDeviationPercentage): Convenience function to calculate
+        the deviation percentage on either end and return the largest deviation.
+        * Animometer/resources/strings.js:
+
+        Allow specifying a regression profile to use instead of taking the one
+        with the lowest error.
+
+        Address an issue in the focus test when the regression calculated ends
+        up overestimating the change point, causing a cascade of tougher
+        ramps. The reason behind this is that at max complexity of an initial
+        ramp, the frame length is very high, and it influences the second
+        segment of the piecewise regression strongly, causing it to be very
+        steep. As a result, the first segment, expected to be flat, ends up
+        covering a higher range of complexity. That makes the change point
+        much higher than it should be. To avoid this, we will add a sanity
+        check on the maximum value of the ramp. If the regression's projected
+        value at the maximum complexity of the current ramp is very slow (less
+        than 20 fps), 1) reduce the maximum complexity by 20%, and 2) do not
+        include the regression's change point in the change point estimator.
+        That estimator is used as the midpoint of the next ramp, and including
+        the change point from a poor regression can bake in the error. The
+        controller already knows how to adjust for ramps that are too easy for
+        the system.
+
+        * Animometer/resources/runner/animometer.js:
+        (this._processData.findRegression): Takes a preferred profile and gives that to
+        Regression.
+        (this._processData.calculateScore): With the ramp controller, take the profile
+        of the ramp that was used the most when calculating the ramp's regression. That
+        profile is what is used for the test's score.
+        * Animometer/resources/statistics.js:
+        (Regression.Utilities.createClass): Update to take an options object which can
+        specify a profile to calculate with. Otherwise it will continue to use both and
+        select the one with the lower error.
+        (_calculateRegression): Fix an issue where we claim 0 error if the regression
+        calculation fails due to divide-by-zero. Instead reject that regression calculation
+        by giving it Number.MAX_VALUE.
+        * Animometer/resources/strings.js: New strings for marking a regression as flat
+        or slope.
+        * Animometer/tests/resources/main.js:
+        (RampController): Rename the thresholds for clarity. Add a threshold that, if
+        exceeded, will lower the maximum complexity of the next ramp.
+        (tune): Relax the cdf check to consider whether the interval definitely falls in
+        the desired frame length threshold.
+        (processSamples): Include the profile in the ramp.
+
+        Update ramp controller test. Increase the length of the test to 30 seconds, and extend
+        the interval to 120 ms during sampling. Improve the estimation of the ramp parameters.
+
+        * Animometer/developer.html: Change default to 30 seconds, and don't show the progress bar
+        by default.
+        * Animometer/resources/runner/animometer.js: Change default to 30 seconds.
+        * Animometer/tests/resources/main.js: A number of improvements to the ramp controller, in
+        the order in which they appear in the patch:
+
+        - With a longer test length use longer ramps with longer intervals to get more data at each
+        complexity. Keep the 100 ms interval length during the ramp up phase since we don't need to
+        spend more time there to find the right order of magnitude, but increase it during the
+        ramps to 120 ms.
+        - The ramp linearly interpolates the complexity to render based on its timestamp, but it would
+        never sample the minimum complexity. Instead of lerping max to min complexity from time
+        0 to t where t is the ramp length, instead lerp from 0 to (t - intervalSampleLength) so that
+        we can have at least one interval sample at the min complexity for that ramp.
+        - Some regression calculations only come out with one line segment rather than the two
+        we expect. This could be due to a noisy ramp or the ramp's range is too narrow. If that's the
+        case, influence the minimum complexity of the next ramp towards the lowest bound of 1, so that
+        we ensure that at least part of the ramp is covering a complexity range that the system can
+        handle at full 60.
+        - Remove an assignment to interpolatedFrameLength since that is never subsequently used.
+
+        Update the format used to serialize the results for analysis.
+
+        Each data point used to have named properties for fields like complexity and frame rate.
+        In addition the serialized numbers had rounding errors that took up many characters.
+        Update the format by introducing a new data container called SampleData, which contains a
+        field map. The map maps a string to an array index. Each data point is an array, so, to
+        get a stat, use the field map to get the array index into the data point. This allows future
+        versions to track other data, and reduces the size of the output string by two-thirds.
+
+        * Animometer/resources/extensions.js:
+        (Utilities.toFixedNumber): Add convenience function that truncates the number to a fixed
+        precision, and converts it back to a number.
+        (SampleData): New class that contains sample data and a field map that maps properties to
+        an array index.
+        (get length): Number of data points.
+        (addField): Add a field to the field map.
+        (push): Add a data point.
+        (sort): Sort the data.
+        (slice): Return new SampleData object with sliced data.
+        (forEach): Iterate over the data with provided function.
+        (createDatum):
+        (getFieldInDatum): Returns the data point associated with the field name by looking it up
+        in the field map in the datum provided, which can be the datum object itself (an array) or
+        an index into the data member variable.
+        (setFieldInDatum): Sets the data point associated with the field name.
+        (at): Returns the data point at the provided index.
+        (toArray): Serializes the data where the field map serves as property names for each point.
+
+        * Animometer/resources/debug-runner/graph.js:
+        (updateGraphData): Remove unused _testData. Convert the data to the old array format for the
+        graph to use, since the old format was much easier to work with when displaying the graphs.
+        (onGraphTypeChanged): For some controllers, no alternative score or mean is provided.
+        * Animometer/resources/runner/animometer.css:
+        * Animometer/resources/runner/animometer.js: Refactor to use SampleData. Update JSON output
+        to only go to 3 digits of precision for purposes of reducing the data size.
+        * Animometer/resources/strings.js: Add new strings to put into the field maps.
+        * Animometer/tests/resources/main.js: Refactor to use SampleData.
+
+        * Animometer/developer.html:
+        * Animometer/index.html: Restructure results table for both pages. Add charset attribute to
+        tests.js include.
+        * Animometer/resources/debug-runner/animometer.css: Clear out styles from release runner.
+        * Animometer/resources/debug-runner/graph.js:
+        (onGraphTypeChanged): Update score and mean if bootstrap results are available from the
+        controller, since not all controllers do bootstrapping.
+        * Animometer/resources/debug-runner/tests.js: Update header text.
+        * Animometer/resources/runner/animometer.css: Include confidence interval in results.
+        * Animometer/resources/runner/animometer.js:
+        (ResultsTable._addHeader): Header contents can be HTML, so use innerHTML instead.
+        (ResultsTable._addBody): Add tbody element.
+        (ResultsTable._addTest): Allow a data cell to invoke a JS function to get its contents.
+        (window.benchmarkController.showResults): Add table that includes tests' confidence intervals.
+        * Animometer/resources/runner/tests.js:
+        (Headers.details.text): Add new details table that includes bootstrap confidence interval.
+        The interval can be asymmetric, but for simplicity, report the maximum deviation percentage
+        on either side of the bootstrap median.
+        * Animometer/resources/statistics.js:
+        (bootstrap): Include the confidence percentage in the return object.
+
+        Report canvas size in results.
+
+        * Animometer/developer.html: Add markup to indicate whether a small, medium, or large
+        canvas was used.
+        * Animometer/index.html: Ditto.
+        * Animometer/resources/debug-runner/animometer.js: Call determineCanvasSize().
+        * Animometer/resources/runner/animometer.css: Update styles to set the canvas based on the
+        body class size.
+        * Animometer/resources/runner/animometer.js:
+        (window.benchmarkController.initialize): Update styles to set the canvas based on the
+        body class size.
+        (window.benchmarkController.determineCanvasSize): Run various media queries and set the body
+        class based on the size of the device.
+
+        * Animometer/developer.html: Refactor to include the main CSS file, and redo
+        the layout so that it doesn't rely on flexbox.
+        * Animometer/resources/debug-runner/animometer.css:
+        * Animometer/resources/debug-runner/animometer.js:
+        (updateDisplay): Since various parts of the script alter the body class, we can't
+        replace the className directly. Instead, remove all display-based values and then add
+        the one that was selected.
+        * Animometer/resources/debug-runner/graph.js:
+        (updateGraphData): To set the size of the graph, use window.innerHeight.
+        * Animometer/resources/runner/animometer.js:
+        (window.sectionsManager.showSection): Since various parts of the script alter the body
+        class, we can't replace the className directly. Remove all of the section classes
+        individually and then add the one desired.
+        * Animometer/tests/resources/stage.css: Remove -apple-system as a font to use in the
+        stage.
+
 2016-06-12  Filip Pizlo  <fpizlo@apple.com>
 
         Fix round-down goof in Air.js's ShuffleCustom.forEachArg