Update data reporting and analysis
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 4 Jan 2016 02:36:57 +0000 (02:36 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 4 Jan 2016 02:36:57 +0000 (02:36 +0000)
https://bugs.webkit.org/show_bug.cgi?id=152670

Reviewed by Simon Fraser.

Show new graph data. Provide controls to show different series data. Provide an
interactive cursor that shows the data at a given sample.

* Animometer/developer.html: Add a nav section in #results. Each part of the graph
has a checkbox for visual toggling, as well as companion spans to contain the data.
        The numbers will always be shown even if the SVG isn't.
* Animometer/resources/debug-runner/animometer.css:
(#suites): Adjust spacing when doing fixed complexity.
(#test-graph nav): Place the nav in the upper right corner.
(#test-graph-data > svg): Fix the FPS scale from 0-60. It makes the raw FPS goes past
that scale. Allow it to show.
(.target-fps): Add a dotted line for where the benchmark is supposed to settle for FPS.
(#cursor line): The cursor contains a line and highlight circles for the data being shown.
(#cursor circle):
(#complexity path): This and rules afterward are named by series type.
(#complexity circle):
(#filteredFPS path):
(#filteredFPS circle):
(#rawFPS path):
(#intervalFPS circle):
(.left-samples): Deleted.
(.right-samples): Deleted.
* Animometer/resources/debug-runner/animometer.js:
(initialize): Add a "changed" listener when the checkboxes change in the nav.
(onBenchmarkOptionsChanged): Renamed.
(showTestGraph): All graph data is passed in as graphData instead of as arguments.
* Animometer/resources/debug-runner/graph.js: Extend BenchmarkController. When showing
a new graph, call updateGraphData(). It creates all of the d3 graphs. onGraphOptionsChanged()
toggles the data on and off.
(updateGraphData): Add the axes. Add the average lines and markers for sample time and
target FPS. Add the cursor group. Use helper function addData() to add the data. On top of
everything add a transparent area which will catch all of the mouse events. When the mouse
moves in the graph, find the closest data point, show the data in the nav area, and highlight
the data points.
(addData): Adds a line and circle for each data point. Also adds a highlight cursor with a
size a little larger than the circle radius for the data points.
(onGraphOptionsChanged): Called when data is visually toggled.
(showOrHideNodes): Helper function to toggle the .hidden class.
* Animometer/resources/extensions.js:
(ResultsDashboard.prototype.get data): Get rid of the arguments for _processData.
(ResultsTable.prototype._addGraphButton): Shove all of the graph data into a singular object.

Producing the JSON can take a while with all of the data. Make it on-demand with a
button.

* Animometer/resources/debug-runner/animometer.js:
(showResults): When showing the results, don't serialize the JSON data. Move that to...
(showJSONResults): ...here. Remove the button.

* Animometer/developer.html: Add a button. The button will remove itself and populate
the textarea with the JSON data.
* Animometer/resources/debug-runner/animometer.css:
(.hidden): Add a universal hidden class.
(#results button.small-button): Promote the small-button styles to the whole results
section for use in the JSON button.
(#results button.small-button:active):
(#results-data button.small-button): Deleted.
(#results-data button.small-button:active): Deleted.

Refactor how Animator does its recording.

* Animometer/tests/resources/math.js: Create a new, simple estimator that just returns
the same interval frame rate for adjustment.
* Animometer/tests/resources/main.js:
(Animator): Remove _dropFrameCount, and make variables more accurate described.
(Animator.prototype.initialize): Use the identity estimator instead of using a bool.
(Animator.prototype._intervalTimeDelta): Rename, only used internally.
(Animator.prototype._shouldRequestAnotherFrame): Assume we drop one frame for adjustment
of the scene. If we are within the number of frames to measure for the interval, just
record the timestamp. Otherwise we are ready to evaluate and adjust the scene. Record
the interval frame rate and the estimator's frame rate.

Avoid processing the data through the Experiment while the test is running. Reconfigure
the sampler to just record the raw samples. After the test is done, run the samples through
the Experiment to get the score.

* Animometer/resources/sampler.js:
(Experiment): Fold _init() into the constructor since nobody else will call it. This is not
needed until the test concludes, so remove startSampling(). Clients should just call sample().
(Sampler): Pre-allocate arrays given the number of data points being recorded, and a capacity
of how many samples will be used. The processor is a called when it's time to process the data
since that is the client also telling the Sampler what to record.
        Introduce the notion of marks as well, which allows the client to mark when an
event occurs. When we mark sample start, we can attach the timestamp there, instead of storing
it separately.
(Sampler.prototype.startSampling): Deleted. Clients should just call record().
(Sampler.prototype.record): The data to record is passed in as variable arguments.
(Sampler.prototype.mark): When adding a mark, a client needs to provide a unique string, and
can provide extra data object for later retrieval.
(Sampler.prototype.process): Renamed from toJSON. Trim the sampling arrays to what was used.
Call the processor to process the samples.
* Animometer/resources/debug-runner/benchmark-runner.js:
(BenchmarkRunner.prototype._runBenchmarkAndRecordResults): Call process().

* Animometer/resources/strings.js: Add some new strings, remove the graph ones since they are
not used.
* Animometer/tests/resources/main.js:
(Benchmark): Create a sampler with 4 series. The maximum number of points expected is the
number of seconds multiplied by 60 fps. Benchmark, as a client of the Sampler, knows about all
of the data being added to the Sampler. It is added through record(), and processed through
processSamples().
(Benchmark.prototype.update): Mark when we've passed warmup and are starting to sample. Include
the timestamp in the custom data for the mark. This avoids the need to store is separately in
the Sampler. Fold what was in record() here, since nothing else needs this functionality.
record() now just relays the information to the sampler.
(Benchmark.prototype.record): Called by Animator, which provides the data to the sampler.
Animator's calls to this is part of a later patch. Requires each stage to return its complexity.
(Benchmark.prototype.processSamples): If the sampling mark exists, add it to the results.
        Go through all of the samples. All samples contain a timestamp and complexity. We
calculate "raw FPS" which is the time differential from the previous sample. At regular intervals
the Kalman-filtered FPS and the interval average FPS are also recorded. We also create two
experiments, to get the scores for the complexity and smoothed FPS, and add those samples to
the experiments. Grab those scores and add them into results also.

Add complexity() to the tests for Benchmark.record().
* Animometer/tests/bouncing-particles/resources/bouncing-particles.js:
* Animometer/tests/misc/resources/canvas-electrons.js:
* Animometer/tests/misc/resources/canvas-stars.js:
* Animometer/tests/text/resources/layering-text.js:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@194523 268f45cc-cd09-0410-ab3c-d52691b4dbfc

15 files changed:
PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/animometer.js
PerformanceTests/Animometer/resources/debug-runner/benchmark-runner.js
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/extensions.js
PerformanceTests/Animometer/resources/sampler.js
PerformanceTests/Animometer/resources/strings.js
PerformanceTests/Animometer/tests/bouncing-particles/resources/bouncing-particles.js
PerformanceTests/Animometer/tests/misc/resources/canvas-electrons.js
PerformanceTests/Animometer/tests/misc/resources/canvas-stars.js
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/Animometer/tests/resources/math.js
PerformanceTests/Animometer/tests/text/resources/layering-text.js
PerformanceTests/ChangeLog

index 86de154..20016fa 100644 (file)
                 <table id="results-header"></table>
             </div>
             <div id="results-json">
-                JSON:
-                <textarea rows=1 onclick="this.focus();this.select()" readonly></textarea>
+                <button class="small-button" onclick="benchmarkController.showJSONResults()">JSON results</button>
+                <div class="hidden">
+                    JSON:
+                    <textarea rows=1 onclick="this.focus();this.select()" readonly></textarea>
+                </div>
             </div>
             <button onclick="benchmarkController.startBenchmark()">Test Again</button>
             <p>'s': Select different data for copy/paste</p>
                 <button onclick="benchmarkController.showResults()">&lt; Results</button>
                 <h1>Graph:</h1>
             </header>
+            <nav>
+                <form name="graph-options">
+                    <ul>
+                        <li><label><input type="checkbox" name="markers" checked> Markers</label>
+                            <span>time: <span class="time"></span></span></li>
+                        <li><label><input type="checkbox" name="averages" checked> Averages</label></li>
+                        <li><label><input type="checkbox" name="complexity" checked> Complexity</label>
+                            <span class="complexity"></span></li>
+                        <li><label><input type="checkbox" name="rawFPS"> Raw FPS</label>
+                            <span class="rawFPS"></span></li>
+                        <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
+                            <span class="filteredFPS"></span></li>
+                        <li><label><input type="checkbox" name="intervalFPS"> Average FPS per sample interval</label>
+                            <span class="intervalFPS"></span></li>
+                    </ul>
+                </form>
+            </nav>
             <p class="score"></p>
             <p class="mean"></p>
             <div id="test-graph-data"></div>
index b4f82be..be05487 100644 (file)
@@ -3,6 +3,10 @@ body {
     color: rgb(235, 235, 235);
 }
 
+.hidden {
+    display: none;
+}
+
 h1 {
     margin: 5vh 0;
 }
@@ -139,6 +143,7 @@ label.tree-label {
 
 #suites {
     padding-left: 15vw;
+    padding-right: 2em;
     flex: 0 1 40%;
 }
 
@@ -149,7 +154,7 @@ label.tree-label {
 #intro input[type="number"] {
     width: 50px;
 }
+
 #suites input[type="number"] {
     display: none;
     float: right;
@@ -266,6 +271,20 @@ label.tree-label {
     font-size: 2em;
 }
 
+#results button.small-button {
+    border: 1px solid rgba(235, 235, 235, .9);
+    color: rgba(235, 235, 235, .9);
+    border-radius: 2px;
+    padding: 1px 4px;
+    margin: 0 0 0 1em;
+    font-size: 9px;
+}
+
+#results button.small-button:active {
+    background-color: rgba(235, 235, 235, .2);
+    color: inherit;
+}
+
 .score {
     font-size: 3em;
 }
@@ -287,20 +306,6 @@ label.tree-label {
     padding-left: .25em;
 }
 
-#results-data button.small-button {
-    border: 1px solid rgba(235, 235, 235, .9);
-    color: rgba(235, 235, 235, .9);
-    border-radius: 2px;
-    padding: 1px 4px;
-    margin: 0 0 0 1em;
-    font-size: 9px;
-}
-
-#results-data button.small-button:active {
-    background-color: rgba(235, 235, 235, .2);
-    color: inherit;
-}
-
 #results-tables td.noisy-results {
     color: rgb(255, 104, 104);
 }
@@ -346,6 +351,36 @@ label.tree-label {
     align-self: stretch;
 }
 
+#test-graph nav {
+    position: absolute;
+    top: 1.5em;
+    right: 0;
+    font-size: .8em;
+    width: 25%;
+}
+
+#test-graph nav ul {
+    margin: 0 30px 0 0;
+    padding: 0;
+    list-style: none;
+}
+
+#test-graph nav ul ul {
+    padding-left: 2em;
+}
+
+#test-graph nav li {
+    padding: .1em 0;
+}
+
+#test-graph nav li > span {
+    float: right;
+}
+
+#test-graph nav.hide-data span {
+    display: none;
+}
+
 /* -------------------------------------------------------------------------- */
 /*                           Graph Section                                    */
 /* -------------------------------------------------------------------------- */
@@ -357,6 +392,7 @@ label.tree-label {
 
 #test-graph-data > svg {
     fill: none;
+    overflow: visible;
 }
 
 .axis path,
@@ -366,16 +402,6 @@ label.tree-label {
     shape-rendering: crispEdges;
 }
 
-.left-samples {
-    stroke: #7ADD49;
-    stroke-width: 1.5px;
-}
-
-.right-samples {
-    stroke: #FA4925;
-    stroke-width: 1.5px;
-}
-
 .sample-time {
     stroke: #5493D6;
 }
@@ -385,7 +411,50 @@ label.tree-label {
     opacity: .8;
 }
 
+.target-fps {
+    stroke: rgba(250, 73, 37, .4);
+    stroke-width: 1px;
+    stroke-dasharray: 10, 10;
+}
+
 .right-mean {
     stroke: #FA4925;
     opacity: .8;
 }
+
+#cursor line {
+    stroke: rgb(250, 250, 250);
+    stroke-width: 1px;
+}
+
+#cursor circle {
+    fill: rgb(250, 250, 250);
+}
+
+#complexity path {
+    stroke: rgba(122, 221, 73, .7);
+    stroke-width: 2px;
+}
+
+#complexity circle {
+    fill: rgb(122, 221, 73);
+}
+
+#filteredFPS path {
+    stroke: rgba(250, 73, 37, .7);
+    stroke-width: 2px;
+}
+
+#filteredFPS circle {
+    fill: rgb(250, 73, 37);
+}
+
+#rawFPS path {
+    stroke: rgba(250, 73, 37, .7);
+    stroke-width: 1px;
+}
+
+#rawFPS circle,
+#intervalFPS circle {
+    fill: rgb(250, 73, 37);
+}
index 48d523a..605bb38 100644 (file)
@@ -308,7 +308,8 @@ window.suitesManager =
 Utilities.extendObject(window.benchmarkController, {
     initialize: function()
     {
-        document.forms["benchmark-options"].addEventListener("change", benchmarkController.onFormChanged, true);
+        document.forms["benchmark-options"].addEventListener("change", benchmarkController.onBenchmarkOptionsChanged, true);
+        document.forms["graph-options"].addEventListener("change", benchmarkController.onGraphOptionsChanged, true);
         optionsManager.updateUIFromLocalStorage();
         suitesManager.createElements();
         suitesManager.updateUIFromLocalStorage();
@@ -316,7 +317,7 @@ Utilities.extendObject(window.benchmarkController, {
         suitesManager.updateEditsElementsState();
     },
 
-    onFormChanged: function(event)
+    onBenchmarkOptionsChanged: function(event)
     {
         if (event.target.name == "adjustment") {
             suitesManager.updateEditsElementsState();
@@ -346,22 +347,28 @@ Utilities.extendObject(window.benchmarkController, {
         sectionsManager.populateTable("results-header", Headers.testName, data);
         sectionsManager.populateTable("results-score", Headers.score, data);
         sectionsManager.populateTable("results-data", Headers.details, data);
+        sectionsManager.showSection("results", true);
+
+        suitesManager.updateLocalStorageFromJSON(data[0]);
+    },
+
+    showJSONResults: function()
+    {
         document.querySelector("#results-json textarea").textContent = JSON.stringify(benchmarkRunnerClient.results.data, function(key, value) {
             if (typeof value == "number")
                 return value.toFixed(2);
             return value;
         });
-        sectionsManager.showSection("results", true);
-
-        suitesManager.updateLocalStorageFromJSON(data[0]);
+        document.querySelector("#results-json button").remove();
+        document.querySelector("#results-json div").classList.remove("hidden");
     },
 
-    showTestGraph: function(testName, score, mean, axes, samples, samplingTimeOffset)
+    showTestGraph: function(testName, score, mean, graphData)
     {
         sectionsManager.setSectionHeader("test-graph", testName);
         sectionsManager.setSectionScore("test-graph", score, mean);
         sectionsManager.showSection("test-graph", true);
-        graph("#test-graph-data", new Insets(10, 20, 30, 40), axes, samples, samplingTimeOffset);
+        this.updateGraphData(graphData);
     }
 });
 
index 376b51b..8fd16f5 100644 (file)
@@ -96,7 +96,7 @@ BenchmarkRunner.prototype = {
         var benchmark = new contentWindow.benchmarkClass(options);
         benchmark.run().then(function(sampler) {
             var samplers = self._suitesSamplers[suite.name] || {};
-            samplers[test.name] = sampler.toJSON(true, true);
+            samplers[test.name] = sampler.process(options);
             self._suitesSamplers[suite.name] = samplers;
 
             if (self._client && self._client.didRunTest)
index 021f5f5..dac213a 100644 (file)
-function graph(selector, margins, axes, samples, samplingTimeOffset)
-{
-    var element = document.querySelector(selector);
-    element.innerHTML = '';
-
-    var size = Point.elementClientSize(element).subtract(margins.size);
-
-    var x = d3.scale.linear()
-            .range([0, size.width])
-            .domain(d3.extent(samples, function(d) { return d.timeOffset; }));            
-
-    var yLeft = d3.scale.linear()
-            .range([size.height, 0])
-            .domain([0, d3.max(samples, function(d) { return d.values[0]; })]);
-
-    var yRight = d3.scale.linear()
-            .range([size.height, 0])
-            .domain([0, d3.max(samples, function(d) { return d.values[1]; })]);
-
-    var xAxis = d3.svg.axis()
-            .scale(x)
-            .orient("bottom");
-
-    var yAxisLeft = d3.svg.axis()
-            .scale(yLeft)
-            .orient("left");
-
-    var yAxisRight = d3.svg.axis()
-            .scale(yRight)
-            .orient("right");
-
-    var lineLeft = d3.svg.line()
-            .x(function(d) { return x(d.timeOffset); })
-            .y(function(d) { return yLeft(d.values[0]); });
-
-    var lineRight = d3.svg.line()
-            .x(function(d) { return x(d.timeOffset); })
-            .y(function(d) { return yRight(d.values[1]); });
-
-    samples.forEach(function(d) {
-        d.timeOffset = +d.timeOffset;
-        d.values[0] = +d.values[0];
-        d.values[1] = +d.values[1];        
-    });    
-
-    var sampledSamples = samples.filter(function(d) {
-        return d.timeOffset >= samplingTimeOffset;
-    });
-    
-    var meanLeft = d3.mean(sampledSamples, function(d) {
-        return +d.values[0];
-    });
-
-    var meanRight = d3.mean(sampledSamples, function(d) {
-        return +d.values[1];
-    });
-                            
-    var svg = d3.select(selector).append("svg")
-        .attr("width", size.width + margins.left + margins.right)
-        .attr("height", size.height + margins.top + margins.bottom)
-        .append("g")
-            .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
-
-    // x-axis
-    svg.append("g")
-        .attr("class", "x axis")
-        .attr("fill", "rgb(235, 235, 235)")
-        .attr("transform", "translate(0," + size.height + ")")
-        .call(xAxis)
-        .append("text")
-            .attr("class", "label")
-            .attr("x", size.width)
-            .attr("y", -6)
+Utilities.extendObject(window.benchmarkController, {
+    updateGraphData: function(graphData)
+    {
+        var element = document.getElementById("test-graph-data");
+        element.innerHTML = "";
+        var margins = new Insets(10, 30, 30, 40);
+        var size = Point.elementClientSize(element).subtract(margins.size);
+
+        var svg = d3.select("#test-graph-data").append("svg")
+            .attr("width", size.width + margins.left + margins.right)
+            .attr("height", size.height + margins.top + margins.bottom)
+            .append("g")
+                .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
+
+        var axes = graphData.axes;
+        var targetFPS = graphData.targetFPS;
+
+        // Axis scales
+        var x = d3.scale.linear()
+                .range([0, size.width])
+                .domain([0, d3.max(graphData.samples, function(s) { return s.time; })]);
+        var yLeft = d3.scale.linear()
+                .range([size.height, 0])
+                .domain([0, d3.max(graphData.samples, function(s) { return s.complexity; })]);
+        var yRight = d3.scale.linear()
+                .range([size.height, 0])
+                .domain([0, 60]);
+
+        // Axes
+        var xAxis = d3.svg.axis()
+                .scale(x)
+                .orient("bottom");
+        var yAxisLeft = d3.svg.axis()
+                .scale(yLeft)
+                .orient("left");
+        var yAxisRight = d3.svg.axis()
+                .scale(yRight)
+                .orient("right");
+
+        // x-axis
+        svg.append("g")
+            .attr("class", "x axis")
             .attr("fill", "rgb(235, 235, 235)")
-            .style("text-anchor", "end")
-            .text("time");
-                 
-    // yLeft-axis
-    svg.append("g")
-        .attr("class", "y axis")
-        .attr("fill", "#7ADD49")
-        .call(yAxisLeft)
-        .append("text")
-            .attr("class", "label")
-            .attr("transform", "rotate(-90)")
-            .attr("y", 6)
+            .attr("transform", "translate(0," + size.height + ")")
+            .call(xAxis)
+            .append("text")
+                .attr("class", "label")
+                .attr("x", size.width)
+                .attr("y", -6)
+                .attr("fill", "rgb(235, 235, 235)")
+                .style("text-anchor", "end")
+                .text("time");
+
+        // yLeft-axis
+        svg.append("g")
+            .attr("class", "y axis")
             .attr("fill", "#7ADD49")
-            .attr("dy", ".71em")
-            .style("text-anchor", "end")
-            .text(axes[0]);
-
-    // yRight-axis
-    svg.append("g")
-        .attr("class", "y axis")
-        .attr("fill", "#FA4925")
-        .attr("transform", "translate(" + size.width + ", 0)")
-        .call(yAxisRight)
-        .append("text")
-            .attr("class", "label")
-            .attr("transform", "rotate(-90)")
-            .attr("y", 6)
+            .call(yAxisLeft)
+            .append("text")
+                .attr("class", "label")
+                .attr("transform", "rotate(-90)")
+                .attr("y", 6)
+                .attr("fill", "#7ADD49")
+                .attr("dy", ".71em")
+                .style("text-anchor", "end")
+                .text(axes[0]);
+
+        // yRight-axis
+        svg.append("g")
+            .attr("class", "y axis")
             .attr("fill", "#FA4925")
-            .attr("dy", ".71em")
-            .style("text-anchor", "end")
-            .text(axes[1]);
-
-    // left-mean
-    svg.append("svg:line")
-        .attr("x1", x(0))
-        .attr("x2", size.width)
-        .attr("y1", yLeft(meanLeft))
-        .attr("y2", yLeft(meanLeft))
-        .attr("class", "left-mean");
-
-    // right-mean
-    svg.append("svg:line")
-        .attr("x1", x(0))
-        .attr("x2", size.width)
-        .attr("y1", yRight(meanRight))
-        .attr("y2", yRight(meanRight))
-        .attr("class", "right-mean");        
-
-    // samplingTimeOffset
-    svg.append("line")
-        .attr("x1", x(samplingTimeOffset))
-        .attr("x2", x(samplingTimeOffset))
-        .attr("y1", yLeft(0))
-        .attr("y2", yLeft(yAxisLeft.scale().domain()[1]))
-        .attr("class", "sample-time");
-
-    // left-samples
-    svg.append("path")
-        .datum(samples)
-        .attr("class", "left-samples")
-        .attr("d", lineLeft);
-        
-    // right-samples
-    svg.append("path")
-        .datum(samples)
-        .attr("class", "right-samples")
-        .attr("d", lineRight);
-}
+            .attr("transform", "translate(" + size.width + ", 0)")
+            .call(yAxisRight)
+            .append("text")
+                .attr("class", "label")
+                .attr("transform", "rotate(-90)")
+                .attr("y", 6)
+                .attr("fill", "#FA4925")
+                .attr("dy", ".71em")
+                .style("text-anchor", "end")
+                .text(axes[1]);
+
+        // samplingTimeOffset
+        svg.append("line")
+            .attr("x1", x(graphData.samplingTimeOffset))
+            .attr("x2", x(graphData.samplingTimeOffset))
+            .attr("y1", yLeft(0))
+            .attr("y2", yLeft(yAxisLeft.scale().domain()[1]))
+            .attr("class", "sample-time marker");
+
+        // left-mean
+        svg.append("line")
+            .attr("x1", x(0))
+            .attr("x2", size.width)
+            .attr("y1", yLeft(graphData.mean[0]))
+            .attr("y2", yLeft(graphData.mean[0]))
+            .attr("class", "left-mean mean");
+
+        // right-mean
+        svg.append("line")
+            .attr("x1", x(0))
+            .attr("x2", size.width)
+            .attr("y1", yRight(graphData.mean[1]))
+            .attr("y2", yRight(graphData.mean[1]))
+            .attr("class", "right-mean mean");
+
+        // right-target
+        if (targetFPS) {
+            svg.append("line")
+                .attr("x1", x(0))
+                .attr("x2", size.width)
+                .attr("y1", yRight(targetFPS))
+                .attr("y2", yRight(targetFPS))
+                .attr("class", "target-fps marker");
+        }
+
+        // Cursor
+        var cursorGroup = svg.append("g").attr("id", "cursor");
+        cursorGroup.append("line")
+            .attr("x1", 0)
+            .attr("x2", 0)
+            .attr("y1", yLeft(0))
+            .attr("y2", yLeft(0));
+
+        // Data
+        var allData = graphData.samples;
+        var filteredData = graphData.samples.filter(function (sample) {
+            return "smoothedFPS" in sample;
+        });
+
+        function addData(name, data, yCoordinateCallback, pointRadius, omitLine) {
+            var svgGroup = svg.append("g").attr("id", name);
+            if (!omitLine) {
+                svgGroup.append("path")
+                    .datum(data)
+                    .attr("d", d3.svg.line()
+                        .x(function(d) { return x(d.time); })
+                        .y(yCoordinateCallback));
+            }
+            svgGroup.selectAll("circle")
+                .data(data)
+                .enter()
+                .append("circle")
+                .attr("cx", function(d) { return x(d.time); })
+                .attr("cy", yCoordinateCallback)
+                .attr("r", pointRadius);
+
+            cursorGroup.append("circle")
+                .attr("class", name)
+                .attr("r", pointRadius + 2);
+        }
+
+        addData("complexity", allData, function(d) { return yLeft(d.complexity); }, 2);
+        addData("rawFPS", allData, function(d) { return yRight(d.fps); }, 1);
+        addData("filteredFPS", filteredData, function(d) { return yRight(d.smoothedFPS); }, 2);
+        addData("intervalFPS", filteredData, function(d) { return yRight(d.intervalFPS); }, 3, true);
+
+        // Area to handle mouse events
+        var area = svg.append("rect")
+            .attr("fill", "transparent")
+            .attr("x", 0)
+            .attr("y", 0)
+            .attr("width", size.x)
+            .attr("height", size.y);
+
+        var timeBisect = d3.bisector(function(d) { return d.time; }).right;
+        var statsToHighlight = ["complexity", "rawFPS", "filteredFPS", "intervalFPS"];
+        area.on("mouseover", function() {
+            document.getElementById("cursor").classList.remove("hidden");
+            document.querySelector("#test-graph nav").classList.remove("hide-data");
+        }).on("mouseout", function() {
+            document.getElementById("cursor").classList.add("hidden");
+            document.querySelector("#test-graph nav").classList.add("hide-data");
+        }).on("mousemove", function() {
+            var form = document.forms["graph-options"].elements;
+
+            var mx_domain = x.invert(d3.mouse(this)[0]);
+            var index = Math.min(timeBisect(allData, mx_domain), allData.length - 1);
+            var data = allData[index];
+            var cursor_x = x(data.time);
+            var cursor_y = yAxisRight.scale().domain()[1];
+            if (form["rawFPS"].checked)
+                cursor_y = Math.max(cursor_y, data.fps);
+            cursorGroup.select("line")
+                .attr("x1", cursor_x)
+                .attr("x2", cursor_x)
+                .attr("y2", yRight(cursor_y));
+
+            document.querySelector("#test-graph nav .time").textContent = data.time.toFixed(3) + "s (" + index + ")";
+            statsToHighlight.forEach(function(name) {
+                var element = document.querySelector("#test-graph nav ." + name);
+                var content = "";
+                var data_y = null;
+                switch (name) {
+                case "complexity":
+                    content = data.complexity;
+                    data_y = yLeft(data.complexity);
+                    break;
+                case "rawFPS":
+                    content = data.fps.toFixed(2);
+                    data_y = yRight(data.fps);
+                    break;
+                case "filteredFPS":
+                    if ("smoothedFPS" in data) {
+                        content = data.smoothedFPS.toFixed(2);
+                        data_y = yRight(data.smoothedFPS);
+                    }
+                    break;
+                case "intervalFPS":
+                    if ("intervalFPS" in data) {
+                        content = data.intervalFPS.toFixed(2);
+                        data_y = yRight(data.intervalFPS);
+                    }
+                    break;
+                }
+
+                element.textContent = content;
+
+                if (form[name].checked && data_y !== null) {
+                    cursorGroup.select("." + name)
+                        .attr("cx", cursor_x)
+                        .attr("cy", data_y);
+                    document.querySelector("#cursor ." + name).classList.remove("hidden");
+                } else
+                    document.querySelector("#cursor ." + name).classList.add("hidden");
+            });
+        });
+        this.onGraphOptionsChanged();
+    },
+
+    onGraphOptionsChanged: function() {
+        var form = document.forms["graph-options"].elements;
+
+        function showOrHideNodes(isShown, selector) {
+            var nodeList = document.querySelectorAll(selector);
+            if (isShown) {
+                for (var i = 0; i < nodeList.length; ++i)
+                    nodeList[i].classList.remove("hidden");
+            } else {
+                for (var i = 0; i < nodeList.length; ++i)
+                    nodeList[i].classList.add("hidden");
+            }
+        }
+
+        showOrHideNodes(form["markers"].checked, ".marker");
+        showOrHideNodes(form["averages"].checked, ".mean");
+        showOrHideNodes(form["complexity"].checked, "#complexity");
+        showOrHideNodes(form["rawFPS"].checked, "#rawFPS");
+        showOrHideNodes(form["filteredFPS"].checked, "#filteredFPS");
+        showOrHideNodes(form["intervalFPS"].checked, "#intervalFPS");
+    }
+});
index 84cc1fd..f39bcaf 100644 (file)
@@ -248,7 +248,7 @@ ResultsDashboard.prototype =
         this._iterationsSamplers.push(suitesSamplers);
     },
 
-    _processData: function(statistics, graph)
+    _processData: function()
     {
         var iterationsResults = [];
         var iterationsScores = [];
@@ -290,7 +290,7 @@ ResultsDashboard.prototype =
     {
         if (this._processedData)
             return this._processedData;
-        this._processData(true, true);
+        this._processData();
         return this._processedData;
     },
 
@@ -346,9 +346,6 @@ ResultsTable.prototype =
         var button = DocumentExtension.createElement("button", { class: "small-button" }, td);
 
         button.addEventListener("click", function() {
-            var samples = data[Strings.json.graph.points];
-            var samplingTimeOffset = data[Strings.json.graph.samplingTimeOffset];
-            var axes = [Strings.text.experiments.complexity, Strings.text.experiments.frameRate];
             var score = testResults[Strings.json.score].toFixed(2);
             var complexity = testResults[Strings.json.experiments.complexity];
             var mean = [
@@ -360,7 +357,19 @@ ResultsTable.prototype =
                 complexity[Strings.json.measurements.percent].toFixed(2),
                 "%), worst 5%: ",
                 complexity[Strings.json.measurements.concern].toFixed(2)].join("");
-            benchmarkController.showTestGraph(testName, score, mean, axes, samples, samplingTimeOffset);
+
+            var graphData = {
+                axes: [Strings.text.experiments.complexity, Strings.text.experiments.frameRate],
+                mean: [
+                    testResults[Strings.json.experiments.complexity][Strings.json.measurements.average],
+                    testResults[Strings.json.experiments.frameRate][Strings.json.measurements.average]
+                ],
+                samples: data,
+                samplingTimeOffset: testResults[Strings.json.samplingTimeOffset]
+            }
+            if (testResults[Strings.json.targetFPS])
+                graphData.targetFPS = testResults[Strings.json.targetFPS];
+            benchmarkController.showTestGraph(testName, score, mean, graphData);
         });
 
         button.textContent = Strings.text.results.graph + "...";
index db0935f..cd1efff 100644 (file)
@@ -22,12 +22,14 @@ var Statistics =
             return 0;
         var roots = values.map(function(value) { return  Math.pow(value, 1 / values.length); })
         return roots.reduce(function(a, b) { return a * b; });
-    }   
+    }
 }
 
 function Experiment()
 {
-    this._init();
+    this._sum = 0;
+    this._squareSum = 0;
+    this._numberOfSamples = 0;
     this._maxHeap = Algorithm.createMaxHeap(Experiment.defaults.CONCERN_SIZE);
 }
 
@@ -39,22 +41,6 @@ Experiment.defaults =
 
 Experiment.prototype =
 {
-    _init: function()
-    {
-        this._sum = 0;
-        this._squareSum = 0;
-        this._numberOfSamples = 0;
-    },
-    
-    // Called after a warm-up period
-    startSampling: function()
-    {
-        var mean = this.mean();
-        this._init();
-        this._maxHeap.init();
-        this.sample(mean);
-    },
-    
     sample: function(value)
     {
         this._sum += value;
@@ -62,91 +48,82 @@ Experiment.prototype =
         this._maxHeap.push(value);
         ++this._numberOfSamples;
     },
-    
+
     mean: function()
     {
         return Statistics.sampleMean(this._numberOfSamples, this._sum);
     },
-    
+
     standardDeviation: function()
     {
         return Statistics.unbiasedSampleStandardDeviation(this._numberOfSamples, this._sum, this._squareSum);
     },
-    
+
     percentage: function()
     {
         var mean = this.mean();
         return mean ? this.standardDeviation() * 100 / mean : 0;
     },
-    
+
     concern: function(percentage)
     {
         var size = Math.ceil(this._numberOfSamples * percentage / 100);
         var values = this._maxHeap.values(size);
         return values.length ? values.reduce(function(a, b) { return a + b; }) / values.length : 0;
     },
-    
+
     score: function(percentage)
     {
         return Statistics.geometricMean([this.mean(), Math.max(this.concern(percentage), 1)]);
     }
 }
 
-function Sampler(count)
+function Sampler(seriesCount, expectedSampleCount, processor)
 {
-    this.experiments = [];
-    while (count--)
-        this.experiments.push(new Experiment());
+    this._processor = processor;
+
     this.samples = [];
-    this.samplingTimeOffset = 0;
+    for (var i = 0; i < seriesCount; ++i) {
+        var array = new Array(expectedSampleCount);
+        array.fill(0);
+        this.samples[i] = array;
+    }
+    this.sampleCount = 0;
+    this.marks = {};
 }
 
 Sampler.prototype =
 {
-    startSampling: function(samplingTimeOffset)
-    {
-        this.experiments.forEach(function(experiment) {
-            experiment.startSampling();
-        });
-            
-        this.samplingTimeOffset = samplingTimeOffset / 1000;
+    record: function() {
+        // Assume that arguments.length == this.samples.length
+        for (var i = 0; i < arguments.length; i++) {
+            this.samples[i][this.sampleCount] = arguments[i];
+        }
+        ++this.sampleCount;
     },
-    
-    sample: function(timeOffset, values)
-    {
-        if (values.length < this.experiments.length)
-            throw "Not enough sample points";
-
-        this.experiments.forEach(function(experiment, index) {
-            experiment.sample(values[index]);
-        });
-                    
-        this.samples.push({ timeOffset: timeOffset / 1000, values: values });
+
+    mark: function(comment, data) {
+        data = data || {};
+        // The mark exists after the last recorded sample
+        data.index = this.sampleCount;
+
+        this.marks[comment] = data;
     },
-    
-    toJSON: function(statistics, graph)
+
+    process: function(options)
     {
         var results = {};
-         
-        results[Strings.json.score] = this.experiments[0].score(Experiment.defaults.CONCERN);
-           
-        if (statistics) {
-            this.experiments.forEach(function(experiment, index) {
-                var jsonExperiment = !index ? Strings.json.experiments.complexity : Strings.json.experiments.frameRate;
-                results[jsonExperiment] = {};
-                results[jsonExperiment][Strings.json.measurements.average] = experiment.mean();
-                results[jsonExperiment][Strings.json.measurements.concern] = experiment.concern(Experiment.defaults.CONCERN);
-                results[jsonExperiment][Strings.json.measurements.stdev] = experiment.standardDeviation();
-                results[jsonExperiment][Strings.json.measurements.percent] = experiment.percentage()
-            });
-        }
-        
-        if (graph) {
-            results[Strings.json.samples] = {};
-            results[Strings.json.samples][Strings.json.graph.points] = this.samples;
-            results[Strings.json.samples][Strings.json.graph.samplingTimeOffset] = this.samplingTimeOffset;
-        }
-        
+
+        if (options["adjustment"] == "adaptive")
+            results[Strings.json.targetFPS] = +options["frame-rate"];
+
+        // Remove unused capacity
+        this.samples = this.samples.map(function(array) {
+            return array.slice(0, this.sampleCount);
+        }, this);
+
+        this._processor.processSamples(results);
+
         return results;
     }
 }
index 81f46be..c53d6e1 100644 (file)
@@ -26,6 +26,9 @@ var Strings = {
         score: "score",
         samples: "samples",
 
+        targetFPS: "targetFPS",
+        samplingTimeOffset: "samplingTimeOffset",
+
         experiments: {
             complexity: "complexity",
             frameRate: "frameRate"
@@ -42,11 +45,6 @@ var Strings = {
             iterations: "iterationsResults",
             suites: "suitesResults",
             tests: "testsResults"
-        },
-
-        graph: {
-            points: "points",
-            samplingTimeOffset: "samplingTimeOffset"
         }
     }
 };
index 50e65cc..e56ffd5 100644 (file)
@@ -113,5 +113,10 @@ BouncingParticlesStage = Utilities.createSubclass(Stage,
 
         this.particles.splice(-count, count);
         return this.particles.length;
+    },
+
+    complexity: function()
+    {
+        return this.particles.length;
     }
 });
index f15804f..f7b0a65 100644 (file)
@@ -95,6 +95,11 @@ CanvasElectronsStage = Utilities.createSubclass(Stage,
         this._electrons.forEach(function(electron) {
             electron.animate(timeDelta);
         });
+    },
+
+    complexity: function()
+    {
+        return this._electrons.length;
     }
 });
 
index e853976..d886eec 100644 (file)
@@ -91,6 +91,11 @@ CanvasStarsStage = Utilities.createSubclass(Stage,
         this._objects.forEach(function(object) {
             object.animate(timeDelta);
         });
+    },
+
+    complexity: function()
+    {
+        return this._objects.length;
     }
 });
 
index 74f64ec..9837c57 100644 (file)
@@ -180,12 +180,10 @@ Stage.prototype =
 
 function Animator()
 {
-    this._frameCount = 0;
-    this._dropFrameCount = 1;
-    this._measureFrameCount = 3;
+    this._intervalFrameCount = 0;
+    this._numberOfFramesToMeasurePerInterval = 3;
     this._referenceTime = 0;
     this._currentTimeOffset = 0;
-    this._estimator = new KalmanEstimator(60);
 }
 
 Animator.prototype =
@@ -193,7 +191,12 @@ Animator.prototype =
     initialize: function(benchmark)
     {
         this._benchmark = benchmark;
-        this._estimateFrameRate = benchmark.options["estimated-frame-rate"];
+
+        // Use Kalman filter to get a more non-fluctuating frame rate.
+        if (benchmark.options["estimated-frame-rate"])
+            this._estimator = new KalmanEstimator(60);
+        else
+            this._estimator = new IdentityEstimator;
     },
 
     get benchmark()
@@ -201,55 +204,55 @@ Animator.prototype =
         return this._benchmark;
     },
 
-    timeDelta: function()
+    _intervalTimeDelta: function()
     {
         return this._currentTimeOffset - this._startTimeOffset;
     },
 
     _shouldRequestAnotherFrame: function()
     {
+        // Cadence is number of frames to measure, then one more frame to adjust the scene, and drop
         var currentTime = performance.now();
-        
+
         if (!this._referenceTime)
             this._referenceTime = currentTime;
-        else
-            this._currentTimeOffset = currentTime - this._referenceTime;
 
-        if (!this._frameCount)
-            this._startTimeOffset = this._currentTimeOffset;
+        this._currentTimeOffset = currentTime - this._referenceTime;
 
-        ++this._frameCount;
+        if (!this._intervalFrameCount)
+            this._startTimeOffset = this._currentTimeOffset;
 
-        // Start measuring after dropping _dropFrameCount frames.
-        if (this._frameCount == this._dropFrameCount)
-            this._measureTimeOffset = this._currentTimeOffset;
+        // Start the work for the next frame.
+        ++this._intervalFrameCount;
 
         // Drop _dropFrameCount frames and measure the average of _measureFrameCount frames.
-        if (this._frameCount < this._dropFrameCount + this._measureFrameCount)
+        if (this._intervalFrameCount <= this._numberOfFramesToMeasurePerInterval) {
+            this._benchmark.record(this._currentTimeOffset, -1, -1);
             return true;
+        }
 
-        // Get the average FPS of _measureFrameCount frames over measureTimeDelta.
-        var measureTimeDelta = this._currentTimeOffset - this._measureTimeOffset;
-        var currentFrameRate = Math.floor(1000 / (measureTimeDelta / this._measureFrameCount));
-
-        // Use Kalman filter to get a more non-fluctuating frame rate.
-        if (this._estimateFrameRate)
-            currentFrameRate = this._estimator.estimate(currentFrameRate);
+        // Get the average FPS of _measureFrameCount frames over intervalTimeDelta.
+        var intervalTimeDelta = this._intervalTimeDelta();
+        var intervalFrameRate = 1000 / (intervalTimeDelta / this._numberOfFramesToMeasurePerInterval);
+        var estimatedIntervalFrameRate = this._estimator.estimate(intervalFrameRate);
+        // Record the complexity of the frame we just rendered. The next frame's time respresents the adjusted
+        // complexity
+        this._benchmark.record(this._currentTimeOffset, estimatedIntervalFrameRate, intervalFrameRate);
 
         // Adjust the test to reach the desired FPS.
-        var result = this._benchmark.update(this._currentTimeOffset, this.timeDelta(), currentFrameRate);
+        var shouldContinueRunning = this._benchmark.update(this._currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
 
         // Start the next drop/measure cycle.
-        this._frameCount = 0;
+        this._intervalFrameCount = 0;
 
-        // If result == 0, no more requestAnimationFrame() will be invoked.
-        return result;
+        // If result is false, no more requestAnimationFrame() will be invoked.
+        return shouldContinueRunning;
     },
 
     animateLoop: function()
     {
         if (this._shouldRequestAnotherFrame()) {
-            this._benchmark.stage.animate(this.timeDelta());
+            this._benchmark.stage.animate(this._intervalTimeDelta());
             requestAnimationFrame(this.animateLoop.bind(this));
         }
     }
@@ -267,7 +270,7 @@ function Benchmark(stage, options)
     this._recordInterval = 200;
     this._isSampling = false;
     this._controller = new PIDController(this._options["frame-rate"]);
-    this._sampler = new Sampler(2);
+    this._sampler = new Sampler(4, 60 * this._options["test-interval"], this);
     this._state = new BenchmarkState(this._options["test-interval"] * 1000);
 }
 
@@ -295,7 +298,7 @@ Benchmark.prototype =
     },
 
     // Called from the animator to adjust the complexity of the test.
-    update: function(currentTimeOffset, timeDelta, currentFrameRate)
+    update: function(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate)
     {
         this._state.update(currentTimeOffset);
 
@@ -306,7 +309,9 @@ Benchmark.prototype =
         }
 
         if (stage == BenchmarkState.stages.SAMPLING && !this._isSampling) {
-            this._sampler.startSampling(this._state.samplingTimeOffset());
+            this._sampler.mark(Strings.json.samplingTimeOffset, {
+                time: this._state.samplingTimeOffset() / 1000
+            });
             this._isSampling = true;
         }
 
@@ -320,27 +325,27 @@ Benchmark.prototype =
         else if (!(this._isSampling && this._options["adjustment"] == "fixed-after-warmup")) {
             // The relationship between frameRate and test complexity is inverse-proportional so we
             // need to use the negative of PIDController.tune() to change the complexity of the test.
-            tuneValue = -this._controller.tune(currentTimeOffset, timeDelta, currentFrameRate);
+            tuneValue = -this._controller.tune(currentTimeOffset, intervalTimeDelta, estimatedIntervalFrameRate);
             tuneValue = tuneValue > 0 ? Math.floor(tuneValue) : Math.ceil(tuneValue);
         }
 
-        var currentComplexity = this._stage.tune(tuneValue);
-        this.record(currentTimeOffset, currentComplexity, currentFrameRate);
-        return true;
-    },
-
-    record: function(currentTimeOffset, currentComplexity, currentFrameRate)
-    {
-        this._sampler.sample(currentTimeOffset, [currentComplexity, currentFrameRate]);
+        this._stage.tune(tuneValue);
 
         if (typeof this._recordTimeOffset == "undefined")
             this._recordTimeOffset = currentTimeOffset;
 
         var stage = this._state.currentStage();
         if (stage != BenchmarkState.stages.FINISHED && currentTimeOffset < this._recordTimeOffset + this._recordInterval)
-            return;
+            return true;
 
         this._recordTimeOffset = currentTimeOffset;
+        return true;
+    },
+
+    record: function(currentTimeOffset, estimatedFrameRate, intervalFrameRate)
+    {
+        // If the frame rate is -1 it means we are still recording for this sample
+        this._sampler.record(currentTimeOffset, this.stage.complexity(), estimatedFrameRate, intervalFrameRate);
     },
 
     run: function()
@@ -357,5 +362,57 @@ Benchmark.prototype =
 
         resolveWhenFinished();
         return promise;
+    },
+
+    processSamples: function(results)
+    {
+        var complexity = new Experiment;
+        var smoothedFPS = new Experiment;
+        var samplingIndex = 0;
+
+        var samplingMark = this._sampler.marks[Strings.json.samplingTimeOffset];
+        if (samplingMark) {
+            samplingIndex = samplingMark.index;
+            results[Strings.json.samplingTimeOffset] = samplingMark.time;
+        }
+
+        results[Strings.json.samples] = this._sampler.samples[0].map(function(d, i) {
+            var result = {
+                // time offsets represented as seconds
+                time: d/1000,
+                complexity: this._sampler.samples[1][i]
+            };
+
+            // time offsets represented as FPS
+            if (i == 0)
+                result.fps = 60;
+            else
+                result.fps = 1000 / (d - this._sampler.samples[0][i - 1]);
+
+            var smoothedFPSresult = this._sampler.samples[2][i];
+            if (smoothedFPSresult != -1) {
+                result.smoothedFPS = smoothedFPSresult;
+                result.intervalFPS = this._sampler.samples[3][i];
+            }
+
+            if (i >= samplingIndex) {
+                complexity.sample(result.complexity);
+                if (smoothedFPSresult != -1) {
+                    smoothedFPS.sample(smoothedFPSresult);
+                }
+            }
+
+            return result;
+        }, this);
+
+        results[Strings.json.score] = complexity.score(Experiment.defaults.CONCERN);
+        [complexity, smoothedFPS].forEach(function(experiment, index) {
+            var jsonExperiment = !index ? Strings.json.experiments.complexity : Strings.json.experiments.frameRate;
+            results[jsonExperiment] = {};
+            results[jsonExperiment][Strings.json.measurements.average] = experiment.mean();
+            results[jsonExperiment][Strings.json.measurements.concern] = experiment.concern(Experiment.defaults.CONCERN);
+            results[jsonExperiment][Strings.json.measurements.stdev] = experiment.standardDeviation();
+            results[jsonExperiment][Strings.json.measurements.percent] = experiment.percentage();
+        });
     }
 };
index 14982dd..677a20b 100644 (file)
@@ -30,7 +30,7 @@ var Matrix =
             out += A[i];
             if (i < n * m - 1)
                 out += ", ";
-        }       
+        }
         return out + "]";
     },
 
@@ -193,7 +193,7 @@ function PIDController(ysp)
 
     this._Kp = 0;
     this._stage = PIDController.stages.WARMING;
-    
+
     this._eold = 0;
     this._I = 0;
 }
@@ -208,12 +208,12 @@ PIDController.yPositions = {
 // The Ziegler–Nichols method for is used tuning the PID controller. The workflow of
 // the tuning is split into four stages. The first two stages determine the values
 // of the PID controller gains. During these two stages we return the proportional
-// term only. The third stage is used to determine the min-max values of the 
+// term only. The third stage is used to determine the min-max values of the
 // saturation actuator. In the last stage back-calculation and tracking are applied
 // to avoid integrator windup. During the last two stages, we return a PID control
 // value.
 PIDController.stages = {
-    WARMING: 0,         // Increase the value of the Kp until the system output reaches ysp. 
+    WARMING: 0,         // Increase the value of the Kp until the system output reaches ysp.
     OVERSHOOT: 1,       // Measure the oscillation period and the overshoot value
     UNDERSHOOT: 2,      // Return PID value and measure the undershoot value
     SATURATE: 3         // Return PID value and apply back-calculation and tracking.
@@ -237,7 +237,7 @@ PIDController.prototype =
     // proportional gain very small but achieves the desired progress. But if y does
     // not change significantly after adding few items, that means we need a much
     // bigger gain. So we need to move over a cubic curve which increases very
-    // slowly with small t values but moves very fast with larger t values. 
+    // slowly with small t values but moves very fast with larger t values.
     // The basic formula is: y = t^3
     // Change the formula to reach y=1 after 1000 ms: y = (t/1000)^3
     // Change the formula to reach y=(ysp - y0) after 1000 ms: y = (ysp - y0) * (t/1000)^3
@@ -257,7 +257,7 @@ PIDController.prototype =
 
     // Decides how much the proportional gain should be increased during the manual
     // gain stage. We choose to use the ratio of the ultimate distance to the current
-    // distance as an indication of how much the system is responsive. We want 
+    // distance as an indication of how much the system is responsive. We want
     // to keep the increment under control so it does not cause the system instability
     // So we choose to take the natural logarithm of this ratio.
     _gainIncrement: function(t, y, e)
@@ -277,7 +277,7 @@ PIDController.prototype =
             if (yPosition == PIDController.yPositions.AFTER_SETPOINT)
                 this._stage = PIDController.stages.OVERSHOOT;
             break;
-        
+
         case PIDController.stages.OVERSHOOT:
             if (yPosition == PIDController.yPositions.BEFORE_SETPOINT)
                 this._stage = PIDController.stages.UNDERSHOOT;
@@ -313,7 +313,7 @@ PIDController.prototype =
         // The ouput is a PID function.
        return P + this._I + D;
     },
-    
+
     // Apply different strategies for the tuning based on the stage of the controller.
     _tune: function(t, h, y, e)
     {
@@ -332,7 +332,7 @@ PIDController.prototype =
                 // set-point yet
                 this._Kp += this._gainIncrement(t, y, e);
             }
-        
+
             return this._tuneP(e);
 
         case PIDController.stages.OVERSHOOT:
@@ -343,41 +343,41 @@ PIDController.prototype =
                 this._t0 = t;
                 this._Kp /= 2;
             }
-        
+
             return this._tuneP(e);
-    
+
         case PIDController.stages.UNDERSHOOT:
             // This is the end of the Zieglerâ\80Nichols method. We need to calculate the
             // integral and derivative periods.
             if (typeof this._Ti == "undefined") {
                 // t is the time of the end of the first overshot
                 var Tu = t - this._t0;
-        
+
                 // Calculate the system parameters from Kp and Tu assuming
                 // a "some overshoot" control type. See:
                 // https://en.wikipedia.org/wiki/Ziegler%E2%80%93Nichols_method
                 this._Ti = Tu / 2;
                 this._Td = Tu / 3;
                 this._Kp = 0.33 * this._Kp;
-        
+
                 // Calculate the tracking time.
                 this._Tt = Math.sqrt(this._Ti * this._Td);
             }
-        
+
             return this._tunePID(h, y, e);
-        
+
         case PIDController.stages.SATURATE:
             return this._tunePID(h, y, e);
         }
-        
+
         return 0;
     },
-    
+
     // Ensures the system does not fluctuates.
     _saturate: function(v, e)
     {
         var u = v;
-        
+
         switch (this._stage) {
         case PIDController.stages.OVERSHOOT:
         case PIDController.stages.UNDERSHOOT:
@@ -389,7 +389,7 @@ PIDController.prototype =
                 this._max = Math.max(this._max, this._out);
             }
             break;
-        
+
         case PIDController.stages.SATURATE:
             const limitPercentage = 0.90;
             var min = this._min > 0 ? Math.min(this._min, this._max * limitPercentage) : this._min;
@@ -399,13 +399,13 @@ PIDController.prototype =
             // Clip the controller output to the min-max values
             out = Math.max(Math.min(max, out), min);
             u = out - this._out;
-    
+
             // Apply the back-calculation and tracking
             if (u != v)
                 u += (this._Kp * this._Tt / this._Ti) * e;
             break;
         }
-        
+
         this._out += u;
         return u;
     },
@@ -415,14 +415,14 @@ PIDController.prototype =
     tune: function(t, h, y)
     {
         this._updateStage(y);
-        
+
         // Current error.
         var e = this._ysp - y;
         var v = this._tune(t, h, y, e);
-        
+
         // Save e for the next call.
         this._eold = e;
-        
+
         // Apply back-calculation and tracking to avoid integrator windup
         return this._saturate(v, e);
     }
@@ -480,3 +480,12 @@ KalmanEstimator.prototype =
         return Vector3.multiplyVector3(this._vecH,  this._vecX_est);
     }
 }
+
+function IdentityEstimator() {}
+IdentityEstimator.prototype =
+{
+    estimate: function(current)
+    {
+        return current;
+    }
+};
index 138b046..27d0141 100644 (file)
@@ -113,6 +113,11 @@ LayeringTextStage = Utilities.createSubclass(Stage,
             this._popTextElement();
 
         return this._textElements.length;
+    },
+
+    complexity: function()
+    {
+        return this._textElements.length;
     }
 });
 
index 9c8f5dd..9568cbe 100644 (file)
@@ -1,3 +1,130 @@
+2016-01-03  Jon Lee  <jonlee@apple.com>
+
+        Update data reporting and analysis
+        https://bugs.webkit.org/show_bug.cgi?id=152670
+
+        Reviewed by Simon Fraser.
+
+        Show new graph data. Provide controls to show different series data. Provide an
+        interactive cursor that shows the data at a given sample.
+
+        * Animometer/developer.html: Add a nav section in #results. Each part of the graph
+        has a checkbox for visual toggling, as well as companion spans to contain the data.
+                The numbers will always be shown even if the SVG isn't.
+        * Animometer/resources/debug-runner/animometer.css:
+        (#suites): Adjust spacing when doing fixed complexity.
+        (#test-graph nav): Place the nav in the upper right corner.
+        (#test-graph-data > svg): Fix the FPS scale from 0-60. It makes the raw FPS goes past
+        that scale. Allow it to show.
+        (.target-fps): Add a dotted line for where the benchmark is supposed to settle for FPS.
+        (#cursor line): The cursor contains a line and highlight circles for the data being shown.
+        (#cursor circle):
+        (#complexity path): This and rules afterward are named by series type.
+        (#complexity circle):
+        (#filteredFPS path):
+        (#filteredFPS circle):
+        (#rawFPS path):
+        (#intervalFPS circle):
+        (.left-samples): Deleted.
+        (.right-samples): Deleted.
+        * Animometer/resources/debug-runner/animometer.js:
+        (initialize): Add a "changed" listener when the checkboxes change in the nav.
+        (onBenchmarkOptionsChanged): Renamed.
+        (showTestGraph): All graph data is passed in as graphData instead of as arguments.
+        * Animometer/resources/debug-runner/graph.js: Extend BenchmarkController. When showing
+        a new graph, call updateGraphData(). It creates all of the d3 graphs. onGraphOptionsChanged()
+        toggles the data on and off.
+        (updateGraphData): Add the axes. Add the average lines and markers for sample time and
+        target FPS. Add the cursor group. Use helper function addData() to add the data. On top of
+        everything add a transparent area which will catch all of the mouse events. When the mouse
+        moves in the graph, find the closest data point, show the data in the nav area, and highlight
+        the data points.
+        (addData): Adds a line and circle for each data point. Also adds a highlight cursor with a
+        size a little larger than the circle radius for the data points.
+        (onGraphOptionsChanged): Called when data is visually toggled.
+        (showOrHideNodes): Helper function to toggle the .hidden class.
+        * Animometer/resources/extensions.js:
+        (ResultsDashboard.prototype.get data): Get rid of the arguments for _processData.
+        (ResultsTable.prototype._addGraphButton): Shove all of the graph data into a singular object.
+
+        Producing the JSON can take a while with all of the data. Make it on-demand with a
+        button.
+
+        * Animometer/resources/debug-runner/animometer.js:
+        (showResults): When showing the results, don't serialize the JSON data. Move that to...
+        (showJSONResults): ...here. Remove the button.
+
+        * Animometer/developer.html: Add a button. The button will remove itself and populate
+        the textarea with the JSON data.
+        * Animometer/resources/debug-runner/animometer.css:
+        (.hidden): Add a universal hidden class.
+        (#results button.small-button): Promote the small-button styles to the whole results
+        section for use in the JSON button.
+        (#results button.small-button:active):
+        (#results-data button.small-button): Deleted.
+        (#results-data button.small-button:active): Deleted.
+
+        Refactor how Animator does its recording.
+
+        * Animometer/tests/resources/math.js: Create a new, simple estimator that just returns
+        the same interval frame rate for adjustment.
+        * Animometer/tests/resources/main.js:
+        (Animator): Remove _dropFrameCount, and make variables more accurate described.
+        (Animator.prototype.initialize): Use the identity estimator instead of using a bool.
+        (Animator.prototype._intervalTimeDelta): Rename, only used internally.
+        (Animator.prototype._shouldRequestAnotherFrame): Assume we drop one frame for adjustment
+        of the scene. If we are within the number of frames to measure for the interval, just
+        record the timestamp. Otherwise we are ready to evaluate and adjust the scene. Record
+        the interval frame rate and the estimator's frame rate.
+
+        Avoid processing the data through the Experiment while the test is running. Reconfigure
+        the sampler to just record the raw samples. After the test is done, run the samples through
+        the Experiment to get the score.
+
+        * Animometer/resources/sampler.js:
+        (Experiment): Fold _init() into the constructor since nobody else will call it. This is not
+        needed until the test concludes, so remove startSampling(). Clients should just call sample().
+        (Sampler): Pre-allocate arrays given the number of data points being recorded, and a capacity
+        of how many samples will be used. The processor is a called when it's time to process the data
+        since that is the client also telling the Sampler what to record.
+                Introduce the notion of marks as well, which allows the client to mark when an
+        event occurs. When we mark sample start, we can attach the timestamp there, instead of storing
+        it separately.
+        (Sampler.prototype.startSampling): Deleted. Clients should just call record().
+        (Sampler.prototype.record): The data to record is passed in as variable arguments.
+        (Sampler.prototype.mark): When adding a mark, a client needs to provide a unique string, and
+        can provide extra data object for later retrieval.
+        (Sampler.prototype.process): Renamed from toJSON. Trim the sampling arrays to what was used.
+        Call the processor to process the samples.
+        * Animometer/resources/debug-runner/benchmark-runner.js:
+        (BenchmarkRunner.prototype._runBenchmarkAndRecordResults): Call process().
+
+        * Animometer/resources/strings.js: Add some new strings, remove the graph ones since they are
+        not used.
+        * Animometer/tests/resources/main.js:
+        (Benchmark): Create a sampler with 4 series. The maximum number of points expected is the
+        number of seconds multiplied by 60 fps. Benchmark, as a client of the Sampler, knows about all
+        of the data being added to the Sampler. It is added through record(), and processed through
+        processSamples().
+        (Benchmark.prototype.update): Mark when we've passed warmup and are starting to sample. Include
+        the timestamp in the custom data for the mark. This avoids the need to store is separately in
+        the Sampler. Fold what was in record() here, since nothing else needs this functionality.
+        record() now just relays the information to the sampler.
+        (Benchmark.prototype.record): Called by Animator, which provides the data to the sampler.
+        Animator's calls to this is part of a later patch. Requires each stage to return its complexity.
+        (Benchmark.prototype.processSamples): If the sampling mark exists, add it to the results.
+                Go through all of the samples. All samples contain a timestamp and complexity. We
+        calculate "raw FPS" which is the time differential from the previous sample. At regular intervals
+        the Kalman-filtered FPS and the interval average FPS are also recorded. We also create two
+        experiments, to get the scores for the complexity and smoothed FPS, and add those samples to
+        the experiments. Grab those scores and add them into results also.
+
+        Add complexity() to the tests for Benchmark.record().
+        * Animometer/tests/bouncing-particles/resources/bouncing-particles.js:
+        * Animometer/tests/misc/resources/canvas-electrons.js:
+        * Animometer/tests/misc/resources/canvas-stars.js:
+        * Animometer/tests/text/resources/layering-text.js:
+
 2015-12-27  Jon Lee  <jonlee@apple.com>
 
         Simplify the test harness