Add a ramp controller
authorjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 10 Feb 2016 19:22:08 +0000 (19:22 +0000)
committerjonlee@apple.com <jonlee@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 10 Feb 2016 19:22:08 +0000 (19:22 +0000)
https://bugs.webkit.org/show_bug.cgi?id=154028

Provisionally reviewed by Said Abou-Hallawa.

Enhance the graph to include a complexity-fps graph, in addition
to the time graph.

* Animometer/developer.html: Add a ramp option.
* Animometer/resources/debug-runner/animometer.css: Update the style.
* Animometer/resources/strings.js: Flatten the Strings.text constants.
* Animometer/resources/debug-runner/animometer.js:
(ResultsTable.call._addGraphButton): Refactor.
(ResultsTable.call._addTest): Add regression data.
(benchmarkController): Add a form that allows the user to switch between the two forms,
Add a form that allows the user to toggle different data. Hide certain header columns
depending on the selected controller.
* Animometer/resources/debug-runner/graph.js: Add the complexity regressions.
* Animometer/resources/debug-runner/tests.js: Add headers for the ramp results.
* Animometer/resources/runner/animometer.js:
(ResultsTable): If a header is disabled don't include them in _flattenedHeaders.
* Animometer/tests/resources/main.js:
(Controller): Allow options to specify the capacity for sample arrays.
(Regression): A piecewise regression that tries to fit a slope and a flat profile.
(_calculateRegression): Options can fix the slope and bias when calculating the minimal
error. Sweep across the samples in time (which could be backward depending on the controller)
and calculate the intersection point.
(RampController): This controller assumes that the target frame rate is below
58 FPS. It runs in two stages. The first stage quickly determines the order of
magnitude of objects needed to stress the system by the setting the complexity
to increasingly difficult tiers. Perform a series of ramps descending from a
high-water mark of complexity. The complexity needed to reach the target frame
length is done by performing a piecewise regression on each ramp, and track a
running average of these values. For the next ramp, make that running average
the center of the ramp. With a minimum complexity of 0, the high-water mark is
twice that average. The score is based on the highest complexity that can
reach 60 fps.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@196372 268f45cc-cd09-0410-ab3c-d52691b4dbfc

PerformanceTests/Animometer/developer.html
PerformanceTests/Animometer/resources/debug-runner/animometer.css
PerformanceTests/Animometer/resources/debug-runner/animometer.js
PerformanceTests/Animometer/resources/debug-runner/graph.js
PerformanceTests/Animometer/resources/debug-runner/tests.js
PerformanceTests/Animometer/resources/runner/animometer.js
PerformanceTests/Animometer/resources/strings.js
PerformanceTests/Animometer/tests/resources/main.js
PerformanceTests/ChangeLog

index c2cece2..e2ad5f3 100644 (file)
@@ -46,6 +46,7 @@
                         <ul>
                             <li><label><input name="adjustment" type="radio" value="step"> Keep at a fixed complexity, then make a big step</label></li>
                             <li><label><input name="adjustment" type="radio" value="adaptive" checked> Maintain target FPS</label></li>
+                            <li><label><input name="adjustment" type="radio" value="ramp"> Ramp</label></li>
                         </ul>
                     </li>
                     <li>
                 <h1>Graph:</h1>
             </header>
             <nav>
+                <form name="graph-type">
+                    <ul>
+                        <li><label><input type="radio" name="graph-type" value="time" checked> Time graph</label></li>
+                        <li><label><input type="radio" name="graph-type" value="complexity"> Complexity graph</label></li>
+                    </ul>
+                </form>
                 <form name="time-graph-options">
                     <ul>
                         <li><label><input type="checkbox" name="markers" checked> Markers</label>
                             <span class="rawFPS"></span></li>
                         <li><label><input type="checkbox" name="filteredFPS" checked> Filtered FPS</label>
                             <span class="filteredFPS"></span></li>
+                        <li><label><input type="checkbox" name="regressions" checked> Regressions</label>
+                    </ul>
+                </form>
+                <form name="complexity-graph-options">
+                    <ul class="series">
+                        <li><label><input type="checkbox" name="series-raw" checked> Series raw</label></li>
+                        <li><label><input type="checkbox" name="series-average" checked> Series average</label></li>
+                    </ul>
+                    <ul>
+                        <li><label><input type="checkbox" name="regression-time-score" checked> Ramp regression score</label>
+                        <li><label><input type="checkbox" name="complexity-regression-aggregate-raw" checked> Regression, series raw</label>
+                        <li><label><input type="checkbox" name="complexity-regression-aggregate-average" checked> Regression, series average</label>
                     </ul>
                 </form>
             </nav>
index 6b81ad8..d847ec8 100644 (file)
@@ -356,20 +356,16 @@ label.tree-label {
     position: absolute;
     top: 1.5em;
     right: 0;
-    font-size: .8em;
-    width: 25%;
+    font-size: .7em;
+    width: 23em;
 }
 
 #test-graph nav ul {
-    margin: 0 30px 0 0;
+    margin: 0 30px 1em 0;
     padding: 0;
     list-style: none;
 }
 
-#test-graph nav ul ul {
-    padding-left: 2em;
-}
-
 #test-graph nav li {
     padding: .1em 0;
 }
@@ -403,6 +399,36 @@ label.tree-label {
     shape-rendering: crispEdges;
 }
 
+.axis text {
+    fill: #999;
+}
+
+.yLeft.axis text {
+    fill: #7add49;
+}
+.yLeft.axis path,
+.yLeft.axis line {
+    stroke: #7add49;
+}
+.yRight.axis text {
+    fill: #fa4925;
+}
+.yRight.axis path,
+.yRight.axis line {
+    stroke: #fa4925;
+}
+
+.axis.complexity .tick line {
+    stroke: rgba(200, 200, 200, .6);
+    stroke-width: 2px;
+}
+
+.axis.complexity .domain,
+.axis.complexity text {
+    stroke: transparent;
+    fill: transparent;
+}
+
 .marker line {
     stroke: #5493D6;
 }
@@ -417,7 +443,7 @@ label.tree-label {
 }
 
 .mean.complexity polygon {
-    fill: hsla(100, 69%, 58%, .1);
+    fill: hsla(100, 69%, 58%, .05);
 }
 
 .target-fps {
@@ -435,6 +461,15 @@ label.tree-label {
     fill: hsla(10, 96%, 56%, .1);
 }
 
+#regressions line {
+    stroke: rgba(200, 200, 200, .8);
+    stroke-width: 2px;
+}
+
+#regressions circle {
+    fill: rgba(200, 200, 200, .8);
+}
+
 .cursor line {
     stroke: rgb(250, 250, 250);
     stroke-width: 1px;
@@ -471,3 +506,25 @@ label.tree-label {
 #rawFPS circle {
     fill: rgb(250, 73, 37);
 }
+
+#complexity-graph .regression line {
+    stroke: rgba(253, 253, 253, .8);
+    stroke-width: 2px;
+}
+
+#complexity-graph .regression circle {
+    fill: rgba(253, 253, 253, .8);
+}
+
+#complexity-graph .regression polygon {
+    fill: rgba(253, 253, 253, .05);
+}
+
+#complexity-graph .series.average circle {
+    fill: hsl(170, 96%, 56%);
+}
+
+#complexity-graph .series.average line {
+    stroke: hsla(170, 96%, 56%, .2);
+    stroke-width: 2px;
+}
index e2c62af..e7635e2 100644 (file)
@@ -35,7 +35,7 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
 
         button.addEventListener("click", function() {
             var graphData = {
-                axes: [Strings.text.experiments.complexity, Strings.text.experiments.frameRate],
+                axes: [Strings.text.complexity, Strings.text.frameRate],
                 samples: data,
                 complexityAverageSamples: testResults[Strings.json.complexityAverageSamples],
                 averages: {},
@@ -48,6 +48,9 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
 
             [
                 Strings.json.score,
+                Strings.json.regressions.timeRegressions,
+                Strings.json.regressions.complexityRegression,
+                Strings.json.regressions.complexityAverageRegression,
                 Strings.json.targetFrameLength
             ].forEach(function(key) {
                 if (testResults[key])
@@ -57,7 +60,7 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
             benchmarkController.showTestGraph(testName, graphData);
         });
 
-        button.textContent = Strings.text.results.graph + "...";
+        button.textContent = Strings.text.graph + "...";
     },
 
     _isNoisyMeasurement: function(jsonExperiment, data, measurement, options)
@@ -105,7 +108,7 @@ DeveloperResultsTable = Utilities.createSubclass(ResultsTable,
             }
 
             var td = Utilities.createElement("td", { class: className }, row);
-            if (header.title == Strings.text.results.graph) {
+            if (header.title == Strings.text.graph) {
                 this._addGraphButton(td, testName, testResults);
             } else if (!("text" in header)) {
                 td.textContent = testResults[header.title];
@@ -451,7 +454,9 @@ Utilities.extendObject(window.benchmarkController, {
     initialize: function()
     {
         document.forms["benchmark-options"].addEventListener("change", benchmarkController.onBenchmarkOptionsChanged, true);
+        document.forms["graph-type"].addEventListener("change", benchmarkController.onGraphTypeChanged, true);
         document.forms["time-graph-options"].addEventListener("change", benchmarkController.onTimeGraphOptionsChanged, true);
+        document.forms["complexity-graph-options"].addEventListener("change", benchmarkController.onComplexityGraphOptionsChanged, true);
         optionsManager.updateUIFromLocalStorage();
         suitesManager.createElements();
         suitesManager.updateUIFromLocalStorage();
@@ -474,6 +479,12 @@ Utilities.extendObject(window.benchmarkController, {
     {
         var options = optionsManager.updateLocalStorageFromUI();
         var suites = suitesManager.updateLocalStorageFromUI();
+        if (options["adjustment"] == "ramp") {
+            Headers.details[2].disabled = true;
+        } else {
+            Headers.details[3].disabled = true;
+            Headers.details[4].disabled = true;
+        }
         this._startBenchmark(suites, options, "running-test");
     },
 
index b386eed..7049463 100644 (file)
@@ -14,6 +14,14 @@ Utilities.extendObject(window.benchmarkController, {
         this.createTimeGraph(graphData, margins, size);
         this.onTimeGraphOptionsChanged();
 
+        var hasComplexityRegression = !!graphData.complexityRegression;
+        this._showOrHideNodes(hasComplexityRegression, "form[name=graph-type]");
+        if (hasComplexityRegression) {
+            document.forms["graph-type"].elements["type"] = "complexity";
+            this.createComplexityGraph(graphData, margins, size);
+            this.onComplexityGraphOptionsChanged();
+        }
+
         this.onGraphTypeChanged();
     },
 
@@ -41,6 +49,165 @@ Utilities.extendObject(window.benchmarkController, {
             .attr("y2", line[3]);
     },
 
+    _addRegression: function(data, svg, xScale, yScale)
+    {
+        svg.append("circle")
+            .attr("cx", xScale(data.segment2[1][0]))
+            .attr("cy", yScale(data.segment2[1][1]))
+            .attr("r", 5);
+        this._addRegressionLine(svg, xScale, yScale, data.segment1, data.stdev);
+        this._addRegressionLine(svg, xScale, yScale, data.segment2, data.stdev);
+    },
+
+    createComplexityGraph: function(graphData, margins, size)
+    {
+        var svg = d3.select("#test-graph-data").append("svg")
+            .attr("id", "complexity-graph")
+            .attr("class", "hidden")
+            .attr("width", size.width + margins.left + margins.right)
+            .attr("height", size.height + margins.top + margins.bottom)
+            .append("g")
+                .attr("transform", "translate(" + margins.left + "," + margins.top + ")");
+
+        var xMin = 100000, xMax = 0;
+        if (graphData.timeRegressions) {
+            graphData.timeRegressions.forEach(function(regression) {
+                for (var i = regression.startIndex; i <= regression.endIndex; ++i) {
+                    xMin = Math.min(xMin, graphData.samples[i].complexity);
+                    xMax = Math.max(xMax, graphData.samples[i].complexity);
+                }
+            });
+        } else {
+            xMin = d3.min(graphData.samples, function(s) { return s.complexity; });
+            xMax = d3.max(graphData.samples, function(s) { return s.complexity; });
+        }
+
+        var xScale = d3.scale.linear()
+            .range([0, size.width])
+            .domain([xMin, xMax]);
+        var yScale = d3.scale.linear()
+                .range([size.height, 0])
+                .domain([1000/20, 1000/60]);
+
+        var xAxis = d3.svg.axis()
+                .scale(xScale)
+                .orient("bottom");
+        var yAxis = d3.svg.axis()
+                .scale(yScale)
+                .tickValues([1000/20, 1000/25, 1000/30, 1000/35, 1000/40, 1000/45, 1000/50, 1000/55, 1000/60])
+                .tickFormat(function(d) { return (1000 / d).toFixed(0); })
+                .orient("left");
+
+        // x-axis
+        svg.append("g")
+            .attr("class", "x axis")
+            .attr("transform", "translate(0," + size.height + ")")
+            .call(xAxis);
+
+        // y-axis
+        svg.append("g")
+            .attr("class", "y axis")
+            .call(yAxis);
+
+        // time-based regression
+        var mean = svg.append("g")
+            .attr("class", "mean complexity");
+        var complexity = graphData.averages[Strings.json.experiments.complexity];
+        this._addRegressionLine(mean, xScale, yScale, [[complexity.average, yScale.domain()[0]], [complexity.average, yScale.domain()[1]]], complexity.stdev, true);
+
+        // regression
+        this._addRegression(graphData.complexityRegression, svg.append("g").attr("class", "regression raw"), xScale, yScale);
+        this._addRegression(graphData.complexityAverageRegression, svg.append("g").attr("class", "regression average"), xScale, yScale);
+
+        var svgGroup = svg.append("g")
+            .attr("class", "series raw");
+        var seriesCounter = 0;
+        graphData.timeRegressions.forEach(function(regression, i) {
+            seriesCounter++;
+            var group = svgGroup.append("g")
+                .attr("class", "series-" + seriesCounter)
+                .attr("fill", "hsl(" + (i / graphData.timeRegressions.length * 360).toFixed(0) + ", 96%, 56%)");
+            group.selectAll("circle")
+                .data(graphData.samples)
+                .enter()
+                .append("circle")
+                .filter(function(d, i) { return i >= regression.startIndex && i <= regression.endIndex; })
+                .attr("cx", function(d) { return xScale(d.complexity); })
+                .attr("cy", function(d) { return yScale(d.frameLength); })
+                .attr("r", 2);
+        });
+
+        group = svg.append("g")
+            .attr("class", "series average")
+            .selectAll("circle")
+                .data(graphData.complexityAverageSamples)
+                .enter();
+        group.append("circle")
+            .attr("cx", function(d) { return xScale(d.complexity); })
+            .attr("cy", function(d) { return yScale(d.frameLength); })
+            .attr("r", 3)
+        group.append("line")
+            .attr("x1", function(d) { return xScale(d.complexity); })
+            .attr("x2", function(d) { return xScale(d.complexity); })
+            .attr("y1", function(d) { return yScale(d.frameLength - d.stdev); })
+            .attr("y2", function(d) { return yScale(d.frameLength + d.stdev); });
+
+        // Cursor
+        var cursorGroup = svg.append("g").attr("class", "cursor hidden");
+        cursorGroup.append("line")
+            .attr("class", "x")
+            .attr("x1", 0)
+            .attr("x2", 0)
+            .attr("y1", yScale(yAxis.scale().domain()[0]) + 10)
+            .attr("y2", yScale(yAxis.scale().domain()[1]));
+        cursorGroup.append("line")
+            .attr("class", "y")
+            .attr("x1", xScale(0) - 10)
+            .attr("x2", xScale(xAxis.scale().domain()[1]))
+            .attr("y1", 0)
+            .attr("y2", 0)
+        cursorGroup.append("text")
+            .attr("class", "label x")
+            .attr("x", 0)
+            .attr("y", yScale(yAxis.scale().domain()[0]) + 15)
+            .attr("baseline-shift", "-100%")
+            .attr("text-anchor", "middle");
+        cursorGroup.append("text")
+            .attr("class", "label y")
+            .attr("x", xScale(0) - 15)
+            .attr("y", 0)
+            .attr("baseline-shift", "-30%")
+            .attr("text-anchor", "end");
+        // Area to handle mouse events
+        var area = svg.append("rect")
+            .attr("fill", "transparent")
+            .attr("x", 0)
+            .attr("y", 0)
+            .attr("width", size.width)
+            .attr("height", size.height);
+
+        area.on("mouseover", function() {
+            document.querySelector("#complexity-graph .cursor").classList.remove("hidden");
+        }).on("mouseout", function() {
+            document.querySelector("#complexity-graph .cursor").classList.add("hidden");
+        }).on("mousemove", function() {
+            var location = d3.mouse(this);
+            var location_domain = [xScale.invert(location[0]), yScale.invert(location[1])];
+            cursorGroup.select("line.x")
+                .attr("x1", location[0])
+                .attr("x2", location[0]);
+            cursorGroup.select("text.x")
+                .attr("x", location[0])
+                .text(location_domain[0].toFixed(1));
+            cursorGroup.select("line.y")
+                .attr("y1", location[1])
+                .attr("y2", location[1]);
+            cursorGroup.select("text.y")
+                .attr("y", location[1])
+                .text((1000 / location_domain[1]).toFixed(1));
+        });
+    },
+
     createTimeGraph: function(graphData, margins, size)
     {
         var svg = d3.select("#test-graph-data").append("svg")
@@ -60,6 +227,11 @@ Utilities.extendObject(window.benchmarkController, {
                     Math.min(d3.min(graphData.samples, function(s) { return s.time; }), 0),
                     d3.max(graphData.samples, function(s) { return s.time; })]);
         var complexityMax = d3.max(graphData.samples, function(s) { return s.complexity; });
+        if (graphData.timeRegressions) {
+            complexityMax = Math.max.apply(Math, graphData.timeRegressions.map(function(regression) {
+                return regression.maxComplexity || 0;
+            }));
+        }
 
         var yLeft = d3.scale.linear()
                 .range([size.height, 0])
@@ -98,7 +270,7 @@ Utilities.extendObject(window.benchmarkController, {
 
         // yLeft-axis
         svg.append("g")
-            .attr("class", "y axis")
+            .attr("class", "yLeft axis")
             .attr("fill", "#7ADD49")
             .call(yAxisLeft)
             .append("text")
@@ -112,17 +284,17 @@ Utilities.extendObject(window.benchmarkController, {
 
         // yRight-axis
         svg.append("g")
-            .attr("class", "y axis")
+            .attr("class", "yRight axis")
             .attr("fill", "#FA4925")
             .attr("transform", "translate(" + size.width + ", 0)")
             .call(yAxisRight)
             .append("text")
                 .attr("class", "label")
-                .attr("transform", "rotate(-90)")
-                .attr("y", 6)
+                .attr("x", 9)
+                .attr("y", -20)
                 .attr("fill", "#FA4925")
                 .attr("dy", ".71em")
-                .style("text-anchor", "end")
+                .style("text-anchor", "start")
                 .text(axes[1]);
 
         // marks
@@ -209,6 +381,41 @@ Utilities.extendObject(window.benchmarkController, {
         addData("rawFPS", allData, function(d) { return yRight(d.frameLength); }, 1);
         addData("filteredFPS", filteredData, function(d) { return yRight(d.smoothedFrameLength); }, 2);
 
+        // regressions
+        var regressionGroup = svg.append("g")
+            .attr("id", "regressions");
+        if (graphData.timeRegressions) {
+            var complexities = [];
+            graphData.timeRegressions.forEach(function (regression) {
+                regressionGroup.append("line")
+                    .attr("x1", x(regression.segment1[0][0]))
+                    .attr("x2", x(regression.segment1[1][0]))
+                    .attr("y1", yRight(regression.segment1[0][1]))
+                    .attr("y2", yRight(regression.segment1[1][1]));
+                regressionGroup.append("line")
+                    .attr("x1", x(regression.segment2[0][0]))
+                    .attr("x2", x(regression.segment2[1][0]))
+                    .attr("y1", yRight(regression.segment2[0][1]))
+                    .attr("y2", yRight(regression.segment2[1][1]));
+                // inflection point
+                regressionGroup.append("circle")
+                    .attr("cx", x(regression.segment1[1][0]))
+                    .attr("cy", yLeft(regression.complexity))
+                    .attr("r", 5);
+                complexities.push(regression.complexity);
+            });
+            if (complexities.length) {
+                var yLeftComplexities = d3.svg.axis()
+                    .scale(yLeft)
+                    .tickValues(complexities)
+                    .tickSize(10)
+                    .orient("left");
+                svg.append("g")
+                    .attr("class", "complexity yLeft axis")
+                    .call(yLeftComplexities);
+            }
+        }
+
         // Area to handle mouse events
         var area = svg.append("rect")
             .attr("fill", "transparent")
@@ -291,6 +498,15 @@ Utilities.extendObject(window.benchmarkController, {
         }
     },
 
+    onComplexityGraphOptionsChanged: function() {
+        var form = document.forms["complexity-graph-options"].elements;
+        benchmarkController._showOrHideNodes(form["series-raw"].checked, "#complexity-graph .series.raw");
+        benchmarkController._showOrHideNodes(form["series-average"].checked, "#complexity-graph .series.average");
+        benchmarkController._showOrHideNodes(form["regression-time-score"].checked, "#complexity-graph .mean.complexity");
+        benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-raw"].checked, "#complexity-graph .regression.raw");
+        benchmarkController._showOrHideNodes(form["complexity-regression-aggregate-average"].checked, "#complexity-graph .regression.average");
+    },
+
     onTimeGraphOptionsChanged: function() {
         var form = document.forms["time-graph-options"].elements;
         benchmarkController._showOrHideNodes(form["markers"].checked, ".marker");
@@ -301,11 +517,14 @@ Utilities.extendObject(window.benchmarkController, {
     },
 
     onGraphTypeChanged: function() {
+        var form = document.forms["graph-type"].elements;
         var graphData = document.getElementById("test-graph-data").graphData;
-        var isTimeSelected = true;
+        var isTimeSelected = form["graph-type"].value == "time";
 
         benchmarkController._showOrHideNodes(isTimeSelected, "#time-graph");
         benchmarkController._showOrHideNodes(isTimeSelected, "form[name=time-graph-options]");
+        benchmarkController._showOrHideNodes(!isTimeSelected, "#complexity-graph");
+        benchmarkController._showOrHideNodes(!isTimeSelected, "form[name=complexity-graph-options]");
 
         var score, mean;
         if (isTimeSelected) {
@@ -326,6 +545,19 @@ Utilities.extendObject(window.benchmarkController, {
                     regression.concern.toFixed(2)]);
             }
             mean = mean.join("");
+        } else {
+            score = [
+                "raw: ",
+                graphData.complexityRegression.complexity.toFixed(2),
+                ", average: ",
+                graphData.complexityAverageRegression.complexity.toFixed(2)].join("");
+
+            mean = [
+                "raw: ±",
+                graphData.complexityRegression.stdev.toFixed(2),
+                "ms, average: ±",
+                graphData.complexityAverageRegression.stdev.toFixed(2),
+                "ms"].join("");
         }
 
         sectionsManager.setSectionScore("test-graph", score, mean);
index d362d35..8b8584a 100644 (file)
@@ -1,10 +1,10 @@
 Utilities.extendObject(Headers, {
     details: [
         {
-            title: Strings.text.results.graph
+            title: Strings.text.graph
         },
         {
-            title: Strings.text.experiments.complexity,
+            title: Strings.text.complexity,
             children:
             [
                 {
@@ -32,7 +32,7 @@ Utilities.extendObject(Headers, {
             ]
         },
         {
-            title: Strings.text.experiments.frameRate,
+            title: Strings.text.frameRate,
             children:
             [
                 {
@@ -65,6 +65,50 @@ Utilities.extendObject(Headers, {
                 }
             ]
         },
+        {
+            title: Strings.text.mergedRawComplexity,
+            children:
+            [
+                {
+                    text: function(data) {
+                        return data[Strings.json.regressions.complexityRegression][Strings.json.regressions.complexity].toFixed(2);
+                    },
+                    className: "average"
+                },
+                {
+                    text: function(data) {
+                        return [
+                            "± ",
+                            data[Strings.json.regressions.complexityRegression][Strings.json.measurements.stdev].toFixed(2),
+                            "ms"
+                        ].join("");
+                    },
+                    className: "stdev"
+                }
+            ]
+        },
+        {
+            title: Strings.text.mergedAverageComplexity,
+            children:
+            [
+                {
+                    text: function(data) {
+                        return data[Strings.json.regressions.complexityAverageRegression][Strings.json.regressions.complexity].toFixed(2);
+                    },
+                    className: "average"
+                },
+                {
+                    text: function(data) {
+                        return [
+                            "± ",
+                            data[Strings.json.regressions.complexityAverageRegression][Strings.json.measurements.stdev].toFixed(2),
+                            "ms"
+                        ].join("");
+                    },
+                    className: "stdev"
+                }
+            ]
+        },
     ]
 })
 
index c15d0af..06e43b4 100644 (file)
@@ -70,12 +70,19 @@ ResultsTable = Utilities.createClass(
 
         this._flattenedHeaders = [];
         this._headers.forEach(function(header) {
+            if (header.disabled)
+                return;
+
             if (header.children)
                 this._flattenedHeaders = this._flattenedHeaders.concat(header.children);
             else
                 this._flattenedHeaders.push(header);
         }, this);
 
+        this._flattenedHeaders = this._flattenedHeaders.filter(function (header) {
+            return !header.disabled;
+        });
+
         this.clear();
     }, {
 
@@ -90,8 +97,11 @@ ResultsTable = Utilities.createClass(
         var row = Utilities.createElement("tr", {}, thead);
 
         this._headers.forEach(function (header) {
+            if (header.disabled)
+                return;
+
             var th = Utilities.createElement("th", {}, row);
-            if (header.title != Strings.text.results.graph)
+            if (header.title != Strings.text.graph)
                 th.textContent = header.title;
             if (header.children)
                 th.colSpan = header.children.length;
index f9ebf67..cc03bf0 100644 (file)
@@ -4,27 +4,16 @@ var Strings = {
         score: "Score",
         samples: "Samples",
 
-        experiments: {
-            complexity: "Complexity",
-            frameRate: "FPS"
-        },
-
-        measurements: {
-            average: "Avg.",
-            concern: "W.5%",
-            stdev: "Std.",
-            percent:  "%"
-        },
-
-        results: {
-            results: "Results",
-            graph: "Graph",
-            json: "JSON"
-        }
+        complexity: "Complexity",
+        frameRate: "FPS",
+        mergedRawComplexity: "Merged raw",
+        mergedAverageComplexity: "Merged average",
+        graph: "Graph"
     },
     json: {
         score: "score",
         samples: "samples",
+        complexityAverageSamples: "complexityAverageSamples",
         marks: "marks",
 
         targetFrameLength: "targetFrameLength",
@@ -36,6 +25,19 @@ var Strings = {
             frameRate: "frameRate"
         },
 
+        regressions: {
+            timeRegressions: "timeRegressions",
+            complexity: "complexity",
+            maxComplexity: "maxComplexity",
+            startIndex: "startIndex",
+            endIndex: "endIndex",
+
+            complexityRegression: "complexityRegression",
+            complexityAverageRegression: "complexityAverageRegression",
+            segment1: "segment1",
+            segment2: "segment2"
+        },
+
         measurements: {
             average: "average",
             concern: "concern",
index 056ff7b..d5f7568 100644 (file)
@@ -6,7 +6,8 @@ Controller = Utilities.createClass(
         this._startTimestamp = 0;
         this._endTimestamp = options["test-interval"];
         // Default data series: timestamp, complexity, estimatedFrameLength
-        this._sampler = new Sampler(options["series-count"] || 3, (60 * options["test-interval"] / 1000), this);
+        var sampleSize = options["sample-capacity"] || (60 * options["test-interval"] / 1000);
+        this._sampler = new Sampler(options["series-count"] || 3, sampleSize, this);
         this._marks = {};
 
         this._frameLengthEstimator = new SimpleKalmanEstimator(options["kalman-process-error"], options["kalman-measurement-error"]);
@@ -258,6 +259,421 @@ AdaptiveController = Utilities.createSubclass(Controller,
     }
 });
 
+Regression = Utilities.createClass(
+    function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+    {
+        var slope = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+            shouldClip: true,
+            s1: 1000/60,
+            t1: 0
+        });
+        var flat = this._calculateRegression(samples, getComplexity, getFrameLength, startIndex, endIndex, {
+            shouldClip: true,
+            t1: 0,
+            t2: 0
+        });
+        var desired;
+        if (slope.error < flat.error)
+            desired = slope;
+        else
+            desired = flat;
+
+        this.startIndex = Math.min(startIndex, endIndex);
+        this.endIndex = Math.max(startIndex, endIndex);
+
+        this.complexity = desired.complexity;
+        this.s1 = desired.s1;
+        this.t1 = desired.t1;
+        this.s2 = desired.s2;
+        this.t2 = desired.t2;
+        this.error = desired.error;
+    }, {
+
+    // A generic two-segment piecewise regression calculator. Based on Kundu/Ubhaya
+    //
+    // Minimize sum of (y - y')^2
+    // where                        y = s1 + t1*x
+    //                              y = s2 + t2*x
+    //                y' = s1 + t1*x' = s2 + t2*x'   if x_0 <= x' <= x_n
+    //
+    // Allows for fixing s1, t1, s2, t2
+    //
+    // x is assumed to be complexity, y is frame length. Can be used for pure complexity-FPS
+    // analysis or for ramp controllers since complexity monotonically decreases with time.
+    _calculateRegression: function(samples, getComplexity, getFrameLength, startIndex, endIndex, options)
+    {
+        var iterationDirection = endIndex > startIndex ? 1 : -1;
+        var lowComplexity = getComplexity(samples, startIndex);
+        var highComplexity = getComplexity(samples, endIndex);
+        var a1 = 0, b1 = 0, c1 = 0, d1 = 0, h1 = 0, k1 = 0;
+        var a2 = 0, b2 = 0, c2 = 0, d2 = 0, h2 = 0, k2 = 0;
+
+        // Iterate from low to high complexity
+        for (var i = startIndex; iterationDirection * (endIndex - i) > -1; i += iterationDirection) {
+            var x = getComplexity(samples, i);
+            var y = getFrameLength(samples, i);
+            a2 += 1;
+            b2 += x;
+            c2 += x * x;
+            d2 += y;
+            h2 += y * x;
+            k2 += y * y;
+        }
+
+        var s1_best, t1_best, s2_best, t2_best, x_best, error_best, x_prime;
+
+        function setBest(s1, t1, s2, t2, error, x_prime, x)
+        {
+            s1_best = s1;
+            t1_best = t1;
+            s2_best = s2;
+            t2_best = t2;
+            error_best = error;
+            if (!options.shouldClip || (x_prime >= lowComplexity && x_prime <= highComplexity))
+                x_best = x_prime;
+            else {
+                // Discontinuous piecewise regression
+                x_best = x;
+            }
+        }
+
+        // Iterate from startIndex to endIndex - 1, inclusive
+        for (var i = startIndex; iterationDirection * (endIndex - i) > 0; i += iterationDirection) {
+            var x = getComplexity(samples, i);
+            var y = getFrameLength(samples, i);
+            var xx = x * x;
+            var yx = y * x;
+            var yy = y * y;
+            // a1, b1, etc. is sum from startIndex to i, inclusive
+            a1 += 1;
+            b1 += x;
+            c1 += xx;
+            d1 += y;
+            h1 += yx;
+            k1 += yy;
+            // a2, b2, etc. is sum from i+1 to endIndex, inclusive
+            a2 -= 1;
+            b2 -= x;
+            c2 -= xx;
+            d2 -= y;
+            h2 -= yx;
+            k2 -= yy;
+
+            var A = c1*d1 - b1*h1;
+            var B = a1*h1 - b1*d1;
+            var C = a1*c1 - b1*b1;
+            var D = c2*d2 - b2*h2;
+            var E = a2*h2 - b2*d2;
+            var F = a2*c2 - b2*b2;
+            var s1 = options.s1 !== undefined ? options.s1 : (A / C);
+            var t1 = options.t1 !== undefined ? options.t1 : (B / C);
+            var s2 = options.s2 !== undefined ? options.s2 : (D / F);
+            var t2 = options.t2 !== undefined ? options.t2 : (E / F);
+            // Assumes that the two segments meet
+            var x_prime = (s1 - s2) / (t2 - t1);
+
+            var error1 = (k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) || 0;
+            var error2 = (k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) || 0;
+
+            if (i == startIndex) {
+                setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
+                continue;
+            }
+
+            // Projected point is not between this and the next sample
+            if (x_prime > getComplexity(samples, i + iterationDirection) || x_prime < x) {
+                // Calculate lambda, which divides the weight of this sample between the two lines
+
+                // These values remove the influence of this sample
+                var I = c1 - 2*b1*x + a1*xx;
+                var H = C - I;
+                var G = A + B*x - C*y;
+
+                var J = D + E*x - F*y;
+                var K = c2 - 2*b2*x + a2*xx;
+
+                var lambda = (G*F + G*K - H*J) / (I*J + G*K);
+                if (lambda > 0 && lambda < 1) {
+                    var lambda1 = 1 - lambda;
+                    s1 = options.s1 !== undefined ? options.s1 : ((A - lambda1*(-h1*x + d1*xx + c1*y - b1*yx)) / (C - lambda1*I));
+                    t1 = options.t1 !== undefined ? options.t1 : ((B - lambda1*(h1 - d1*x - b1*y + a1*yx)) / (C - lambda1*I));
+                    s2 = options.s2 !== undefined ? options.s2 : ((D + lambda1*(-h2*x + d2*xx + c2*y - b2*yx)) / (F + lambda1*K));
+                    t2 = options.t2 !== undefined ? options.t2 : ((E + lambda1*(h2 - d2*x - b2*y + a2*yx)) / (F + lambda1*K));
+                    x_prime = (s1 - s2) / (t2 - t1);
+
+                    error1 = ((k1 + a1*s1*s1 + c1*t1*t1 - 2*d1*s1 - 2*h1*t1 + 2*b1*s1*t1) - lambda1 * Math.pow(y - (s1 + t1*x), 2)) || 0;
+                    error2 = ((k2 + a2*s2*s2 + c2*t2*t2 - 2*d2*s2 - 2*h2*t2 + 2*b2*s2*t2) + lambda1 * Math.pow(y - (s2 + t2*x), 2)) || 0;
+                }
+            }
+
+            if (error1 + error2 < error_best)
+                setBest(s1, t1, s2, t2, error1 + error2, x_prime, x);
+        }
+
+        return {
+            complexity: x_best,
+            s1: s1_best,
+            t1: t1_best,
+            s2: s2_best,
+            t2: t2_best,
+            error: error_best
+        };
+    }
+});
+
+RampController = Utilities.createSubclass(Controller,
+    function(benchmark, options)
+    {
+        // The tier warmup takes at most 5 seconds
+        options["sample-capacity"] = (options["test-interval"] / 1000 + 5) * 60;
+        Controller.call(this, benchmark, options);
+
+        // Initially start with a tier test to find the bounds
+        // The number of objects in a tier test is 10^|_tier|
+        this._tier = 0;
+        // The timestamp is first set after the first interval completes
+        this._tierStartTimestamp = 0;
+        // If the engine can handle the tier's complexity at 60 FPS, test for a short
+        // period, then move on to the next tier
+        this._tierFastTestLength = 250;
+        // If the engine is under stress, let the test run a little longer to let
+        // the measurement settle
+        this._tierSlowTestLength = 750;
+        this._maximumComplexity = 0;
+        this._minimumTier = 0;
+
+        // After the tier range is determined, figure out the number of ramp iterations
+        var minimumRampLength = 3000;
+        var totalRampIterations = Math.max(1, Math.floor(this._endTimestamp / minimumRampLength));
+        // Give a little extra room to run since the ramps won't be exactly this length
+        this._rampLength = Math.floor((this._endTimestamp - totalRampIterations * this._intervalLength) / totalRampIterations);
+        this._rampWarmupLength = 200;
+        this._rampDidWarmup = false;
+        this._rampRegressions = [];
+
+        // Add some tolerance; frame lengths shorter than this are considered to be @ 60 fps
+        this._fps60Threshold = 1000/58;
+        // We are looking for the complexity that will get us at least as slow this threshold
+        this._fpsLowestThreshold = 1000/30;
+
+        this._finishedTierSampling = false;
+        this._startedRamps = false;
+        this._complexityPrime = new Experiment;
+    }, {
+
+    start: function(startTimestamp, stage)
+    {
+        Controller.prototype.start.call(this, startTimestamp, stage);
+        this._rampStartTimestamp = 0;
+    },
+
+    didFinishInterval: function(timestamp, stage, intervalAverageFrameLength)
+    {
+        if (!this._finishedTierSampling) {
+            if (this._tierStartTimestamp > 0 && timestamp < this._tierStartTimestamp + this._tierFastTestLength)
+                return;
+
+            var currentComplexity = stage.complexity();
+            var currentFrameLength = this._frameLengthEstimator.estimate;
+            if (currentFrameLength < this._fpsLowestThreshold) {
+                var isAnimatingAt60FPS = currentFrameLength < this._fps60Threshold;
+                var hasFinishedSlowTierTest = timestamp > this._tierStartTimestamp + this._tierSlowTestLength;
+
+                // We're measuring at 60 fps, so quickly move on to the next tier, or
+                // we've slower than 60 fps, but we've let this tier run long enough to
+                // get an estimate
+                if (currentFrameLength < this._fps60Threshold || timestamp > this._tierStartTimestamp + this._tierSlowTestLength) {
+                    this._lastComplexity = currentComplexity;
+                    this._lastFrameLength = currentFrameLength;
+
+                    this._tierStartTimestamp = timestamp;
+                    this._tier += .5;
+                    var nextTierComplexity = Math.round(Math.pow(10, this._tier));
+                    this.mark("Complexity: " + nextTierComplexity, timestamp);
+
+                    stage.tune(nextTierComplexity - currentComplexity);
+                }
+                return;
+            } else if (timestamp < this._tierStartTimestamp + this._tierSlowTestLength)
+                return;
+
+            this._finishedTierSampling = true;
+            // Extend the test length so that the full test length is made of the ramps
+            this._endTimestamp += timestamp;
+            this.mark(Strings.json.samplingStartTimeOffset, timestamp);
+
+            // Sometimes this last tier will drop the frame length well below the threshold
+            // Avoid going down that far since it means fewer measurements are taken in the 60 fps area
+            // Interpolate a maximum complexity that gets us around the lowest threshold
+            this._maximumComplexity = Math.floor(this._lastComplexity + (this._fpsLowestThreshold - this._lastFrameLength) / (currentFrameLength - this._lastFrameLength) * (currentComplexity - this._lastComplexity));
+            stage.tune(this._maximumComplexity - currentComplexity);
+            this._rampStartTimestamp = timestamp;
+            this._rampDidWarmup = false;
+            this.isFrameLengthEstimatorEnabled = false;
+            this._intervalCount = 0;
+            return;
+        }
+
+        if ((timestamp - this._rampStartTimestamp) < this._rampWarmupLength)
+            return;
+
+        if (this._rampDidWarmup)
+            return;
+
+        this._rampDidWarmup = true;
+        this._currentRampLength = this._rampStartTimestamp + this._rampLength - timestamp;
+        // Start timestamp represents start of ramp down, after warm up
+        this._rampStartTimestamp = timestamp;
+        this._rampStartIndex = this._sampler.sampleCount;
+    },
+
+    tune: function(timestamp, stage, didFinishInterval)
+    {
+        if (!didFinishInterval || !this._rampDidWarmup)
+            return;
+
+        var progress = (timestamp - this._rampStartTimestamp) / this._currentRampLength;
+
+        if (progress < 1) {
+            stage.tune(Math.round((1 - progress) * this._maximumComplexity) - stage.complexity());
+            return;
+        }
+
+        var regression = new Regression(this._sampler.samples, this._getComplexity, this._getFrameLength,
+            this._sampler.sampleCount - 1, this._rampStartIndex);
+        this._rampRegressions.push(regression);
+
+        this._complexityPrime.sample(regression.complexity);
+        this._maximumComplexity = Math.max(5, Math.round(this._complexityPrime.mean() * 2));
+
+        // Next ramp
+        this._rampDidWarmup = false;
+        // Start timestamp represents start of ramp iteration and warm up
+        this._rampStartTimestamp = timestamp;
+        stage.tune(this._maximumComplexity - stage.complexity());
+    },
+
+    _getComplexity: function(samples, i) {
+        return samples[1][i];
+    },
+
+    _getFrameLength: function(samples, i) {
+        return samples[0][i] - samples[0][i - 1];
+    },
+
+    processSamples: function(results)
+    {
+        Controller.prototype.processSamples.call(this, results);
+
+        // Have samplingTimeOffset represent time 0
+        var startTimestamp = this._marks[Strings.json.samplingStartTimeOffset].time;
+        results[Strings.json.samples].forEach(function(sample) {
+            sample.time -= startTimestamp;
+        });
+        for (var markName in results[Strings.json.marks]) {
+            results[Strings.json.marks][markName].time -= startTimestamp;
+        }
+
+        var samples = results[Strings.json.samples];
+        results[Strings.json.regressions.timeRegressions] = [];
+        var complexityRegressionSamples = [];
+        var timeComplexityScore = new Experiment;
+        this._rampRegressions.forEach(function(ramp) {
+            var startIndex = ramp.startIndex, endIndex = ramp.endIndex;
+            var startTime = samples[startIndex].time, endTime = samples[endIndex].time;
+            var startComplexity = samples[startIndex].complexity, endComplexity = samples[endIndex].complexity;
+
+            timeComplexityScore.sample(ramp.complexity);
+
+            var regression = {};
+            results[Strings.json.regressions.timeRegressions].push(regression);
+
+            var percentage = (ramp.complexity - startComplexity) / (endComplexity - startComplexity);
+            var inflectionTime = startTime + percentage * (endTime - startTime);
+
+            regression[Strings.json.regressions.segment1] = [
+                [startTime, ramp.s2 + ramp.t2 * startComplexity],
+                [inflectionTime, ramp.s2 + ramp.t2 * ramp.complexity]
+            ];
+            regression[Strings.json.regressions.segment2] = [
+                [inflectionTime, ramp.s1 + ramp.t1 * ramp.complexity],
+                [endTime, ramp.s1 + ramp.t1 * endComplexity]
+            ];
+            regression[Strings.json.regressions.complexity] = ramp.complexity;
+            regression[Strings.json.regressions.maxComplexity] = Math.max(startComplexity, endComplexity);
+            regression[Strings.json.regressions.startIndex] = startIndex;
+            regression[Strings.json.regressions.endIndex] = endIndex;
+
+            for (var j = startIndex; j <= endIndex; ++j)
+                complexityRegressionSamples.push(samples[j]);
+        });
+
+        // Aggregate all of the ramps into one big dataset and calculate a regression from this
+        complexityRegressionSamples.sort(function(a, b) {
+            return a.complexity - b.complexity;
+        });
+
+        // Samples averaged based on complexity
+        results[Strings.json.complexityAverageSamples] = [];
+        var currentComplexity = -1;
+        var experimentAtComplexity;
+        function addSample() {
+            results[Strings.json.complexityAverageSamples].push({
+                complexity: currentComplexity,
+                frameLength: experimentAtComplexity.mean(),
+                stdev: experimentAtComplexity.standardDeviation(),
+            });
+        }
+        complexityRegressionSamples.forEach(function(sample) {
+            if (sample.complexity != currentComplexity) {
+                if (currentComplexity > -1)
+                    addSample();
+
+                currentComplexity = sample.complexity;
+                experimentAtComplexity = new Experiment;
+            }
+            experimentAtComplexity.sample(sample.frameLength);
+        });
+        // Finish off the last one
+        addSample();
+
+        function calculateRegression(samples, key) {
+            var complexityRegression = new Regression(
+                samples,
+                function (samples, i) { return samples[i].complexity; },
+                function (samples, i) { return samples[i].frameLength; },
+                0, samples.length - 1
+            );
+            var minComplexity = samples[0].complexity;
+            var maxComplexity = samples[samples.length - 1].complexity;
+            var regression = {};
+            results[key] = regression;
+            regression[Strings.json.regressions.segment1] = [
+                [minComplexity, complexityRegression.s1 + complexityRegression.t1 * minComplexity],
+                [complexityRegression.complexity, complexityRegression.s1 + complexityRegression.t1 * complexityRegression.complexity]
+            ];
+            regression[Strings.json.regressions.segment2] = [
+                [complexityRegression.complexity, complexityRegression.s2 + complexityRegression.t2 * complexityRegression.complexity],
+                [maxComplexity, complexityRegression.s2 + complexityRegression.t2 * maxComplexity]
+            ];
+            regression[Strings.json.regressions.complexity] = complexityRegression.complexity;
+            regression[Strings.json.measurements.stdev] = Math.sqrt(complexityRegression.error / samples.length);
+        }
+
+        calculateRegression(complexityRegressionSamples, Strings.json.regressions.complexityRegression);
+        calculateRegression(results[Strings.json.complexityAverageSamples], Strings.json.regressions.complexityAverageRegression);
+
+        // Frame rate experiment result is unneeded
+        delete results[Strings.json.experiments.frameRate];
+
+        results[Strings.json.score] = timeComplexityScore.mean();
+        results[Strings.json.experiments.complexity] = {};
+        results[Strings.json.experiments.complexity][Strings.json.measurements.average] = timeComplexityScore.mean();
+        results[Strings.json.experiments.complexity][Strings.json.measurements.stdev] = timeComplexityScore.standardDeviation();
+        results[Strings.json.experiments.complexity][Strings.json.measurements.percent] = timeComplexityScore.percentage();
+    }
+});
+
 Stage = Utilities.createClass(
     function()
     {
@@ -435,9 +851,11 @@ Benchmark = Utilities.createClass(
             this._controller = new StepController(this, options);
             break;
         case "adaptive":
-        default:
             this._controller = new AdaptiveController(this, options);
             break;
+        case "ramp":
+            this._controller = new RampController(this, options);
+            break;
         }
     }, {
 
index 10724fd..c81bc4c 100644 (file)
@@ -1,5 +1,45 @@
 2016-02-08  Jon Lee  <jonlee@apple.com>
 
+        Add a ramp controller
+        https://bugs.webkit.org/show_bug.cgi?id=154028
+
+        Provisionally reviewed by Said Abou-Hallawa.
+
+        Enhance the graph to include a complexity-fps graph, in addition
+        to the time graph.
+
+        * Animometer/developer.html: Add a ramp option.
+        * Animometer/resources/debug-runner/animometer.css: Update the style.
+        * Animometer/resources/strings.js: Flatten the Strings.text constants.
+        * Animometer/resources/debug-runner/animometer.js:
+        (ResultsTable.call._addGraphButton): Refactor.
+        (ResultsTable.call._addTest): Add regression data.
+        (benchmarkController): Add a form that allows the user to switch between the two forms,
+        Add a form that allows the user to toggle different data. Hide certain header columns
+        depending on the selected controller.
+        * Animometer/resources/debug-runner/graph.js: Add the complexity regressions.
+        * Animometer/resources/debug-runner/tests.js: Add headers for the ramp results.
+        * Animometer/resources/runner/animometer.js:
+        (ResultsTable): If a header is disabled don't include them in _flattenedHeaders.
+        * Animometer/tests/resources/main.js:
+        (Controller): Allow options to specify the capacity for sample arrays.
+        (Regression): A piecewise regression that tries to fit a slope and a flat profile.
+        (_calculateRegression): Options can fix the slope and bias when calculating the minimal
+        error. Sweep across the samples in time (which could be backward depending on the controller)
+        and calculate the intersection point.
+        (RampController): This controller assumes that the target frame rate is below
+        58 FPS. It runs in two stages. The first stage quickly determines the order of
+        magnitude of objects needed to stress the system by the setting the complexity
+        to increasingly difficult tiers. Perform a series of ramps descending from a
+        high-water mark of complexity. The complexity needed to reach the target frame
+        length is done by performing a piecewise regression on each ramp, and track a
+        running average of these values. For the next ramp, make that running average
+        the center of the ramp. With a minimum complexity of 0, the high-water mark is
+        twice that average. The score is based on the highest complexity that can
+        reach 60 fps.
+
+2016-02-08  Jon Lee  <jonlee@apple.com>
+
         Address Said's comments on the benchmark, and do some clean up.
 
         * Animometer/developer.html: