Relationship between A/B testing results are unclear
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 19 Feb 2015 22:18:17 +0000 (22:18 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 19 Feb 2015 22:18:17 +0000 (22:18 +0000)
https://bugs.webkit.org/show_bug.cgi?id=141810

Reviewed by Andreas Kling.

Show a "reference chart" indicating which two points have been tested in each test group pane.

Now the chart shown at the top of an analysis task page is called the "overview pane", and we use the pane
and the domain used in this chart to show charts in each test group.

Also renamed an array of revisions used in the A/B test results tables from 'revisions' to 'revisionList'.

* public/v2/analysis.js:
(App.TestGroup._fetchTestResults): Renamed from _fetchChartData. Set 'testResults' instead of 'chartData'
since this is the results of A/B testing results, not the data for charts shown next to them.

* public/v2/app.css: Added CSS rules for reference charts.

* public/v2/app.js:
(App.AnalysisTaskController.paneDomain): Set 'overviewPane' and 'overviewDomain' on each test group pane.
(App.TestGroupPane._populate): Updated per 'chartData' to 'testResults' rename.
(App.TestGroupPane._updateReferenceChart): Get the chart data via the overview pane and find points that
identically matches root sets. If one of configuration used a set of revisions for which no measurement
was made in the original chart, don't show the reference chart as that would be misleading / confusing.
(App.TestGroupPane._computeRepositoryList): Updated per 'chartData' to 'testResults' rename.
(App.TestGroupPane._createConfigurationSummary): Ditto. Also renamed 'revisions' to 'revisionList'.
In addition, renamed 'buildNumber' to 'buildLabel' and prefixed it with "Build ".

* public/v2/data.js:
(Measurement.prototype.revisionForRepository): Added.
(Measurement.prototype.commitTimeForRepository): Cleanup.
(TimeSeries.prototype.findPointByRevisions): Added. Finds a point based on a set of revisions.

* public/v2/index.html: Added the reference chart. Streamlined the status label for each build request
by including the build number in the title attribute instead of in the markup.

* public/v2/interactive-chart.js:
(App.InteractiveChartComponent._updateDomain): Fixed a typo introduced as a consequence of r179913.
(App.InteractiveChartComponent._computeYAxisDomain): Expand the y-axis to show the highlighted points.
(App.InteractiveChartComponent._highlightedItemsChanged): Adjust the y-axis as needed.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@180365 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/v2/analysis.js
Websites/perf.webkit.org/public/v2/app.css
Websites/perf.webkit.org/public/v2/app.js
Websites/perf.webkit.org/public/v2/data.js
Websites/perf.webkit.org/public/v2/index.html
Websites/perf.webkit.org/public/v2/interactive-chart.js

index 9e1c6c7..dbc9248 100644 (file)
@@ -1,3 +1,46 @@
+2015-02-19  Ryosuke Niwa  <rniwa@webkit.org>
+
+        Relationship between A/B testing results are unclear
+        https://bugs.webkit.org/show_bug.cgi?id=141810
+
+        Reviewed by Andreas Kling.
+
+        Show a "reference chart" indicating which two points have been tested in each test group pane.
+
+        Now the chart shown at the top of an analysis task page is called the "overview pane", and we use the pane
+        and the domain used in this chart to show charts in each test group.
+
+        Also renamed an array of revisions used in the A/B test results tables from 'revisions' to 'revisionList'.
+
+        * public/v2/analysis.js:
+        (App.TestGroup._fetchTestResults): Renamed from _fetchChartData. Set 'testResults' instead of 'chartData'
+        since this is the results of A/B testing results, not the data for charts shown next to them.
+
+        * public/v2/app.css: Added CSS rules for reference charts.
+
+        * public/v2/app.js:
+        (App.AnalysisTaskController.paneDomain): Set 'overviewPane' and 'overviewDomain' on each test group pane.
+        (App.TestGroupPane._populate): Updated per 'chartData' to 'testResults' rename.
+        (App.TestGroupPane._updateReferenceChart): Get the chart data via the overview pane and find points that
+        identically matches root sets. If one of configuration used a set of revisions for which no measurement
+        was made in the original chart, don't show the reference chart as that would be misleading / confusing.
+        (App.TestGroupPane._computeRepositoryList): Updated per 'chartData' to 'testResults' rename.
+        (App.TestGroupPane._createConfigurationSummary): Ditto. Also renamed 'revisions' to 'revisionList'.
+        In addition, renamed 'buildNumber' to 'buildLabel' and prefixed it with "Build ".
+
+        * public/v2/data.js:
+        (Measurement.prototype.revisionForRepository): Added.
+        (Measurement.prototype.commitTimeForRepository): Cleanup.
+        (TimeSeries.prototype.findPointByRevisions): Added. Finds a point based on a set of revisions.
+
+        * public/v2/index.html: Added the reference chart. Streamlined the status label for each build request
+        by including the build number in the title attribute instead of in the markup.
+
+        * public/v2/interactive-chart.js:
+        (App.InteractiveChartComponent._updateDomain): Fixed a typo introduced as a consequence of r179913.
+        (App.InteractiveChartComponent._computeYAxisDomain): Expand the y-axis to show the highlighted points.
+        (App.InteractiveChartComponent._highlightedItemsChanged): Adjust the y-axis as needed.
+
 2015-02-18  Ryosuke Niwa  <rniwa@webkit.org>
 
         Analysis task pages are unusable
index 98bc74a..443cb96 100644 (file)
@@ -87,7 +87,7 @@ App.TestGroup = App.NameLabelModel.extend({
     createdAt: DS.attr('date'),
     buildRequests: DS.hasMany('buildRequests'),
     rootSets: DS.hasMany('rootSets'),
-    _fetchChartData: function ()
+    _fetchTestResults: function ()
     {
         var task = this.get('task');
         if (!task)
@@ -95,7 +95,7 @@ App.TestGroup = App.NameLabelModel.extend({
         var self = this;
         return App.Manifest.fetchRunsWithPlatformAndMetric(this.store,
             task.get('platform').get('id'), task.get('metric').get('id'), this.get('id')).then(
-            function (result) { self.set('chartData', result.data); },
+            function (result) { self.set('testResults', result.data); },
             function (error) {
                 // FIXME: Somehow this never gets called.
                 alert('Failed to fetch the results:' + error);
index b7eca0a..d7ee9f7 100755 (executable)
@@ -472,6 +472,29 @@ table.dashboard tbody td .failure {
     display: none;
 }
 
+.analysis-group {
+    display: block;
+    position: relative;
+    min-height: 6.5rem;
+}
+
+.analysis-group .results {
+    margin-right: 20rem;
+}
+
+.analysis-group .reference-chart {
+    position: absolute;
+    top: 1.2rem;
+    right: -0.7rem;
+    width: 19rem;
+    height: 5rem;
+}
+
+.analysis-group .reference-chart .chart {
+    width: 100%;
+    height: 100%;
+}
+
 .box-plot {
     display: inline-block;
     width: 100px;
index 50c5c18..640ac69 100755 (executable)
@@ -1048,8 +1048,16 @@ App.AnalysisTaskController = Ember.Controller.extend({
         this.set('highlightedItems', highlightedItems);
         this.set('analysisPoints', formatedPoints);
 
-        return [start.time - margin, +end.time + margin];
-    }.property('pane.chartData', 'model', 'model'),
+        var paneDomain = [start.time - margin, +end.time + margin];
+
+        var testGroupPanes = this.get('testGroupPanes');
+        if (testGroupPanes) {
+            testGroupPanes.setEach('overviewPane', pane);
+            testGroupPanes.setEach('overviewDomain', paneDomain);
+        }
+
+        return paneDomain;
+    }.property('pane.chartData'),
     testSets: function ()
     {
         var analysisPoints = this.get('analysisPoints');
@@ -1166,8 +1174,8 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
     _populate: function ()
     {
         var buildRequests = this.get('buildRequests');
-        var chartData = this.get('chartData');
-        if (!buildRequests || !chartData)
+        var testResults = this.get('testResults');
+        if (!buildRequests || !testResults)
             return [];
 
         var repositories = this._computeRepositoryList();
@@ -1188,7 +1196,41 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
         range.min -= margin;
 
         this.set('configurations', configurations);
-    }.observes('chartData', 'buildRequests'),
+    }.observes('testResults', 'buildRequests'),
+    _updateReferenceChart: function ()
+    {
+        var configurations = this.get('configurations');
+        var chartData = this.get('overviewPane') ? this.get('overviewPane').get('chartData') : null;
+        if (!configurations || !chartData || this.get('referenceChart'))
+            return;
+
+        var currentTimeSeries = chartData.current;
+        if (!currentTimeSeries)
+            return;
+
+        var repositories = this.get('repositories');
+        var highlightedItems = {};
+        var failedToFindPoint = false;
+        configurations.forEach(function (config) {
+            var revisions = {};
+            config.get('rootSet').get('roots').forEach(function (root) {
+                revisions[root.get('repository').get('id')] = root.get('revision');
+            });
+            var point = currentTimeSeries.findPointByRevisions(revisions);
+            if (!point) {
+                failedToFindPoint = true;
+                return;
+            }
+            highlightedItems[point.measurement.id()] = true;
+        });
+        if (failedToFindPoint)
+            return;
+
+        this.set('referenceChart', {
+            data: chartData,
+            highlightedItems: highlightedItems,
+        });
+    }.observes('configurations', 'overviewPane.chartData'),
     _computeRepositoryList: function ()
     {
         var specifiedRepositories = new Ember.Set();
@@ -1198,9 +1240,9 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
             });
         });
         var reportedRepositories = new Ember.Set();
-        var chartData = this.get('chartData');
+        var testResults = this.get('testResults');
         (this.get('buildRequests') || []).forEach(function (request) {
-            var point = chartData.current.findPointByBuild(request.get('build'));
+            var point = testResults.current.findPointByBuild(request.get('build'));
             if (!point)
                 return;
 
@@ -1228,19 +1270,19 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
     _createConfigurationSummary: function (buildRequests, configLetter, range)
     {
         var repositories = this.get('repositories');
-        var chartData = this.get('chartData');
+        var testResults = this.get('testResults');
         var requests = buildRequests.map(function (originalRequest) {
-            var point = chartData.current.findPointByBuild(originalRequest.get('build'));
+            var point = testResults.current.findPointByBuild(originalRequest.get('build'));
             var revisionByRepositoryId = point ? point.measurement.formattedRevisions() : {};
             return Ember.ObjectProxy.create({
                 content: originalRequest,
-                revisions: repositories.map(function (repository, index) {
+                revisionList: repositories.map(function (repository, index) {
                     return (revisionByRepositoryId[repository.get('id')] || {label:null}).label;
                 }),
                 value: point ? point.value : null,
                 valueRange: range,
-                formattedValue: point ? chartData.formatWithUnit(point.value) : null,
-                buildNumber: point ? point.measurement.buildNumber() : null,
+                formattedValue: point ? testResults.formatWithUnit(point.value) : null,
+                buildLabel: point ? 'Build ' + point.measurement.buildNumber() : null,
             });
         });
 
@@ -1248,15 +1290,15 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
         var summaryRevisions = repositories.map(function (repository, index) {
             var revision = rootSet ? rootSet.revisionForRepository(repository) : null;
             if (!revision)
-                return requests[0].get('revisions')[index];
+                return requests[0].get('revisionList')[index];
             return Measurement.formatRevisionRange(revision).label;
         });
 
         requests.forEach(function (request) {
-            var revisions = request.get('revisions');
+            var revisionList = request.get('revisionList');
             repositories.forEach(function (repository, index) {
-                if (revisions[index] == summaryRevisions[index])
-                    revisions[index] = null;
+                if (revisionList[index] == summaryRevisions[index])
+                    revisionList[index] = null;
             });
         });
 
@@ -1275,15 +1317,15 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
         var summary = Ember.Object.create({
             isAverage: true,
             configLetter: configLetter,
-            revisions: summaryRevisions,
-            formattedValue: isNaN(mean) ? null : chartData.formatWithDeltaAndUnit(mean, ciDelta),
+            revisionList: summaryRevisions,
+            formattedValue: isNaN(mean) ? null : testResults.formatWithDeltaAndUnit(mean, ciDelta),
             value: mean,
             confidenceIntervalDelta: ciDelta,
             valueRange: range,
             statusLabel: App.BuildRequest.aggregateStatuses(requests),
         });
 
-        return Ember.Object.create({summary: summary, items: requests});
+        return Ember.Object.create({summary: summary, items: requests, rootSet: rootSet});
     },
 });
 
index 94d0fb2..9e52113 100755 (executable)
@@ -152,13 +152,18 @@ function Measurement(rawData)
     this._formattedRevisions = undefined;
 }
 
+Measurement.prototype.revisionForRepository = function (repositoryId)
+{
+    var revisions = this._raw['revisions'];
+    var rawData = revisions[repositoryId];
+    return rawData ? rawData[0] : null;
+}
+
 Measurement.prototype.commitTimeForRepository = function (repositoryId)
 {
     var revisions = this._raw['revisions'];
     var rawData = revisions[repositoryId];
-    if (!rawData)
-        return null;
-    return new Date(rawData[1]);
+    return rawData ? new Date(rawData[1]) : null;
 }
 
 Measurement.prototype.formattedRevisions = function (previousMeasurement)
@@ -368,6 +373,17 @@ TimeSeries.prototype.findPointByBuild = function (buildId)
     return this._series.find(function (point) { return point.measurement.buildId() == buildId; })
 }
 
+TimeSeries.prototype.findPointByRevisions = function (revisions)
+{
+    return this._series.find(function (point, index) {
+        for (var repositoryId in revisions) {
+            if (point.measurement.revisionForRepository(repositoryId) != revisions[repositoryId])
+                return false;
+        }
+        return true;
+    });
+}
+
 TimeSeries.prototype.findPointByMeasurementId = function (measurementId)
 {
     return this._series.find(function (point) { return point.measurement.id() == measurementId; });
index bdfc99b..6c4cadf 100755 (executable)
                     </tbody>
                 {{/each}}
             </table>
+            <div class="reference-chart">
+                {{#if referenceChart}}
+                    {{interactive-chart
+                        chartData=referenceChart.data
+                        domain=overviewDomain
+                        chartPointRadius=2
+                        showYAxis=false
+                        enableSelection=false
+                        highlightedItems=referenceChart.highlightedItems}}
+                {{/if}}
+            </div>
         </section>
     </script>
 
     <script type="text/x-handlebars" data-template-name="testGroupRow">
-        {{#each revisions}}
+        {{#each revisionList}}
             <td>{{this}}</td>
         {{/each}}
         <td>
             {{formattedValue}}
         </td>
         <td>
-            {{#if buildNumber}}
-                 {{statusLabel}} / <a {{bind-attr href=url}}>Build {{buildNumber}}</a>
-            {{else}}
-                <a {{bind-attr href=url}}>{{statusLabel}}</a>
-            {{/if}}
+            <a {{bind-attr href=url title=buildLabel}}>{{statusLabel}}</a>
         </td>
     </script>
 
index 0567680..40ed2d0 100644 (file)
@@ -208,7 +208,7 @@ App.InteractiveChartComponent = Ember.Component.extend({
         var currentYDomain = this._y.domain();
         if (currentXDomain && App.domainsAreEqual(currentXDomain, xDomain)
             && currentYDomain && App.domainsAreEqual(currentYDomain, yDomain))
-            return currentDomain;
+            return currentXDomain;
 
         this._x.domain(xDomain);
         this._y.domain(yDomain);
@@ -343,6 +343,16 @@ App.InteractiveChartComponent = Ember.Component.extend({
         var range = this._minMaxForAllTimeSeries(startTime, endTime, !shouldShowFullYAxis);
         var min = range[0];
         var max = range[1];
+
+        var highlightedItems = this.get('highlightedItems');
+        if (highlightedItems) {
+            var data = this._currentTimeSeriesData
+                .filter(function (point) { return startTime <= point.time && point.time <= endTime && highlightedItems[point.measurement.id()]; })
+                .map(function (point) { return point.value });
+            min = Math.min(min, Statistics.min(data));
+            max = Math.max(max, Statistics.max(data));
+        }
+
         if (max < min)
             min = max = 0;
         else if (shouldShowFullYAxis)
@@ -615,7 +625,7 @@ App.InteractiveChartComponent = Ember.Component.extend({
                 .attr("class", "highlight")
                 .attr("r", (this.get('chartPointRadius') || 1) * 1.8);
 
-        this._updateHighlightPositions();
+        this._domainChanged();
     }.observes('highlightedItems'),
     _rangesChanged: function ()
     {