Relationship between A/B testing results are unclear
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 19 Feb 2015 22:18:17 +0000 (22:18 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 19 Feb 2015 22:18:17 +0000 (22:18 +0000)
https://bugs.webkit.org/show_bug.cgi?id=141810

Reviewed by Andreas Kling.

Show a "reference chart" indicating which two points have been tested in each test group pane.

Now the chart shown at the top of an analysis task page is called the "overview pane", and we use the pane
and the domain used in this chart to show charts in each test group.

Also renamed an array of revisions used in the A/B test results tables from 'revisions' to 'revisionList'.

* public/v2/analysis.js:
(App.TestGroup._fetchTestResults): Renamed from _fetchChartData. Set 'testResults' instead of 'chartData'
since this is the results of A/B testing results, not the data for charts shown next to them.

* public/v2/app.css: Added CSS rules for reference charts.

* public/v2/app.js:
(App.AnalysisTaskController.paneDomain): Set 'overviewPane' and 'overviewDomain' on each test group pane.
(App.TestGroupPane._populate): Updated per 'chartData' to 'testResults' rename.
(App.TestGroupPane._updateReferenceChart): Get the chart data via the overview pane and find points that
identically matches root sets. If one of configuration used a set of revisions for which no measurement
was made in the original chart, don't show the reference chart as that would be misleading / confusing.
(App.TestGroupPane._computeRepositoryList): Updated per 'chartData' to 'testResults' rename.
(App.TestGroupPane._createConfigurationSummary): Ditto. Also renamed 'revisions' to 'revisionList'.
In addition, renamed 'buildNumber' to 'buildLabel' and prefixed it with "Build ".

* public/v2/data.js:
(Measurement.prototype.revisionForRepository): Added.
(Measurement.prototype.commitTimeForRepository): Cleanup.
(TimeSeries.prototype.findPointByRevisions): Added. Finds a point based on a set of revisions.

* public/v2/index.html: Added the reference chart. Streamlined the status label for each build request
by including the build number in the title attribute instead of in the markup.

* public/v2/interactive-chart.js:
(App.InteractiveChartComponent._updateDomain): Fixed a typo introduced as a consequence of r179913.
(App.InteractiveChartComponent._computeYAxisDomain): Expand the y-axis to show the highlighted points.
(App.InteractiveChartComponent._highlightedItemsChanged): Adjust the y-axis as needed.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@180365 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/v2/analysis.js
Websites/perf.webkit.org/public/v2/app.css
Websites/perf.webkit.org/public/v2/app.js
Websites/perf.webkit.org/public/v2/data.js
Websites/perf.webkit.org/public/v2/index.html
Websites/perf.webkit.org/public/v2/interactive-chart.js

index 9e1c6c766d5bd244e4b4311bcb2cef7340ca5b48..dbc92480f5eb41942b9c76b02e6f9498d5b5ff84 100644 (file)
@@ -1,3 +1,46 @@
+2015-02-19  Ryosuke Niwa  <rniwa@webkit.org>
+
+        Relationship between A/B testing results are unclear
+        https://bugs.webkit.org/show_bug.cgi?id=141810
+
+        Reviewed by Andreas Kling.
+
+        Show a "reference chart" indicating which two points have been tested in each test group pane.
+
+        Now the chart shown at the top of an analysis task page is called the "overview pane", and we use the pane
+        and the domain used in this chart to show charts in each test group.
+
+        Also renamed an array of revisions used in the A/B test results tables from 'revisions' to 'revisionList'.
+
+        * public/v2/analysis.js:
+        (App.TestGroup._fetchTestResults): Renamed from _fetchChartData. Set 'testResults' instead of 'chartData'
+        since this is the results of A/B testing results, not the data for charts shown next to them.
+
+        * public/v2/app.css: Added CSS rules for reference charts.
+
+        * public/v2/app.js:
+        (App.AnalysisTaskController.paneDomain): Set 'overviewPane' and 'overviewDomain' on each test group pane.
+        (App.TestGroupPane._populate): Updated per 'chartData' to 'testResults' rename.
+        (App.TestGroupPane._updateReferenceChart): Get the chart data via the overview pane and find points that
+        identically matches root sets. If one of configuration used a set of revisions for which no measurement
+        was made in the original chart, don't show the reference chart as that would be misleading / confusing.
+        (App.TestGroupPane._computeRepositoryList): Updated per 'chartData' to 'testResults' rename.
+        (App.TestGroupPane._createConfigurationSummary): Ditto. Also renamed 'revisions' to 'revisionList'.
+        In addition, renamed 'buildNumber' to 'buildLabel' and prefixed it with "Build ".
+
+        * public/v2/data.js:
+        (Measurement.prototype.revisionForRepository): Added.
+        (Measurement.prototype.commitTimeForRepository): Cleanup.
+        (TimeSeries.prototype.findPointByRevisions): Added. Finds a point based on a set of revisions.
+
+        * public/v2/index.html: Added the reference chart. Streamlined the status label for each build request
+        by including the build number in the title attribute instead of in the markup.
+
+        * public/v2/interactive-chart.js:
+        (App.InteractiveChartComponent._updateDomain): Fixed a typo introduced as a consequence of r179913.
+        (App.InteractiveChartComponent._computeYAxisDomain): Expand the y-axis to show the highlighted points.
+        (App.InteractiveChartComponent._highlightedItemsChanged): Adjust the y-axis as needed.
+
 2015-02-18  Ryosuke Niwa  <rniwa@webkit.org>
 
         Analysis task pages are unusable
index 98bc74a0255209b8ee913f3ae273b09325f50189..443cb96b154136da7ee0004706fc82dc010dfa3d 100644 (file)
@@ -87,7 +87,7 @@ App.TestGroup = App.NameLabelModel.extend({
     createdAt: DS.attr('date'),
     buildRequests: DS.hasMany('buildRequests'),
     rootSets: DS.hasMany('rootSets'),
-    _fetchChartData: function ()
+    _fetchTestResults: function ()
     {
         var task = this.get('task');
         if (!task)
@@ -95,7 +95,7 @@ App.TestGroup = App.NameLabelModel.extend({
         var self = this;
         return App.Manifest.fetchRunsWithPlatformAndMetric(this.store,
             task.get('platform').get('id'), task.get('metric').get('id'), this.get('id')).then(
-            function (result) { self.set('chartData', result.data); },
+            function (result) { self.set('testResults', result.data); },
             function (error) {
                 // FIXME: Somehow this never gets called.
                 alert('Failed to fetch the results:' + error);
index b7eca0aa7f3f954dfa622c66fd77411b91620415..d7ee9f75bbac72e8731e20ba95917b3890e9477f 100755 (executable)
@@ -472,6 +472,29 @@ table.dashboard tbody td .failure {
     display: none;
 }
 
+.analysis-group {
+    display: block;
+    position: relative;
+    min-height: 6.5rem;
+}
+
+.analysis-group .results {
+    margin-right: 20rem;
+}
+
+.analysis-group .reference-chart {
+    position: absolute;
+    top: 1.2rem;
+    right: -0.7rem;
+    width: 19rem;
+    height: 5rem;
+}
+
+.analysis-group .reference-chart .chart {
+    width: 100%;
+    height: 100%;
+}
+
 .box-plot {
     display: inline-block;
     width: 100px;
index 50c5c1897df9707f2cf87699621c0f063e6eba91..640ac69a7ac37f34c4500ee1ea2421a653e8235f 100755 (executable)
@@ -1048,8 +1048,16 @@ App.AnalysisTaskController = Ember.Controller.extend({
         this.set('highlightedItems', highlightedItems);
         this.set('analysisPoints', formatedPoints);
 
-        return [start.time - margin, +end.time + margin];
-    }.property('pane.chartData', 'model', 'model'),
+        var paneDomain = [start.time - margin, +end.time + margin];
+
+        var testGroupPanes = this.get('testGroupPanes');
+        if (testGroupPanes) {
+            testGroupPanes.setEach('overviewPane', pane);
+            testGroupPanes.setEach('overviewDomain', paneDomain);
+        }
+
+        return paneDomain;
+    }.property('pane.chartData'),
     testSets: function ()
     {
         var analysisPoints = this.get('analysisPoints');
@@ -1166,8 +1174,8 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
     _populate: function ()
     {
         var buildRequests = this.get('buildRequests');
-        var chartData = this.get('chartData');
-        if (!buildRequests || !chartData)
+        var testResults = this.get('testResults');
+        if (!buildRequests || !testResults)
             return [];
 
         var repositories = this._computeRepositoryList();
@@ -1188,7 +1196,41 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
         range.min -= margin;
 
         this.set('configurations', configurations);
-    }.observes('chartData', 'buildRequests'),
+    }.observes('testResults', 'buildRequests'),
+    _updateReferenceChart: function ()
+    {
+        var configurations = this.get('configurations');
+        var chartData = this.get('overviewPane') ? this.get('overviewPane').get('chartData') : null;
+        if (!configurations || !chartData || this.get('referenceChart'))
+            return;
+
+        var currentTimeSeries = chartData.current;
+        if (!currentTimeSeries)
+            return;
+
+        var repositories = this.get('repositories');
+        var highlightedItems = {};
+        var failedToFindPoint = false;
+        configurations.forEach(function (config) {
+            var revisions = {};
+            config.get('rootSet').get('roots').forEach(function (root) {
+                revisions[root.get('repository').get('id')] = root.get('revision');
+            });
+            var point = currentTimeSeries.findPointByRevisions(revisions);
+            if (!point) {
+                failedToFindPoint = true;
+                return;
+            }
+            highlightedItems[point.measurement.id()] = true;
+        });
+        if (failedToFindPoint)
+            return;
+
+        this.set('referenceChart', {
+            data: chartData,
+            highlightedItems: highlightedItems,
+        });
+    }.observes('configurations', 'overviewPane.chartData'),
     _computeRepositoryList: function ()
     {
         var specifiedRepositories = new Ember.Set();
@@ -1198,9 +1240,9 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
             });
         });
         var reportedRepositories = new Ember.Set();
-        var chartData = this.get('chartData');
+        var testResults = this.get('testResults');
         (this.get('buildRequests') || []).forEach(function (request) {
-            var point = chartData.current.findPointByBuild(request.get('build'));
+            var point = testResults.current.findPointByBuild(request.get('build'));
             if (!point)
                 return;
 
@@ -1228,19 +1270,19 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
     _createConfigurationSummary: function (buildRequests, configLetter, range)
     {
         var repositories = this.get('repositories');
-        var chartData = this.get('chartData');
+        var testResults = this.get('testResults');
         var requests = buildRequests.map(function (originalRequest) {
-            var point = chartData.current.findPointByBuild(originalRequest.get('build'));
+            var point = testResults.current.findPointByBuild(originalRequest.get('build'));
             var revisionByRepositoryId = point ? point.measurement.formattedRevisions() : {};
             return Ember.ObjectProxy.create({
                 content: originalRequest,
-                revisions: repositories.map(function (repository, index) {
+                revisionList: repositories.map(function (repository, index) {
                     return (revisionByRepositoryId[repository.get('id')] || {label:null}).label;
                 }),
                 value: point ? point.value : null,
                 valueRange: range,
-                formattedValue: point ? chartData.formatWithUnit(point.value) : null,
-                buildNumber: point ? point.measurement.buildNumber() : null,
+                formattedValue: point ? testResults.formatWithUnit(point.value) : null,
+                buildLabel: point ? 'Build ' + point.measurement.buildNumber() : null,
             });
         });
 
@@ -1248,15 +1290,15 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
         var summaryRevisions = repositories.map(function (repository, index) {
             var revision = rootSet ? rootSet.revisionForRepository(repository) : null;
             if (!revision)
-                return requests[0].get('revisions')[index];
+                return requests[0].get('revisionList')[index];
             return Measurement.formatRevisionRange(revision).label;
         });
 
         requests.forEach(function (request) {
-            var revisions = request.get('revisions');
+            var revisionList = request.get('revisionList');
             repositories.forEach(function (repository, index) {
-                if (revisions[index] == summaryRevisions[index])
-                    revisions[index] = null;
+                if (revisionList[index] == summaryRevisions[index])
+                    revisionList[index] = null;
             });
         });
 
@@ -1275,15 +1317,15 @@ App.TestGroupPane = Ember.ObjectProxy.extend({
         var summary = Ember.Object.create({
             isAverage: true,
             configLetter: configLetter,
-            revisions: summaryRevisions,
-            formattedValue: isNaN(mean) ? null : chartData.formatWithDeltaAndUnit(mean, ciDelta),
+            revisionList: summaryRevisions,
+            formattedValue: isNaN(mean) ? null : testResults.formatWithDeltaAndUnit(mean, ciDelta),
             value: mean,
             confidenceIntervalDelta: ciDelta,
             valueRange: range,
             statusLabel: App.BuildRequest.aggregateStatuses(requests),
         });
 
-        return Ember.Object.create({summary: summary, items: requests});
+        return Ember.Object.create({summary: summary, items: requests, rootSet: rootSet});
     },
 });
 
index 94d0fb24bc4cd2ed172f11f6214a1b99c0c818d1..9e521130f258453a07217020fad275a8377c2ab0 100755 (executable)
@@ -152,13 +152,18 @@ function Measurement(rawData)
     this._formattedRevisions = undefined;
 }
 
+Measurement.prototype.revisionForRepository = function (repositoryId)
+{
+    var revisions = this._raw['revisions'];
+    var rawData = revisions[repositoryId];
+    return rawData ? rawData[0] : null;
+}
+
 Measurement.prototype.commitTimeForRepository = function (repositoryId)
 {
     var revisions = this._raw['revisions'];
     var rawData = revisions[repositoryId];
-    if (!rawData)
-        return null;
-    return new Date(rawData[1]);
+    return rawData ? new Date(rawData[1]) : null;
 }
 
 Measurement.prototype.formattedRevisions = function (previousMeasurement)
@@ -368,6 +373,17 @@ TimeSeries.prototype.findPointByBuild = function (buildId)
     return this._series.find(function (point) { return point.measurement.buildId() == buildId; })
 }
 
+TimeSeries.prototype.findPointByRevisions = function (revisions)
+{
+    return this._series.find(function (point, index) {
+        for (var repositoryId in revisions) {
+            if (point.measurement.revisionForRepository(repositoryId) != revisions[repositoryId])
+                return false;
+        }
+        return true;
+    });
+}
+
 TimeSeries.prototype.findPointByMeasurementId = function (measurementId)
 {
     return this._series.find(function (point) { return point.measurement.id() == measurementId; });
index bdfc99bec348ff665079f9ca372793414f106e45..6c4cadf3d6996d306e02102cdc1c4d8ea0359a04 100755 (executable)
                     </tbody>
                 {{/each}}
             </table>
+            <div class="reference-chart">
+                {{#if referenceChart}}
+                    {{interactive-chart
+                        chartData=referenceChart.data
+                        domain=overviewDomain
+                        chartPointRadius=2
+                        showYAxis=false
+                        enableSelection=false
+                        highlightedItems=referenceChart.highlightedItems}}
+                {{/if}}
+            </div>
         </section>
     </script>
 
     <script type="text/x-handlebars" data-template-name="testGroupRow">
-        {{#each revisions}}
+        {{#each revisionList}}
             <td>{{this}}</td>
         {{/each}}
         <td>
             {{formattedValue}}
         </td>
         <td>
-            {{#if buildNumber}}
-                 {{statusLabel}} / <a {{bind-attr href=url}}>Build {{buildNumber}}</a>
-            {{else}}
-                <a {{bind-attr href=url}}>{{statusLabel}}</a>
-            {{/if}}
+            <a {{bind-attr href=url title=buildLabel}}>{{statusLabel}}</a>
         </td>
     </script>
 
index 0567680d704b1824c7d8107d29b6801578f4bf46..40ed2d00bf04d51b6d83582a6529924f084ea358 100644 (file)
@@ -208,7 +208,7 @@ App.InteractiveChartComponent = Ember.Component.extend({
         var currentYDomain = this._y.domain();
         if (currentXDomain && App.domainsAreEqual(currentXDomain, xDomain)
             && currentYDomain && App.domainsAreEqual(currentYDomain, yDomain))
-            return currentDomain;
+            return currentXDomain;
 
         this._x.domain(xDomain);
         this._y.domain(yDomain);
@@ -343,6 +343,16 @@ App.InteractiveChartComponent = Ember.Component.extend({
         var range = this._minMaxForAllTimeSeries(startTime, endTime, !shouldShowFullYAxis);
         var min = range[0];
         var max = range[1];
+
+        var highlightedItems = this.get('highlightedItems');
+        if (highlightedItems) {
+            var data = this._currentTimeSeriesData
+                .filter(function (point) { return startTime <= point.time && point.time <= endTime && highlightedItems[point.measurement.id()]; })
+                .map(function (point) { return point.value });
+            min = Math.min(min, Statistics.min(data));
+            max = Math.max(max, Statistics.max(data));
+        }
+
         if (max < min)
             min = max = 0;
         else if (shouldShowFullYAxis)
@@ -615,7 +625,7 @@ App.InteractiveChartComponent = Ember.Component.extend({
                 .attr("class", "highlight")
                 .attr("r", (this.get('chartPointRadius') || 1) * 1.8);
 
-        this._updateHighlightPositions();
+        this._domainChanged();
     }.observes('highlightedItems'),
     _rangesChanged: function ()
     {