Selecting revisions for A/B testing is hard
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 20 Feb 2015 21:39:01 +0000 (21:39 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 20 Feb 2015 21:39:01 +0000 (21:39 +0000)
https://bugs.webkit.org/show_bug.cgi?id=141824

Reviewed by Andreas Kling.

Update the revisions used in A/B testing based on the selection in the overview chart. This allows users to
intuitively select revisions based on points shown in the chart. Removed the old select elements used to
select A/B testing points manually.

Also renamed 'testSets' to 'configurations', 'roots' to 'rootConfigurations', and 'revisions' in each root's
sets to 'options' for clarity.

* public/v2/app.css: Reorganized style rules.

* public/v2/app.js:
(App.AnalysisTaskController):
(App.AnalysisTaskController._taskUpdated): Merged updateTestGroupPanes.
(App.AnalysisTaskController._chartDataChanged): Renamed from paneDomain. It's now an observer instead of
a property, which sets 'overviewDomain' property as well as other properties.
(App.AnalysisTaskController.updateRootConfigurations): Renamed from updateRoots.
(App.AnalysisTaskController._updateRootsBySelectedPoints): Added. Select roots based on the selected points
in the overview chart.

* public/v2/chart-pane.css: Added arrows next to the configuration names (e.g. 'A') to indicate whether
individual build requests / test results are shown or not.

* public/v2/index.html: Removed the select element per configuration column. Also moved the select element
for the number of runs as it doesn't belong in the same table as the one that lists repositories and roots.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@180451 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/v2/app.css
Websites/perf.webkit.org/public/v2/app.js
Websites/perf.webkit.org/public/v2/chart-pane.css
Websites/perf.webkit.org/public/v2/index.html

index 3f20b4b0660cccc8cf8bedbc696547525af1e474..f23c4e87f045c0a81356e4f0ce256687664daba2 100644 (file)
@@ -1,3 +1,34 @@
+2015-02-20  Ryosuke Niwa  <rniwa@webkit.org>
+
+        Selecting revisions for A/B testing is hard
+        https://bugs.webkit.org/show_bug.cgi?id=141824
+
+        Reviewed by Andreas Kling.
+
+        Update the revisions used in A/B testing based on the selection in the overview chart. This allows users to
+        intuitively select revisions based on points shown in the chart. Removed the old select elements used to
+        select A/B testing points manually.
+
+        Also renamed 'testSets' to 'configurations', 'roots' to 'rootConfigurations', and 'revisions' in each root's
+        sets to 'options' for clarity.
+
+        * public/v2/app.css: Reorganized style rules. 
+
+        * public/v2/app.js:
+        (App.AnalysisTaskController):
+        (App.AnalysisTaskController._taskUpdated): Merged updateTestGroupPanes.
+        (App.AnalysisTaskController._chartDataChanged): Renamed from paneDomain. It's now an observer instead of
+        a property, which sets 'overviewDomain' property as well as other properties.
+        (App.AnalysisTaskController.updateRootConfigurations): Renamed from updateRoots.
+        (App.AnalysisTaskController._updateRootsBySelectedPoints): Added. Select roots based on the selected points
+        in the overview chart.
+
+        * public/v2/chart-pane.css: Added arrows next to the configuration names (e.g. 'A') to indicate whether
+        individual build requests / test results are shown or not.
+
+        * public/v2/index.html: Removed the select element per configuration column. Also moved the select element
+        for the number of runs as it doesn't belong in the same table as the one that lists repositories and roots.
+
 2015-02-20  Ryosuke Niwa  <rniwa@webkit.org>
 
         Unreviewed test fixes after r179037, r179591, and r179763.
index d7ee9f75bbac72e8731e20ba95917b3890e9477f..b2c11ddabc1600c945f3a527f7fb0fb3c60332c9 100755 (executable)
@@ -421,6 +421,21 @@ table.dashboard tbody td .failure {
     vertical-align: middle;
 }
 
+.chart-pane,
+.analysis-group {
+    border: 1px solid #bbb;
+    border-radius: 0.5rem;
+    box-shadow: rgba(0, 0, 0, 0.03) 1px 1px 0px 0px;
+    padding: 0;
+    margin-bottom: 1rem;
+    outline: none;
+    position: relative;
+}
+
+form .analysis-group > * {
+    margin: 0.5rem;
+}
+
 #analysis-tasks,
 .analysis-group > table {
     border: solid 0px #999;
@@ -472,20 +487,15 @@ table.dashboard tbody td .failure {
     display: none;
 }
 
-.analysis-group {
-    display: block;
-    position: relative;
-    min-height: 6.5rem;
-}
-
 .analysis-group .results {
+    margin: 0.5rem;
     margin-right: 20rem;
 }
 
 .analysis-group .reference-chart {
     position: absolute;
-    top: 1.2rem;
-    right: -0.7rem;
+    top: 1.8rem;
+    right: 0rem;
     width: 19rem;
     height: 5rem;
 }
@@ -532,17 +542,6 @@ table.dashboard tbody td .failure {
     color: #333;
 }
 
-.analysis-group {
-    border: 1px solid #bbb;
-    border-radius: 0.5rem;
-    box-shadow: rgba(0, 0, 0, 0.03) 1px 1px 0px 0px;
-    margin-bottom: 1.5rem;
-}
-
-.analysis-group > * {
-    margin: 0.5rem;
-}
-
 .analysis-group > h1 {
     font-size: 1.1rem;
     font-weight: normal;
index 640ac69a7ac37f34c4500ee1ea2421a653e8235f..91838d521bfdf4adb8cc24a0a582fee9f2e8deee 100755 (executable)
@@ -979,7 +979,6 @@ App.AnalysisTaskController = Ember.Controller.extend({
     platform: Ember.computed.alias('model.platform'),
     metric: Ember.computed.alias('model.metric'),
     details: Ember.computed.alias('pane.details'),
-    testSets: [],
     roots: [],
     bugTrackers: [],
     possibleRepetitionCounts: [1, 2, 3, 4, 5, 6],
@@ -995,7 +994,12 @@ App.AnalysisTaskController = Ember.Controller.extend({
             platformId: model.get('platform').get('id'),
             metricId: model.get('metric').get('id'),
         }));
-    }.observes('model').on('init'),
+
+        var self = this;
+        model.get('testGroups').then(function (groups) {
+            self.set('testGroupPanes', groups.map(function (group) { return App.TestGroupPane.create({content: group}); }));
+        });
+    }.observes('model', 'model.testGroups').on('init'),
     _fetchedManifest: function ()
     {
         var trackerIdToBugNumber = {};
@@ -1012,7 +1016,7 @@ App.AnalysisTaskController = Ember.Controller.extend({
             });
         }));
     },
-    paneDomain: function ()
+    _chartDataChanged: function ()
     {
         var pane = this.get('pane');
         if (!pane)
@@ -1046,52 +1050,20 @@ App.AnalysisTaskController = Ember.Controller.extend({
 
         var margin = (end.time - start.time) * 0.1;
         this.set('highlightedItems', highlightedItems);
+        this.set('overviewEndPoints', [start, end]);
         this.set('analysisPoints', formatedPoints);
 
-        var paneDomain = [start.time - margin, +end.time + margin];
+        var overviewDomain = [start.time - margin, +end.time + margin];
 
         var testGroupPanes = this.get('testGroupPanes');
         if (testGroupPanes) {
             testGroupPanes.setEach('overviewPane', pane);
-            testGroupPanes.setEach('overviewDomain', paneDomain);
+            testGroupPanes.setEach('overviewDomain', overviewDomain);
         }
 
-        return paneDomain;
-    }.property('pane.chartData'),
-    testSets: function ()
-    {
-        var analysisPoints = this.get('analysisPoints');
-        if (!analysisPoints)
-            return;
-        var pointOptions = [{value: ' ', label: 'None'}]
-            .concat(analysisPoints.map(function (point) { return {value: point.id, label: point.label}; }));
-        return [
-            Ember.Object.create({name: "A", options: pointOptions, selection: pointOptions[1]}),
-            Ember.Object.create({name: "B", options: pointOptions, selection: pointOptions[pointOptions.length - 1]}),
-        ];
-    }.property('analysisPoints'),
-    _rootChangedForTestSet: function ()
-    {
-        var sets = this.get('testSets');
-        var roots = this.get('roots');
-        if (!sets || !roots)
-            return;
-
-        sets.forEach(function (testSet, setIndex) {
-            var currentSelection = testSet.get('selection');
-            if (currentSelection == testSet.get('previousSelection'))
-                return;
-            testSet.set('previousSelection', currentSelection);
-            var pointIndex = testSet.get('options').indexOf(currentSelection);
-
-            roots.forEach(function (root) {
-                var set = root.sets[setIndex];
-                set.set('selection', set.revisions[pointIndex]);
-            });
-        });
-
-    }.observes('testSets.@each.selection'),
-    updateRoots: function ()
+        this.set('overviewDomain', overviewDomain);
+    }.observes('pane.chartData'),
+    updateRootConfigurations: function ()
     {
         var analysisPoints = this.get('analysisPoints');
         if (!analysisPoints)
@@ -1115,33 +1087,25 @@ App.AnalysisTaskController = Ember.Controller.extend({
             if (!triggerable)
                 return;
 
-            self.set('roots', triggerable.get('acceptedRepositories').map(function (repository) {
+            self.set('configurations', ['A', 'B']);
+            self.set('rootConfigurations', triggerable.get('acceptedRepositories').map(function (repository) {
                 var repositoryId = repository.get('id');
-                var revisions = [{value: ' ', label: 'None'}].concat(repositoryToRevisions[repositoryId]);
+                var options = [{value: ' ', label: 'None'}].concat(repositoryToRevisions[repositoryId]);
                 return Ember.Object.create({
+                    repository: repository,
                     name: repository.get('name'),
                     sets: [
                         Ember.Object.create({name: 'A[' + repositoryId + ']',
-                            revisions: revisions,
-                            selection: revisions[1]}),
+                            options: options,
+                            selection: options[1]}),
                         Ember.Object.create({name: 'B[' + repositoryId + ']',
-                            revisions: revisions,
-                            selection: revisions[revisions.length - 1]}),
+                            options: options,
+                            selection: options[options.length - 1]}),
                     ],
                 });
             }));
         });
     }.observes('analysisPoints'),
-    updateTestGroupPanes: function ()
-    {
-        var model = this.get('model');
-        if (!model)
-            return;
-        var self = this;
-        model.get('testGroups').then(function (groups) {
-            self.set('testGroupPanes', groups.map(function (group) { return App.TestGroupPane.create({content: group}); }));
-        });
-    }.observes('model'),
     actions: {
         associateBug: function (bugTracker, bugNumber)
         {
@@ -1156,11 +1120,12 @@ App.AnalysisTaskController = Ember.Controller.extend({
         createTestGroup: function (name, repetitionCount)
         {
             var roots = {};
-            this.get('roots').map(function (root) {
+            this.get('rootConfigurations').map(function (root) {
                 roots[root.get('name')] = root.get('sets').map(function (item) { return item.get('selection').value; });
             });
             App.TestGroup.create(this.get('model'), name, roots, repetitionCount).then(function () {
-                
+            }, function (error) {
+                alert('Failed to create a new test group:' + error);
             });
         },
         toggleShowRequestList: function (configuration)
@@ -1168,6 +1133,35 @@ App.AnalysisTaskController = Ember.Controller.extend({
             configuration.toggleProperty('showRequestList');
         }
     },
+    _updateRootsBySelectedPoints: function ()
+    {
+        var rootConfigurations = this.get('rootConfigurations');
+        var pane = this.get('pane');
+        if (!rootConfigurations || !pane)
+            return;
+
+        var rootSetPoints;
+        var selectedPoints = pane.get('selectedPoints');
+        if (selectedPoints && selectedPoints.length >= 2)
+            rootSetPoints = [selectedPoints[0], selectedPoints[selectedPoints.length - 1]];
+        else
+            rootSetPoints = this.get('overviewEndPoints');
+        if (!rootSetPoints)
+            return;
+
+        rootConfigurations.forEach(function (root) {
+            root.get('sets').forEach(function (set, setIndex) {
+                if (setIndex >= rootSetPoints.length)
+                    return;
+                var targetRevision = rootSetPoints[setIndex].measurement.revisionForRepository(root.get('repository').get('id'));
+                var selectedOption;
+                if (targetRevision)
+                    selectedOption = set.get('options').find(function (option) { return option.value == targetRevision; });
+                set.set('selection', selectedOption || sets[i].get('options')[0]);
+            });
+        });
+
+    }.observes('pane.selectedPoints'),
 });
 
 App.TestGroupPane = Ember.ObjectProxy.extend({
index 131d567ca5b3b20767236e6c26bec76a83960fc8..4da24550dcbde3b28b83e9ef095c2059dcb13a07 100755 (executable)
@@ -1,12 +1,3 @@
-.chart-pane {
-    border: 1px solid #bbb;
-    border-radius: 0.5rem;
-    box-shadow: rgba(0, 0, 0, 0.03) 1px 1px 0px 0px;
-    padding: 0;
-    margin-bottom: 1rem;
-    outline: none;
-    position: relative;
-}
 .chart-pane header {
     background: #fff;
     margin: 0;
     cursor: pointer;
 }
 
-.chart-pane .commits-viewer caption:before {
+.chart-pane .commits-viewer caption:before,
+.analysis-group .results .summary .config-letter:before {
     display: inline-block;
     width: 0.8rem;
     content: "\25BE"; /* Down arrow */
 }
 
-.chart-pane .commits-viewer.hidden caption:before {
+.chart-pane .commits-viewer.hidden caption:before,
+.analysis-group .results .hideRequests .summary .config-letter:before {
     display: inline-block;
     width: 0.8rem;
     content: "\25B8"; /* Right arrow */
index 6c4cadf3d6996d306e02102cdc1c4d8ea0359a04..f330d1e7582dd7030d9548d1c4c0cf27c3909ba7 100755 (executable)
                     {{interactive-chart
                         chartData=pane.chartData
                         ranges=pane.analyticRanges
-                        domain=paneDomain
+                        domain=overviewDomain
                         interactive=true
                         chartPointRadius=2
                         currentItem=pane.hoveredOrSelectedItem
     </script>
 
     <script type="text/x-handlebars" data-template-name="testGroupForm">
-    {{#if roots}}
         <form method="POST" {{action "createTestGroup" newTestGroupName repetitionCount on="submit"}}>
             <section class="analysis-group">
                 <h1>{{input name="name" value=newTestGroupName placeholder="Test group name" required=true type="text"}}</h1>
                 <table>
                     <thead>
                         <tr>
-                            <th>Root</th>
-                            {{#each testSets}}
-                                <th>
-                                    {{name}}
-                                    {{view Ember.Select
-                                        content=options
-                                        optionValuePath="content.value"
-                                        optionLabelPath="content.label"
-                                        selection=selection}}
-                                </th>
+                            <th>Configuration</th>
+                            {{#each configurations}}
+                                <th>{{this}}</th>
                             {{/each}}
                         </tr>
                     </thead>
                     <tbody>
-                        {{#each roots}}
+                        {{#each rootConfigurations}}
                             <tr>
                                 <th>{{name}}</th>
                                 {{#each sets}}
-                                    <td>{{view Ember.Select name=name content=revisions disabled=true
+                                    <td>{{view Ember.Select name=name content=options disabled=true
                                         optionValuePath="content.value" optionLabelPath="content.label"
                                         selection=selection}}</td>
                                 {{/each}}
                             </tr>
                         {{/each}}
                     </tbody>
-                    <tbody>
-                        <tr>
-                            <th>Number of runs</th>
-                            <td colspan=2>
-                                {{view Ember.Select content=possibleRepetitionCounts value=repetitionCount}}
-                            </td>
-                        </tr>
-                    </tbody>
                 </table>
-
+                <label>Number of runs {{view Ember.Select content=possibleRepetitionCounts value=repetitionCount}}</label>
                 <button type="submit">Start A/B testing</button>
             </section>
         </form>
-    {{/if}}
     </script>
 
 </head>