Selecting revisions for A/B testing is hard
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 20 Feb 2015 21:39:01 +0000 (21:39 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 20 Feb 2015 21:39:01 +0000 (21:39 +0000)
https://bugs.webkit.org/show_bug.cgi?id=141824

Reviewed by Andreas Kling.

Update the revisions used in A/B testing based on the selection in the overview chart. This allows users to
intuitively select revisions based on points shown in the chart. Removed the old select elements used to
select A/B testing points manually.

Also renamed 'testSets' to 'configurations', 'roots' to 'rootConfigurations', and 'revisions' in each root's
sets to 'options' for clarity.

* public/v2/app.css: Reorganized style rules.

* public/v2/app.js:
(App.AnalysisTaskController):
(App.AnalysisTaskController._taskUpdated): Merged updateTestGroupPanes.
(App.AnalysisTaskController._chartDataChanged): Renamed from paneDomain. It's now an observer instead of
a property, which sets 'overviewDomain' property as well as other properties.
(App.AnalysisTaskController.updateRootConfigurations): Renamed from updateRoots.
(App.AnalysisTaskController._updateRootsBySelectedPoints): Added. Select roots based on the selected points
in the overview chart.

* public/v2/chart-pane.css: Added arrows next to the configuration names (e.g. 'A') to indicate whether
individual build requests / test results are shown or not.

* public/v2/index.html: Removed the select element per configuration column. Also moved the select element
for the number of runs as it doesn't belong in the same table as the one that lists repositories and roots.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@180451 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/v2/app.css
Websites/perf.webkit.org/public/v2/app.js
Websites/perf.webkit.org/public/v2/chart-pane.css
Websites/perf.webkit.org/public/v2/index.html

index 3f20b4b..f23c4e8 100644 (file)
@@ -1,5 +1,36 @@
 2015-02-20  Ryosuke Niwa  <rniwa@webkit.org>
 
+        Selecting revisions for A/B testing is hard
+        https://bugs.webkit.org/show_bug.cgi?id=141824
+
+        Reviewed by Andreas Kling.
+
+        Update the revisions used in A/B testing based on the selection in the overview chart. This allows users to
+        intuitively select revisions based on points shown in the chart. Removed the old select elements used to
+        select A/B testing points manually.
+
+        Also renamed 'testSets' to 'configurations', 'roots' to 'rootConfigurations', and 'revisions' in each root's
+        sets to 'options' for clarity.
+
+        * public/v2/app.css: Reorganized style rules. 
+
+        * public/v2/app.js:
+        (App.AnalysisTaskController):
+        (App.AnalysisTaskController._taskUpdated): Merged updateTestGroupPanes.
+        (App.AnalysisTaskController._chartDataChanged): Renamed from paneDomain. It's now an observer instead of
+        a property, which sets 'overviewDomain' property as well as other properties.
+        (App.AnalysisTaskController.updateRootConfigurations): Renamed from updateRoots.
+        (App.AnalysisTaskController._updateRootsBySelectedPoints): Added. Select roots based on the selected points
+        in the overview chart.
+
+        * public/v2/chart-pane.css: Added arrows next to the configuration names (e.g. 'A') to indicate whether
+        individual build requests / test results are shown or not.
+
+        * public/v2/index.html: Removed the select element per configuration column. Also moved the select element
+        for the number of runs as it doesn't belong in the same table as the one that lists repositories and roots.
+
+2015-02-20  Ryosuke Niwa  <rniwa@webkit.org>
+
         Unreviewed test fixes after r179037, r179591, and r179763.
 
         * tests/admin-regenerate-manifest.js:
index d7ee9f7..b2c11dd 100755 (executable)
@@ -421,6 +421,21 @@ table.dashboard tbody td .failure {
     vertical-align: middle;
 }
 
+.chart-pane,
+.analysis-group {
+    border: 1px solid #bbb;
+    border-radius: 0.5rem;
+    box-shadow: rgba(0, 0, 0, 0.03) 1px 1px 0px 0px;
+    padding: 0;
+    margin-bottom: 1rem;
+    outline: none;
+    position: relative;
+}
+
+form .analysis-group > * {
+    margin: 0.5rem;
+}
+
 #analysis-tasks,
 .analysis-group > table {
     border: solid 0px #999;
@@ -472,20 +487,15 @@ table.dashboard tbody td .failure {
     display: none;
 }
 
-.analysis-group {
-    display: block;
-    position: relative;
-    min-height: 6.5rem;
-}
-
 .analysis-group .results {
+    margin: 0.5rem;
     margin-right: 20rem;
 }
 
 .analysis-group .reference-chart {
     position: absolute;
-    top: 1.2rem;
-    right: -0.7rem;
+    top: 1.8rem;
+    right: 0rem;
     width: 19rem;
     height: 5rem;
 }
@@ -532,17 +542,6 @@ table.dashboard tbody td .failure {
     color: #333;
 }
 
-.analysis-group {
-    border: 1px solid #bbb;
-    border-radius: 0.5rem;
-    box-shadow: rgba(0, 0, 0, 0.03) 1px 1px 0px 0px;
-    margin-bottom: 1.5rem;
-}
-
-.analysis-group > * {
-    margin: 0.5rem;
-}
-
 .analysis-group > h1 {
     font-size: 1.1rem;
     font-weight: normal;
index 640ac69..91838d5 100755 (executable)
@@ -979,7 +979,6 @@ App.AnalysisTaskController = Ember.Controller.extend({
     platform: Ember.computed.alias('model.platform'),
     metric: Ember.computed.alias('model.metric'),
     details: Ember.computed.alias('pane.details'),
-    testSets: [],
     roots: [],
     bugTrackers: [],
     possibleRepetitionCounts: [1, 2, 3, 4, 5, 6],
@@ -995,7 +994,12 @@ App.AnalysisTaskController = Ember.Controller.extend({
             platformId: model.get('platform').get('id'),
             metricId: model.get('metric').get('id'),
         }));
-    }.observes('model').on('init'),
+
+        var self = this;
+        model.get('testGroups').then(function (groups) {
+            self.set('testGroupPanes', groups.map(function (group) { return App.TestGroupPane.create({content: group}); }));
+        });
+    }.observes('model', 'model.testGroups').on('init'),
     _fetchedManifest: function ()
     {
         var trackerIdToBugNumber = {};
@@ -1012,7 +1016,7 @@ App.AnalysisTaskController = Ember.Controller.extend({
             });
         }));
     },
-    paneDomain: function ()
+    _chartDataChanged: function ()
     {
         var pane = this.get('pane');
         if (!pane)
@@ -1046,52 +1050,20 @@ App.AnalysisTaskController = Ember.Controller.extend({
 
         var margin = (end.time - start.time) * 0.1;
         this.set('highlightedItems', highlightedItems);
+        this.set('overviewEndPoints', [start, end]);
         this.set('analysisPoints', formatedPoints);
 
-        var paneDomain = [start.time - margin, +end.time + margin];
+        var overviewDomain = [start.time - margin, +end.time + margin];
 
         var testGroupPanes = this.get('testGroupPanes');
         if (testGroupPanes) {
             testGroupPanes.setEach('overviewPane', pane);
-            testGroupPanes.setEach('overviewDomain', paneDomain);
+            testGroupPanes.setEach('overviewDomain', overviewDomain);
         }
 
-        return paneDomain;
-    }.property('pane.chartData'),
-    testSets: function ()
-    {
-        var analysisPoints = this.get('analysisPoints');
-        if (!analysisPoints)
-            return;
-        var pointOptions = [{value: ' ', label: 'None'}]
-            .concat(analysisPoints.map(function (point) { return {value: point.id, label: point.label}; }));
-        return [
-            Ember.Object.create({name: "A", options: pointOptions, selection: pointOptions[1]}),
-            Ember.Object.create({name: "B", options: pointOptions, selection: pointOptions[pointOptions.length - 1]}),
-        ];
-    }.property('analysisPoints'),
-    _rootChangedForTestSet: function ()
-    {
-        var sets = this.get('testSets');
-        var roots = this.get('roots');
-        if (!sets || !roots)
-            return;
-
-        sets.forEach(function (testSet, setIndex) {
-            var currentSelection = testSet.get('selection');
-            if (currentSelection == testSet.get('previousSelection'))
-                return;
-            testSet.set('previousSelection', currentSelection);
-            var pointIndex = testSet.get('options').indexOf(currentSelection);
-
-            roots.forEach(function (root) {
-                var set = root.sets[setIndex];
-                set.set('selection', set.revisions[pointIndex]);
-            });
-        });
-
-    }.observes('testSets.@each.selection'),
-    updateRoots: function ()
+        this.set('overviewDomain', overviewDomain);
+    }.observes('pane.chartData'),
+    updateRootConfigurations: function ()
     {
         var analysisPoints = this.get('analysisPoints');
         if (!analysisPoints)
@@ -1115,33 +1087,25 @@ App.AnalysisTaskController = Ember.Controller.extend({
             if (!triggerable)
                 return;
 
-            self.set('roots', triggerable.get('acceptedRepositories').map(function (repository) {
+            self.set('configurations', ['A', 'B']);
+            self.set('rootConfigurations', triggerable.get('acceptedRepositories').map(function (repository) {
                 var repositoryId = repository.get('id');
-                var revisions = [{value: ' ', label: 'None'}].concat(repositoryToRevisions[repositoryId]);
+                var options = [{value: ' ', label: 'None'}].concat(repositoryToRevisions[repositoryId]);
                 return Ember.Object.create({
+                    repository: repository,
                     name: repository.get('name'),
                     sets: [
                         Ember.Object.create({name: 'A[' + repositoryId + ']',
-                            revisions: revisions,
-                            selection: revisions[1]}),
+                            options: options,
+                            selection: options[1]}),
                         Ember.Object.create({name: 'B[' + repositoryId + ']',
-                            revisions: revisions,
-                            selection: revisions[revisions.length - 1]}),
+                            options: options,
+                            selection: options[options.length - 1]}),
                     ],
                 });
             }));
         });
     }.observes('analysisPoints'),
-    updateTestGroupPanes: function ()
-    {
-        var model = this.get('model');
-        if (!model)
-            return;
-        var self = this;
-        model.get('testGroups').then(function (groups) {
-            self.set('testGroupPanes', groups.map(function (group) { return App.TestGroupPane.create({content: group}); }));
-        });
-    }.observes('model'),
     actions: {
         associateBug: function (bugTracker, bugNumber)
         {
@@ -1156,11 +1120,12 @@ App.AnalysisTaskController = Ember.Controller.extend({
         createTestGroup: function (name, repetitionCount)
         {
             var roots = {};
-            this.get('roots').map(function (root) {
+            this.get('rootConfigurations').map(function (root) {
                 roots[root.get('name')] = root.get('sets').map(function (item) { return item.get('selection').value; });
             });
             App.TestGroup.create(this.get('model'), name, roots, repetitionCount).then(function () {
-                
+            }, function (error) {
+                alert('Failed to create a new test group:' + error);
             });
         },
         toggleShowRequestList: function (configuration)
@@ -1168,6 +1133,35 @@ App.AnalysisTaskController = Ember.Controller.extend({
             configuration.toggleProperty('showRequestList');
         }
     },
+    _updateRootsBySelectedPoints: function ()
+    {
+        var rootConfigurations = this.get('rootConfigurations');
+        var pane = this.get('pane');
+        if (!rootConfigurations || !pane)
+            return;
+
+        var rootSetPoints;
+        var selectedPoints = pane.get('selectedPoints');
+        if (selectedPoints && selectedPoints.length >= 2)
+            rootSetPoints = [selectedPoints[0], selectedPoints[selectedPoints.length - 1]];
+        else
+            rootSetPoints = this.get('overviewEndPoints');
+        if (!rootSetPoints)
+            return;
+
+        rootConfigurations.forEach(function (root) {
+            root.get('sets').forEach(function (set, setIndex) {
+                if (setIndex >= rootSetPoints.length)
+                    return;
+                var targetRevision = rootSetPoints[setIndex].measurement.revisionForRepository(root.get('repository').get('id'));
+                var selectedOption;
+                if (targetRevision)
+                    selectedOption = set.get('options').find(function (option) { return option.value == targetRevision; });
+                set.set('selection', selectedOption || sets[i].get('options')[0]);
+            });
+        });
+
+    }.observes('pane.selectedPoints'),
 });
 
 App.TestGroupPane = Ember.ObjectProxy.extend({
index 131d567..4da2455 100755 (executable)
@@ -1,12 +1,3 @@
-.chart-pane {
-    border: 1px solid #bbb;
-    border-radius: 0.5rem;
-    box-shadow: rgba(0, 0, 0, 0.03) 1px 1px 0px 0px;
-    padding: 0;
-    margin-bottom: 1rem;
-    outline: none;
-    position: relative;
-}
 .chart-pane header {
     background: #fff;
     margin: 0;
     cursor: pointer;
 }
 
-.chart-pane .commits-viewer caption:before {
+.chart-pane .commits-viewer caption:before,
+.analysis-group .results .summary .config-letter:before {
     display: inline-block;
     width: 0.8rem;
     content: "\25BE"; /* Down arrow */
 }
 
-.chart-pane .commits-viewer.hidden caption:before {
+.chart-pane .commits-viewer.hidden caption:before,
+.analysis-group .results .hideRequests .summary .config-letter:before {
     display: inline-block;
     width: 0.8rem;
     content: "\25B8"; /* Right arrow */
index 6c4cadf..f330d1e 100755 (executable)
                     {{interactive-chart
                         chartData=pane.chartData
                         ranges=pane.analyticRanges
-                        domain=paneDomain
+                        domain=overviewDomain
                         interactive=true
                         chartPointRadius=2
                         currentItem=pane.hoveredOrSelectedItem
     </script>
 
     <script type="text/x-handlebars" data-template-name="testGroupForm">
-    {{#if roots}}
         <form method="POST" {{action "createTestGroup" newTestGroupName repetitionCount on="submit"}}>
             <section class="analysis-group">
                 <h1>{{input name="name" value=newTestGroupName placeholder="Test group name" required=true type="text"}}</h1>
                 <table>
                     <thead>
                         <tr>
-                            <th>Root</th>
-                            {{#each testSets}}
-                                <th>
-                                    {{name}}
-                                    {{view Ember.Select
-                                        content=options
-                                        optionValuePath="content.value"
-                                        optionLabelPath="content.label"
-                                        selection=selection}}
-                                </th>
+                            <th>Configuration</th>
+                            {{#each configurations}}
+                                <th>{{this}}</th>
                             {{/each}}
                         </tr>
                     </thead>
                     <tbody>
-                        {{#each roots}}
+                        {{#each rootConfigurations}}
                             <tr>
                                 <th>{{name}}</th>
                                 {{#each sets}}
-                                    <td>{{view Ember.Select name=name content=revisions disabled=true
+                                    <td>{{view Ember.Select name=name content=options disabled=true
                                         optionValuePath="content.value" optionLabelPath="content.label"
                                         selection=selection}}</td>
                                 {{/each}}
                             </tr>
                         {{/each}}
                     </tbody>
-                    <tbody>
-                        <tr>
-                            <th>Number of runs</th>
-                            <td colspan=2>
-                                {{view Ember.Select content=possibleRepetitionCounts value=repetitionCount}}
-                            </td>
-                        </tr>
-                    </tbody>
                 </table>
-
+                <label>Number of runs {{view Ember.Select content=possibleRepetitionCounts value=repetitionCount}}</label>
                 <button type="submit">Start A/B testing</button>
             </section>
         </form>
-    {{/if}}
     </script>
 
 </head>