Perf dashboard should be able to trigger A/B testing jobs for iOS
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 6 Apr 2015 21:08:26 +0000 (21:08 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 6 Apr 2015 21:08:26 +0000 (21:08 +0000)
https://bugs.webkit.org/show_bug.cgi?id=143398

Reviewed by Chris Dumez.

Fix various bugs in the perf dashboard so that it can schedule A/B testing jobs for iOS.

Also generalized sync-with-buildbot.py slightly to meet the requirements of iOS builders.

* public/api/triggerables.php:
(main): Avoid spitting a warning when $id_to_triggerable doesn't contain the triggerable.
* public/v2/analysis.js:
(App.AnalysisTask.triggerable): Log an error when failed to fetch triggerables for debugging purposes.
* public/v2/app.js:
(App.AnalysisTaskController.updateRootConfigurations): Show 'None' when a revision is missing from
some of the data points. This will happen when we modify the list of projects we build for iOS.
(App.AnalysisTaskController.actions.createTestGroup): Gracefully fail by showing alerts when an user
attempts to create an invalid test group; when there is already another test group of the same or when
only either configuration specifies the revision for some repository.
(App.AnalysisTaskController._updateRootsBySelectedPoints): Fixed a typo: sets[i] -> set.
* public/v2/index.html: Don't show the form to create a new test group if it's not available.
* tools/sync-with-buildbot.py:
(find_request_updates):
(schedule_request): iOS builders take a JSON that contains the list of roots. Generate this JSON when
a dictionary of the form {rootsExcluding: ["WebKit"]} is specified. Also replaced the way we refer to
a revision from $-based text replacements to an explicit dictionary of the form {root: "WebKit"}.
(request_id_from_build): Don't hard code the parameter name here. Retrieve the name from the config.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@182438 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/api/triggerables.php
Websites/perf.webkit.org/public/v2/analysis.js
Websites/perf.webkit.org/public/v2/app.js
Websites/perf.webkit.org/public/v2/index.html
Websites/perf.webkit.org/tools/sync-with-buildbot.py

index 608d015..66d9434 100644 (file)
@@ -1,5 +1,35 @@
 2015-04-03  Ryosuke Niwa  <rniwa@webkit.org>
 
+        Perf dashboard should be able to trigger A/B testing jobs for iOS
+        https://bugs.webkit.org/show_bug.cgi?id=143398
+
+        Reviewed by Chris Dumez.
+
+        Fix various bugs in the perf dashboard so that it can schedule A/B testing jobs for iOS.
+
+        Also generalized sync-with-buildbot.py slightly to meet the requirements of iOS builders.
+
+        * public/api/triggerables.php:
+        (main): Avoid spitting a warning when $id_to_triggerable doesn't contain the triggerable.
+        * public/v2/analysis.js:
+        (App.AnalysisTask.triggerable): Log an error when failed to fetch triggerables for debugging purposes.
+        * public/v2/app.js:
+        (App.AnalysisTaskController.updateRootConfigurations): Show 'None' when a revision is missing from
+        some of the data points. This will happen when we modify the list of projects we build for iOS.
+        (App.AnalysisTaskController.actions.createTestGroup): Gracefully fail by showing alerts when an user
+        attempts to create an invalid test group; when there is already another test group of the same or when
+        only either configuration specifies the revision for some repository.
+        (App.AnalysisTaskController._updateRootsBySelectedPoints): Fixed a typo: sets[i] -> set.
+        * public/v2/index.html: Don't show the form to create a new test group if it's not available.
+        * tools/sync-with-buildbot.py:
+        (find_request_updates):
+        (schedule_request): iOS builders take a JSON that contains the list of roots. Generate this JSON when
+        a dictionary of the form {rootsExcluding: ["WebKit"]} is specified. Also replaced the way we refer to
+        a revision from $-based text replacements to an explicit dictionary of the form {root: "WebKit"}.
+        (request_id_from_build): Don't hard code the parameter name here. Retrieve the name from the config.
+
+2015-04-03  Ryosuke Niwa  <rniwa@webkit.org>
+
         Add time series segmentation algorithms as moving averages
         https://bugs.webkit.org/show_bug.cgi?id=143362
 
index 3183910..44f4c97 100644 (file)
@@ -27,7 +27,7 @@ function main($path) {
     }
 
     foreach ($db->select_rows('triggerable_repositories', 'trigrepo', array()) as $row) {
-        $triggerable = $id_to_triggerable[$row['trigrepo_triggerable']];
+        $triggerable = array_get($id_to_triggerable, $row['trigrepo_triggerable']);
         if ($triggerable)
             array_push($triggerable['acceptedRepositories'], $row['trigrepo_repository']);
     }
index 443cb96..f3f3578 100644 (file)
@@ -12,7 +12,8 @@ App.AnalysisTask = App.NameLabelModel.extend({
     triggerable: function () {
         return this.store.find('triggerable', {task: this.get('id')}).then(function (triggerables) {
             return triggerables.objectAt(0);
-        }, function () {
+        }, function (error) {
+            console.log('Failed to fetch triggerables', error);
             return null;
         });
     }.property(),
index 6a53a32..72d21b2 100755 (executable)
@@ -1149,7 +1149,11 @@ App.AnalysisTaskController = Ember.Controller.extend({
             self.set('configurations', ['A', 'B']);
             self.set('rootConfigurations', triggerable.get('acceptedRepositories').map(function (repository) {
                 var repositoryId = repository.get('id');
-                var options = [{value: ' ', label: 'None'}].concat(repositoryToRevisions[repositoryId]);
+                var options = [{label: 'None'}].concat((repositoryToRevisions[repositoryId] || []).map(function (option, index) {
+                    if (!option || !option['value'])
+                        return {value: '', label: analysisPoints[index].label + ': None'}; 
+                    return option;
+                }));
                 return Ember.Object.create({
                     repository: repository,
                     name: repository.get('name'),
@@ -1178,11 +1182,27 @@ App.AnalysisTaskController = Ember.Controller.extend({
         },
         createTestGroup: function (name, repetitionCount)
         {
+            var analysisTask = this.get('model');
+            if (analysisTask.get('testGroups').isAny('name', name)) {
+                alert('Cannot create two test groups of the same name.');
+                return;
+            }
+
             var roots = {};
-            this.get('rootConfigurations').map(function (root) {
-                roots[root.get('name')] = root.get('sets').map(function (item) { return item.get('selection').value; });
-            });
-            App.TestGroup.create(this.get('model'), name, roots, repetitionCount).then(function () {
+            var rootConfigurations = this.get('rootConfigurations').toArray();
+            for (var root of rootConfigurations) {
+                var sets = root.get('sets');
+                var hasSelection = function (item) { return item.get('selection') && item.get('selection').value; };
+                if (!sets.any(hasSelection))
+                    continue;
+                if (!sets.every(hasSelection)) {
+                    alert('Only some configuration specifies ' + root.get('name'));
+                    return;
+                }
+                roots[root.get('name')] = sets.map(function (item) { return item.get('selection').value; });                
+            }
+
+            App.TestGroup.create(analysisTask, name, roots, repetitionCount).then(function () {
             }, function (error) {
                 alert('Failed to create a new test group:' + error);
             });
@@ -1216,7 +1236,7 @@ App.AnalysisTaskController = Ember.Controller.extend({
                 var selectedOption;
                 if (targetRevision)
                     selectedOption = set.get('options').find(function (option) { return option.value == targetRevision; });
-                set.set('selection', selectedOption || sets[i].get('options')[0]);
+                set.set('selection', selectedOption || set.get('options')[0]);
             });
         });
 
index c7cd030..be4e7cc 100755 (executable)
     </script>
 
     <script type="text/x-handlebars" data-template-name="testGroupForm">
+    {{#if rootConfigurations}}
         <form method="POST" {{action "createTestGroup" newTestGroupName repetitionCount on="submit"}}>
             <section class="analysis-group">
                 <h1>{{input name="name" value=newTestGroupName placeholder="Test group name" required=true type="text"}}</h1>
                             <tr>
                                 <th>{{name}}</th>
                                 {{#each sets}}
-                                    <td>{{view Ember.Select name=name content=options disabled=true
+                                    <td>{{view Ember.Select name=name content=options
                                         optionValuePath="content.value" optionLabelPath="content.label"
                                         selection=selection}}</td>
                                 {{/each}}
                 <button type="submit">Start A/B testing</button>
             </section>
         </form>
+    {{/if}}
     </script>
 
 </head>
index 12adac2..8196472 100755 (executable)
@@ -72,7 +72,7 @@ def find_request_updates(configurations, lookback_count):
     for config in configurations:
         try:
             pending_builds = fetch_json(config['jsonURL'] + 'pendingBuilds')
-            scheduled_requests = filter(None, [request_id_from_build(build) for build in pending_builds])
+            scheduled_requests = filter(None, [request_id_from_build(config, build) for build in pending_builds])
             for request_id in scheduled_requests:
                 request_updates[request_id] = {'status': 'scheduled', 'url': config['url']}
             config['scheduledRequests'] = set(scheduled_requests)
@@ -85,7 +85,7 @@ def find_request_updates(configurations, lookback_count):
             build_index = -i
             try:
                 build = fetch_json(config['jsonURL'] + 'builds/%d' % build_index)
-                request_id = request_id_from_build(build)
+                request_id = request_id_from_build(config, build)
                 if not request_id:
                     continue
 
@@ -133,14 +133,25 @@ def find_stale_request_updates(configurations, open_requests, requests_on_buildb
 
 
 def schedule_request(config, request, root_sets):
-    replacements = root_sets.get(request['rootSet'], {})
-    replacements['buildRequest'] = request['id']
+    roots = root_sets.get(request['rootSet'], {})
+    replacements = roots.copy()
 
     payload = {}
     for property_name, property_value in config['arguments'].iteritems():
-        for key, value in replacements.iteritems():
-            property_value = property_value.replace('$' + key, value)
-        payload[property_name] = property_value
+        if not isinstance(property_value, dict):
+            payload[property_name] = property_value
+        elif 'root' in property_value:
+            payload[property_name] = replacements[property_value['root']]
+        elif 'rootsExcluding' in property_value:
+            excluded_roots = property_value['rootsExcluding']
+            filtered_roots = {}
+            for root_name in roots:
+                if root_name not in excluded_roots:
+                    filtered_roots[root_name] = roots[root_name]
+            payload[property_name] = json.dumps(filtered_roots)
+        else:
+            print >> sys.stderr, "Failed to process an argument %s: %s" % (property_name, property_value)
+    payload[config['buildRequestArgument']] = request['id']
 
     try:
         urllib2.urlopen(urllib2.Request(config['url'] + 'force'), urllib.urlencode(payload))
@@ -174,8 +185,8 @@ def property_value_from_build(build, name):
     return None
 
 
-def request_id_from_build(build):
-    job_id = property_value_from_build(build, 'jobid')
+def request_id_from_build(config, build):
+    job_id = property_value_from_build(build, config['buildRequestArgument'])
     return int(job_id) if job_id and job_id.isdigit() else None