Perf dashboard should show the results of A/B testing
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 12 Feb 2015 18:50:53 +0000 (18:50 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 12 Feb 2015 18:50:53 +0000 (18:50 +0000)
https://bugs.webkit.org/show_bug.cgi?id=141500

Reviewed by Chris Dumez.

Added the support for fetching test_runs for a specific test group in /api/runs/, and used it in the
analysis task page to fetch results for each test group.

Merged App.createChartData into App.Manifest.fetchRunsWithPlatformAndMetric so that App.BuildRequest
can use the formatter.

* public/api/runs.php:
(fetch_runs_for_config_and_test_group): Added.
(fetch_runs_for_config): Just return the fetched rows since main will format them with RunsGenerator.
(main): Use fetch_runs_for_config_and_test_group to fetch rows when a test group id is specified. Also
use RunsGenerator to format results.
(RunsGenerator): Added.
(RunsGenerator::__construct): Added.
(RunsGenerator::add_runs): Added.
(RunsGenerator::format_run): Moved.
(RunsGenerator::parse_revisions_array): Moved.

* public/v2/analysis.js:
(App.TestGroup): Fixed a typo. The property on a test group that refers to an analysis task is "task".
(App.TestGroup._fetchChartData): Added. Fetches all A/B testing results for this group.
(App.BuildRequest.configLetter): Renamed from config since this returns a letter that identifies the
configuration associated with this build request such as "A" and "B".
(App.BuildRequest.statusLabel): Added the missing label for failed build requests.
(App.BuildRequest.url): Added. Returns the URL associated with this build request.
(App.BuildRequest._meanFetched): Added. Retrieve the mean and the build number for this request via
_fetchChartData.

* public/v2/app.js:
(App.Pane._fetch): Set chartData directly here.
(App.Pane._updateMovingAverageAndEnvelope): Renamed from _computeChartData. No longer sets chartData
now that it's done in App.Pane._fetch.
(App.AnalysisTaskController._fetchedRuns): Updated per createChartData merge.

* public/v2/data.js:
(Measurement.prototype.buildId): Added.
(TimeSeries.prototype.findPointByBuild): Added.

* public/v2/index.html: Fixed a bug that build status URL was broken. We can't use link-to helper since
url is not an Ember routed path.

* public/v2/manifest.js:
(App.Manifest.fetchRunsWithPlatformAndMetric): Takes testGroupId as the third argument. Merged
App.createChartData here so that App.BuildRequest can use the formatter

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@180000 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/api/runs.php
Websites/perf.webkit.org/public/v2/analysis.js
Websites/perf.webkit.org/public/v2/app.js
Websites/perf.webkit.org/public/v2/data.js
Websites/perf.webkit.org/public/v2/index.html
Websites/perf.webkit.org/public/v2/manifest.js

index 9a95fd7..0f4d537 100644 (file)
@@ -1,5 +1,56 @@
 2015-02-12  Ryosuke Niwa  <rniwa@webkit.org>
 
+        Perf dashboard should show the results of A/B testing
+        https://bugs.webkit.org/show_bug.cgi?id=141500
+
+        Reviewed by Chris Dumez.
+
+        Added the support for fetching test_runs for a specific test group in /api/runs/, and used it in the
+        analysis task page to fetch results for each test group.
+
+        Merged App.createChartData into App.Manifest.fetchRunsWithPlatformAndMetric so that App.BuildRequest
+        can use the formatter.
+
+        * public/api/runs.php:
+        (fetch_runs_for_config_and_test_group): Added.
+        (fetch_runs_for_config): Just return the fetched rows since main will format them with RunsGenerator.
+        (main): Use fetch_runs_for_config_and_test_group to fetch rows when a test group id is specified. Also
+        use RunsGenerator to format results.
+        (RunsGenerator): Added.
+        (RunsGenerator::__construct): Added.
+        (RunsGenerator::add_runs): Added.
+        (RunsGenerator::format_run): Moved.
+        (RunsGenerator::parse_revisions_array): Moved.
+
+        * public/v2/analysis.js:
+        (App.TestGroup): Fixed a typo. The property on a test group that refers to an analysis task is "task".
+        (App.TestGroup._fetchChartData): Added. Fetches all A/B testing results for this group.
+        (App.BuildRequest.configLetter): Renamed from config since this returns a letter that identifies the
+        configuration associated with this build request such as "A" and "B".
+        (App.BuildRequest.statusLabel): Added the missing label for failed build requests.
+        (App.BuildRequest.url): Added. Returns the URL associated with this build request.
+        (App.BuildRequest._meanFetched): Added. Retrieve the mean and the build number for this request via
+        _fetchChartData.
+
+        * public/v2/app.js:
+        (App.Pane._fetch): Set chartData directly here.
+        (App.Pane._updateMovingAverageAndEnvelope): Renamed from _computeChartData. No longer sets chartData
+        now that it's done in App.Pane._fetch.
+        (App.AnalysisTaskController._fetchedRuns): Updated per createChartData merge.
+
+        * public/v2/data.js:
+        (Measurement.prototype.buildId): Added.
+        (TimeSeries.prototype.findPointByBuild): Added.
+
+        * public/v2/index.html: Fixed a bug that build status URL was broken. We can't use link-to helper since
+        url is not an Ember routed path.
+
+        * public/v2/manifest.js:
+        (App.Manifest.fetchRunsWithPlatformAndMetric): Takes testGroupId as the third argument. Merged
+        App.createChartData here so that App.BuildRequest can use the formatter
+
+2015-02-12  Ryosuke Niwa  <rniwa@webkit.org>
+
         v2 UI should adjust the number of ticks on dashboards based on screen size
         https://bugs.webkit.org/show_bug.cgi?id=141502
 
index d2e8664..3516de4 100644 (file)
@@ -2,50 +2,25 @@
 
 require('../include/json-header.php');
 
+function fetch_runs_for_config_and_test_group($db, $config, $test_group_id) {
+    return $db->query_and_fetch_all('
+        SELECT test_runs.*, builds.*, array_agg((commit_repository, commit_revision, commit_time)) AS revisions
+            FROM builds
+                LEFT OUTER JOIN build_commits ON commit_build = build_id
+                LEFT OUTER JOIN commits ON build_commit = commit_id,
+                test_runs, build_requests, analysis_test_groups
+            WHERE run_build = build_id AND run_config = $1 AND request_build = build_id AND request_group = $2
+            GROUP BY build_id, run_id', array($config['config_id'], $test_group_id));
+}
+
 function fetch_runs_for_config($db, $config) {
-    $raw_runs = $db->query_and_fetch_all('
+    return $db->query_and_fetch_all('
         SELECT test_runs.*, builds.*, array_agg((commit_repository, commit_revision, commit_time)) AS revisions
             FROM builds
                 LEFT OUTER JOIN build_commits ON commit_build = build_id
                 LEFT OUTER JOIN commits ON build_commit = commit_id, test_runs
             WHERE run_build = build_id AND run_config = $1 AND NOT EXISTS (SELECT * FROM build_requests WHERE request_build = build_id)
             GROUP BY build_id, run_id', array($config['config_id']));
-
-    $formatted_runs = array();
-    if (!$raw_runs)
-        return $formatted_runs;
-
-    foreach ($raw_runs as $run)
-        array_push($formatted_runs, format_run($run));
-
-    return $formatted_runs;
-}
-
-function parse_revisions_array($postgres_array) {
-    // e.g. {"(WebKit,131456,\"2012-10-16 14:53:00\")","(Chromium,162004,)"}
-    $outer_array = json_decode('[' . trim($postgres_array, '{}') . ']');
-    $revisions = array();
-    foreach ($outer_array as $item) {
-        $name_and_revision = explode(',', trim($item, '()'));
-        if (!$name_and_revision[0])
-            continue;
-        $time = strtotime(trim($name_and_revision[2], '"')) * 1000;
-        $revisions[trim($name_and_revision[0], '"')] = array(trim($name_and_revision[1], '"'), $time);
-    }
-    return $revisions;
-}
-
-function format_run($run) {
-    return array(
-        'id' => intval($run['run_id']),
-        'mean' => floatval($run['run_mean_cache']),
-        'iterationCount' => intval($run['run_iteration_count_cache']),
-        'sum' => floatval($run['run_sum_cache']),
-        'squareSum' => floatval($run['run_square_sum_cache']),
-        'revisions' => parse_revisions_array($run['revisions']),
-        'buildTime' => strtotime($run['build_time']) * 1000,
-        'buildNumber' => intval($run['build_number']),
-        'builder' => $run['build_builder']);
 }
 
 function main($path) {
@@ -60,11 +35,6 @@ function main($path) {
     if (!$db->connect())
         exit_with_error('DatabaseConnectionFailure');
 
-    // FIXME: We should support revalication as well as caching results in the server side.
-    $maxage = config('jsonCacheMaxAge');
-    header('Expires: ' . gmdate('D, d M Y H:i:s', time() + $maxage) . ' GMT');
-    header("Cache-Control: maxage=$maxage");
-
     $platform_id = intval($parts[0]);
     $metric_id = intval($parts[1]);
     $config_rows = $db->query_and_fetch_all('SELECT config_id, config_type, config_platform, config_metric
@@ -72,13 +42,73 @@ function main($path) {
     if (!$config_rows)
         exit_with_error('ConfigurationNotFound');
 
-    $results = array();
+    $test_group_id = array_get($_GET, 'testGroup');
+    if ($test_group_id)
+        $test_group_id = intval($test_group_id);
+    else {
+        // FIXME: We should support revalication as well as caching results in the server side.
+        $maxage = config('jsonCacheMaxAge');
+        header('Expires: ' . gmdate('D, d M Y H:i:s', time() + $maxage) . ' GMT');
+        header("Cache-Control: maxage=$maxage");
+    }
+
+    $generator = new RunsGenerator();
+
     foreach ($config_rows as $config) {
-        if ($runs = fetch_runs_for_config($db, $config))
-            $results[$config['config_type']] = $runs;
+        if ($test_group_id)
+            $raw_runs = fetch_runs_for_config_and_test_group($db, $config, $test_group_id);
+        else
+            $raw_runs = fetch_runs_for_config($db, $config);
+        $generator->add_runs($config['config_type'], $raw_runs);
+    }
+
+    exit_with_success($generator->results());
+}
+
+class RunsGenerator {
+    function __construct() {
+        $this->results = array();
     }
 
-    exit_with_success($results);
+    function &results() { return $this->results; }
+
+    function add_runs($name, $raw_runs) {
+        $formatted_runs = array();
+        if ($raw_runs) {
+            foreach ($raw_runs as $run)
+                array_push($formatted_runs, self::format_run($run));
+        }
+        $this->results[$name] = $formatted_runs;
+        return $formatted_runs;
+    }
+
+    private static function format_run($run) {
+        return array(
+            'id' => intval($run['run_id']),
+            'mean' => floatval($run['run_mean_cache']),
+            'iterationCount' => intval($run['run_iteration_count_cache']),
+            'sum' => floatval($run['run_sum_cache']),
+            'squareSum' => floatval($run['run_square_sum_cache']),
+            'revisions' => self::parse_revisions_array($run['revisions']),
+            'build' => $run['build_id'],
+            'buildTime' => strtotime($run['build_time']) * 1000,
+            'buildNumber' => intval($run['build_number']),
+            'builder' => $run['build_builder']);
+    }
+
+    private static function parse_revisions_array($postgres_array) {
+        // e.g. {"(WebKit,131456,\"2012-10-16 14:53:00\")","(Chromium,162004,)"}
+        $outer_array = json_decode('[' . trim($postgres_array, '{}') . ']');
+        $revisions = array();
+        foreach ($outer_array as $item) {
+            $name_and_revision = explode(',', trim($item, '()'));
+            if (!$name_and_revision[0])
+                continue;
+            $time = strtotime(trim($name_and_revision[2], '"')) * 1000;
+            $revisions[trim($name_and_revision[0], '"')] = array(trim($name_and_revision[1], '"'), $time);
+        }
+        return $revisions;
+    }
 }
 
 main(array_key_exists('PATH_INFO', $_SERVER) ? explode('/', trim($_SERVER['PATH_INFO'], '/')) : array());
index 067d0d6..82e045f 100644 (file)
@@ -66,7 +66,7 @@ App.BugAdapter = DS.RESTAdapter.extend({
 });
 
 App.TestGroup = App.NameLabelModel.extend({
-    analysisTask: DS.belongsTo('analysisTask'),
+    task: DS.belongsTo('analysisTask'),
     author: DS.attr('string'),
     createdAt: DS.attr('date'),
     buildRequests: DS.hasMany('buildRequests'),
@@ -80,6 +80,21 @@ App.TestGroup = App.NameLabelModel.extend({
         });
         return rootSetIds;
     }.property('buildRequests'),
+    _fetchChartData: function ()
+    {
+        var task = this.get('task');
+        if (!task)
+            return null;
+        var self = this;
+        return App.Manifest.fetchRunsWithPlatformAndMetric(this.store,
+            task.get('platform').get('id'), task.get('metric').get('id'), this.get('id')).then(
+            function (result) { self.set('chartData', result.data); },
+            function (error) {
+                // FIXME: Somehow this never gets called.
+                alert('Failed to fetch the results:' + error);
+                return null;
+            });
+    }.observes('task', 'task.platform', 'task.metric').on('init'),
 });
 
 App.TestGroup.create = function (analysisTask, name, rootSets, repetitionCount)
@@ -130,7 +145,7 @@ App.BuildRequest = App.Model.extend({
         return this.get('order') + 1;
     }.property('order'),
     rootSet: DS.attr('number'),
-    config: function ()
+    configLetter: function ()
     {
         var rootSets = this.get('testGroup').get('rootSets');
         var index = rootSets.indexOf(this.get('rootSet'));
@@ -146,9 +161,27 @@ App.BuildRequest = App.Model.extend({
             return 'Scheduled';
         case 'running':
             return 'Running';
+        case 'failed':
+            return 'Failed';
         case 'completed':
             return 'Finished';
         }
     }.property('status'),
+    url: DS.attr('string'),
     build: DS.attr('number'),
+    _fetchMean: function ()
+    {
+        var testGroup = this.get('testGroup');
+        if (!testGroup)
+            return;
+        var chartData = testGroup.get('chartData');
+        if (!chartData)
+            return;
+
+        var point = chartData.current.findPointByBuild(this.get('build'));
+        if (!point)
+            return;
+        this.set('mean', chartData.formatter(point.value) + (chartData.unit ? ' ' + chartData.unit : ''));
+        this.set('buildNumber', point.measurement.buildNumber());
+    }.observes('build', 'testGroup', 'testGroup.chartData').on('init'),
 });
index 4b065ad..d3e36f9 100755 (executable)
@@ -352,8 +352,8 @@ App.Pane = Ember.Object.extend({
             App.Manifest.fetchRunsWithPlatformAndMetric(this.get('store'), platformId, metricId).then(function (result) {
                 self.set('platform', result.platform);
                 self.set('metric', result.metric);
-                self.set('fetchedData', result);
-                self._computeChartData();
+                self.set('chartData', result.data);
+                self._updateMovingAverageAndEnvelope();
             }, function (result) {
                 if (!result || typeof(result) === "string")
                     self.set('failure', 'Failed to fetch the JSON with an error: ' + result);
@@ -471,13 +471,12 @@ App.Pane = Ember.Object.extend({
 
         return chosenStrategy;
     },
-    _computeChartData: function ()
+    _updateMovingAverageAndEnvelope: function ()
     {
-        if (!this.get('fetchedData'))
+        var chartData = this.get('chartData');
+        if (!chartData)
             return;
 
-        var chartData = App.createChartData(this.get('fetchedData'));
-
         var movingAverageStrategy = this.get('chosenMovingAverageStrategy');
         this._updateStrategyConfigIfNeeded(movingAverageStrategy, 'movingAverageConfig');
 
@@ -485,8 +484,6 @@ App.Pane = Ember.Object.extend({
         this._updateStrategyConfigIfNeeded(envelopingStrategy, 'envelopingConfig');
 
         chartData.movingAverage = this._computeMovingAverageAndOutliers(chartData, movingAverageStrategy, envelopingStrategy);
-
-        this.set('chartData', chartData);
     }.observes('chosenMovingAverageStrategy', 'chosenMovingAverageStrategy.parameterList.@each.value',
         'chosenEnvelopingStrategy', 'chosenEnvelopingStrategy.parameterList.@each.value'),
     _computeMovingAverageAndOutliers: function (chartData, movingAverageStrategy, envelopingStrategy)
@@ -543,20 +540,6 @@ App.Pane = Ember.Object.extend({
     },
 });
 
-App.createChartData = function (data)
-{
-    var runs = data.runs;
-    return {
-        current: runs.current.timeSeriesByCommitTime(),
-        baseline: runs.baseline ? runs.baseline.timeSeriesByCommitTime() : null,
-        target: runs.target ? runs.target.timeSeriesByCommitTime() : null,
-        unit: data.unit,
-        formatter: data.useSI ? d3.format('.4s') : d3.format('.4g'),
-        deltaFormatter: data.useSI ? d3.format('+.2s') : d3.format('+.2g'),
-        smallerIsBetter: data.smallerIsBetter,
-    };
-}
-
 App.encodePrettifiedJSON = function (plain)
 {
     function numberIfPossible(string) {
@@ -1027,11 +1010,10 @@ App.AnalysisTaskController = Ember.Controller.extend({
             });
         }));
     },
-    _fetchedRuns: function (data)
+    _fetchedRuns: function (result)
     {
-        var runs = data.runs;
-
-        var currentTimeSeries = runs.current.timeSeriesByCommitTime();
+        var chartData = result.data;
+        var currentTimeSeries = chartData.current;
         if (!currentTimeSeries)
             return; // FIXME: Report an error.
 
@@ -1044,13 +1026,12 @@ App.AnalysisTaskController = Ember.Controller.extend({
         highlightedItems[start.measurement.id()] = true;
         highlightedItems[end.measurement.id()] = true;
 
-        var chartData = App.createChartData(data);
         var formatedPoints = currentTimeSeries.seriesBetweenPoints(start, end).map(function (point, index) {
             return {
                 id: point.measurement.id(),
                 measurement: point.measurement,
                 label: 'Point ' + (index + 1),
-                value: chartData.formatter(point.value) + (data.unit ? ' ' + data.unit : ''),
+                value: chartData.formatter(point.value) + (chartData.unit ? ' ' + chartData.unit : ''),
             };
         });
 
index 208575e..dd86d95 100755 (executable)
@@ -242,6 +242,11 @@ Measurement.prototype.latestCommitTime = function()
     return this._latestCommitTime || this._buildTime;
 }
 
+Measurement.prototype.buildId = function()
+{
+    return this._raw['build'];
+}
+
 Measurement.prototype.buildNumber = function ()
 {
     return this._raw['buildNumber'];
@@ -315,10 +320,13 @@ RunsData.prototype.timeSeriesByBuildTime = function ()
 
 // FIXME: We need to devise a way to fetch runs in multiple chunks so that
 // we don't have to fetch the entire time series to just show the last 3 days.
-RunsData.fetchRuns = function (platformId, metricId)
+RunsData.fetchRuns = function (platformId, metricId, testGroupId)
 {
     var filename = platformId + '-' + metricId + '.json';
 
+    if (testGroupId)
+        filename += '?testGroup=' + testGroupId;
+
     return new Ember.RSVP.Promise(function (resolve, reject) {
         $.getJSON('../api/runs/' + filename, function (data) {
             if (data.status != 'OK') {
@@ -355,6 +363,11 @@ function TimeSeries(series)
     this._max = max;
 }
 
+TimeSeries.prototype.findPointByBuild = function (buildId)
+{
+    return this._series.find(function (point) { return point.measurement.buildId() == buildId; })
+}
+
 TimeSeries.prototype.findPointByMeasurementId = function (measurementId)
 {
     return this._series.find(function (point) { return point.measurement.id() == measurementId; });
index ced6008..30f4078 100755 (executable)
                             {{#each buildRequests}}
                                 <tr>
                                     <td>{{orderLabel}}</td>
-                                    <td>{{config}}</td>
-                                    <td>{{#if url}}{{#link-to url}}{{statusLabel}}{{/link-to}}{{else}}{{statusLabel}}{{/if}}</td>
-                                    <td>{{build}}</td>
+                                    <td>{{configLetter}}</td>
+                                    <td><a {{bind-attr href=url}}>{{statusLabel}}</a></td>
+                                    <td>{{buildNumber}}</td>
                                     <td>{{mean}}</td>
                                 </tr>
                             {{/each}}
index 5de4649..87d803e 100755 (executable)
@@ -279,10 +279,10 @@ App.Manifest = Ember.Controller.extend({
         dashboards.forEach(function (dashboard) { self._dashboardByName[dashboard.get('name')] = dashboard; });
         this._defaultDashboardName = dashboards.length ? dashboards[0].get('name') : null;
     },
-    fetchRunsWithPlatformAndMetric: function (store, platformId, metricId)
+    fetchRunsWithPlatformAndMetric: function (store, platformId, metricId, testGroupId)
     {
         return Ember.RSVP.all([
-            RunsData.fetchRuns(platformId, metricId),
+            RunsData.fetchRuns(platformId, metricId, testGroupId),
             this.fetch(store),
         ]).then(function (values) {
             var runs = values[0];
@@ -301,7 +301,20 @@ App.Manifest = Ember.Controller.extend({
             }[suffix];
             var smallerIsBetter = unit != 'fps' && unit != '/s'; // Assume smaller is better for unit-less metrics.
 
-            return {platform: platform, metric: metric, runs: runs, unit: unit, useSI: unit == 'bytes', smallerIsBetter: smallerIsBetter};
+            var useSI = unit == 'bytes';
+            return {
+                platform: platform,
+                metric: metric,
+                data: {
+                    current: runs.current.timeSeriesByCommitTime(),
+                    baseline: runs.baseline ? runs.baseline.timeSeriesByCommitTime() : null,
+                    target: runs.target ? runs.target.timeSeriesByCommitTime() : null,
+                    unit: unit,
+                    formatter: useSI ? d3.format('.4s') : d3.format('.4g'),
+                    deltaFormatter: useSI ? d3.format('+.2s') : d3.format('+.2g'),
+                    smallerIsBetter: smallerIsBetter,
+                }
+            };
         });
     },
 }).create();