https://bugs.webkit.org/show_bug.cgi?id=154090
Reviewed by Chris Dumez.
Added a button to re-try an existing A/B testing group with a custom repetition count. The same button functions
as a way of confirming the progression/regression when there have been no A/B testing scheduled in the task.
Also fixed the bug that A/B testing groups that have been waiting for other test groups will be shown as "running".
* public/v3/components/results-table.js:
(ResultsTable.cssTemplate): Don't pad the list of extra repositories when it's empty.
* public/v3/components/test-group-results-table.js:
(TestGroupResultsTable.prototype.buildRowGroups): Use TestGroup.labelForRootSet instead of manually
computing the letter for each configuration set.
* public/v3/models/build-request.js:
(BuildRequest.prototype.hasStarted): Added.
* public/v3/models/data-model.js:
(DataModelObject.ensureSingleton): Added.
(DataModelObject.cachedFetch): Added noCache option. This is used when re-fetching the test groups after
creating one.
* public/v3/models/measurement-cluster.js:
(MeasurementCluster.prototype.startTime): Added.
* public/v3/models/measurement-set.js:
(MeasurementSet.prototype.hasFetchedRange): Added. Returns true only if there are no "holes" (cluster
yet to be fetched) between the specified time range. This was added to fix a bug in AnalysisTaskPage's
_didFetchMeasurement.
* public/v3/models/test-group.js:
(TestGroup): Added this._rootSetToLabel.
(TestGroup.prototype.addBuildRequest): Reset this._rootSetToLabel along with this._requestedRootSets.
(TestGroup.prototype.repetitionCount): Added. Returns the number of iterations executed per set. We assume that
every root set in the test group shares a single repetition count.
(TestGroup.prototype.requestedRootSets): Now populates this._rootSetToLabel for labelForRootSet.
(TestGroup.prototype.labelForRootSet): Added.
(TestGroup.prototype.hasStarted): Added.
(TestGroup.prototype.compareTestResults): Use 'running' and 'pending' to differentiate test groups that are waiting
for other groups to finish running from the ones that are actually running ('incomplete' before this patch).
(TestGroup.fetchByTask):
(TestGroup.createAndRefetchTestGroups): Added. Creates a new test group using the privileged-api/create-test-group
and fetches the list of test groups for the specified analysis task.
(TestGroup._createModelsFromFetchedTestGroups): Extracted from TestGroup.fetchByTask.
* public/v3/pages/analysis-task-page.js:
(AnalysisTaskPage): Initialize _renderedCurrentTestGroup to undefined so that we'd always can differentiate
the initial call to AnalysisTaskPage.render and subsequent calls in which it's identical to _currentTestGroup.
(AnalysisTaskPage.prototype._didFetchMeasurement): Fixed a bug that we don't exit early even when some
clusters in between startPoint and endPoint are still being fetched via newly added hasFetchedRange.
(AnalysisTaskPage.prototype.render): Update the default repetition count based on the current test group.
Also update the label of the button to "Confirm the change" if there is no A/B testing in this task.
(AnalysisTaskPage.prototype._retryCurrentTestGroup): Added. Re-triggers an existing A/B testing group or creates
the A/B testing for the entire range of the analysis task.
(AnalysisTaskPage.prototype._hasDuplicateTestGroupName): Added.
(AnalysisTaskPage.prototype._createRetryNameForTestGroup): Added.
(AnalysisTaskPage.htmlTemplate): Added form controls to re-trigger A/B testing.
(AnalysisTaskPage.cssTemplate): Updated the style.
git-svn-id: http://svn.webkit.org/repository/webkit/trunk@196440
268f45cc-cd09-0410-ab3c-
d52691b4dbfc
2016-02-10 Ryosuke Niwa <rniwa@webkit.org>
+ Perf dashboard should have UI to retry A/B testing
+ https://bugs.webkit.org/show_bug.cgi?id=154090
+
+ Reviewed by Chris Dumez.
+
+ Added a button to re-try an existing A/B testing group with a custom repetition count. The same button functions
+ as a way of confirming the progression/regression when there have been no A/B testing scheduled in the task.
+
+ Also fixed the bug that A/B testing groups that have been waiting for other test groups will be shown as "running".
+
+ * public/v3/components/results-table.js:
+ (ResultsTable.cssTemplate): Don't pad the list of extra repositories when it's empty.
+
+ * public/v3/components/test-group-results-table.js:
+ (TestGroupResultsTable.prototype.buildRowGroups): Use TestGroup.labelForRootSet instead of manually
+ computing the letter for each configuration set.
+
+ * public/v3/models/build-request.js:
+ (BuildRequest.prototype.hasStarted): Added.
+
+ * public/v3/models/data-model.js:
+ (DataModelObject.ensureSingleton): Added.
+ (DataModelObject.cachedFetch): Added noCache option. This is used when re-fetching the test groups after
+ creating one.
+
+ * public/v3/models/measurement-cluster.js:
+ (MeasurementCluster.prototype.startTime): Added.
+
+ * public/v3/models/measurement-set.js:
+ (MeasurementSet.prototype.hasFetchedRange): Added. Returns true only if there are no "holes" (cluster
+ yet to be fetched) between the specified time range. This was added to fix a bug in AnalysisTaskPage's
+ _didFetchMeasurement.
+
+ * public/v3/models/test-group.js:
+ (TestGroup): Added this._rootSetToLabel.
+ (TestGroup.prototype.addBuildRequest): Reset this._rootSetToLabel along with this._requestedRootSets.
+ (TestGroup.prototype.repetitionCount): Added. Returns the number of iterations executed per set. We assume that
+ every root set in the test group shares a single repetition count.
+ (TestGroup.prototype.requestedRootSets): Now populates this._rootSetToLabel for labelForRootSet.
+ (TestGroup.prototype.labelForRootSet): Added.
+ (TestGroup.prototype.hasStarted): Added.
+ (TestGroup.prototype.compareTestResults): Use 'running' and 'pending' to differentiate test groups that are waiting
+ for other groups to finish running from the ones that are actually running ('incomplete' before this patch).
+ (TestGroup.fetchByTask):
+ (TestGroup.createAndRefetchTestGroups): Added. Creates a new test group using the privileged-api/create-test-group
+ and fetches the list of test groups for the specified analysis task.
+ (TestGroup._createModelsFromFetchedTestGroups): Extracted from TestGroup.fetchByTask.
+
+ * public/v3/pages/analysis-task-page.js:
+ (AnalysisTaskPage): Initialize _renderedCurrentTestGroup to undefined so that we'd always can differentiate
+ the initial call to AnalysisTaskPage.render and subsequent calls in which it's identical to _currentTestGroup.
+ (AnalysisTaskPage.prototype._didFetchMeasurement): Fixed a bug that we don't exit early even when some
+ clusters in between startPoint and endPoint are still being fetched via newly added hasFetchedRange.
+ (AnalysisTaskPage.prototype.render): Update the default repetition count based on the current test group.
+ Also update the label of the button to "Confirm the change" if there is no A/B testing in this task.
+ (AnalysisTaskPage.prototype._retryCurrentTestGroup): Added. Re-triggers an existing A/B testing group or creates
+ the A/B testing for the entire range of the analysis task.
+ (AnalysisTaskPage.prototype._hasDuplicateTestGroupName): Added.
+ (AnalysisTaskPage.prototype._createRetryNameForTestGroup): Added.
+ (AnalysisTaskPage.htmlTemplate): Added form controls to re-trigger A/B testing.
+ (AnalysisTaskPage.cssTemplate): Updated the style.
+
+2016-02-10 Ryosuke Niwa <rniwa@webkit.org>
+
Removed the duplicated definition of ChartPaneBase.
* public/v3/components/chart-pane-base.js:
font-size: 0.8rem;
}
+ .results-table-extra-repositories:empty {
+ padding: 0;
+ }
+
.results-table-extra-repositories li {
display: inline;
}
return [];
var rootSets = this._testGroup.requestedRootSets();
- var groups = rootSets.map(function (rootSet, setIndex) {
+ var groups = rootSets.map(function (rootSet) {
var rows = [new ResultsTableRow('Mean', rootSet)];
var results = [];
if (!isNaN(aggregatedResult.value))
rows[0].setResult(aggregatedResult);
- return {heading: String.fromCharCode('A'.charCodeAt(0) + setIndex), rows:rows};
+ return {heading: testGroup.labelForRootSet(rootSet), rows:rows};
});
var comparisonRows = [];
for (var i = 0; i < rootSets.length; i++) {
for (var j = i + 1; j < rootSets.length; j++) {
- var startConfig = String.fromCharCode('A'.charCodeAt(0) + i);
- var endConfig = String.fromCharCode('A'.charCodeAt(0) + j);
+ var startConfig = testGroup.labelForRootSet(rootSets[i]);
+ var endConfig = testGroup.labelForRootSet(rootSets[j]);
var result = this._testGroup.compareTestResults(rootSets[i], rootSets[j]);
- if (result.status == 'incomplete' || result.status == 'failed')
+ if (result.status == 'pending' || result.status == 'running' || result.status == 'failed')
continue;
var row = new ResultsTableRow(`${startConfig} to ${endConfig}`, null);
}
}
- groups.push({heading: '', rows: comparisonRows});
+ groups.unshift({heading: '', rows: comparisonRows});
return groups;
}
rootSet() { return this._rootSet; }
hasCompleted() { return this._status == 'failed' || this._status == 'completed'; }
+ hasStarted() { return this._status != 'pending'; }
statusLabel()
{
switch (this._status) {
}
id() { return this._id; }
+ static ensureSingleton(id, object)
+ {
+ var singleton = this.findById(id);
+ if (singleton)
+ return singleton;
+ return new (this)(id, object);
+ }
+
static namedStaticMap(name)
{
var staticMap = this[DataModelObject.StaticMapSymbol];
return list;
}
- static cachedFetch(path, params)
+ static cachedFetch(path, params, noCache)
{
var query = [];
if (params) {
if (query.length)
path += '?' + query.join('&');
+ if (noCache)
+ return getJSONWithStatus(path);
+
var cacheMap = this.ensureNamedStaticMap(DataModelObject.CacheMapSymbol);
if (!cacheMap[path])
cacheMap[path] = getJSONWithStatus(path);
}
startTime() { return this._response['startTime']; }
+ endTime() { return this._response['endTime']; }
addToSeries(series, configType, includeOutliers, idMap)
{
});
}
+ hasFetchedRange(startTime, endTime)
+ {
+ console.assert(startTime < endTime);
+ var hasHole = false;
+ var previousEndTime = null;
+ for (var cluster of this._sortedClusters) {
+ if (cluster.startTime() < startTime && startTime < cluster.endTime())
+ hasHole = false;
+ if (previousEndTime !== null && previousEndTime != cluster.startTime())
+ hasHole = true;
+ if (cluster.startTime() < endTime && endTime < cluster.endTime())
+ break;
+ previousEndTime = cluster.endTime();
+ }
+ return !hasHole;
+ }
+
fetchedTimeSeries(configType, includeOutliers, extendToFuture)
{
Instrumentation.startMeasuringTime('MeasurementSet', 'fetchedTimeSeries');
this._requestsAreInOrder = false;
this._repositories = null;
this._requestedRootSets = null;
+ this._rootSetToLabel = new Map;
this._allRootSets = null;
console.assert(!object.platform || object.platform instanceof Platform);
this._platform = object.platform;
this._buildRequests.push(request);
this._requestsAreInOrder = false;
this._requestedRootSets = null;
+ this._rootSetToLabel = null;
+ }
+
+ repetitionCount()
+ {
+ if (!this._buildRequests.length)
+ return 0;
+ var rootSet = this._buildRequests[0].rootSet();
+ var count = 0;
+ for (var request of this._buildRequests) {
+ if (request.rootSet() == rootSet)
+ count++;
+ }
+ return count;
}
requestedRootSets()
this._requestedRootSets.push(set);
}
this._requestedRootSets.sort(function (a, b) { return a.latestCommitTime() - b.latestCommitTime(); });
+ var setIndex = 0;
+ for (var set of this._requestedRootSets) {
+ this._rootSetToLabel.set(set, String.fromCharCode('A'.charCodeAt(0) + setIndex));
+ setIndex++;
+ }
+
}
return this._requestedRootSets;
}
return this._buildRequests.filter(function (request) { return request.rootSet() == rootSet; });
}
+ labelForRootSet(rootSet)
+ {
+ console.assert(this._requestedRootSets);
+ return this._rootSetToLabel.get(rootSet);
+ }
+
_orderBuildRequests()
{
if (this._requestsAreInOrder)
return this._buildRequests.every(function (request) { return request.hasCompleted(); });
}
+ hasStarted()
+ {
+ return this._buildRequests.some(function (request) { return request.hasStarted(); });
+ }
+
compareTestResults(rootSetA, rootSetB)
{
var beforeValues = this._valuesForRootSet(rootSetA);
}
if (!this.hasCompleted()) {
- result.status = 'incomplete';
- result.label = 'Running';
- result.fullLabel = 'Running';
+ if (this.hasStarted()) {
+ result.status = 'running';
+ result.label = 'Running';
+ result.fullLabel = 'Running';
+ } else {
+ result.status = 'pending';
+ result.label = 'Pending';
+ result.fullLabel = 'Pending';
+ }
} else if (result.changeType) {
var significance = result.isStatisticallySignificant ? 'significant' : 'insignificant';
result.fullLabel = `${result.label} (statistically ${significance})`;
return values;
}
- static fetchByTask(taskId)
+ static createAndRefetchTestGroups(task, name, repetitionCount, rootSets)
{
- return this.cachedFetch('../api/test-groups', {task: taskId}).then(function (data) {
- var testGroups = data['testGroups'].map(function (row) {
- row.platform = Platform.findById(row.platform);
- return new TestGroup(row.id, row);
- });
-
- var rootIdMap = {};
- for (var root of data['roots'])
- rootIdMap[root.id] = root;
-
- var rootSets = data['rootSets'].map(function (row) {
- row.roots = row.roots.map(function (rootId) { return rootIdMap[rootId]; });
- row.testGroup = RootSet.findById(row.testGroup);
- return new RootSet(row.id, row);
- });
-
- var buildRequests = data['buildRequests'].map(function (rawData) {
- rawData.testGroup = TestGroup.findById(rawData.testGroup);
- rawData.rootSet = RootSet.findById(rawData.rootSet);
- return new BuildRequest(rawData.id, rawData);
- });
-
- return testGroups;
+ var self = this;
+ return PrivilegedAPI.sendRequest('create-test-group', {
+ task: task.id(),
+ name: name,
+ repetitionCount: repetitionCount,
+ rootSets: rootSets,
+ }).then(function (data) {
+ return self.cachedFetch('../api/test-groups', {task: task.id()}, true).then(self._createModelsFromFetchedTestGroups.bind(self));
});
}
+ static fetchByTask(taskId)
+ {
+ return this.cachedFetch('../api/test-groups', {task: taskId}).then(this._createModelsFromFetchedTestGroups.bind(this));
+ }
+
+ static _createModelsFromFetchedTestGroups(data)
+ {
+ var testGroups = data['testGroups'].map(function (row) {
+ row.platform = Platform.findById(row.platform);
+ return TestGroup.ensureSingleton(row.id, row);
+ });
+
+ var rootIdMap = {};
+ for (var root of data['roots'])
+ rootIdMap[root.id] = root;
+
+ var rootSets = data['rootSets'].map(function (row) {
+ row.roots = row.roots.map(function (rootId) { return rootIdMap[rootId]; });
+ row.testGroup = RootSet.findById(row.testGroup);
+ return RootSet.ensureSingleton(row.id, row);
+ });
+
+ var buildRequests = data['buildRequests'].map(function (rawData) {
+ rawData.testGroup = TestGroup.findById(rawData.testGroup);
+ rawData.rootSet = RootSet.findById(rawData.rootSet);
+ return BuildRequest.ensureSingleton(rawData.id, rawData);
+ });
+
+ return testGroups;
+ }
}
this._task = null;
this._testGroups = null;
this._renderedTestGroups = null;
- this._renderedCurrentTestGroup = null;
+ this._renderedCurrentTestGroup = undefined;
this._analysisResults = null;
this._measurementSet = null;
this._startPoint = null;
this._analysisResultsViewer = this.content().querySelector('analysis-results-viewer').component();
this._analysisResultsViewer.setTestGroupCallback(this._showTestGroup.bind(this));
this._testGroupResultsTable = this.content().querySelector('test-group-results-table').component();
+
+ this.content().querySelector('.test-group-retry-form').onsubmit = this._retryCurrentTestGroup.bind(this);
}
title() { return this._task ? this._task.label() : 'Analysis Task'; }
var series = this._measurementSet.fetchedTimeSeries('current', false, false);
var startPoint = series.findById(this._task.startMeasurementId());
var endPoint = series.findById(this._task.endMeasurementId());
- if (!startPoint || !endPoint)
+ if (!startPoint || !endPoint || !this._measurementSet.hasFetchedRange(startPoint.time, endPoint.time))
return;
this._analysisResultsViewer.setPoints(startPoint, endPoint);
var v2URL = `/v2/#/analysis/task/${this._taskId}`;
this.content().querySelector('.error-message').innerHTML +=
- `<p>This page is read only for now. To schedule a new A/B testing job, use <a href="${v2URL}">v2 page</a>.</p>`;
+ `<p>To schedule a custom A/B testing, use <a href="${v2URL}">v2 UI</a>.</p>`;
this._chartPane.render();
}));
this._renderedCurrentTestGroup = null;
}
- if (this._renderedCurrentTestGroup != this._currentTestGroup) {
+
+ if (this._renderedCurrentTestGroup !== this._currentTestGroup) {
if (this._renderedCurrentTestGroup) {
var element = this.content().querySelector('.test-group-list-' + this._renderedCurrentTestGroup.id());
if (element)
if (element)
element.classList.add('selected');
}
+
+ this.content().querySelector('.test-group-retry-button').textContent = this._currentTestGroup ? 'Retry' : 'Confirm the change';
+
+ var repetitionCount = this._currentTestGroup ? this._currentTestGroup.repetitionCount() : 4;
+ var repetitionCountController = this.content().querySelector('.test-group-retry-repetition-count');
+ repetitionCountController.value = repetitionCount;
+
this._renderedCurrentTestGroup = this._currentTestGroup;
}
+ this.content().querySelector('.test-group-retry-button').disabled = !(this._currentTestGroup || this._startPoint);
+
this._testGroupResultsTable.render();
Instrumentation.endMeasuringTime('AnalysisTaskPage', 'render');
this.render();
}
+ _retryCurrentTestGroup(event)
+ {
+ event.preventDefault();
+ console.assert(this._currentTestGroup || this._startPoint);
+
+ var testGroupName;
+ var rootSetList;
+ var rootSetLabels;
+
+ if (this._currentTestGroup) {
+ var testGroup = this._currentTestGroup;
+ testGroupName = this._createRetryNameForTestGroup(testGroup.name());
+ rootSetList = testGroup.requestedRootSets();
+ rootSetLabels = rootSetList.map(function (rootSet) { return testGroup.labelForRootSet(rootSet); });
+ } else {
+ testGroupName = 'Confirming the change';
+ rootSetList = [this._startPoint.rootSet(), this._endPoint.rootSet()];
+ rootSetLabels = ['Point 0', `Point ${this._endPoint.seriesIndex - this._startPoint.seriesIndex}`];
+ }
+
+ var rootSetsByName = {};
+ for (var repository of rootSetList[0].repositories())
+ rootSetsByName[repository.name()] = [];
+
+ var setIndex = 0;
+ for (var rootSet of rootSetList) {
+ for (var repository of rootSet.repositories()) {
+ var list = rootSetsByName[repository.name()];
+ if (!list) {
+ alert(`Set ${rootSetLabels[setIndex]} specifies ${repository.label()} but set ${rootSetLabels[0]} does not.`);
+ return null;
+ }
+ list.push(rootSet.commitForRepository(repository).revision());
+ }
+ setIndex++;
+ for (var name in rootSetsByName) {
+ var list = rootSetsByName[name];
+ if (list.length < setIndex) {
+ alert(`Set ${rootSetLabels[0]} specifies ${repository.label()} but set ${rootSetLabels[setIndex]} does not.`);
+ return null;
+ }
+ }
+ }
+
+ var repetitionCount = this.content().querySelector('.test-group-retry-repetition-count').value;
+
+ TestGroup.createAndRefetchTestGroups(this._task, testGroupName, repetitionCount, rootSetsByName)
+ .then(this._didFetchTestGroups.bind(this), function (error) {
+ alert('Failed to create a new test group: ' + error);
+ });
+ }
+
+ _createRetryNameForTestGroup(name)
+ {
+ var nameWithNumberMatch = name.match(/(.+?)\s*\(\s*(\d+)\s*\)\s*$/);
+ var number = 1;
+ if (nameWithNumberMatch) {
+ name = nameWithNumberMatch[1];
+ number = parseInt(nameWithNumberMatch[2]);
+ }
+
+ var newName;
+ do {
+ number++;
+ newName = `${name} (${number})`;
+ } while (this._hasDuplicateTestGroupName(newName));
+
+ return newName;
+ }
+
+ _hasDuplicateTestGroupName(name)
+ {
+ console.assert(this._testGroups);
+ for (var group of this._testGroups) {
+ if (group.name() == name)
+ return true;
+ }
+ return false;
+ }
+
static htmlTemplate()
{
return `
</section>
<section class="test-group-view">
<ul class="test-group-list"></ul>
- <div class="test-group-details"><test-group-results-table></test-group-results-table></div>
+ <div class="test-group-details">
+ <test-group-results-table></test-group-results-table>
+ <form class="test-group-retry-form">
+ <button class="test-group-retry-button" type="submit">Retry</button>
+ with
+ <select class="test-group-retry-repetition-count">
+ <option>1</option>
+ <option>2</option>
+ <option>3</option>
+ <option>4</option>
+ <option>5</option>
+ <option>6</option>
+ <option>7</option>
+ <option>8</option>
+ <option>9</option>
+ <option>10</option>
+ </select>
+ iterations per set
+ </form>
+ </div>
</section>
</div>
</div>
.test-group-view {
display: table;
margin: 0 1rem;
+ margin-bottom: 2rem;
}
.test-group-details {
display: table-cell;
margin-bottom: 1rem;
+ padding: 0;
+ margin: 0;
}
- .test-group-list {
- display: table-cell;
+ .test-group-retry-form {
+ padding: 0;
+ margin: 0.5rem;
}
- .test-group-list:not(:empty) {
+ .test-group-list {
+ display: table-cell;
margin: 0;
padding: 0.2rem 0;
list-style: none;
white-space: nowrap;
}
+ .test-group-list:empty {
+ margin: 0;
+ padding: 0;
+ border-right: none;
+ }
+
.test-group-list li {
display: block;
}
git://git.webkit.org / WebKit.git / commitdiff