Update syncing script to be able to build binary for commit set with owned commits.
authordewei_zhu@apple.com <dewei_zhu@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 29 Sep 2017 07:34:41 +0000 (07:34 +0000)
committerdewei_zhu@apple.com <dewei_zhu@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 29 Sep 2017 07:34:41 +0000 (07:34 +0000)
https://bugs.webkit.org/show_bug.cgi?id=177225

Reviewed by Ryosuke Niwa.

Added support for syncing script to be able to schedule builds to build binary for owned commits.
Introduces 'ifRepositorySet' and 'ownedRevisions' in 'buildProperties'.
'ifRepositorySet' will conditionaly set a build property if at least one of the repositories it specified requires build.
'ownedRevisions' specifies owned commits revision informations.

* public/v3/models/commit-set.js:
(CommitSet): Added '_ownerRepositoryToOwnedRepositoriesMap'.
(CommitSet.prototype.updateSingleton): Reset '_ownerRepositoryToOwnedRepositoriesMap'.
(CommitSet.prototype._updateFromObject): Only update '_repositoryToCommitOwnerMap' and '_ownerRepositoryToOwnedRepositoriesMap' when 'commitOwner' exists.
(CommitSet.prototype.ownerCommitForRepository): Returns a sorted list of top level repositories.
(CommitSet.prototype.ownedRepositoriesForOwnerRepository): Returns owned repositories given a owner repository.
(CustomCommitSet.prototype.ownerCommitForRepository): Returns a sorted list of top level repositories.
* public/v3/models/triggerable.js:
(prototype.accepts): It should only check against top-level repositories. Removed a deprecated 'FIXME'.
* server-tests/tools-sync-buildbot-integration-tests.js: Added unit test for building owned commits binary.
(createTriggerable): Added conditional 'ifRepositorySet' and 'ownedRevisions' in the repository groups.
* tools/js/buildbot-syncer.js:
(BuildbotSyncer.prototype._propertiesForBuildRequest):
    Added logic to conditionaly create build property for 'ifRepositorySet'.
    Added logic to create 'ownedRevisions' based on the owner repositories it specified.
(BuildbotSyncer._parseRepositoryGroup): Build property template should be able to handle 'ifRepositorySet' and 'ownedRevisions'.
* unit-tests/buildbot-syncer-tests.js: Added unit tests for 'ifRepositorySet' and 'ownedRevisions'.
* unit-tests/commit-set-tests.js: Added unit tests for 'topLevelRepositoriesSortedByNamePreferringOnesWithURL'.
* unit-tests/resources/mock-v3-models.js: Added a repository group contains 'ios', 'webkit' and 'ownerRepository'.

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@222648 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Websites/perf.webkit.org/ChangeLog
Websites/perf.webkit.org/public/v3/models/commit-set.js
Websites/perf.webkit.org/public/v3/models/triggerable.js
Websites/perf.webkit.org/server-tests/tools-sync-buildbot-integration-tests.js
Websites/perf.webkit.org/tools/js/buildbot-syncer.js
Websites/perf.webkit.org/unit-tests/buildbot-syncer-tests.js
Websites/perf.webkit.org/unit-tests/commit-set-tests.js
Websites/perf.webkit.org/unit-tests/resources/mock-v3-models.js

index ef6028a..4bea1fd 100644 (file)
@@ -1,3 +1,35 @@
+2017-09-28  Dewei Zhu  <dewei_zhu@apple.com>
+
+        Update syncing script to be able to build binary for commit set with owned commits.
+        https://bugs.webkit.org/show_bug.cgi?id=177225
+
+        Reviewed by Ryosuke Niwa.
+
+        Added support for syncing script to be able to schedule builds to build binary for owned commits.
+        Introduces 'ifRepositorySet' and 'ownedRevisions' in 'buildProperties'.
+        'ifRepositorySet' will conditionaly set a build property if at least one of the repositories it specified requires build.
+        'ownedRevisions' specifies owned commits revision informations.
+
+        * public/v3/models/commit-set.js:
+        (CommitSet): Added '_ownerRepositoryToOwnedRepositoriesMap'.
+        (CommitSet.prototype.updateSingleton): Reset '_ownerRepositoryToOwnedRepositoriesMap'.
+        (CommitSet.prototype._updateFromObject): Only update '_repositoryToCommitOwnerMap' and '_ownerRepositoryToOwnedRepositoriesMap' when 'commitOwner' exists.
+        (CommitSet.prototype.ownerCommitForRepository): Returns a sorted list of top level repositories.
+        (CommitSet.prototype.ownedRepositoriesForOwnerRepository): Returns owned repositories given a owner repository.
+        (CustomCommitSet.prototype.ownerCommitForRepository): Returns a sorted list of top level repositories.
+        * public/v3/models/triggerable.js:
+        (prototype.accepts): It should only check against top-level repositories. Removed a deprecated 'FIXME'.
+        * server-tests/tools-sync-buildbot-integration-tests.js: Added unit test for building owned commits binary.
+        (createTriggerable): Added conditional 'ifRepositorySet' and 'ownedRevisions' in the repository groups.
+        * tools/js/buildbot-syncer.js:
+        (BuildbotSyncer.prototype._propertiesForBuildRequest):
+            Added logic to conditionaly create build property for 'ifRepositorySet'.
+            Added logic to create 'ownedRevisions' based on the owner repositories it specified.
+        (BuildbotSyncer._parseRepositoryGroup): Build property template should be able to handle 'ifRepositorySet' and 'ownedRevisions'.
+        * unit-tests/buildbot-syncer-tests.js: Added unit tests for 'ifRepositorySet' and 'ownedRevisions'.
+        * unit-tests/commit-set-tests.js: Added unit tests for 'topLevelRepositoriesSortedByNamePreferringOnesWithURL'.
+        * unit-tests/resources/mock-v3-models.js: Added a repository group contains 'ios', 'webkit' and 'ownerRepository'.
+
 2017-09-19  Dewei Zhu  <dewei_zhu@apple.com>
 
         Use 'owned commit' instead of 'sub commit' whenever refers to a commit owned by another commit.
index cb8e6f2..b178ac8 100644 (file)
@@ -11,6 +11,7 @@ class CommitSet extends DataModelObject {
         this._repositoryToRootMap = new Map;
         this._repositoryToCommitOwnerMap = new Map;
         this._repositoryRequiresBuildMap = new Map;
+        this._ownerRepositoryToOwnedRepositoriesMap = new Map;
         this._latestCommitTime = null;
         this._customRoots = [];
         this._allRootFiles = [];
@@ -28,6 +29,7 @@ class CommitSet extends DataModelObject {
         this._repositoryToRootMap.clear();
         this._repositoryToCommitOwnerMap.clear();
         this._repositoryRequiresBuildMap.clear();
+        this._ownerRepositoryToOwnedRepositoriesMap.clear();
         this._repositories = [];
         this._updateFromObject(object);
     }
@@ -44,7 +46,14 @@ class CommitSet extends DataModelObject {
             const repository = commit.repository();
             this._repositoryToCommitMap.set(repository, commit);
             this._repositoryToPatchMap.set(repository, item.patch);
-            this._repositoryToCommitOwnerMap.set(repository, item.commitOwner);
+            if (item.commitOwner) {
+                this._repositoryToCommitOwnerMap.set(repository, item.commitOwner);
+                const ownerRepository = item.commitOwner.repository();
+                if (!this._ownerRepositoryToOwnedRepositoriesMap.get(ownerRepository))
+                    this._ownerRepositoryToOwnedRepositoriesMap.set(ownerRepository, [repository]);
+                else
+                    this._ownerRepositoryToOwnedRepositoriesMap.get(ownerRepository).push(repository);
+            }
             this._repositoryRequiresBuildMap.set(repository, item.requiresBuild);
             this._repositoryToRootMap.set(repository, item.rootFile);
             if (item.rootFile)
@@ -59,6 +68,10 @@ class CommitSet extends DataModelObject {
     customRoots() { return this._customRoots; }
     allRootFiles() { return this._allRootFiles; }
     commitForRepository(repository) { return this._repositoryToCommitMap.get(repository); }
+    ownerCommitForRepository(repository) { return this._repositoryToCommitOwnerMap.get(repository); }
+    topLevelRepositories() { return Repository.sortByNamePreferringOnesWithURL(this._repositories.filter((repository) => !this.ownerRevisionForRepository(repository))); }
+
+    ownedRepositoriesForOwnerRepository(repository) { return this._ownerRepositoryToOwnedRepositoriesMap.get(repository); }
 
     revisionForRepository(repository)
     {
@@ -204,6 +217,7 @@ class CustomCommitSet {
     }
 
     repositories() { return Array.from(this._revisionListByRepository.keys()); }
+    topLevelRepositories() { return Repository.sortByNamePreferringOnesWithURL(this.repositories().filter((repository) => !this.ownerRevisionForRepository(repository))); }
     revisionForRepository(repository)
     {
         const entry = this._revisionListByRepository.get(repository);
index 3ea8e21..3cb0f37 100644 (file)
@@ -65,8 +65,7 @@ class TriggerableRepositoryGroup extends LabeledObject {
 
     accepts(commitSet)
     {
-        // FIXME: Add a check for patch.
-        const commitSetRepositories = Repository.sortByNamePreferringOnesWithURL(commitSet.repositories());
+        const commitSetRepositories = commitSet.topLevelRepositories();
         if (this._repositories.length != commitSetRepositories.length)
             return false;
         for (let i = 0; i < this._repositories.length; i++) {
index bada57a..b0c084a 100644 (file)
@@ -26,7 +26,9 @@ function createTriggerable()
                 'webkit': {
                     repositories: {'WebKit': {acceptsPatch: true}},
                     testProperties: {'wk': {'revision': 'WebKit'}, 'roots': {'roots': {}}},
-                    buildProperties: {'wk': {'revision': 'WebKit'}, 'wk-patch': {'patch': 'WebKit'}},
+                    buildProperties: {'wk': {'revision': 'WebKit'}, 'wk-patch': {'patch': 'WebKit'},
+                        'checkbox': {'ifRepositorySet': ['WebKit'], 'value': 'build-wk'},
+                        'owned-commits': {'ownedRevisions': 'WebKit'}},
                     acceptsRoots: true,
                 }
             },
@@ -84,6 +86,22 @@ function createTestGroupWihPatch()
     })
 }
 
+function createTestGroupWihOwnedCommit()
+{
+    const someTest = Test.findById(MockData.someTestId());
+    const webkit = Repository.findById(MockData.webkitRepositoryId());
+    const ownedSJC = Repository.findById(MockData.ownedJSCRepositoryId());
+    const set1 = new CustomCommitSet;
+    set1.setRevisionForRepository(webkit, '191622');
+    set1.setRevisionForRepository(ownedSJC, 'owned-jsc-6161', null, '191622');
+    const set2 = new CustomCommitSet;
+    set2.setRevisionForRepository(webkit, '192736');
+    set2.setRevisionForRepository(ownedSJC, 'owned-jsc-9191', null, '192736');
+    return TestGroup.createWithTask('custom task', Platform.findById(MockData.somePlatformId()), someTest, 'some group', 2, [set1, set2]).then((task) => {
+        return TestGroup.findAllByTask(task.id())[0];
+    });
+}
+
 function uploadRoot(buildRequestId, buildNumber)
 {
     return TemporaryFile.makeTemporaryFile(`root${buildNumber}.dat`, `root for build ${buildNumber}`).then((rootFile) => {
@@ -174,7 +192,7 @@ describe('sync-buildbot', function () {
             assert.equal(requests.length, 7);
             assertAndResolveRequest(requests[6], 'POST', '/builders/some%20builder/force', 'OK');
             assert.deepEqual(requests[6].data, {'wk': '191622', 'wk-patch': RemoteAPI.url('/api/uploaded-file/1.dat'),
-                'build-request-id': '1', 'forcescheduler': 'force-ab-builds'});
+                'build-request-id': '1', 'forcescheduler': 'force-ab-builds', 'checkbox': 'build-wk'});
             return MockRemoteAPI.waitForRequest();
         }).then(() => {
             assert.equal(requests.length, 10);
@@ -286,7 +304,7 @@ describe('sync-buildbot', function () {
         }).then(() => {
             assert.equal(requests.length, 7);
             assertAndResolveRequest(requests[6], 'POST', '/builders/some%20builder/force', 'OK');
-            assert.deepEqual(requests[6].data, {'wk': '191622', 'build-request-id': '2', 'forcescheduler': 'force-ab-builds'});
+            assert.deepEqual(requests[6].data, {'wk': '191622', 'build-request-id': '2', 'forcescheduler': 'force-ab-builds', 'checkbox': 'build-wk'});
             return MockRemoteAPI.waitForRequest();
         }).then(() => {
             assert.equal(requests.length, 10);
@@ -658,4 +676,286 @@ describe('sync-buildbot', function () {
         });
     });
 
+    it('should schedule a build to build binary for owned commits', () => {
+        const requests = MockRemoteAPI.requests;
+        let triggerable;
+        let taskId = null;
+        let syncPromise;
+        return createTriggerable().then((newTriggerable) => {
+            triggerable = newTriggerable;
+            return createTestGroupWihOwnedCommit();
+        }).then((testGroup) => {
+            taskId = testGroup.task().id();
+            const webkit = Repository.findById(MockData.webkitRepositoryId());
+            const ownedJSC = Repository.findById(MockData.ownedJSCRepositoryId());
+            assert.equal(testGroup.buildRequests().length, 6);
+
+            const buildRequest = testGroup.buildRequests()[0];
+            assert(buildRequest.isBuild());
+            assert(!buildRequest.isTest());
+            assert.equal(buildRequest.statusLabel(), 'Waiting');
+            assert.equal(buildRequest.statusUrl(), null);
+            assert.equal(buildRequest.buildId(), null);
+
+            const commitSet = buildRequest.commitSet();
+            assert.equal(commitSet.revisionForRepository(webkit), '191622');
+            assert.equal(commitSet.patchForRepository(webkit), null);
+            assert.equal(commitSet.rootForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(ownedJSC), commitSet.revisionForRepository(webkit));
+            assert.deepEqual(commitSet.allRootFiles(), []);
+
+            const otherBuildRequest = testGroup.buildRequests()[1];
+            assert(otherBuildRequest.isBuild());
+            assert(!otherBuildRequest.isTest());
+            assert.equal(otherBuildRequest.statusLabel(), 'Waiting');
+            assert.equal(otherBuildRequest.statusUrl(), null);
+            assert.equal(otherBuildRequest.buildId(), null);
+
+            const otherCommitSet = otherBuildRequest.commitSet();
+            assert.equal(otherCommitSet.revisionForRepository(webkit), '192736');
+            assert.equal(otherCommitSet.patchForRepository(webkit), null);
+            assert.equal(otherCommitSet.rootForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(ownedJSC), otherCommitSet.revisionForRepository(webkit));
+            assert.deepEqual(otherCommitSet.allRootFiles(), []);
+
+            syncPromise = triggerable.initSyncers().then(() => triggerable.syncOnce());
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 3);
+            assertAndResolveRequest(requests[0], 'GET', '/json/builders/some%20tester/pendingBuilds', []);
+            assertAndResolveRequest(requests[1], 'GET', '/json/builders/some%20builder/pendingBuilds', []);
+            assertAndResolveRequest(requests[2], 'GET', '/json/builders/other%20builder/pendingBuilds', []);
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 6);
+            assertAndResolveRequest(requests[3], 'GET', '/json/builders/some%20tester/builds/?select=-1&select=-2', {});
+            assertAndResolveRequest(requests[4], 'GET', '/json/builders/some%20builder/builds/?select=-1&select=-2', {});
+            assertAndResolveRequest(requests[5], 'GET', '/json/builders/other%20builder/builds/?select=-1&select=-2', {});
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 7);
+            assertAndResolveRequest(requests[6], 'POST', '/builders/some%20builder/force', 'OK');
+            assert.deepEqual(requests[6].data, {'wk': '191622', 'build-request-id': '1', 'forcescheduler': 'force-ab-builds', 'owned-commits': `{"WebKit":[{"revision":"owned-jsc-6161","repository":"JavaScriptCore","ownerRevision":"191622"}]}`});
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 10);
+            assertAndResolveRequest(requests[7], 'GET', '/json/builders/some%20tester/pendingBuilds', []);
+            assertAndResolveRequest(requests[8], 'GET', '/json/builders/some%20builder/pendingBuilds',
+                [MockData.pendingBuild({builder: 'some builder', buildRequestId: 1})]);
+            assertAndResolveRequest(requests[9], 'GET', '/json/builders/other%20builder/pendingBuilds', []);
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 13);
+            assertAndResolveRequest(requests[10], 'GET', '/json/builders/some%20tester/builds/?select=-1&select=-2', {});
+            assertAndResolveRequest(requests[11], 'GET', '/json/builders/some%20builder/builds/?select=-1&select=-2', {
+                [-1]: MockData.runningBuild({builder: 'some builder', buildRequestId: 1})
+            });
+            assertAndResolveRequest(requests[12], 'GET', '/json/builders/other%20builder/builds/?select=-1&select=-2', {});
+            return syncPromise;
+        }).then(() => {
+            return TestGroup.fetchForTask(taskId, true);
+        }).then((testGroups) => {
+            assert.equal(testGroups.length, 1);
+            const testGroup = testGroups[0];
+            const webkit = Repository.findById(MockData.webkitRepositoryId());
+            const ownedJSC = Repository.findById(MockData.ownedJSCRepositoryId());
+            assert.equal(testGroup.buildRequests().length, 6);
+
+            const buildRequest = testGroup.buildRequests()[0];
+            assert(buildRequest.isBuild());
+            assert(!buildRequest.isTest());
+            assert.equal(buildRequest.statusLabel(), 'Running');
+            assert.equal(buildRequest.statusUrl(), 'http://build.webkit.org/builders/some%20builder/builds/124');
+            assert.equal(buildRequest.buildId(), null);
+
+            const commitSet = buildRequest.commitSet();
+            assert.equal(commitSet.revisionForRepository(webkit), '191622');
+            assert.equal(commitSet.patchForRepository(webkit), null);
+            assert.equal(commitSet.rootForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(ownedJSC), commitSet.revisionForRepository(webkit));
+            assert.deepEqual(commitSet.allRootFiles(), []);
+
+            const otherBuildRequest = testGroup.buildRequests()[1];
+            assert(otherBuildRequest.isBuild());
+            assert(!otherBuildRequest.isTest());
+            assert.equal(otherBuildRequest.statusLabel(), 'Waiting');
+            assert.equal(otherBuildRequest.statusUrl(), null);
+            assert.equal(otherBuildRequest.buildId(), null);
+
+            const otherCommitSet = otherBuildRequest.commitSet();
+            assert.equal(otherCommitSet.revisionForRepository(webkit), '192736');
+            assert.equal(otherCommitSet.patchForRepository(webkit), null);
+            assert.equal(otherCommitSet.rootForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(ownedJSC), otherCommitSet.revisionForRepository(webkit));
+            assert.deepEqual(otherCommitSet.allRootFiles(), []);
+
+            return uploadRoot(buildRequest.id(), 123);
+        }).then(() => {
+            return TestGroup.fetchForTask(taskId, true);
+        }).then((testGroups) => {
+            assert.equal(testGroups.length, 1);
+            const testGroup = testGroups[0];
+            const webkit = Repository.findById(MockData.webkitRepositoryId());
+            const ownedJSC = Repository.findById(MockData.ownedJSCRepositoryId());
+            assert.equal(testGroup.buildRequests().length, 6);
+
+            const buildRequest = testGroup.buildRequests()[0];
+            assert(buildRequest.isBuild());
+            assert(!buildRequest.isTest());
+            assert.equal(buildRequest.statusLabel(), 'Completed');
+            assert.equal(buildRequest.statusUrl(), 'http://build.webkit.org/builders/some%20builder/builds/124');
+            assert.notEqual(buildRequest.buildId(), null);
+
+            const commitSet = buildRequest.commitSet();
+            assert.equal(commitSet.revisionForRepository(webkit), '191622');
+            assert.equal(commitSet.patchForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(ownedJSC), commitSet.revisionForRepository(webkit));
+            const webkitRoot = commitSet.rootForRepository(webkit);
+            assert(webkitRoot instanceof UploadedFile);
+            assert.equal(webkitRoot.filename(), 'root123.dat');
+            assert.deepEqual(commitSet.allRootFiles(), [webkitRoot]);
+
+            const otherBuildRequest = testGroup.buildRequests()[1];
+            assert(otherBuildRequest.isBuild());
+            assert(!otherBuildRequest.isTest());
+            assert.equal(otherBuildRequest.statusLabel(), 'Waiting');
+            assert.equal(otherBuildRequest.statusUrl(), null);
+            assert.equal(otherBuildRequest.buildId(), null);
+
+            const otherCommitSet = otherBuildRequest.commitSet();
+            assert.equal(otherCommitSet.revisionForRepository(webkit), '192736');
+            assert.equal(otherCommitSet.patchForRepository(webkit), null);
+            assert.equal(otherCommitSet.rootForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(ownedJSC), otherCommitSet.revisionForRepository(webkit));
+            assert.deepEqual(otherCommitSet.allRootFiles(), []);
+
+            MockRemoteAPI.reset();
+            syncPromise = triggerable.initSyncers().then(() => triggerable.syncOnce());
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 3);
+            assertAndResolveRequest(requests[0], 'GET', '/json/builders/some%20tester/pendingBuilds', []);
+            assertAndResolveRequest(requests[1], 'GET', '/json/builders/some%20builder/pendingBuilds', []);
+            assertAndResolveRequest(requests[2], 'GET', '/json/builders/other%20builder/pendingBuilds', []);
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 6);
+            assertAndResolveRequest(requests[3], 'GET', '/json/builders/some%20tester/builds/?select=-1&select=-2', {});
+            assertAndResolveRequest(requests[4], 'GET', '/json/builders/some%20builder/builds/?select=-1&select=-2', {
+                [-1]: MockData.finishedBuild({builder: 'some builder', buildRequestId: 1})
+            });
+            assertAndResolveRequest(requests[5], 'GET', '/json/builders/other%20builder/builds/?select=-1&select=-2', {});
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 7);
+            assertAndResolveRequest(requests[6], 'POST', '/builders/some%20builder/force', 'OK');
+            assert.deepEqual(requests[6].data, {'wk': '192736', 'build-request-id': '2', 'forcescheduler': 'force-ab-builds', 'owned-commits': `{"WebKit":[{"revision":"owned-jsc-9191","repository":"JavaScriptCore","ownerRevision":"192736"}]}`});
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 10);
+            assertAndResolveRequest(requests[7], 'GET', '/json/builders/some%20tester/pendingBuilds', []);
+            assertAndResolveRequest(requests[8], 'GET', '/json/builders/some%20builder/pendingBuilds', []);
+            assertAndResolveRequest(requests[9], 'GET', '/json/builders/other%20builder/pendingBuilds', []);
+            return MockRemoteAPI.waitForRequest();
+        }).then(() => {
+            assert.equal(requests.length, 13);
+            assertAndResolveRequest(requests[10], 'GET', '/json/builders/some%20tester/builds/?select=-1&select=-2', {});
+            assertAndResolveRequest(requests[11], 'GET', '/json/builders/some%20builder/builds/?select=-1&select=-2', {
+                [-1]: MockData.runningBuild({builder: 'some builder', buildRequestId: 2}),
+                [-2]: MockData.finishedBuild({builder: 'some builder', buildRequestId: 1}),
+            });
+            assertAndResolveRequest(requests[12], 'GET', '/json/builders/other%20builder/builds/?select=-1&select=-2', {});
+            return syncPromise;
+        }).then(() => {
+            return TestGroup.fetchForTask(taskId, true);
+        }).then((testGroups) => {
+            assert.equal(testGroups.length, 1);
+            const testGroup = testGroups[0];
+            const webkit = Repository.findById(MockData.webkitRepositoryId());
+            const ownedJSC = Repository.findById(MockData.ownedJSCRepositoryId());
+            assert.equal(testGroup.buildRequests().length, 6);
+
+            const buildRequest = testGroup.buildRequests()[0];
+            assert(buildRequest.isBuild());
+            assert(!buildRequest.isTest());
+            assert.equal(buildRequest.statusLabel(), 'Completed');
+            assert.equal(buildRequest.statusUrl(), 'http://build.webkit.org/builders/some%20builder/builds/124');
+            assert.notEqual(buildRequest.buildId(), null);
+
+            const commitSet = buildRequest.commitSet();
+            assert.equal(commitSet.revisionForRepository(webkit), '191622');
+            assert.equal(commitSet.patchForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(ownedJSC), commitSet.revisionForRepository(webkit));
+            const webkitRoot = commitSet.rootForRepository(webkit);
+            assert(webkitRoot instanceof UploadedFile);
+            assert.equal(webkitRoot.filename(), 'root123.dat');
+            assert.deepEqual(commitSet.allRootFiles(), [webkitRoot]);
+
+            const otherBuildRequest = testGroup.buildRequests()[1];
+            assert(otherBuildRequest.isBuild());
+            assert(!otherBuildRequest.isTest());
+            assert.equal(otherBuildRequest.statusLabel(), 'Running');
+            assert.equal(otherBuildRequest.statusUrl(), 'http://build.webkit.org/builders/some%20builder/builds/124');
+            assert.equal(otherBuildRequest.buildId(), null);
+
+            const otherCommitSet = otherBuildRequest.commitSet();
+            assert.equal(otherCommitSet.revisionForRepository(webkit), '192736');
+            assert.equal(otherCommitSet.patchForRepository(webkit), null);
+            assert.equal(otherCommitSet.rootForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(ownedJSC), otherCommitSet.revisionForRepository(webkit));
+            assert.deepEqual(otherCommitSet.allRootFiles(), []);
+
+            return uploadRoot(otherBuildRequest.id(), 124);
+        }).then(() => {
+            return TestGroup.fetchForTask(taskId, true);
+        }).then((testGroups) => {
+            assert.equal(testGroups.length, 1);
+            const testGroup = testGroups[0];
+            const webkit = Repository.findById(MockData.webkitRepositoryId());
+            const ownedJSC = Repository.findById(MockData.ownedJSCRepositoryId());
+            assert.equal(testGroup.buildRequests().length, 6);
+
+            const buildRequest = testGroup.buildRequests()[0];
+            assert(buildRequest.isBuild());
+            assert(!buildRequest.isTest());
+            assert.equal(buildRequest.statusLabel(), 'Completed');
+            assert.equal(buildRequest.statusUrl(), 'http://build.webkit.org/builders/some%20builder/builds/124');
+            assert.notEqual(buildRequest.buildId(), null);
+
+            const commitSet = buildRequest.commitSet();
+            assert.equal(commitSet.revisionForRepository(webkit), '191622');
+            assert.equal(commitSet.patchForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(commitSet.ownerRevisionForRepository(ownedJSC), commitSet.revisionForRepository(webkit));
+            const webkitRoot = commitSet.rootForRepository(webkit);
+            assert(webkitRoot instanceof UploadedFile);
+            assert.equal(webkitRoot.filename(), 'root123.dat');
+            assert.deepEqual(commitSet.allRootFiles(), [webkitRoot]);
+
+            const otherBuildRequest = testGroup.buildRequests()[1];
+            assert(otherBuildRequest.isBuild());
+            assert(!otherBuildRequest.isTest());
+            assert.equal(otherBuildRequest.statusLabel(), 'Completed');
+            assert.equal(otherBuildRequest.statusUrl(), 'http://build.webkit.org/builders/some%20builder/builds/124');
+            assert.notEqual(otherBuildRequest.buildId(), null);
+
+            const otherCommitSet = otherBuildRequest.commitSet();
+            assert.equal(otherCommitSet.revisionForRepository(webkit), '192736');
+            assert.equal(otherCommitSet.patchForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(webkit), null);
+            assert.equal(otherCommitSet.ownerRevisionForRepository(ownedJSC), otherCommitSet.revisionForRepository(webkit));
+            const otherWebkitRoot = otherCommitSet.rootForRepository(webkit);
+            assert(otherWebkitRoot instanceof UploadedFile);
+            assert.equal(otherWebkitRoot.filename(), 'root124.dat');
+            assert.deepEqual(otherCommitSet.allRootFiles(), [otherWebkitRoot]);
+        });
+    });
 });
index 588af90..47a0885 100644 (file)
@@ -260,12 +260,32 @@ class BuildbotSyncer {
                     continue;
                 value = JSON.stringify(rootFiles.map((file) => ({url: file.url()})));
                 break;
+            case 'ownedRevisions':
+                const ownedRepositories = commitSet.ownedRepositoriesForOwnerRepository(value.ownerRepository);
+                if (!ownedRepositories)
+                    continue;
+
+                const revisionInfo = {};
+                revisionInfo[value.ownerRepository.name()] = ownedRepositories.map((ownedRepository) => {
+                    return {
+                        'revision': commitSet.revisionForRepository(ownedRepository),
+                        'repository': ownedRepository.name(),
+                        'ownerRevision': commitSet.ownerRevisionForRepository(ownedRepository)
+                    };
+                });
+                value = JSON.stringify(revisionInfo);
+                break;
             case 'conditional':
                 switch (value.condition) {
                 case 'built':
                     if (!requestsInGroup.some((otherRequest) => otherRequest.isBuild() && otherRequest.commitSet() == buildRequest.commitSet()))
                         continue;
                     break;
+                case 'requiresBuild':
+                    const requiresBuild = value.repositoriesToCheck.some((repository) => commitSet.requiresBuildForRepository(repository));
+                    if (!requiresBuild)
+                        continue;
+                    break;
                 }
                 value = value.value;
             }
@@ -438,17 +458,24 @@ class BuildbotSyncer {
             assert(group.acceptsRoots, `Repository group "${name}" specifies the properties for building but does not accept roots in testing`);
             const revisionRepositories = new Set;
             const patchRepositories = new Set;
-            buildPropertiesTemplate = this._parseRepositoryGroupPropertyTemplate('build', name, group.buildProperties, (type, value) => {
+            buildPropertiesTemplate = this._parseRepositoryGroupPropertyTemplate('build', name, group.buildProperties, (type, value, condition) => {
                 assert(type != 'roots', `Repository group "${name}" specifies roots in the properties for building`);
-                const repository = resolveRepository(value);
+                let repository = null;
                 switch (type) {
                 case 'patch':
+                    repository = resolveRepository(value);
                     assert(patchAcceptingRepositoryList.has(repository), `Repository group "${name}" specifies a patch for "${value}" but it does not accept a patch`);
                     patchRepositories.add(repository);
                     return {type, repository};
                 case 'revision':
+                    repository = resolveRepository(value);
                     revisionRepositories.add(repository);
                     return {type, repository};
+                case 'ownedRevisions':
+                    return {type, ownerRepository: resolveRepository(value)};
+                case 'ifRepositorySet':
+                    assert(condition, 'condition must set if type is "ifRepositorySet"');
+                    return {type: 'conditional', condition: 'requiresBuild', value, repositoriesToCheck: condition.map(resolveRepository)};
                 }
                 return null;
             });
@@ -483,10 +510,22 @@ class BuildbotSyncer {
             }
 
             const keys = Object.keys(value);
-            assert.equal(keys.length, 1,
-                `Repository group "${groupName}" specifies more than one type in property "${propertyName}": "${keys.join('", "')}"`);
-            const type = keys[0];
-            const option = makeOption(type, value[type]);
+            assert(keys.length == 1 || keys.length == 2,
+                `Repository group "${groupName}" specifies more than two types in property "${propertyName}": "${keys.join('", "')}"`);
+            let type;
+            let condition = null;
+            let optionValue;
+            if (keys.length == 2) {
+                assert(keys.includes('value'), `Repository group "${groupName}" with two types in property "${propertyName}": "${keys.join('", "')}" should contains 'value' as one type`);
+                type = keys.find((key) => key != 'value');
+                optionValue = value.value;
+                condition = value[type];
+            }
+            else {
+                type = keys[0];
+                optionValue = value[type];
+            }
+            const option = makeOption(type, optionValue, condition);
             assert(option, `Repository group "${groupName}" specifies an invalid type "${type}" in property "${propertyName}"`);
             propertiesTemplate[propertyName] = option;
         }
index f0d125e..30bf2cb 100644 (file)
@@ -235,13 +235,50 @@ function createSampleBuildRequestWithPatch(platform, test, order)
     const root = new UploadedFile(456, {'createdAt': new Date('2017-05-01T21:03:27Z'), 'filename': 'root.dat', 'extension': '.dat', 'author': 'some user',
         size: 16452234, sha256: '03eed7a8494ab8794c44b7d4308e55448fc56f4d6c175809ba968f78f656d58d'});
 
-    const commitSet = CommitSet.ensureSingleton('53246456', {customRoots: [root], revisionItems: [{commit: webkit197463, patch}, {commit: shared111237}, {commit: ios13A452}]});
+    const commitSet = CommitSet.ensureSingleton('53246456', {customRoots: [root], revisionItems: [{commit: webkit197463, patch, requiresBuild: true}, {commit: shared111237}, {commit: ios13A452}]});
 
     return BuildRequest.ensureSingleton(`6345645376-${order}`, {'triggerable': MockModels.triggerable,
         repositoryGroup: MockModels.svnRepositoryGroup,
         'commitSet': commitSet, 'status': 'pending', 'platform': platform, 'test': test, 'order': order});
 }
 
+function createSampleBuildRequestWithOwnedCommit(platform, test, order)
+{
+    assert(platform instanceof Platform);
+    assert(!test || test instanceof Test);
+
+    const webkit197463 = CommitLog.ensureSingleton('111127', {'id': '111127', 'time': 1456955807334, 'repository': MockModels.webkit, 'revision': '197463'});
+    const owner111289 = CommitLog.ensureSingleton('111289', {'id': '111289', 'time': 1456931874000, 'repository': MockModels.ownerRepository, 'revision': 'owner-001'});
+    const owned111222 = CommitLog.ensureSingleton('111222', {'id': '111222', 'time': 1456932774000, 'repository': MockModels.ownedRepository, 'revision': 'owned-002'});
+    const ios13A452 = CommitLog.ensureSingleton('88930', {'id': '88930', 'time': 0, 'repository': MockModels.ios, 'revision': '13A452'});
+
+    const commitSet = CommitSet.ensureSingleton('53246486', {customRoots: [], revisionItems: [{commit: webkit197463}, {commit: owner111289}, {commit: owned111222, commitOwner: owner111289, requiresBuild: true}, {commit: ios13A452}]});
+
+    return BuildRequest.ensureSingleton(`6345645370-${order}`, {'triggerable': MockModels.triggerable,
+        repositoryGroup: MockModels.svnRepositoryWithOwnedRepositoryGroup,
+        'commitSet': commitSet, 'status': 'pending', 'platform': platform, 'test': test, 'order': order});
+}
+
+function createSampleBuildRequestWithOwnedCommitAndPatch(platform, test, order)
+{
+    assert(platform instanceof Platform);
+    assert(!test || test instanceof Test);
+
+    const webkit197463 = CommitLog.ensureSingleton('111127', {'id': '111127', 'time': 1456955807334, 'repository': MockModels.webkit, 'revision': '197463'});
+    const owner111289 = CommitLog.ensureSingleton('111289', {'id': '111289', 'time': 1456931874000, 'repository': MockModels.ownerRepository, 'revision': 'owner-001'});
+    const owned111222 = CommitLog.ensureSingleton('111222', {'id': '111222', 'time': 1456932774000, 'repository': MockModels.ownedRepository, 'revision': 'owned-002'});
+    const ios13A452 = CommitLog.ensureSingleton('88930', {'id': '88930', 'time': 0, 'repository': MockModels.ios, 'revision': '13A452'});
+
+    const patch = new UploadedFile(453, {'createdAt': new Date('2017-05-01T19:16:53Z'), 'filename': 'patch.dat', 'extension': '.dat', 'author': 'some user',
+        size: 534637, sha256: '169463c8125e07c577110fe144ecd63942eb9472d438fc0014f474245e5df8a1'});
+
+    const commitSet = CommitSet.ensureSingleton('53246486', {customRoots: [], revisionItems: [{commit: webkit197463, patch, requiresBuild: true}, {commit: owner111289}, {commit: owned111222, commitOwner: owner111289, requiresBuild: true}, {commit: ios13A452}]});
+
+    return BuildRequest.ensureSingleton(`6345645370-${order}`, {'triggerable': MockModels.triggerable,
+        repositoryGroup: MockModels.svnRepositoryWithOwnedRepositoryGroup,
+        'commitSet': commitSet, 'status': 'pending', 'platform': platform, 'test': test, 'order': order});
+}
+
 function samplePendingBuild(buildRequestId, buildTime, slaveName)
 {
     return {
@@ -967,6 +1004,7 @@ describe('BuildbotSyncer', () => {
                 'buildProperties': {
                     'webkit': {'revision': 'WebKit'},
                     'webkit-patch': {'patch': 'WebKit'},
+                    'checkbox': {'ifRepositorySet': ['WebKit'], 'value': 'build-webkit'},
                     'shared': {'revision': 'Shared'},
                 },
                 'acceptsRoots': true,
@@ -976,6 +1014,7 @@ describe('BuildbotSyncer', () => {
             const properties = syncers[2]._propertiesForBuildRequest(request, [request]);
             assert.equal(properties['webkit'], '197463');
             assert.equal(properties['webkit-patch'], 'http://build.webkit.org/api/uploaded-file/453.dat');
+            assert.equal(properties['checkbox'], 'build-webkit');
         });
 
         it('should resolve "ifBuilt"', () => {
@@ -1017,6 +1056,64 @@ describe('BuildbotSyncer', () => {
 
         });
 
+        it('should resolve "ifRepositorySet" and "requiresBuild"', () => {
+            const config = sampleiOSConfig();
+            config.repositoryGroups['ios-svn-webkit-with-owned-commit'] = {
+                'repositories': {'WebKit': {'acceptsPatch': true}, 'Owner Repository': {}, 'iOS': {}},
+                'testProperties': {
+                    'os': {'revision': 'iOS'},
+                    'webkit': {'revision': 'WebKit'},
+                    'owner-repo': {'revision': 'Owner Repository'},
+                    'roots': {'roots': {}},
+                },
+                'buildProperties': {
+                    'webkit': {'revision': 'WebKit'},
+                    'webkit-patch': {'patch': 'WebKit'},
+                    'owner-repo': {'revision': 'Owner Repository'},
+                    'checkbox': {'ifRepositorySet': ['WebKit'], 'value': 'build-webkit'},
+                    'owned-commits': {'ownedRevisions': 'Owner Repository'}
+                },
+                'acceptsRoots': true,
+            };
+            const syncers = BuildbotSyncer._loadConfig(RemoteAPI, config);
+            const request = createSampleBuildRequestWithOwnedCommit(MockModels.iphone, null, -1);
+            const properties = syncers[2]._propertiesForBuildRequest(request, [request]);
+            assert.equal(properties['webkit'], '197463');
+            assert.equal(properties['owner-repo'], 'owner-001');
+            assert.equal(properties['checkbox'], undefined);
+            assert.deepEqual(JSON.parse(properties['owned-commits']), {'Owner Repository': [{revision: 'owned-002', repository: 'Owned Repository', ownerRevision: 'owner-001'}]});
+        });
+
+        it('should resolve "patch", "ifRepositorySet" and "requiresBuild"', () => {
+
+            const config = sampleiOSConfig();
+            config.repositoryGroups['ios-svn-webkit-with-owned-commit'] = {
+                'repositories': {'WebKit': {'acceptsPatch': true}, 'Owner Repository': {}, 'iOS': {}},
+                'testProperties': {
+                    'os': {'revision': 'iOS'},
+                    'webkit': {'revision': 'WebKit'},
+                    'owner-repo': {'revision': 'Owner Repository'},
+                    'roots': {'roots': {}},
+                },
+                'buildProperties': {
+                    'webkit': {'revision': 'WebKit'},
+                    'webkit-patch': {'patch': 'WebKit'},
+                    'owner-repo': {'revision': 'Owner Repository'},
+                    'checkbox': {'ifRepositorySet': ['WebKit'], 'value': 'build-webkit'},
+                    'owned-commits': {'ownedRevisions': 'Owner Repository'}
+                },
+                'acceptsRoots': true,
+            };
+            const syncers = BuildbotSyncer._loadConfig(RemoteAPI, config);
+            const request = createSampleBuildRequestWithOwnedCommitAndPatch(MockModels.iphone, null, -1);
+            const properties = syncers[2]._propertiesForBuildRequest(request, [request]);
+            assert.equal(properties['webkit'], '197463');
+            assert.equal(properties['owner-repo'], 'owner-001');
+            assert.equal(properties['checkbox'], 'build-webkit');
+            assert.equal(properties['webkit-patch'], 'http://build.webkit.org/api/uploaded-file/453.dat');
+            assert.deepEqual(JSON.parse(properties['owned-commits']), {'Owner Repository': [{revision: 'owned-002', repository: 'Owned Repository', ownerRevision: 'owner-001'}]});
+        });
+
         it('should set the property for the build request id', () => {
             const syncers = BuildbotSyncer._loadConfig(RemoteAPI, sampleiOSConfig());
             const request = createSampleBuildRequest(MockModels.iphone, MockModels.speedometer);
index 27c2d9e..ab76d58 100644 (file)
@@ -77,6 +77,11 @@ describe('CustomCommitSet', () => {
             const commitSet = customCommitSetWithoutOwnedCommit();
             assert.deepEqual(commitSet.repositories(), [MockModels.osx, MockModels.webkit]);
         });
+
+        it('should return only top level repositories', () => {
+            const commitSet = customCommitSetWithoutOwnedCommit();
+            assert.deepEqual(commitSet.topLevelRepositories(), [MockModels.osx, MockModels.webkit]);
+        });
     });
 
     describe('Test custom commit set with owned commit', () => {
@@ -105,6 +110,11 @@ describe('CustomCommitSet', () => {
             const commitSet = customCommitSetWithOwnedCommit();
             assert.deepEqual(commitSet.repositories(), [MockModels.osx, MockModels.ownerRepository, MockModels.ownedRepository]);
         });
+
+        it('should return only top level repositories', () => {
+            const commitSet = customCommitSetWithOwnedCommit();
+            assert.deepEqual(commitSet.topLevelRepositories(), [MockModels.osx, MockModels.ownerRepository]);
+        });
     });
 
     describe('Test custom commit set with patch', () => {
@@ -130,6 +140,11 @@ describe('CustomCommitSet', () => {
             const commitSet = customCommitSetWithPatch();
             assert.deepEqual(commitSet.repositories(), [MockModels.osx, MockModels.webkit]);
         });
+
+        it('should return only top level repositories', () => {
+            const commitSet = customCommitSetWithPatch();
+            assert.deepEqual(commitSet.topLevelRepositories(), [MockModels.osx, MockModels.webkit]);
+        });
     });
 
     describe('Test custom commit set with owned repository has same name as non-owned repository',  () => {
@@ -158,6 +173,11 @@ describe('CustomCommitSet', () => {
             const commitSet = customCommitSetWithOwnedRepositoryHasSameNameAsNotOwnedRepository();
             assert.deepEqual(commitSet.repositories(), [MockModels.osx, MockModels.webkit, MockModels.ownedWebkit]);
         });
+
+        it('should return only top level repositories', () => {
+            const commitSet = customCommitSetWithOwnedRepositoryHasSameNameAsNotOwnedRepository();
+            assert.deepEqual(commitSet.topLevelRepositories(), [MockModels.osx, MockModels.webkit]);
+        });
     });
 
     describe('Test custom commit set equality function', () => {
index 96aec2b..c4277b0 100644 (file)
@@ -57,8 +57,13 @@ var MockModels = {
                 name: 'ios-git-webkit',
                 repositories: [{repository: MockModels.ios}, {repository: MockModels.webkitGit}, {repository: MockModels.sharedRepository}]
             });
+            MockModels.svnRepositoryWithOwnedRepositoryGroup = new TriggerableRepositoryGroup(34, {
+                name: 'ios-svn-webkit-with-owned-commit',
+                repositories: [{repository: MockModels.ios}, {repository: MockModels.webkit, acceptsPatch: true}, {repository: MockModels.ownerRepository}],
+                acceptsCustomRoots: true,
+            });
             MockModels.triggerable = new Triggerable(3, {name: 'build-webkit',
-                repositoryGroups: [MockModels.osRepositoryGroup, MockModels.svnRepositoryGroup, MockModels.gitRepositoryGroup],
+                repositoryGroups: [MockModels.osRepositoryGroup, MockModels.svnRepositoryGroup, MockModels.gitRepositoryGroup, MockModels.svnRepositoryWithOwnedRepositoryGroup],
                 configurations: [{test: MockModels.iPhonePLT, platform: MockModels.iphone}]});
 
         });