Chromium testers should extract builds instead of building on their own
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 3 Apr 2012 18:58:21 +0000 (18:58 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 3 Apr 2012 18:58:21 +0000 (18:58 +0000)
https://bugs.webkit.org/show_bug.cgi?id=82996

Reviewed by Tony Chang.

Make Chromium builders triggger Chromium testers and make testers download and extract builds
from the buildbot master instead of building binaries on their own.

Add download-built-product to wrap curl used in the download-built-product step since Windows
does not provide "curl" natively.

* BuildSlaveSupport/build.webkit.org-config/config.json: Add new triggerables for Chromium testers and make
Chromium builders trigger them. Also change the type of Chromium testers from NewBuildAndTest to Test.
* BuildSlaveSupport/build.webkit.org-config/master.cfg:
(DownloadBuiltProduct): Call download-built-product instead of curl.
* BuildSlaveSupport/built-product-archive:
(createZip): Don't zip the parent configuration build directory like "release" and "debug". While Mac port
needs this behavior for compatibility reasons, Chromium Mac port doesn't want this behavior.
(archiveBuiltProduct): Zip the configuration build directory on Mac port.
(unzipArchive): Extracted. Use ditto on Mac, unzip on linux and cygwin, and zipfile package on Windows.
(extractBuiltProduct): Refactor the code to use removeDirectoryIfExists and unzipArchive. Support Chromium.
* BuildSlaveSupport/download-built-product: Added to wrap curl which isn't available on Windows.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@113067 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/BuildSlaveSupport/build.webkit.org-config/config.json
Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg
Tools/BuildSlaveSupport/built-product-archive
Tools/BuildSlaveSupport/download-built-product [new file with mode: 0644]
Tools/ChangeLog

index 5c6bd8f9fbd321efe131f7462a5649fe086940e2..631b74ecaabd8888b06c0670dd41ddb17054a883 100644 (file)
                     {
                       "name": "Chromium Win Release", "type": "Build", "builddir": "chromium-win-release",
                       "platform": "chromium-win", "configuration": "release", "architectures": ["i386"],
-                      "upload": true,
+                      "triggers": ["chromium-win-release-tests"],
                       "slavenames": ["google-windows-1"]
                     },
                     {
-                      "name": "Chromium Win Release (Tests)", "type": "NewBuildAndTest", "builddir": "chromium-win-release-tests",
+                      "name": "Chromium Win Release (Tests)", "type": "Test", "builddir": "chromium-win-release-tests",
                       "platform": "chromium-win", "configuration": "release", "architectures": ["i386"],
                       "slavenames": ["google-windows-2"]
                     },
                     {
                       "name": "Chromium Mac Release", "type": "Build", "builddir": "chromium-mac-release",
                       "platform": "chromium-mac", "configuration": "release", "architectures": ["i386"],
-                      "upload": true,
+                      "triggers": ["chromium-mac-release-tests"],
                       "slavenames": ["google-mac-1", "google-mac-3"]
                     },
                     {
-                      "name": "Chromium Mac Release (Tests)", "type": "NewBuildAndTest", "builddir": "chromium-mac-release-tests",
+                      "name": "Chromium Mac Release (Tests)", "type": "Test", "builddir": "chromium-mac-release-tests",
                       "platform": "chromium-mac", "configuration": "release", "architectures": ["i386"],
                       "slavenames": ["google-mac-2", "google-mac-4"]
                     },
                     {
                       "name": "Chromium Linux Release", "type": "Build", "builddir": "chromium-linux-release",
                       "platform": "chromium-linux", "configuration": "release", "architectures": ["i386"],
-                      "upload": true,
+                      "triggers": ["chromium-linux-release-tests"],
                       "slavenames": ["google-linux-1"]
                     },
                     {
-                      "name": "Chromium Linux Release (Tests)", "type": "NewBuildAndTest", "builddir": "chromium-linux-release-tests",
+                      "name": "Chromium Linux Release (Tests)", "type": "Test", "builddir": "chromium-linux-release-tests",
                       "platform": "chromium-linux", "configuration": "release", "architectures": ["i386"],
                       "slavenames": ["google-linux-2"]
                     },
                     },
                     { "type": "Triggerable", "name": "win-release-tests-wk2",
                       "builderNames": ["Windows 7 Release (WebKit2 Tests)"]
+                    },
+                    { "type": "Triggerable", "name": "chromium-win-release-tests",
+                      "builderNames": ["Chromium Win Release (Tests)"]
+                    },
+                    { "type": "Triggerable", "name": "chromium-mac-release-tests",
+                      "builderNames": ["Chromium Mac Release (Tests)"]
+                    },
+                    { "type": "Triggerable", "name": "chromium-linux-release-tests",
+                      "builderNames": ["Chromium Linux Release (Tests)"]
                     }
                   ]
 }
index cc3c023749885bec13bf11b5d411a86968602efc..b5735e6bfc9f332429b37e4dccf94b44029bf758 100644 (file)
@@ -234,7 +234,9 @@ class UploadBuiltProduct(transfer.FileUpload):
 
 
 class DownloadBuiltProduct(shell.ShellCommand):
-    command = ["curl", "--fail", "--output", WithProperties("WebKitBuild/%(configuration)s.zip"), WithProperties(c["buildbotURL"] + "archives/%(fullPlatform)s-%(architecture)s-%(configuration)s/%(got_revision)s.zip")]
+    command = ["python", "./Tools/BuildSlaveSupport/download-built-product",
+        WithProperties("--platform=%(platform)s"), WithProperties("--%(configuration)s"),
+        WithProperties(c["buildbotURL"] + "archives/%(fullPlatform)s-%(architecture)s-%(configuration)s/%(got_revision)s.zip")]
     name = "download-built-product"
     description = ["downloading built product"]
     descriptionDone = ["downloaded built product"]
index ce52638960e07748c011c2b5ae6d3a66033be31a..810109babe7da4c63aa2ed70991311142b3f36b5 100644 (file)
@@ -105,7 +105,7 @@ def createZip(directoryToZip, configuration):
             raise
 
     if sys.platform == 'darwin':
-        return subprocess.call(["ditto", "-c", "-k", "--keepParent", "--sequesterRsrc", directoryToZip, archiveFile])
+        return subprocess.call(["ditto", "-c", "-k", "--sequesterRsrc", directoryToZip, archiveFile])
     elif sys.platform == 'cygwin':
         return subprocess.call(["zip", "-r", archiveFile, "bin"], cwd=directoryToZip)
     elif sys.platform == 'win32':
@@ -121,7 +121,7 @@ def archiveBuiltProduct(configuration, platform):
     configurationBuildDirectory = os.path.join(_buildDirectory, configuration.title())
 
     if platform == 'mac':
-        return createZip(configurationBuildDirectory, configuration)
+        return createZip(_buildDirectory, configuration)
     elif platform == 'win':
         binDirectory = os.path.join(configurationBuildDirectory, "bin")
         thinDirectory = os.path.join(configurationBuildDirectory, "thin")
@@ -167,27 +167,39 @@ def archiveBuiltProduct(configuration, platform):
         copyBuildFiles(configurationBuildDirectory, thinDirectory)
         if createZip(thinDirectory, configuration):
             return 1
-        
+
+
+def unzipArchive(directoryToExtractTo, configuration):
+    archiveDir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "WebKitBuild"))
+    assert os.path.isdir(archiveDir)
+    archiveFile = os.path.join(archiveDir, configuration + ".zip")
+
+    if sys.platform == 'darwin':
+        if subprocess.call(["ditto", "-x", "-k", archiveFile, directoryToExtractTo]):
+            return 1
+    elif sys.platform == 'cygwin' or sys.platform.startswith('linux'):
+        if subprocess.call(["unzip", "-o", archiveFile], cwd=directoryToExtractTo):
+            return 1
+    elif sys.platform == 'win32':
+        archive = zipfile.ZipFile(archiveFile, "r")
+        archive.extractall(directoryToExtractTo)
+
+    os.unlink(archiveFile)
+
 
 def extractBuiltProduct(configuration, platform):
-    assert platform in ('mac', 'win', 'qt', 'gtk', 'efl')
+    assert platform in ('mac', 'win', 'qt', 'gtk', 'efl', 'chromium')
 
     archiveFile = os.path.join(_buildDirectory, configuration + ".zip")
     configurationBuildDirectory = os.path.join(_buildDirectory, configuration.title())
 
-    if platform == 'mac':
-        if os.path.isdir(configurationBuildDirectory):
-            shutil.rmtree(configurationBuildDirectory)
-
-        if subprocess.call(["ditto", "-x", "-k", archiveFile, _buildDirectory]):
-            return 1
-        os.unlink(archiveFile)
+    removeDirectoryIfExists(_buildDirectory)
+    os.makedirs(_buildDirectory)
 
+    if platform == 'mac':
+        return unzipArchive(_buildDirectory, configuration)
     elif platform == 'win':
         binDirectory = os.path.join(configurationBuildDirectory, "bin")
-        if os.path.isdir(binDirectory):
-            shutil.rmtree(binDirectory)
-
         os.makedirs(binDirectory)
 
         safariPath = subprocess.Popen('cygpath -w "$PROGRAMFILES"/Safari',
@@ -196,16 +208,11 @@ def extractBuiltProduct(configuration, platform):
         if subprocess.call('cp -R "%s"/*.dll "%s"/*.resources %s' % (safariPath, safariPath, binDirectory), shell=True):
             return 1
 
-        if subprocess.call(["unzip", "-o", archiveFile], cwd=configurationBuildDirectory):
-            return 1
-
-    elif platform == 'qt' or platform == 'gtk' or platform == 'efl':
-        if os.path.isdir(configurationBuildDirectory):
-            shutil.rmtree(configurationBuildDirectory)
+        return unzipArchive(configurationBuildDirectory, configuration)
+    elif platform == 'qt' or platform == 'gtk' or platform == 'efl' or platform == 'chromium':
+        os.makedirs(configurationBuildDirectory)
+        return unzipArchive(configurationBuildDirectory, configuration)
 
-        if subprocess.call(["unzip", "-o", archiveFile, "-d", configurationBuildDirectory], cwd=_buildDirectory):
-            return 1
-        os.unlink(archiveFile)
 
 if __name__ == '__main__':
     sys.exit(main())
diff --git a/Tools/BuildSlaveSupport/download-built-product b/Tools/BuildSlaveSupport/download-built-product
new file mode 100644 (file)
index 0000000..4be30a1
--- /dev/null
@@ -0,0 +1,56 @@
+#!/usr/bin/python
+#
+# Copyright (C) 2009 Apple Inc.  All rights reserved.
+# Copyright (C) 2012 Google Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# 1.  Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer. 
+# 2.  Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in the
+#     documentation and/or other materials provided with the distribution. 
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE AND ITS CONTRIBUTORS "AS IS" AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+import optparse
+import os
+import subprocess
+import sys
+import urllib
+
+
+def main():
+    parser = optparse.OptionParser("usage: %prog [options] [url]")
+    parser.add_option("--platform", dest="platform")
+    parser.add_option("--debug", action="store_const", const="debug", dest="configuration")
+    parser.add_option("--release", action="store_const", const="release", dest="configuration")
+
+    options, (url, ) = parser.parse_args()
+
+    archiveDir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "WebKitBuild"))
+    if not os.path.isdir(archiveDir):
+        os.makedirs(archiveDir)
+    archivePath = os.path.join(archiveDir, "%s.zip" % options.configuration)
+
+    if sys.platform == 'win32':  # curl is not availble on Windows (outside of cygwin)
+        urllib.urlretrieve(url, archivePath)
+        return 0
+
+    return subprocess.call(["curl", "--fail", "--output", archivePath, url])
+
+
+if __name__ == '__main__':
+    sys.exit(main())
index 7653a1ae4adf93c0b94a5b00a8b0970eb7692069..afface4a9881c866dea3c62e23f20b5b21206435 100644 (file)
@@ -1,3 +1,28 @@
+2012-04-02  Ryosuke Niwa  <rniwa@webkit.org>
+
+        Chromium testers should extract builds instead of building on their own
+        https://bugs.webkit.org/show_bug.cgi?id=82996
+
+        Reviewed by Tony Chang.
+
+        Make Chromium builders triggger Chromium testers and make testers download and extract builds
+        from the buildbot master instead of building binaries on their own.
+
+        Add download-built-product to wrap curl used in the download-built-product step since Windows
+        does not provide "curl" natively.
+
+        * BuildSlaveSupport/build.webkit.org-config/config.json: Add new triggerables for Chromium testers and make
+        Chromium builders trigger them. Also change the type of Chromium testers from NewBuildAndTest to Test.
+        * BuildSlaveSupport/build.webkit.org-config/master.cfg:
+        (DownloadBuiltProduct): Call download-built-product instead of curl.
+        * BuildSlaveSupport/built-product-archive:
+        (createZip): Don't zip the parent configuration build directory like "release" and "debug". While Mac port
+        needs this behavior for compatibility reasons, Chromium Mac port doesn't want this behavior.
+        (archiveBuiltProduct): Zip the configuration build directory on Mac port.
+        (unzipArchive): Extracted. Use ditto on Mac, unzip on linux and cygwin, and zipfile package on Windows.
+        (extractBuiltProduct): Refactor the code to use removeDirectoryIfExists and unzipArchive. Support Chromium.
+        * BuildSlaveSupport/download-built-product: Added to wrap curl which isn't available on Windows.
+
 2012-04-03  Martin Robinson  <mrobinson@igalia.com>
 
         [GTK] Tools/Scripts/run-gtk-tests should not force you to use the WebKit jhbuild