W3C test importer should generate the modules installed dynamically to run wpt tests
authoryouenn.fablet@crf.canon.fr <youenn.fablet@crf.canon.fr@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 6 Apr 2015 09:06:03 +0000 (09:06 +0000)
committeryouenn.fablet@crf.canon.fr <youenn.fablet@crf.canon.fr@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 6 Apr 2015 09:06:03 +0000 (09:06 +0000)
https://bugs.webkit.org/show_bug.cgi?id=142738

Reviewed by Ryosuke Niwa.

LayoutTests/imported/w3c:

Renamed WPTModules to web-platform-test-modules.json
Updated TestRepositories to ask the importer to generate web-platform-test-modules.json at import time.

* resources/TestRepositories:
* resources/WPTModules: Removed.
* resources/web-platform-tests-modules.json: Added.

Tools:

The test importer can now generate the submodules description file based on information extracted from the corresponding git repository.
The implementation is done within TestDownloader and it is activated for the WPT repository.
LayoutTests/imported/w3c/resources/WPTModules is renamed as LayoutTests/imported/w3c/resources/web-platform-tests-modules.json for that reason.

* Scripts/webkitpy/layout_tests/servers/web_platform_test_server.py:
(WebPlatformTestServer._install_modules): Updated to cope with path as array.
* Scripts/webkitpy/w3c/test_downloader.py:
(TestDownloader._git_submodules_status): Added so that it can be overriden for unit tests.
(TestDownloader):
(TestDownloader._git_submodules_description): Computes submodule description.
(TestDownloader.generate_git_submodules_description): Write submodule description in a file.
* Scripts/webkitpy/w3c/test_importer.py:
(TestImporter.do_import): Added the possibility to post process the tests.
(TestImporter):
(TestImporter.generate_git_submodules_description_for_all_repositories): Enable generating module description file.
(TestImporter.should_convert_test_harness_links): Updated according new options format.
* Scripts/webkitpy/w3c/test_importer_unittest.py:
(TestImporterTest.import_downloaded_tests): Making use of TestDownloaderMock.
(TestImporterTest.import_downloaded_tests.TestDownloaderMock): Added to override submodule status gathering.
(TestImporterTest.import_downloaded_tests.TestDownloaderMock.__init__):
(TestImporterTest.import_downloaded_tests.TestDownloaderMock._git_submodules_status):
(TestImporterTest.test_submodules_generation): Added to check that modules description files works.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@182386 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/imported/w3c/ChangeLog
LayoutTests/imported/w3c/resources/TestRepositories
LayoutTests/imported/w3c/resources/WPTModules [deleted file]
LayoutTests/imported/w3c/resources/web-platform-tests-modules.json [new file with mode: 0644]
Tools/ChangeLog
Tools/Scripts/webkitpy/layout_tests/servers/web_platform_test_server.py
Tools/Scripts/webkitpy/w3c/test_downloader.py
Tools/Scripts/webkitpy/w3c/test_importer.py
Tools/Scripts/webkitpy/w3c/test_importer_unittest.py

index bc4764c..e7260a5 100644 (file)
@@ -1,3 +1,17 @@
+2015-04-06  Youenn Fablet  <youenn.fablet@crf.canon.fr>
+
+        W3C test importer should generate the modules installed dynamically to run wpt tests
+        https://bugs.webkit.org/show_bug.cgi?id=142738
+
+        Reviewed by Ryosuke Niwa.
+
+        Renamed WPTModules to web-platform-test-modules.json
+        Updated TestRepositories to ask the importer to generate web-platform-test-modules.json at import time.  
+
+        * resources/TestRepositories:
+        * resources/WPTModules: Removed.
+        * resources/web-platform-tests-modules.json: Added.
+
 2015-03-13  Youenn Fablet  <youenn.fablet@crf.canon.fr>
 
         WebKit test infrastructure should automate the process of cloning W3C test suite and importing tests from it
index 9568e20..f28a077 100644 (file)
@@ -10,7 +10,7 @@
         "paths_to_import": [
             "support"
         ],
-        "convert_test_harness_links": true
+        "import_options": ["convert_test_harness_links"]
     },
     {
         "name": "web-platform-tests",
@@ -30,6 +30,6 @@
             "config.default.json",
             "serve.py"
         ],
-        "convert_test_harness_links": false
+        "import_options": ["generate_git_submodules_description"]
     }
 ]
diff --git a/LayoutTests/imported/w3c/resources/WPTModules b/LayoutTests/imported/w3c/resources/WPTModules
deleted file mode 100644 (file)
index 62845fa..0000000
+++ /dev/null
@@ -1,32 +0,0 @@
-[
-{
-    "name": "html5lib",
-    "path": "tools",
-    "url": "https://github.com/html5lib/html5lib-python/archive/7cce65bbaa78411f98b8b37eeefc9db03c580097.tar.gz",
-    "url_subpath": "html5lib-python-7cce65bbaa78411f98b8b37eeefc9db03c580097"
-},
-{
-    "name": "pywebsocket",
-    "path": "tools",
-    "url": "https://github.com/w3c/pywebsocket/archive/8c285d9015121e8c1c40be852439fc03b4a18112.tar.gz",
-    "url_subpath": "pywebsocket-8c285d9015121e8c1c40be852439fc03b4a18112"
-},
-{
-    "name": "six",
-    "path": "tools",
-    "url": "https://github.com/jgraham/six/archive/3b6173c833d217ab0186c355804f5925cbcfca47.tar.gz",
-    "url_subpath": "six-3b6173c833d217ab0186c355804f5925cbcfca47"
-},
-{
-    "name": "wptserve",
-    "path": "tools",
-    "url": "https://github.com/w3c/wptserve/archive/91e764ffa0f587090341a26af457dfdc6762eb0d.tar.gz",
-    "url_subpath": "wptserve-91e764ffa0f587090341a26af457dfdc6762eb0d"
-},
-{
-    "name": "resources",
-    "path": ".",
-    "url": "https://github.com/w3c/testharness.js/archive/74ba13d2fbb19c6e7494960bd7724873e88c523c.tar.gz",
-    "url_subpath": "testharness.js-74ba13d2fbb19c6e7494960bd7724873e88c523c"
-}
-]
diff --git a/LayoutTests/imported/w3c/resources/web-platform-tests-modules.json b/LayoutTests/imported/w3c/resources/web-platform-tests-modules.json
new file mode 100644 (file)
index 0000000..768169c
--- /dev/null
@@ -0,0 +1,42 @@
+[
+    {
+        "name": "resources", 
+        "path": [
+            "."
+        ], 
+        "url": "https://github.com/w3c/testharness.js/archive/2a1da264f6718db04c3925a9956b635426827aef.tar.gz", 
+        "url_subpath": "testharness.js-2a1da264f6718db04c3925a9956b635426827aef"
+    }, 
+    {
+        "name": "wptserve", 
+        "path": [
+            "tools"
+        ], 
+        "url": "https://github.com/w3c/wptserve/archive/91e764ffa0f587090341a26af457dfdc6762eb0d.tar.gz", 
+        "url_subpath": "wptserve-91e764ffa0f587090341a26af457dfdc6762eb0d"
+    }, 
+    {
+        "name": "pywebsocket", 
+        "path": [
+            "tools"
+        ], 
+        "url": "https://github.com/w3c/pywebsocket/archive/8c285d9015121e8c1c40be852439fc03b4a18112.tar.gz", 
+        "url_subpath": "pywebsocket-8c285d9015121e8c1c40be852439fc03b4a18112"
+    }, 
+    {
+        "name": "html5lib", 
+        "path": [
+            "tools"
+        ], 
+        "url": "https://github.com/html5lib/html5lib-python/archive/7cce65bbaa78411f98b8b37eeefc9db03c580097.tar.gz", 
+        "url_subpath": "html5lib-python-7cce65bbaa78411f98b8b37eeefc9db03c580097"
+    }, 
+    {
+        "name": "six", 
+        "path": [
+            "tools"
+        ], 
+        "url": "https://github.com/jgraham/six/archive/3b6173c833d217ab0186c355804f5925cbcfca47.tar.gz", 
+        "url_subpath": "six-3b6173c833d217ab0186c355804f5925cbcfca47"
+    }
+]
\ No newline at end of file
index 8a045ba..4dd0ea8 100644 (file)
@@ -1,3 +1,33 @@
+2015-04-06  Youenn Fablet  <youenn.fablet@crf.canon.fr>
+
+        W3C test importer should generate the modules installed dynamically to run wpt tests
+        https://bugs.webkit.org/show_bug.cgi?id=142738
+
+        Reviewed by Ryosuke Niwa.
+
+        The test importer can now generate the submodules description file based on information extracted from the corresponding git repository.
+        The implementation is done within TestDownloader and it is activated for the WPT repository.
+        LayoutTests/imported/w3c/resources/WPTModules is renamed as LayoutTests/imported/w3c/resources/web-platform-tests-modules.json for that reason.
+
+        * Scripts/webkitpy/layout_tests/servers/web_platform_test_server.py:
+        (WebPlatformTestServer._install_modules): Updated to cope with path as array.
+        * Scripts/webkitpy/w3c/test_downloader.py:
+        (TestDownloader._git_submodules_status): Added so that it can be overriden for unit tests.
+        (TestDownloader):
+        (TestDownloader._git_submodules_description): Computes submodule description.
+        (TestDownloader.generate_git_submodules_description): Write submodule description in a file.
+        * Scripts/webkitpy/w3c/test_importer.py:
+        (TestImporter.do_import): Added the possibility to post process the tests.
+        (TestImporter):
+        (TestImporter.generate_git_submodules_description_for_all_repositories): Enable generating module description file.
+        (TestImporter.should_convert_test_harness_links): Updated according new options format.
+        * Scripts/webkitpy/w3c/test_importer_unittest.py:
+        (TestImporterTest.import_downloaded_tests): Making use of TestDownloaderMock. 
+        (TestImporterTest.import_downloaded_tests.TestDownloaderMock): Added to override submodule status gathering.
+        (TestImporterTest.import_downloaded_tests.TestDownloaderMock.__init__):
+        (TestImporterTest.import_downloaded_tests.TestDownloaderMock._git_submodules_status):
+        (TestImporterTest.test_submodules_generation): Added to check that modules description files works.
+
 2015-04-05  Antti Koivisto  <antti@apple.com>
 
         Bloom filter should support longer hashes
index 041a9fe..26f9f9a 100755 (executable)
@@ -79,13 +79,13 @@ class WebPlatformTestServer(http_server_base.HttpServerBase):
         self._doc_root_path = port_obj.path_from_webkit_base("LayoutTests", self._doc_root)
 
     def _install_modules(self):
-        modules_file_path = self._filesystem.join(self._layout_root, "imported", "w3c", "resources", "WPTModules")
+        modules_file_path = self._filesystem.join(self._layout_root, "imported", "w3c", "resources", "web-platform-tests-modules.json")
         if not self._filesystem.isfile(modules_file_path):
             _log.warning("Cannot read " + modules_file_path)
             return
         modules = json.loads(self._filesystem.read_text_file(modules_file_path))
         for module in modules:
-            AutoInstaller(target_dir=self._filesystem.join(self._doc_root, module["path"])).install(url=module["url"], url_subpath=module["url_subpath"], target_name=module["name"])
+            AutoInstaller(target_dir=self._filesystem.join(self._doc_root, self._filesystem.sep.join(module["path"]))).install(url=module["url"], url_subpath=module["url_subpath"], target_name=module["name"])
 
     def _copy_webkit_test_files(self):
         _log.debug('Copying WebKit resources files')
index 5adba2c..3feacb8 100644 (file)
@@ -149,6 +149,47 @@ class TestDownloader(object):
             if self._filesystem.isfile(destination_path):
                 self._filesystem.remove(destination_path)
 
+    def _git_submodules_status(self, repository_directory):
+        return self.git(repository_directory)._run_git(['submodule', 'status'])
+
+    def _git_submodules_description(self, test_repository):
+        submodules = []
+        repository_directory = self._filesystem.join(self.repository_directory, test_repository['name'])
+        if self._filesystem.isfile(self._filesystem.join(repository_directory, '.gitmodules')):
+            submodule = {}
+            for line in self._filesystem.read_text_file(self._filesystem.join(repository_directory, '.gitmodules')).splitlines():
+                line = line.strip()
+                if line.startswith('path = '):
+                    submodule['path'] = line[7:]
+                elif line.startswith('url = '):
+                    submodule['url'] = line[6:]
+                    if not submodule['url'].startswith('https://github.com/'):
+                        _log.warning('Submodule %s is not hosted on github' % submodule['path'])
+                        _log.warning('Please ensure that generated URL points to an archive of the module or manually edit its value after the import')
+                    submodules.append(submodule)
+                    submodule = {}
+
+        submodules_status = [line[1:].split(' ') for line in self._git_submodules_status(repository_directory).splitlines()]
+        for submodule in submodules:
+            for status in submodules_status:
+                if submodule['path'] == status[1]:
+                    url = submodule['url'][:-4]
+                    version = status[0]
+                    repository_name = url.split('/').pop()
+                    submodule['url'] = url + '/archive/' + version + '.tar.gz'
+                    submodule['url_subpath'] = repository_name + '-' + version
+            if '/' in submodule['path']:
+                steps = submodule['path'].split('/')
+                submodule['name'] = steps.pop()
+                submodule['path'] = steps
+            else:
+                submodule['name'] = submodule['path']
+                submodule['path'] = ['.']
+        return submodules
+
+    def generate_git_submodules_description(self, test_repository, filepath):
+        self._filesystem.write_text_file(filepath, json.dumps(self._git_submodules_description(test_repository), sort_keys=True, indent=4))
+
     def download_tests(self, destination_directory, test_paths=[]):
         for test_repository in self.test_repositories:
             self.checkout_test_repository(test_repository['revision'], test_repository['url'], self._filesystem.join(self.repository_directory, test_repository['name']))
index 06172c3..a4a3af5 100644 (file)
@@ -190,6 +190,16 @@ class TestImporter(object):
 
         self.import_tests()
 
+        if self._importing_downloaded_tests:
+            self.generate_git_submodules_description_for_all_repositories()
+
+    def generate_git_submodules_description_for_all_repositories(self):
+        for test_repository in self._test_downloader.test_repositories:
+            if 'generate_git_submodules_description' in test_repository['import_options']:
+                self.filesystem.maybe_make_directory(self.filesystem.join(self.destination_directory, 'resources'))
+                self._test_downloader.generate_git_submodules_description(test_repository, self.filesystem.join(self.destination_directory, 'resources', test_repository['name'] + '-modules.json'))
+            # FIXME: Generate WPT .gitignore and  main __init__.py
+
     def test_downloader(self):
         if not self._test_downloader:
             download_options = TestDownloader.default_options()
@@ -281,7 +291,7 @@ class TestImporter(object):
         if self._importing_downloaded_tests:
             for test_repository in self.test_downloader().test_repositories:
                 if test.startswith(test_repository['name']):
-                    return test_repository['convert_test_harness_links']
+                    return 'convert_test_harness_links' in test_repository['import_options']
             return True
         return self.options.convert_test_harness_links
 
index 90e5fee..0ae3660 100644 (file)
@@ -34,6 +34,7 @@ from webkitpy.common.host_mock import MockHost
 from webkitpy.common.system.filesystem_mock import MockFileSystem
 from webkitpy.common.system.executive_mock import MockExecutive2, ScriptError
 from webkitpy.common.system.outputcapture import OutputCapture
+from webkitpy.w3c.test_downloader import TestDownloader
 from webkitpy.w3c.test_importer import parse_args, TestImporter
 
 
@@ -92,12 +93,21 @@ class TestImporterTest(unittest.TestCase):
 
     def import_downloaded_tests(self, args, files):
         # files are passed as parameter as we cannot clone/fetch/checkout a repo in mock system.
+
+        class TestDownloaderMock(TestDownloader):
+            def __init__(self, repository_directory, host, options):
+                TestDownloader.__init__(self, repository_directory, host, options)
+
+            def _git_submodules_status(self, repository_directory):
+                return 'adb4d391a69877d4a1eaaf51d1725c99a5b8ed84 tools/resources'
+
         host = MockHost()
         host.executive = MockExecutive2()
         host.filesystem = MockFileSystem(files=files)
 
         options, args = parse_args(args)
         importer = TestImporter(host, None, options)
+        importer._test_downloader = TestDownloaderMock(importer.tests_download_path, importer.host, importer.options)
         importer.do_import()
         return host.filesystem
 
@@ -116,4 +126,17 @@ class TestImporterTest(unittest.TestCase):
         self.assertTrue('src="/resources/testharness.js"' in fs.read_text_file('/mock-checkout/LayoutTests/w3c/web-platform-tests/t/test.html'))
         self.assertTrue('src="../' in fs.read_text_file('/mock-checkout/LayoutTests/w3c/csswg-tests/t/test.html'))
 
+    def test_submodules_generation(self):
+        FAKE_FILES = {
+            '/mock-checkout/WebKitBuild/w3c-tests/csswg-tests/.gitmodules': '[submodule "tools/resources"]\n   path = tools/resources\n        url = https://github.com/w3c/testharness.js.git\n  ignore = dirty\n',
+            '/mock-checkout/WebKitBuild/w3c-tests/web-platform-tests/.gitmodules': '[submodule "tools/resources"]\n    path = tools/resources\n        url = https://github.com/w3c/testharness.js.git\n  ignore = dirty\n',
+        }
+
+        fs = self.import_downloaded_tests(['--no-fetch', '--import-all', '-d', 'w3c'], FAKE_FILES)
+
+        self.assertFalse(fs.exists('/mock-checkout/LayoutTests/w3c/resources/csswg-tests-modules.json'))
+        self.assertTrue(fs.exists('/mock-checkout/LayoutTests/w3c/resources/web-platform-tests-modules.json'))
+        self.assertTrue('https://github.com/w3c/testharness.js/archive/db4d391a69877d4a1eaaf51d1725c99a5b8ed84.tar.gz' in fs.read_text_file('/mock-checkout/LayoutTests/w3c/resources/web-platform-tests-modules.json'))
+
+
     # FIXME: Needs more tests.