Add run-web-platform-tests script
authorzandobersek@gmail.com <zandobersek@gmail.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 12 Jun 2018 07:55:42 +0000 (07:55 +0000)
committerzandobersek@gmail.com <zandobersek@gmail.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 12 Jun 2018 07:55:42 +0000 (07:55 +0000)
https://bugs.webkit.org/show_bug.cgi?id=183356

Reviewed by Carlos Alberto Lopez Perez.

.:

Store port-specific test expectations and test manifest files for the
web-platform-tests test runs under the top-level WebPlatformTests
directory.

TestExpectations.json file lists all the failing or disabled tests
or subtests that are to be expected in the test run. This information is
parsed in the run-web-platform-tests script and used to build
test-specific metadata files (ending with .ini) inside a temporary
directory that is then used during the test run. JSON format is used to
follow the WebDriver test suite in how it manages expectations, and to
avoid having to manually manage .ini files for every deviant test case.

TestManifest.ini file is used to fine-tune which tests are to be enabled
by default. For the GTK+ port, we currently disable all tests by default
but then specifically enable tests under the 2dcontext and WebCryptoAPI
directories. This will allow for gradual enabling of further tests.

* WebPlatformTests/gtk/TestExpectations.json: Added.
* WebPlatformTests/gtk/TestManifest.ini: Added.

Tools:

Add the run-web-platform-tests script that allows running WebKit builds
against the external web-platform-tests project.

The script sets up all the port-specific information, including the
display driver. If no web-platform-tests repository location is
specified, the TestDownloader utility is used to clone the repository,
and path to that clone is leveraged instead.

Port-specific TestManifest.ini and TestExpectations.json file paths are
then constructed, bailing if any of the files is missing. The JSON file
is then used to construct the metadata file tree, creating an .ini file
for each test that is disabled, has a certain expectation, or has
subtests that themselves have certain expectations. This metadata tree
is built under the build directory, and is purged and reconstructed for
each separate run.

Lastly, the WPT python module is imported from the web-platform-tests
source location, and wpt.main() is invoked along with all the necessary
arguments that set up a web-platform-tests suite run using the WebKit
product.

* Scripts/run-web-platform-tests: Added.
* Scripts/webkitpy/common/config/ports_mock.py:
(MockPort.run_bindings_tests_command):
(MockPort):
(MockPort.wpt_metadata_directory):
(MockPort.wpt_manifest_file):
* Scripts/webkitpy/port/base.py:
(Port.wpt_metadata_directory):
(Port):
(Port.wpt_manifest_file):
* Scripts/webkitpy/w3c/common.py:
(is_file_exportable):
(WPTPaths):
(WPTPaths.checkout_directory):
(WPTPaths.wpt_checkout_path):
* Scripts/webkitpy/w3c/test_downloader.py:
(TestDownloader.clone_tests):
(TestDownloader):
(TestDownloader.download_tests):
* Scripts/webkitpy/w3c/test_exporter.py:
(WebPlatformTestExporter.__init__):
* Scripts/webkitpy/w3c/wpt_runner.py: Added.
(main):
(parse_args):
(create_webdriver):
(spawn_wpt):
(WPTRunner):
(WPTRunner.__init__):
(WPTRunner._prepare_wpt_checkout):
(WPTRunner._generate_metadata_directory):
(WPTRunner.run):
* Scripts/webkitpy/w3c/wpt_runner_unittest.py: Added.
(WPTRunnerTest):
(WPTRunnerTest.MockTestDownloader):
(WPTRunnerTest.MockTestDownloader.default_options):
(WPTRunnerTest.MockTestDownloader.__init__):
(WPTRunnerTest.MockTestDownloader.clone_tests):
(WPTRunnerTest.MockWebDriver):
(WPTRunnerTest.MockWebDriver.create):
(WPTRunnerTest.MockWebDriver.binary_path):
(WPTRunnerTest.MockWebDriver.browser_path):
(WPTRunnerTest.MockWebDriver.browser_args):
(WPTRunnerTest.MockSpawnWPT):
(WPTRunnerTest.MockSpawnWPT.__init__):
(WPTRunnerTest.MockSpawnWPT.__call__):
(WPTRunnerTest.TestInstance):
(WPTRunnerTest.TestInstance.__init__):
(WPTRunnerTest.TestInstance.prepare_mock_files_for_run):
(WPTRunnerTest.test_prepare_wpt_checkout):
(WPTRunnerTest.test_prepare_wpt_checkout_specified_path):
(WPTRunnerTest.test_generate_metadata_directory):
(WPTRunnerTest.test_run):
(WPTRunnerTest.test_run_with_specified_options):
(WPTRunnerTest.test_run_with_args):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@232746 268f45cc-cd09-0410-ab3c-d52691b4dbfc

12 files changed:
ChangeLog
Tools/ChangeLog
Tools/Scripts/run-web-platform-tests [new file with mode: 0755]
Tools/Scripts/webkitpy/common/config/ports_mock.py
Tools/Scripts/webkitpy/port/base.py
Tools/Scripts/webkitpy/w3c/common.py
Tools/Scripts/webkitpy/w3c/test_downloader.py
Tools/Scripts/webkitpy/w3c/test_exporter.py
Tools/Scripts/webkitpy/w3c/wpt_runner.py [new file with mode: 0644]
Tools/Scripts/webkitpy/w3c/wpt_runner_unittest.py [new file with mode: 0644]
WebPlatformTests/gtk/TestExpectations.json [new file with mode: 0644]
WebPlatformTests/gtk/TestManifest.ini [new file with mode: 0644]

index 927c2cc..b6c82cc 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,30 @@
+2018-06-12  Zan Dobersek  <zdobersek@igalia.com>
+
+        Add run-web-platform-tests script
+        https://bugs.webkit.org/show_bug.cgi?id=183356
+
+        Reviewed by Carlos Alberto Lopez Perez.
+
+        Store port-specific test expectations and test manifest files for the
+        web-platform-tests test runs under the top-level WebPlatformTests
+        directory.
+
+        TestExpectations.json file lists all the failing or disabled tests
+        or subtests that are to be expected in the test run. This information is
+        parsed in the run-web-platform-tests script and used to build
+        test-specific metadata files (ending with .ini) inside a temporary
+        directory that is then used during the test run. JSON format is used to
+        follow the WebDriver test suite in how it manages expectations, and to
+        avoid having to manually manage .ini files for every deviant test case.
+
+        TestManifest.ini file is used to fine-tune which tests are to be enabled
+        by default. For the GTK+ port, we currently disable all tests by default
+        but then specifically enable tests under the 2dcontext and WebCryptoAPI
+        directories. This will allow for gradual enabling of further tests.
+
+        * WebPlatformTests/gtk/TestExpectations.json: Added.
+        * WebPlatformTests/gtk/TestManifest.ini: Added.
+
 2018-06-11  Carlos Garcia Campos  <cgarcia@igalia.com>
 
         Unreviewed. Update OptionsGTK.cmake and NEWS for 2.21.4 release.
index d0b5d1c..fcbeb49 100644 (file)
@@ -1,3 +1,86 @@
+2018-06-12  Zan Dobersek  <zdobersek@igalia.com>
+
+        Add run-web-platform-tests script
+        https://bugs.webkit.org/show_bug.cgi?id=183356
+
+        Reviewed by Carlos Alberto Lopez Perez.
+
+        Add the run-web-platform-tests script that allows running WebKit builds
+        against the external web-platform-tests project.
+
+        The script sets up all the port-specific information, including the
+        display driver. If no web-platform-tests repository location is
+        specified, the TestDownloader utility is used to clone the repository,
+        and path to that clone is leveraged instead.
+
+        Port-specific TestManifest.ini and TestExpectations.json file paths are
+        then constructed, bailing if any of the files is missing. The JSON file
+        is then used to construct the metadata file tree, creating an .ini file
+        for each test that is disabled, has a certain expectation, or has
+        subtests that themselves have certain expectations. This metadata tree
+        is built under the build directory, and is purged and reconstructed for
+        each separate run.
+
+        Lastly, the WPT python module is imported from the web-platform-tests
+        source location, and wpt.main() is invoked along with all the necessary
+        arguments that set up a web-platform-tests suite run using the WebKit
+        product.
+
+        * Scripts/run-web-platform-tests: Added.
+        * Scripts/webkitpy/common/config/ports_mock.py:
+        (MockPort.run_bindings_tests_command):
+        (MockPort):
+        (MockPort.wpt_metadata_directory):
+        (MockPort.wpt_manifest_file):
+        * Scripts/webkitpy/port/base.py:
+        (Port.wpt_metadata_directory):
+        (Port):
+        (Port.wpt_manifest_file):
+        * Scripts/webkitpy/w3c/common.py:
+        (is_file_exportable):
+        (WPTPaths):
+        (WPTPaths.checkout_directory):
+        (WPTPaths.wpt_checkout_path):
+        * Scripts/webkitpy/w3c/test_downloader.py:
+        (TestDownloader.clone_tests):
+        (TestDownloader):
+        (TestDownloader.download_tests):
+        * Scripts/webkitpy/w3c/test_exporter.py:
+        (WebPlatformTestExporter.__init__):
+        * Scripts/webkitpy/w3c/wpt_runner.py: Added.
+        (main):
+        (parse_args):
+        (create_webdriver):
+        (spawn_wpt):
+        (WPTRunner):
+        (WPTRunner.__init__):
+        (WPTRunner._prepare_wpt_checkout):
+        (WPTRunner._generate_metadata_directory):
+        (WPTRunner.run):
+        * Scripts/webkitpy/w3c/wpt_runner_unittest.py: Added.
+        (WPTRunnerTest):
+        (WPTRunnerTest.MockTestDownloader):
+        (WPTRunnerTest.MockTestDownloader.default_options):
+        (WPTRunnerTest.MockTestDownloader.__init__):
+        (WPTRunnerTest.MockTestDownloader.clone_tests):
+        (WPTRunnerTest.MockWebDriver):
+        (WPTRunnerTest.MockWebDriver.create):
+        (WPTRunnerTest.MockWebDriver.binary_path):
+        (WPTRunnerTest.MockWebDriver.browser_path):
+        (WPTRunnerTest.MockWebDriver.browser_args):
+        (WPTRunnerTest.MockSpawnWPT):
+        (WPTRunnerTest.MockSpawnWPT.__init__):
+        (WPTRunnerTest.MockSpawnWPT.__call__):
+        (WPTRunnerTest.TestInstance):
+        (WPTRunnerTest.TestInstance.__init__):
+        (WPTRunnerTest.TestInstance.prepare_mock_files_for_run):
+        (WPTRunnerTest.test_prepare_wpt_checkout):
+        (WPTRunnerTest.test_prepare_wpt_checkout_specified_path):
+        (WPTRunnerTest.test_generate_metadata_directory):
+        (WPTRunnerTest.test_run):
+        (WPTRunnerTest.test_run_with_specified_options):
+        (WPTRunnerTest.test_run_with_args):
+
 2018-06-11  Mark Lam  <mark.lam@apple.com>
 
         Add support for webkit-test-runner jscOptions in DumpRenderTree and WebKitTestRunner.
diff --git a/Tools/Scripts/run-web-platform-tests b/Tools/Scripts/run-web-platform-tests
new file mode 100755 (executable)
index 0000000..dcd2515
--- /dev/null
@@ -0,0 +1,30 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2018 Igalia S.L.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in the
+#     documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import sys
+
+from webkitpy.w3c import wpt_runner
+
+
+wpt_runner.main(sys.argv[0], sys.argv[1:])
index ca1b18a..cb83b55 100644 (file)
@@ -58,3 +58,9 @@ class MockPort(object):
 
     def run_bindings_tests_command(self):
         return ['mock-run-bindings-tests']
+
+    def wpt_metadata_directory(self):
+        return '/mock-path/mock-wpt-tests-metadata'
+
+    def wpt_manifest_file(self):
+        return '/mock-path/mock-wpt-manifest.json'
index 0a83813..87caa97 100644 (file)
@@ -871,6 +871,12 @@ class Port(object):
         # to have multiple copies of webkit checked out and built.
         return self._build_path('layout-test-results')
 
+    def wpt_metadata_directory(self):
+        return self._build_path('web-platform-tests-metadata')
+
+    def wpt_manifest_file(self):
+        return self._build_path('web-platform-tests-manifest.json')
+
     def setup_test_run(self, device_class=None):
         """Perform port-specific work at the beginning of a test run."""
         pass
index 29ce3be..f823cfa 100644 (file)
@@ -98,3 +98,16 @@ def is_file_exportable(path):
     assert path.startswith(CHROMIUM_WPT_DIR)
     basename = path[path.rfind('/') + 1:]
     return not is_basename_skipped(basename)
+
+
+class WPTPaths:
+    CHECKOUT_DIRECTORY = ["WebKitBuild", "w3c-tests"]
+    WPT_CHECKOUT_PATH = CHECKOUT_DIRECTORY + ["web-platform-tests"]
+
+    @staticmethod
+    def checkout_directory(finder):
+        return finder.path_from_webkit_base(*WPTPaths.CHECKOUT_DIRECTORY)
+
+    @staticmethod
+    def wpt_checkout_path(finder):
+        return finder.path_from_webkit_base(*WPTPaths.WPT_CHECKOUT_PATH)
index d17fa84..49d78f9 100644 (file)
@@ -219,7 +219,10 @@ class TestDownloader(object):
             rules.append('/'.join(path[:-1]) + '/.' + path[-1] + '.url')
         self._filesystem.write_text_file(self._filesystem.join(destination_directory, test_repository['name'], '.gitignore'), '\n'.join(rules))
 
-    def download_tests(self, destination_directory, test_paths=[], use_tip_of_tree=False):
+    def clone_tests(self, use_tip_of_tree=False):
         for test_repository in self.test_repositories:
             self.checkout_test_repository(test_repository['revision'] if not use_tip_of_tree else 'origin/master', test_repository['url'], self._filesystem.join(self.repository_directory, test_repository['name']))
+
+    def download_tests(self, destination_directory, test_paths=[], use_tip_of_tree=False):
+        self.clone_tests(use_tip_of_tree)
         self.copy_tests(destination_directory, test_paths)
index 9bf2008..6a72d15 100644 (file)
@@ -37,7 +37,7 @@ from webkitpy.common.net.bugzilla import Bugzilla
 from webkitpy.common.webkit_finder import WebKitFinder
 from webkitpy.w3c.wpt_github import WPTGitHub
 from webkitpy.w3c.wpt_linter import WPTLinter
-from webkitpy.w3c.common import WPT_GH_ORG, WPT_GH_REPO_NAME, WPT_GH_URL
+from webkitpy.w3c.common import WPT_GH_ORG, WPT_GH_REPO_NAME, WPT_GH_URL, WPTPaths
 from webkitpy.common.memoized import memoized
 
 _log = logging.getLogger(__name__)
@@ -69,7 +69,7 @@ class WebPlatformTestExporter(object):
 
         if not self._options.repository_directory:
             webkit_finder = WebKitFinder(self._filesystem)
-            self._options.repository_directory = webkit_finder.path_from_webkit_base('WebKitBuild', 'w3c-tests', 'web-platform-tests')
+            self._options.repository_directory = WPTPaths.wpt_checkout_path(webkit_finder)
 
         self._linter = WPTLinterClass(self._options.repository_directory, host.filesystem)
 
diff --git a/Tools/Scripts/webkitpy/w3c/wpt_runner.py b/Tools/Scripts/webkitpy/w3c/wpt_runner.py
new file mode 100644 (file)
index 0000000..22a528a
--- /dev/null
@@ -0,0 +1,201 @@
+# Copyright (C) 2018 Igalia S.L.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in the
+#     documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import argparse
+import json
+import logging
+import optparse
+import os
+import sys
+
+from webkitpy.common.host import Host
+from webkitpy.common.system.logutils import configure_logging
+from webkitpy.common.webkit_finder import WebKitFinder
+from webkitpy.w3c.common import WPTPaths
+from webkitpy.w3c.test_downloader import TestDownloader
+from webkitpy.webdriver_tests.webdriver_driver import create_driver
+
+_log = logging.getLogger(__name__)
+
+
+def main(script_name, argv):
+    options, args = parse_args(argv)
+
+    configure_logging(logging.DEBUG if options.verbose else logging.INFO)
+
+    # Determine Port for the specified platform.
+    host = Host()
+    try:
+        port = host.port_factory.get(options.platform, options)
+    except NotImplementedError, e:
+        _log.error(str(e))
+        sys.exit(-1)
+
+    # If necessary, inject the jhbuild wrapper.
+    if port.name() in ['gtk', 'wpe']:
+        filesystem = host.filesystem
+        top_level_directory = filesystem.normpath(filesystem.join(filesystem.dirname(__file__), '..', '..', '..', '..'))
+        sys.path.insert(0, filesystem.join(top_level_directory, 'Tools', 'jhbuild'))
+        import jhbuildutils
+
+        if not jhbuildutils.enter_jhbuild_environment_if_available(port.name()):
+            _log.warning('jhbuild environment not present. Run update-webkitgtk-libs before build-webkit to ensure proper testing.')
+
+    # Create the Port-specific driver.
+    port._display_server = options.display_server
+    display_driver = port.create_driver(worker_number=0, no_timeout=True)._make_driver(pixel_tests=False)
+    if not display_driver.check_driver(port):
+        raise RuntimeError("Failed to check driver %s" % display_driver.__class__.__name__)
+    os.environ.update(display_driver._setup_environ_for_test())
+
+    if not options.child_processes:
+        options.child_processes = os.environ.get("WEBKIT_TEST_CHILD_PROCESSES",
+                                                 str(port.default_child_processes()))
+
+    runner = WPTRunner(port, port.host, script_name, options)
+    if not runner.run(args):
+        sys.exit(1)
+
+
+def parse_args(args):
+    option_parser = optparse.OptionParser(usage='usage: %prog [options] [test...]')
+    option_parser.add_option('--verbose', action='store_true',
+                             help='Show debug message')
+    option_parser.add_option('--platform', action='store',
+                             help='Platform to use (e.g., "gtk")')
+    option_parser.add_option('--gtk', action='store_const', dest='platform', const='gtk',
+                             help='Alias for --platform=gtk')
+    option_parser.add_option('--child-processes',
+                             help='Number of tests to run in parallel'),
+    option_parser.add_option('--wpt-checkout', default=None,
+                             help='web-platform-tests repository location')
+    option_parser.add_option('--display-server', choices=['xvfb', 'xorg', 'weston', 'wayland'], default='xvfb',
+                             help='"xvfb": Use a virtualized X11 server. "xorg": Use the current X11 session. '
+                                  '"weston": Use a virtualized Weston server. "wayland": Use the current wayland session.')
+
+    return option_parser.parse_args(args)
+
+
+def create_webdriver(port):
+    return create_driver(port)
+
+
+def spawn_wpt(script_name, wpt_checkout, wpt_args):
+    # Import the WPT python module.
+    sys.path.insert(0, wpt_checkout)
+    try:
+        from tools.wpt import wpt
+    except ImportError:
+        _log.error("Failed to import wpt Python modules from the web-platform-tests directory")
+        sys.exit(-1)
+
+    wpt.main(script_name, wpt_args)
+
+
+class WPTRunner(object):
+    def __init__(self, port, host, script_name, options, downloader_class=TestDownloader,
+        create_webdriver_func=create_webdriver, spawn_wpt_func=spawn_wpt):
+        self._port = port
+        self._host = host
+        self._finder = WebKitFinder(self._host.filesystem)
+
+        self._script_name = script_name
+        self._options = options
+
+        self._downloader_class = downloader_class
+        self._create_webdriver_func = create_webdriver_func
+        self._spawn_wpt_func = spawn_wpt_func
+
+    def _prepare_wpt_checkout(self):
+        if not self._options.wpt_checkout:
+            test_downloader = self._downloader_class(WPTPaths.checkout_directory(self._finder),
+                self._host, self._downloader_class.default_options())
+            test_downloader.clone_tests()
+            self._options.wpt_checkout = WPTPaths.wpt_checkout_path(self._finder)
+
+        if not self._options.wpt_checkout or not self._host.filesystem.exists(self._options.wpt_checkout):
+            _log.error("Valid web-platform-tests directory required")
+            return False
+        return True
+
+    def _generate_metadata_directory(self, metadata_path):
+        expectations_file = self._finder.path_from_webkit_base("WebPlatformTests", self._port.name(), "TestExpectations.json")
+        if not self._host.filesystem.exists(expectations_file):
+            _log.error("Port-specific expectation .json file does not exist")
+            return False
+
+        with self._host.filesystem.open_text_file_for_reading(expectations_file) as fd:
+            expectations = json.load(fd)
+
+        for test_name, test_data in expectations.iteritems():
+            ini_file = self._host.filesystem.join(metadata_path, test_name + ".ini")
+            self._host.filesystem.maybe_make_directory(self._host.filesystem.dirname(ini_file))
+            with self._host.filesystem.open_text_file_for_writing(ini_file) as fd:
+                fd.write("[{}]\n".format(test_data.get("test_name", self._host.filesystem.basename(test_name))))
+                if "disabled" in test_data:
+                    fd.write("    disabled: {}\n".format(test_data["disabled"]))
+                elif "expected" in test_data:
+                    fd.write("    expected: {}\n".format(test_data["expected"]))
+                elif "subtests" in test_data:
+                    for subtest_name, subtest_data in test_data["subtests"].iteritems():
+                        fd.write("    [{}]\n".format(subtest_name))
+                        if "expected" in subtest_data:
+                            fd.write("        expected: {}\n".format(subtest_data["expected"]))
+                        else:
+                            _log.error("Invalid format of TestExpectations.json")
+                            return False
+                else:
+                    _log.error("Invalid format of TestExpectations.json")
+                    return False
+
+        return True
+
+    def run(self, args):
+        if not self._prepare_wpt_checkout():
+            return False
+
+        # Parse the test expectations JSON and construct corresponding metadata files.
+        metadata_path = self._port.wpt_metadata_directory()
+        if not self._generate_metadata_directory(metadata_path):
+            return False
+
+        include_manifest_path = self._finder.path_from_webkit_base("WebPlatformTests", self._port.name(), "TestManifest.ini")
+        if not self._host.filesystem.exists(include_manifest_path):
+            _log.error("Port-specific manifest .ini file does not exist")
+            return False
+
+        # Construct the WebDriver instance and run WPT tests via the 'webkit' product.
+        manifest_path = self._port.wpt_manifest_file()
+        webdriver = self._create_webdriver_func(self._port)
+
+        wpt_args = ["run", "--webkit-port={}".format(self._port.name()),
+            "--processes={}".format(self._options.child_processes),
+            "--metadata={}".format(metadata_path),
+            "--manifest={}".format(manifest_path),
+            "--include-manifest={}".format(include_manifest_path),
+            "--webdriver-binary={}".format(webdriver.binary_path()),
+            "--binary={}".format(webdriver.browser_path())]
+        wpt_args += ["--binary-arg={}".format(arg) for arg in webdriver.browser_args()]
+        wpt_args += ["webkit"] + args
+
+        self._spawn_wpt_func(self._script_name, self._options.wpt_checkout, wpt_args)
+        return True
diff --git a/Tools/Scripts/webkitpy/w3c/wpt_runner_unittest.py b/Tools/Scripts/webkitpy/w3c/wpt_runner_unittest.py
new file mode 100644 (file)
index 0000000..da399bf
--- /dev/null
@@ -0,0 +1,270 @@
+# Copyright (C) 2018 Igalia S.L.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in the
+#     documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import unittest
+
+from webkitpy.common.config.ports_mock import MockPort
+from webkitpy.common.host_mock import MockHost
+from webkitpy.w3c.wpt_runner import WPTRunner, parse_args
+
+
+TEST_EXPECTATIONS_JSON_CONTENT = """{
+    "passing_test.html": { "expected": "PASS" },
+    "failing_test.html": { "expected": "FAIL" },
+    "disabled_test.html": { "disabled": "Test Name #1" },
+    "custom_test_name.html": {
+        "test_name": "Custom Test Name",
+        "expected": "FAIL"
+    },
+    "test_with_subtests.html": {
+        "subtests": {
+            "Subtest #1": { "expected": "PASS" },
+            "Subtest #2": { "expected": "TIMEOUT" }
+        }
+    },
+    "nested/test_file.html": { "expected": "PASS" },
+    "nested/nested/test_file.html": {
+        "test_name": "Deeper-nested test case",
+        "subtests": {
+            "Only Subtest": { "expected": "PASS" }
+        }
+    }
+}"""
+
+EXPECTED_TEST_MANIFESTS = {
+    "passing_test.html.ini":
+"""[passing_test.html]
+    expected: PASS
+""",
+
+    "failing_test.html.ini":
+"""[failing_test.html]
+    expected: FAIL
+""",
+
+    "disabled_test.html.ini":
+"""[disabled_test.html]
+    disabled: Test Name #1
+""",
+
+    "custom_test_name.html.ini":
+"""[Custom Test Name]
+    expected: FAIL
+""",
+
+    "test_with_subtests.html.ini":
+"""[test_with_subtests.html]
+    [Subtest #1]
+        expected: PASS
+    [Subtest #2]
+        expected: TIMEOUT
+""",
+
+    "nested/test_file.html.ini":
+"""[test_file.html]
+    expected: PASS
+""",
+
+    "nested/nested/test_file.html.ini":
+"""[Deeper-nested test case]
+    [Only Subtest]
+        expected: PASS
+"""
+}
+
+
+class WPTRunnerTest(unittest.TestCase):
+
+    class MockTestDownloader(object):
+        @staticmethod
+        def default_options():
+            return {}
+
+        def __init__(self, repository_directory, host, options):
+            self._repository_directory = repository_directory
+            self._host = host
+
+        def clone_tests(self):
+            self._host.filesystem.maybe_make_directory(self._repository_directory, "web-platform-tests")
+
+    class MockWebDriver(object):
+        @staticmethod
+        def create(port):
+            return WPTRunnerTest.MockWebDriver()
+
+        def binary_path(self):
+            return "/mock-webdriver/bin/webdriver"
+
+        def browser_path(self):
+            return "/mock-webdriver/bin/browser"
+
+        def browser_args(self):
+            return ["webdriver_arg1", "webdriver_arg2"]
+
+    class MockSpawnWPT(object):
+        def __init__(self, test_case, expected_wpt_checkout=None, expected_wpt_args=None):
+            self._test_case = test_case
+            self._expected_wpt_checkout = expected_wpt_checkout
+            self._expected_wpt_args = expected_wpt_args
+
+        def __call__(self, script_name, wpt_checkout, wpt_args):
+            self._test_case.assertEquals(script_name, "wptrunner_unittest")
+            self._test_case.assertEquals(wpt_checkout, self._expected_wpt_checkout)
+            self._test_case.assertEquals(wpt_args, self._expected_wpt_args)
+
+    class TestInstance(object):
+        def __init__(self, options, spawn_wpt_func=None):
+            self.port = MockPort()
+            self.host = MockHost()
+
+            # In non-test environments, this value is by default set to the WEBKIT_TEST_CHILD_PROCESSES
+            # env value or, if that's not present, to the default number of child processes. Here we
+            # manually set it to 4 for consistency, unless the test case specifies it.
+            if not options.child_processes:
+                options.child_processes = 4
+
+            self.runner = WPTRunner(self.port, self.host, "wptrunner_unittest", options,
+                WPTRunnerTest.MockTestDownloader, WPTRunnerTest.MockWebDriver.create, spawn_wpt_func)
+
+        def prepare_mock_files_for_run(self):
+            self.host.filesystem.maybe_make_directory(self.port.wpt_metadata_directory())
+            self.host.filesystem.write_text_file(
+                "/mock-checkout/WebPlatformTests/MockPort/TestExpectations.json", "{}")
+            self.host.filesystem.write_text_file(
+                "/mock-checkout/WebPlatformTests/MockPort/TestManifest.ini", "{}")
+
+    def test_prepare_wpt_checkout(self):
+        # Tests the _prepare_wpt_checkout() method with no WPT checkout specified in options.
+
+        options, _ = parse_args([])
+        instance = WPTRunnerTest.TestInstance(options)
+
+        self.assertTrue(instance.runner._prepare_wpt_checkout())
+
+        expected_wpt_checkout = "/mock-checkout/WebKitBuild/w3c-tests/web-platform-tests"
+        self.assertEquals(instance.runner._options.wpt_checkout, expected_wpt_checkout)
+        self.assertTrue(instance.host.filesystem.isdir(expected_wpt_checkout))
+
+    def test_prepare_wpt_checkout_specified_path(self):
+        # Tests the _prepare_wpt_checkout() method with WPT checkout specified in options.
+
+        specified_wpt_checkout = "/mock-path/web-platform-tests"
+        options, _ = parse_args(["--wpt-checkout", specified_wpt_checkout])
+        instance = WPTRunnerTest.TestInstance(options)
+        instance.host.filesystem.maybe_make_directory(specified_wpt_checkout)
+
+        self.assertTrue(instance.runner._prepare_wpt_checkout())
+        self.assertEquals(instance.runner._options.wpt_checkout, specified_wpt_checkout)
+
+    def test_generate_metadata_directory(self):
+        # Tests the _generate_metadata_directory() method. TestExpectations.json file is mocked with
+        # TEST_EXPECTATIONS_JSON_CONTENT value as contents. Expected metadata is generated under
+        # /mock-metadata and should correspond to files and files' content in the
+        # EXPECTED_TEST_MANIFESTS dictionary.
+
+        expectations_json_path = "/mock-checkout/WebPlatformTests/MockPort/TestExpectations.json"
+        metadata_path = "/mock-metadata"
+
+        options, _ = parse_args([])
+        instance = WPTRunnerTest.TestInstance(options)
+
+        instance.host.filesystem.write_text_file(
+            "/mock-checkout/WebPlatformTests/MockPort/TestExpectations.json",
+            TEST_EXPECTATIONS_JSON_CONTENT)
+
+        self.assertTrue(instance.runner._generate_metadata_directory(metadata_path))
+        for path, content in EXPECTED_TEST_MANIFESTS.items():
+            manifest_path = instance.host.filesystem.join(metadata_path, path)
+            self.assertTrue(instance.host.filesystem.isfile(manifest_path))
+            self.assertEquals(instance.host.filesystem.read_text_file(manifest_path), content)
+
+    def test_run(self):
+        # Tests the run() method. Files are mocked to the point that helper methods don't fail.
+        # Goal of this test is for the WPT spawn command to match the desired WPT directory and
+        # arguments. No options or arguments are used.
+
+        spawn_wpt_func = WPTRunnerTest.MockSpawnWPT(self,
+            "/mock-checkout/WebKitBuild/w3c-tests/web-platform-tests",
+            ["run", "--webkit-port=MockPort", "--processes=4",
+                "--metadata=/mock-path/mock-wpt-tests-metadata",
+                "--manifest=/mock-path/mock-wpt-manifest.json",
+                "--include-manifest=/mock-checkout/WebPlatformTests/MockPort/TestManifest.ini",
+                "--webdriver-binary=/mock-webdriver/bin/webdriver",
+                "--binary=/mock-webdriver/bin/browser",
+                "--binary-arg=webdriver_arg1", "--binary-arg=webdriver_arg2", "webkit"])
+
+        options, _ = parse_args([])
+        instance = WPTRunnerTest.TestInstance(options, spawn_wpt_func)
+        instance.prepare_mock_files_for_run()
+
+        self.assertTrue(instance.runner.run([]))
+
+    def test_run_with_specified_options(self):
+        # Tests the run() method. Files are mocked to the point that helper methods don't fail.
+        # Goal of this test is for the WPT spawn command to match the desired WPT directory and
+        # arguments. Custom WPT checkout and child process count are specified. Note that the
+        # WPT checkout doesn't have an impact on the resulting WPT argument list, as intended.
+        specified_wpt_checkout = "/mock-path/web-platform-tests"
+        specified_child_processes = 16
+
+        spawn_wpt_func = WPTRunnerTest.MockSpawnWPT(self, specified_wpt_checkout,
+            ["run", "--webkit-port=MockPort", "--processes=16",
+                "--metadata=/mock-path/mock-wpt-tests-metadata",
+                "--manifest=/mock-path/mock-wpt-manifest.json",
+                "--include-manifest=/mock-checkout/WebPlatformTests/MockPort/TestManifest.ini",
+                "--webdriver-binary=/mock-webdriver/bin/webdriver",
+                "--binary=/mock-webdriver/bin/browser",
+                "--binary-arg=webdriver_arg1", "--binary-arg=webdriver_arg2", "webkit"])
+
+        options, _ = parse_args(["--wpt-checkout", specified_wpt_checkout,
+            "--child-processes", specified_child_processes])
+        instance = WPTRunnerTest.TestInstance(options, spawn_wpt_func)
+        instance.prepare_mock_files_for_run()
+
+        # Also create the mock WPT checkout directory.
+        instance.host.filesystem.maybe_make_directory(specified_wpt_checkout)
+
+        self.assertTrue(instance.runner.run([]))
+
+    def test_run_with_args(self):
+        # Tests the run() method. Files are mocked to the point that helper methods don't fail.
+        # Goal of this test is for the WPT spawn command to match the desired WPT directory and
+        # arguments. A custom two-element argument list is used. It's expected to be appended
+        # to the resulting WPT argument list.
+
+        specified_args = ["test1.html", "test2.html"]
+
+        spawn_wpt_func = WPTRunnerTest.MockSpawnWPT(self,
+            "/mock-checkout/WebKitBuild/w3c-tests/web-platform-tests",
+            ["run", "--webkit-port=MockPort", "--processes=4",
+                "--metadata=/mock-path/mock-wpt-tests-metadata",
+                "--manifest=/mock-path/mock-wpt-manifest.json",
+                "--include-manifest=/mock-checkout/WebPlatformTests/MockPort/TestManifest.ini",
+                "--webdriver-binary=/mock-webdriver/bin/webdriver",
+                "--binary=/mock-webdriver/bin/browser",
+                "--binary-arg=webdriver_arg1", "--binary-arg=webdriver_arg2", "webkit"] + specified_args)
+
+        options, _ = parse_args([])
+        instance = WPTRunnerTest.TestInstance(options, spawn_wpt_func)
+        instance.prepare_mock_files_for_run()
+
+        self.assertTrue(instance.runner.run(specified_args))
diff --git a/WebPlatformTests/gtk/TestExpectations.json b/WebPlatformTests/gtk/TestExpectations.json
new file mode 100644 (file)
index 0000000..a73b63f
--- /dev/null
@@ -0,0 +1,559 @@
+{
+    "2dcontext/building-paths/canvas_complexshapes_arcto_001.htm": {
+        "expected": "FAIL"
+    },
+    "2dcontext/building-paths/canvas_complexshapes_beziercurveto_001.htm": {
+        "expected": "FAIL"
+    },
+    "2dcontext/compositing/2d.composite.globalAlpha.canvascopy.html": {
+        "subtests": {
+            "Canvas test: 2d.composite.globalAlpha.canvascopy": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/compositing/2d.composite.operation.darker.html": {
+        "subtests": {
+            "Canvas test: 2d.composite.operation.darker": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/2d.drawImage.animated.apng.html": {
+        "subtests": {
+            "drawImage() of an APNG with no poster frame draws the first frame": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/2d.drawImage.animated.gif.html": {
+        "subtests": {
+            "drawImage() of an animated GIF draws the first frame": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/2d.drawImage.animated.poster.html": {
+        "subtests": {
+            "drawImage() of an APNG draws the poster frame": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/2d.drawImage.zerosource.html": {
+        "subtests": {
+            "drawImage with zero-sized source rectangle draws nothing without exception": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/2d.drawImage.zerosource.image.html": {
+        "subtests": {
+            "drawImage with zero-sized source rectangle from image throws INDEX_SIZE_ERR": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/drawimage_canvas.html": {
+        "subtests": {
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 0,20 should be blue.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 20,0 should be blue.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 20,20 should be blue.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 24,24 should be blue.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 0,0 should be black.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 0,19 should be black.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 19,0 should be black.": { "expected": "FAIL" },
+            "Test scenario 8: sx = 25, sy = 25, sw = 50, sh = 50, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 19,19 should be black.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 50,50 should be blue.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 50,99 should be blue.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 99,50 should be blue.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 99,99 should be blue.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 55,55 should be black.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 55,94 should be black.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 94,55 should be black.": { "expected": "FAIL" },
+            "Test scenario 9: sx = 0, sy = 0, sw = 50, sh = 50, dx = 100, dy = 100, dw = -50, dh = -50 --- Pixel 94,94 should be black.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 0,0 should be blue.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 1,1 should be blue.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 23,23 should be blue.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 24,24 should be blue.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 3,3 should be black.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 3,21 should be black.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 21,3 should be black.": { "expected": "FAIL" },
+            "Test scenario 11: sx = 0, sy = 0, sw = 100, sh = 100, dx = 0, dy = 0, dw = 50, dh = 50 --- Pixel 21,21 should be black.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 70,70 should be blue.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 70,99 should be blue.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 99,70 should be blue.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 82,82 should be blue.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 84,84 should be black.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 84,99 should be black.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 99,84 should be black.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 99,99 should be black.": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/drawimage_html_image.html": {
+        "subtests": {
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 70,99 should be light purple.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 99,70 should be light purple.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 99,99 should be light purple.": { "expected": "FAIL" },
+            "Test scenario 12: sx = -20, sy = -20, sw = 50, sh = 50, dx = 20, dy = 20, dw = 125, dh = 125 --- Pixel 69,69 should be red.": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-images-to-the-canvas/drawimage_svg_image_1.html": {
+        "subtests": {
+            "Load a 100x100 image to a SVG image and draw it to a 100x100 canvas.": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-paths-to-the-canvas/canvas_focus_drawCustomFocusRing_001.html": {
+        "subtests": {
+            "drawCustomFocusRing must return false for an element that is not focused.": { "expected": "FAIL" },
+            "drawCustomFocusRing must return false for an element that is not a descendant of the canvas element.": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/drawing-text-to-the-canvas/2d.text.draw.fontface.notinpage.html": {
+        "subtests": {
+            "@font-face fonts should work even if they are not used in the page": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-1.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-1": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-2.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-2": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-3.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-3": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-4.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-4": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-5.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-5": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-6.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-6": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-7.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-7": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-8.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-8": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsl-9.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsl-9": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-1.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-1": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-2.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-2": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-3.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-3": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-4.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-4": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-5.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-5": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-6.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-6": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-7.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-7": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-8.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-8": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-hsla-9.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-hsla-9": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgb-1.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgb-1": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgb-2.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgb-2": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgb-3.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgb-3": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgb-4.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgb-4": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgb-5.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgb-5": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgb-6.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgb-6": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgba-1.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgba-1": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgba-3.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgba-3": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgba-4.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgba-4": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgba-5.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgba-5": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.fillStyle.parse.css-color-4-rgba-6.html": {
+        "subtests": {
+            "Canvas test: 2d.fillStyle.parse.css-color-4-rgba-6": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.pattern.animated.gif.html": {
+        "subtests": {
+            "createPattern() of an animated GIF draws the first frame": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/fill-and-stroke-styles/2d.pattern.image.broken.html": {
+        "subtests": {
+            "Canvas test: 2d.pattern.image.broken": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/hit-regions/addHitRegions-NotSupportedError-01.html": {
+        "subtests": {
+            "fillRect should not affect current default path and NotSupportedError should be thrown.": { "expected": "FAIL" },
+            "strokeRect should not affect current default path and NotSupportedError should be thrown.": { "expected": "FAIL" },
+            "fillText should not affect current default path and NotSupportedError should be thrown.": { "expected": "FAIL" },
+            "strokeText should not affect current default path and NotSupportedError shuld be thrown.": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/hit-regions/hitregions-members-exist.html": {
+        "subtests": {
+            "context.addHitRegion Exists": { "expected": "FAIL" },
+            "context.removeHitRegion Exists": { "expected": "FAIL" },
+            "context.clearHitRegions Exists": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/image-smoothing/imagesmoothing.html": {
+        "subtests": {
+            "Test that imageSmoothingEnabled = false (nearest-neighbor interpolation) works with fillRect and createPattern().": { "expected": "FAIL" },
+            "Test that imageSmoothingEnabled = false (nearest-neighbor interpolation) works with fill() and createPattern().": { "expected": "FAIL" },
+            "Test that imageSmoothingEnabled = false (nearest-neighbor interpolation) works with stroke() and createPattern().": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/imagebitmap/createImageBitmap-drawImage.html": {
+        "subtests": {
+            "createImageBitmap from an HTMLCanvasElement with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an HTMLVideoElement with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an HTMLVideoElement from a data URL with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a bitmap HTMLImageElement with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a vector HTMLImageElement with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a bitmap SVGImageElement, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a bitmap SVGImageElement scaled down, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a bitmap SVGImageElement scaled up, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a bitmap SVGImageElement resized, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a bitmap SVGImageElement with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a vector SVGImageElement, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a vector SVGImageElement scaled down, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a vector SVGImageElement scaled up, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a vector SVGImageElement resized, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a vector SVGImageElement with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an OffscreenCanvas, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an OffscreenCanvas scaled down, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an OffscreenCanvas scaled up, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an OffscreenCanvas resized, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an OffscreenCanvas with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an ImageData, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an ImageData scaled down, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an ImageData scaled up, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an ImageData resized, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an ImageData with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from an ImageBitmap with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a Blob, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a Blob scaled down, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a Blob scaled up, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a Blob resized, and drawImage on the created ImageBitmap": { "expected": "FAIL" },
+            "createImageBitmap from a Blob with negative sw/sh, and drawImage on the created ImageBitmap": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/imagebitmap/createImageBitmap-invalid-args.html": {
+        "subtests": {
+            "createImageBitmap with a an HTMLCanvasElement source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a an HTMLVideoElement source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a an HTMLVideoElement from a data URL source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a a bitmap HTMLImageElement source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a a vector HTMLImageElement source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a a bitmap SVGImageElement source and sw set to 0": { "expected": "FAIL" },
+            "createImageBitmap with a a bitmap SVGImageElement source and sh set to 0": { "expected": "FAIL" },
+            "createImageBitmap with a a bitmap SVGImageElement source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a a vector SVGImageElement source and sw set to 0": { "expected": "FAIL" },
+            "createImageBitmap with a a vector SVGImageElement source and sh set to 0": { "expected": "FAIL" },
+            "createImageBitmap with a a vector SVGImageElement source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a an OffscreenCanvas source and sw set to 0": { "expected": "FAIL" },
+            "createImageBitmap with a an OffscreenCanvas source and sh set to 0": { "expected": "FAIL" },
+            "createImageBitmap with a an OffscreenCanvas source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a an ImageData source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a an ImageBitmap source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with a a Blob source and oversized (unallocatable) crop region": { "expected": "FAIL" },
+            "createImageBitmap with an invalid OffscreenCanvas source.": { "expected": "FAIL" },
+            "createImageBitmap with an undecodable blob source.": { "expected": "FAIL" },
+            "createImageBitmap with an available but undecodable image source.": { "expected": "FAIL" },
+            "createImageBitmap with an available but zero height image source.": { "expected": "FAIL" },
+            "createImageBitmap with an available but zero width image source.": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/imagebitmap/createImageBitmap-origin.sub.html": {
+        "subtests": {
+            "cross-origin SVGImageElement": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/imagebitmap/createImageBitmap-sizeOverflow.html": {
+        "subtests": {
+            "createImageBitmap does not crash or reject the promise when passing very large sx": { "expected": "FAIL" },
+            "createImageBitmap does not crash or reject the promise when passing very large sy": { "expected": "FAIL" },
+            "createImageBitmap does not crash or reject the promise when passing very large sw": { "expected": "FAIL" },
+            "createImageBitmap does not crash or reject the promise when passing very large sh": { "expected": "FAIL" },
+            "createImageBitmap does not crash or reject the promise when passing very large sx, sy, sw and sh": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/pixel-manipulation/2d.imageData.create2.double.html": {
+        "subtests": {
+            "createImageData(w, h) double is converted to long": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/pixel-manipulation/2d.imageData.create2.zero.html": {
+        "subtests": {
+            "createImageData(sw, sh) throws INDEX_SIZE_ERR if size is zero": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/pixel-manipulation/2d.imageData.get.double.html": {
+        "subtests": {
+            "createImageData(w, h) double is converted to long": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/pixel-manipulation/2d.imageData.get.zero.html": {
+        "subtests": {
+            "getImageData() throws INDEX_SIZE_ERR if size is zero": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/pixel-manipulation/2d.imageData.object.ctor.array.html": {
+        "subtests": {
+            "ImageData has a usable constructor": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.baseline.bottom.html": {
+        "subtests": {
+            "textBaseline bottom is the bottom of the em square (not the bounding box)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.baseline.hanging.html": {
+        "subtests": {
+            "Canvas test: 2d.text.draw.baseline.hanging": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.baseline.ideographic.html": {
+        "subtests": {
+            "Canvas test: 2d.text.draw.baseline.ideographic": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.baseline.middle.html": {
+        "subtests": {
+            "textBaseline middle is the middle of the em square (not the bounding box)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.baseline.top.html": {
+        "subtests": {
+            "textBaseline top is the top of the em square (not the bounding box)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.space.collapse.end.html": {
+        "subtests": {
+            "Space characters at the end of a line are collapsed (per CSS)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.space.collapse.other.html": {
+        "subtests": {
+            "Space characters are converted to U+0020, and collapsed (per CSS)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.space.collapse.space.html": {
+        "subtests": {
+            "Space characters are converted to U+0020, and collapsed (per CSS)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.draw.space.collapse.start.html": {
+        "subtests": {
+            "Space characters at the start of a line are collapsed (per CSS)": { "expected": "FAIL" }
+        }
+    },
+    "2dcontext/text-styles/2d.text.measure.width.space.html": {
+        "subtests": {
+            "Space characters are converted to U+0020 and collapsed (per CSS)": { "expected": "FAIL" }
+        }
+    },
+
+    "WebCryptoAPI/derive_bits_keys/ecdh_bits.https.worker.js": {
+        "test_name": "ecdh_bits.https.worker.html",
+        "subtests": {
+            "P-256 non-multiple of 8 bits": { "expected": "FAIL" },
+            "P-384 non-multiple of 8 bits": { "expected": "FAIL" },
+            "P-521 non-multiple of 8 bits": { "expected": "FAIL" }
+        }
+    },
+    "WebCryptoAPI/derive_bits_keys/hkdf.https.worker.js": {
+        "test_name": "hkdf.https.worker.html",
+        "disabled": "Lousy HKDF support"
+    },
+    "WebCryptoAPI/derive_bits_keys/pbkdf2.https.worker.js": {
+        "test_name": "pbkdf2.https.worker.html",
+        "disabled": "Lousy PBKDF2 support"
+    },
+    "WebCryptoAPI/derive_bits_keys/test_ecdh_bits.https.html": {
+        "subtests": {
+            "P-256 non-multiple of 8 bits": { "expected": "FAIL" },
+            "P-384 non-multiple of 8 bits": { "expected": "FAIL" },
+            "P-521 non-multiple of 8 bits": { "expected": "FAIL" }
+        }
+    },
+    "WebCryptoAPI/derive_bits_keys/test_hkdf.https.html": {
+        "disabled": "Lousy HKDF support"
+    },
+    "WebCryptoAPI/derive_bits_keys/test_pbkdf2_empty_empty.https.html": {
+        "disabled": "Lousy PBKDF2 support"
+    },
+    "WebCryptoAPI/derive_bits_keys/test_pbkdf2_empty_long.https.html": {
+        "disabled": "Lousy PBKDF2 support"
+    },
+    "WebCryptoAPI/derive_bits_keys/test_pbkdf2_empty_short.https.html": {
+        "disabled": "Lousy PBKDF2 support"
+    },
+    "WebCryptoAPI/derive_bits_keys/test_pbkdf2_long_empty.https.html": {
+        "disabled": "Lousy PBKDF2 support"
+    },
+    "WebCryptoAPI/derive_bits_keys/test_pbkdf2_short_empty.https.html": {
+        "disabled": "Lousy PBKDF2 support"
+    },
+    "WebCryptoAPI/encrypt_decrypt/aes_gcm.https.worker.js": {
+        "test_name": "aes_gcm.https.worker.html",
+        "subtests": {
+            "AES-GCM 128-bit key, illegal tag length 256-bits": { "expected": "FAIL" },
+            "AES-GCM 192-bit key, illegal tag length 256-bits": { "expected": "FAIL" },
+            "AES-GCM 256-bit key, illegal tag length 256-bits": { "expected": "FAIL" },
+            "AES-GCM 128-bit key, illegal tag length 256-bits decryption": { "expected": "FAIL" },
+            "AES-GCM 192-bit key, illegal tag length 256-bits decryption": { "expected": "FAIL" },
+            "AES-GCM 256-bit key, illegal tag length 256-bits decryption": { "expected": "FAIL" }
+        }
+    },
+    "WebCryptoAPI/encrypt_decrypt/test_aes_gcm.https.html": {
+        "subtests": {
+            "AES-GCM 128-bit key, illegal tag length 256-bits": { "expected": "FAIL" },
+            "AES-GCM 192-bit key, illegal tag length 256-bits": { "expected": "FAIL" },
+            "AES-GCM 256-bit key, illegal tag length 256-bits": { "expected": "FAIL" },
+            "AES-GCM 128-bit key, illegal tag length 256-bits decryption": { "expected": "FAIL" },
+            "AES-GCM 192-bit key, illegal tag length 256-bits decryption": { "expected": "FAIL" },
+            "AES-GCM 256-bit key, illegal tag length 256-bits decryption": { "expected": "FAIL" }
+        }
+    },
+    "WebCryptoAPI/idlharness.https.worker.js": {
+        "test_name": "idlharness.https.worker.html",
+        "subtests": {
+            "ArrayBuffer interface: existence and properties of interface object": { "expected": "FAIL" }
+        }
+    },
+    "WebCryptoAPI/secure_context/crypto-subtle-non-secure-context-not-available.sub.html": {
+        "subtests": {
+            "Non-secure context window does not have access to crypto.subtle": { "expected": "FAIL" },
+            "Non-secure context worker does not have access to crypto.subtle": { "expected": "FAI:" }
+        }
+    },
+    "WebCryptoAPI/wrapKey_unwrapKey/test_wrapKey_unwrapKey.https.html": {
+        "subtests": {
+            "Can wrap and unwrap ECDSA public key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDSA private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDSA private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDH public key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDH private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDH private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CTR keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CTR keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CBC keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CBC keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-GCM keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-GCM keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-KW keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-KW keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap HMAC keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap HMAC keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSASSA-PKCS1-v1_5 private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSASSA-PKCS1-v1_5 private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-PSS private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-PSS private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-OAEP private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-OAEP private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" }
+        }
+    },
+    "WebCryptoAPI/wrapKey_unwrapKey/wrapKey_unwrapKey.https.worker.js": {
+        "test_name": "wrapKey_unwrapKey.https.worker.html",
+        "subtests": {
+            "Can wrap and unwrap ECDSA public key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDSA private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDSA private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDH public key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDH private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap ECDH private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CTR keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CTR keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CBC keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-CBC keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-GCM keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-GCM keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-KW keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap AES-KW keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap HMAC keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap HMAC keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSASSA-PKCS1-v1_5 private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSASSA-PKCS1-v1_5 private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-PSS private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-PSS private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-OAEP private key keys using jwk and AES-KW": { "expected": "FAIL" },
+            "Can wrap and unwrap RSA-OAEP private key keys as non-extractable using jwk and AES-KW": { "expected": "FAIL" }
+        }
+    }
+}
+
diff --git a/WebPlatformTests/gtk/TestManifest.ini b/WebPlatformTests/gtk/TestManifest.ini
new file mode 100644 (file)
index 0000000..69fb98a
--- /dev/null
@@ -0,0 +1,7 @@
+skip: true
+
+[2dcontext]
+    skip: false
+
+[WebCryptoAPI]
+    skip: false