Add run-web-platform-tests script
authorzandobersek@gmail.com <zandobersek@gmail.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 12 Jun 2018 07:55:42 +0000 (07:55 +0000)
committerzandobersek@gmail.com <zandobersek@gmail.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 12 Jun 2018 07:55:42 +0000 (07:55 +0000)
commit915e129245b394172e88856da032df7c35320ac0
tree2140d8e110e9f3621a8ff863be310137c61d879d
parent0a7fa32583f70fb881b6fb1862460d28936ae839
Add run-web-platform-tests script
https://bugs.webkit.org/show_bug.cgi?id=183356

Reviewed by Carlos Alberto Lopez Perez.

.:

Store port-specific test expectations and test manifest files for the
web-platform-tests test runs under the top-level WebPlatformTests
directory.

TestExpectations.json file lists all the failing or disabled tests
or subtests that are to be expected in the test run. This information is
parsed in the run-web-platform-tests script and used to build
test-specific metadata files (ending with .ini) inside a temporary
directory that is then used during the test run. JSON format is used to
follow the WebDriver test suite in how it manages expectations, and to
avoid having to manually manage .ini files for every deviant test case.

TestManifest.ini file is used to fine-tune which tests are to be enabled
by default. For the GTK+ port, we currently disable all tests by default
but then specifically enable tests under the 2dcontext and WebCryptoAPI
directories. This will allow for gradual enabling of further tests.

* WebPlatformTests/gtk/TestExpectations.json: Added.
* WebPlatformTests/gtk/TestManifest.ini: Added.

Tools:

Add the run-web-platform-tests script that allows running WebKit builds
against the external web-platform-tests project.

The script sets up all the port-specific information, including the
display driver. If no web-platform-tests repository location is
specified, the TestDownloader utility is used to clone the repository,
and path to that clone is leveraged instead.

Port-specific TestManifest.ini and TestExpectations.json file paths are
then constructed, bailing if any of the files is missing. The JSON file
is then used to construct the metadata file tree, creating an .ini file
for each test that is disabled, has a certain expectation, or has
subtests that themselves have certain expectations. This metadata tree
is built under the build directory, and is purged and reconstructed for
each separate run.

Lastly, the WPT python module is imported from the web-platform-tests
source location, and wpt.main() is invoked along with all the necessary
arguments that set up a web-platform-tests suite run using the WebKit
product.

* Scripts/run-web-platform-tests: Added.
* Scripts/webkitpy/common/config/ports_mock.py:
(MockPort.run_bindings_tests_command):
(MockPort):
(MockPort.wpt_metadata_directory):
(MockPort.wpt_manifest_file):
* Scripts/webkitpy/port/base.py:
(Port.wpt_metadata_directory):
(Port):
(Port.wpt_manifest_file):
* Scripts/webkitpy/w3c/common.py:
(is_file_exportable):
(WPTPaths):
(WPTPaths.checkout_directory):
(WPTPaths.wpt_checkout_path):
* Scripts/webkitpy/w3c/test_downloader.py:
(TestDownloader.clone_tests):
(TestDownloader):
(TestDownloader.download_tests):
* Scripts/webkitpy/w3c/test_exporter.py:
(WebPlatformTestExporter.__init__):
* Scripts/webkitpy/w3c/wpt_runner.py: Added.
(main):
(parse_args):
(create_webdriver):
(spawn_wpt):
(WPTRunner):
(WPTRunner.__init__):
(WPTRunner._prepare_wpt_checkout):
(WPTRunner._generate_metadata_directory):
(WPTRunner.run):
* Scripts/webkitpy/w3c/wpt_runner_unittest.py: Added.
(WPTRunnerTest):
(WPTRunnerTest.MockTestDownloader):
(WPTRunnerTest.MockTestDownloader.default_options):
(WPTRunnerTest.MockTestDownloader.__init__):
(WPTRunnerTest.MockTestDownloader.clone_tests):
(WPTRunnerTest.MockWebDriver):
(WPTRunnerTest.MockWebDriver.create):
(WPTRunnerTest.MockWebDriver.binary_path):
(WPTRunnerTest.MockWebDriver.browser_path):
(WPTRunnerTest.MockWebDriver.browser_args):
(WPTRunnerTest.MockSpawnWPT):
(WPTRunnerTest.MockSpawnWPT.__init__):
(WPTRunnerTest.MockSpawnWPT.__call__):
(WPTRunnerTest.TestInstance):
(WPTRunnerTest.TestInstance.__init__):
(WPTRunnerTest.TestInstance.prepare_mock_files_for_run):
(WPTRunnerTest.test_prepare_wpt_checkout):
(WPTRunnerTest.test_prepare_wpt_checkout_specified_path):
(WPTRunnerTest.test_generate_metadata_directory):
(WPTRunnerTest.test_run):
(WPTRunnerTest.test_run_with_specified_options):
(WPTRunnerTest.test_run_with_args):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@232746 268f45cc-cd09-0410-ab3c-d52691b4dbfc
12 files changed:
ChangeLog
Tools/ChangeLog
Tools/Scripts/run-web-platform-tests [new file with mode: 0755]
Tools/Scripts/webkitpy/common/config/ports_mock.py
Tools/Scripts/webkitpy/port/base.py
Tools/Scripts/webkitpy/w3c/common.py
Tools/Scripts/webkitpy/w3c/test_downloader.py
Tools/Scripts/webkitpy/w3c/test_exporter.py
Tools/Scripts/webkitpy/w3c/wpt_runner.py [new file with mode: 0644]
Tools/Scripts/webkitpy/w3c/wpt_runner_unittest.py [new file with mode: 0644]
WebPlatformTests/gtk/TestExpectations.json [new file with mode: 0644]
WebPlatformTests/gtk/TestManifest.ini [new file with mode: 0644]