Test infrastructure should allow to filter out text output before doing a comparison...
authorcommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 14 Dec 2016 12:22:47 +0000 (12:22 +0000)
committercommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 14 Dec 2016 12:22:47 +0000 (12:22 +0000)
commitd166bc3259005a6ae43ed72e2a2d5234397390ab
tree8fa53ba0e70883bbfbfff2069f6b7b3e3d6a4caa
parentf38b95a8bee635522fbef85428e5138c6c2d68f7
Test infrastructure should allow to filter out text output before doing a comparison with the baseline
https://bugs.webkit.org/show_bug.cgi?id=161310

Patch by Youenn Fablet <youenn@apple.com> on 2016-12-14
Reviewed by Ryosuke Niwa.

LayoutTests/imported/w3c:

* web-platform-tests/fetch/api/cors/cors-preflight.html: Adding a console log line to ensure the DumpJSConsoleLogInStdErr modifier works.

Tools:

Adding --dump-jsconsolelog-in-stderr test option for DRT and WTR.
When set, WTR and DRT output JS console log in the stderr and not the test output.
This allows removing potentially flaky information while still being able to read it for investigation.

Updated webkitpy infrastructure to use this test option in case of DumpJSConsoleLogInStdErr modifier.

Changes are covered by updating a fetch test to output console log and set its option to --dump-jsconsolelog-in-stderr in TestExpectations.
This test should have this option when fetch API will start logging resource loader errors.

* DumpRenderTree/DumpRenderTree.h:
(TestCommand::TestCommand): Deleted.
* DumpRenderTree/DumpRenderTreeCommon.cpp:
(parseInputLine):
* DumpRenderTree/TestRunner.h:
(TestRunner::setDumpJSConsoleLogInStdErr):
(TestRunner::dumpJSConsoleLogInStdErr):
* DumpRenderTree/mac/DumpRenderTree.mm:
(runTest):
* DumpRenderTree/mac/UIDelegate.mm:
(-[UIDelegate webView:addMessageToConsole:withSource:]):
* DumpRenderTree/win/DumpRenderTree.cpp:
(runTest):
* DumpRenderTree/win/UIDelegate.cpp:
(UIDelegate::webViewAddMessageToConsole):
* Scripts/webkitpy/layout_tests/models/test_expectations.py:
(TestExpectationParser._tokenize_line):
* Scripts/webkitpy/port/driver.py:
(Driver._command_from_driver_input):
* WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:
(WTR::InjectedBundle::didReceiveMessageToPage):
(WTR::InjectedBundle::dumpToStdErr):
* WebKitTestRunner/InjectedBundle/InjectedBundle.h:
(WTR::InjectedBundle::dumpJSConsoleLogInStdErr):
* WebKitTestRunner/InjectedBundle/InjectedBundlePage.cpp:
(WTR::InjectedBundlePage::willAddMessageToConsole):
* WebKitTestRunner/TestController.cpp:
(WTR::parseInputLine):
(WTR::TestController::runTest):
(WTR::TestCommand::TestCommand): Deleted.
* WebKitTestRunner/TestInvocation.cpp:
(WTR::TestInvocation::invoke):
(WTR::TestInvocation::didReceiveMessageFromInjectedBundle):
* WebKitTestRunner/TestInvocation.h:
(WTR::TestInvocation::setDumpJSConsoleLogInStdErr):

LayoutTests:

* TestExpectations: Adding DumpJSConsoleLogInStdErr to the fetch modified test to ensure this modifier works as expected.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@209798 268f45cc-cd09-0410-ab3c-d52691b4dbfc
23 files changed:
LayoutTests/ChangeLog
LayoutTests/TestExpectations
LayoutTests/imported/w3c/ChangeLog
LayoutTests/imported/w3c/web-platform-tests/fetch/api/cors/cors-preflight.html
Tools/ChangeLog
Tools/DumpRenderTree/DumpRenderTree.h
Tools/DumpRenderTree/DumpRenderTreeCommon.cpp
Tools/DumpRenderTree/TestRunner.h
Tools/DumpRenderTree/mac/DumpRenderTree.mm
Tools/DumpRenderTree/mac/UIDelegate.mm
Tools/DumpRenderTree/win/DumpRenderTree.cpp
Tools/DumpRenderTree/win/UIDelegate.cpp
Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py
Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py
Tools/Scripts/webkitpy/layout_tests/models/test_input.py
Tools/Scripts/webkitpy/port/driver.py
Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.cpp
Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.h
Tools/WebKitTestRunner/InjectedBundle/InjectedBundlePage.cpp
Tools/WebKitTestRunner/TestController.cpp
Tools/WebKitTestRunner/TestInvocation.cpp
Tools/WebKitTestRunner/TestInvocation.h