Test infrastructure should allow to filter out text output before doing a comparison...
authorcommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 14 Dec 2016 12:22:47 +0000 (12:22 +0000)
committercommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 14 Dec 2016 12:22:47 +0000 (12:22 +0000)
https://bugs.webkit.org/show_bug.cgi?id=161310

Patch by Youenn Fablet <youenn@apple.com> on 2016-12-14
Reviewed by Ryosuke Niwa.

LayoutTests/imported/w3c:

* web-platform-tests/fetch/api/cors/cors-preflight.html: Adding a console log line to ensure the DumpJSConsoleLogInStdErr modifier works.

Tools:

Adding --dump-jsconsolelog-in-stderr test option for DRT and WTR.
When set, WTR and DRT output JS console log in the stderr and not the test output.
This allows removing potentially flaky information while still being able to read it for investigation.

Updated webkitpy infrastructure to use this test option in case of DumpJSConsoleLogInStdErr modifier.

Changes are covered by updating a fetch test to output console log and set its option to --dump-jsconsolelog-in-stderr in TestExpectations.
This test should have this option when fetch API will start logging resource loader errors.

* DumpRenderTree/DumpRenderTree.h:
(TestCommand::TestCommand): Deleted.
* DumpRenderTree/DumpRenderTreeCommon.cpp:
(parseInputLine):
* DumpRenderTree/TestRunner.h:
(TestRunner::setDumpJSConsoleLogInStdErr):
(TestRunner::dumpJSConsoleLogInStdErr):
* DumpRenderTree/mac/DumpRenderTree.mm:
(runTest):
* DumpRenderTree/mac/UIDelegate.mm:
(-[UIDelegate webView:addMessageToConsole:withSource:]):
* DumpRenderTree/win/DumpRenderTree.cpp:
(runTest):
* DumpRenderTree/win/UIDelegate.cpp:
(UIDelegate::webViewAddMessageToConsole):
* Scripts/webkitpy/layout_tests/models/test_expectations.py:
(TestExpectationParser._tokenize_line):
* Scripts/webkitpy/port/driver.py:
(Driver._command_from_driver_input):
* WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:
(WTR::InjectedBundle::didReceiveMessageToPage):
(WTR::InjectedBundle::dumpToStdErr):
* WebKitTestRunner/InjectedBundle/InjectedBundle.h:
(WTR::InjectedBundle::dumpJSConsoleLogInStdErr):
* WebKitTestRunner/InjectedBundle/InjectedBundlePage.cpp:
(WTR::InjectedBundlePage::willAddMessageToConsole):
* WebKitTestRunner/TestController.cpp:
(WTR::parseInputLine):
(WTR::TestController::runTest):
(WTR::TestCommand::TestCommand): Deleted.
* WebKitTestRunner/TestInvocation.cpp:
(WTR::TestInvocation::invoke):
(WTR::TestInvocation::didReceiveMessageFromInjectedBundle):
* WebKitTestRunner/TestInvocation.h:
(WTR::TestInvocation::setDumpJSConsoleLogInStdErr):

LayoutTests:

* TestExpectations: Adding DumpJSConsoleLogInStdErr to the fetch modified test to ensure this modifier works as expected.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@209798 268f45cc-cd09-0410-ab3c-d52691b4dbfc

23 files changed:
LayoutTests/ChangeLog
LayoutTests/TestExpectations
LayoutTests/imported/w3c/ChangeLog
LayoutTests/imported/w3c/web-platform-tests/fetch/api/cors/cors-preflight.html
Tools/ChangeLog
Tools/DumpRenderTree/DumpRenderTree.h
Tools/DumpRenderTree/DumpRenderTreeCommon.cpp
Tools/DumpRenderTree/TestRunner.h
Tools/DumpRenderTree/mac/DumpRenderTree.mm
Tools/DumpRenderTree/mac/UIDelegate.mm
Tools/DumpRenderTree/win/DumpRenderTree.cpp
Tools/DumpRenderTree/win/UIDelegate.cpp
Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py
Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py
Tools/Scripts/webkitpy/layout_tests/models/test_input.py
Tools/Scripts/webkitpy/port/driver.py
Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.cpp
Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.h
Tools/WebKitTestRunner/InjectedBundle/InjectedBundlePage.cpp
Tools/WebKitTestRunner/TestController.cpp
Tools/WebKitTestRunner/TestInvocation.cpp
Tools/WebKitTestRunner/TestInvocation.h

index 31eccfd..1ed70df 100644 (file)
@@ -1,3 +1,12 @@
+2016-12-14  Youenn Fablet  <youenn@apple.com>
+
+        Test infrastructure should allow to filter out text output before doing a comparison with the baseline
+        https://bugs.webkit.org/show_bug.cgi?id=161310
+
+        Reviewed by Ryosuke Niwa.
+
+        * TestExpectations: Adding DumpJSConsoleLogInStdErr to the fetch modified test to ensure this modifier works as expected.
+
 2016-12-14  Enrique Ocaña González  <eocanha@igalia.com>
 
         REGRESSION(r207879-207891): [GStreamer] Introduced many layout test failures and crashes, bots exiting early
index 681cb6d..1ab3406 100644 (file)
@@ -298,6 +298,8 @@ webkit.org/b/157068 imported/w3c/web-platform-tests/fetch/nosniff/importscripts.
 webkit.org/b/157145 imported/w3c/web-platform-tests/fetch/nosniff/stylesheet.html [ Failure Pass ]
 webkit.org/b/161312 imported/w3c/web-platform-tests/html/semantics/document-metadata/the-link-element/document-without-browsing-context.html [ Failure Pass ]
 
+imported/w3c/web-platform-tests/fetch/api/cors/cors-preflight.html [ DumpJSConsoleLogInStdErr ]
+
 webkit.org/b/159724 imported/w3c/web-platform-tests/XMLHttpRequest/send-redirect-post-upload.htm [ Failure Pass ]
 
 # New W3C ref tests that are failing.
index 9136cc2..e1bf5c0 100644 (file)
@@ -1,3 +1,12 @@
+2016-12-14  Youenn Fablet  <youenn@apple.com>
+
+        Test infrastructure should allow to filter out text output before doing a comparison with the baseline
+        https://bugs.webkit.org/show_bug.cgi?id=161310
+
+        Reviewed by Ryosuke Niwa.
+
+        * web-platform-tests/fetch/api/cors/cors-preflight.html: Adding a console log line to ensure the DumpJSConsoleLogInStdErr modifier works.
+
 2016-12-12  Darin Adler  <darin@apple.com>
 
         Remove bindings generation support for legacy WebCore::Dictionary
index 036f2f5..e3b1352 100644 (file)
     <script src="../resources/utils.js"></script>
     <script src="/common/get-host-info.sub.js"></script>
     <script src="/common/utils.js"></script>
+    <script>
+// To be removed, see https://bugs.webkit.org/show_bug.cgi?id=161310
+console.log("Doing some logging once")
+console.log("Doing some logging again")
+    </script>
     <script src="cors-preflight.js"></script>
   </body>
 </html>
index 12cf00a..cc2f9b6 100644 (file)
@@ -1,3 +1,56 @@
+2016-12-14  Youenn Fablet  <youenn@apple.com>
+
+        Test infrastructure should allow to filter out text output before doing a comparison with the baseline
+        https://bugs.webkit.org/show_bug.cgi?id=161310
+
+        Reviewed by Ryosuke Niwa.
+
+        Adding --dump-jsconsolelog-in-stderr test option for DRT and WTR.
+        When set, WTR and DRT output JS console log in the stderr and not the test output.
+        This allows removing potentially flaky information while still being able to read it for investigation.
+
+        Updated webkitpy infrastructure to use this test option in case of DumpJSConsoleLogInStdErr modifier.
+
+        Changes are covered by updating a fetch test to output console log and set its option to --dump-jsconsolelog-in-stderr in TestExpectations.
+        This test should have this option when fetch API will start logging resource loader errors.
+
+        * DumpRenderTree/DumpRenderTree.h:
+        (TestCommand::TestCommand): Deleted.
+        * DumpRenderTree/DumpRenderTreeCommon.cpp:
+        (parseInputLine):
+        * DumpRenderTree/TestRunner.h:
+        (TestRunner::setDumpJSConsoleLogInStdErr):
+        (TestRunner::dumpJSConsoleLogInStdErr):
+        * DumpRenderTree/mac/DumpRenderTree.mm:
+        (runTest):
+        * DumpRenderTree/mac/UIDelegate.mm:
+        (-[UIDelegate webView:addMessageToConsole:withSource:]):
+        * DumpRenderTree/win/DumpRenderTree.cpp:
+        (runTest):
+        * DumpRenderTree/win/UIDelegate.cpp:
+        (UIDelegate::webViewAddMessageToConsole):
+        * Scripts/webkitpy/layout_tests/models/test_expectations.py:
+        (TestExpectationParser._tokenize_line):
+        * Scripts/webkitpy/port/driver.py:
+        (Driver._command_from_driver_input):
+        * WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:
+        (WTR::InjectedBundle::didReceiveMessageToPage):
+        (WTR::InjectedBundle::dumpToStdErr):
+        * WebKitTestRunner/InjectedBundle/InjectedBundle.h:
+        (WTR::InjectedBundle::dumpJSConsoleLogInStdErr):
+        * WebKitTestRunner/InjectedBundle/InjectedBundlePage.cpp:
+        (WTR::InjectedBundlePage::willAddMessageToConsole):
+        * WebKitTestRunner/TestController.cpp:
+        (WTR::parseInputLine):
+        (WTR::TestController::runTest):
+        (WTR::TestCommand::TestCommand): Deleted.
+        * WebKitTestRunner/TestInvocation.cpp:
+        (WTR::TestInvocation::invoke):
+        (WTR::TestInvocation::didReceiveMessageFromInjectedBundle):
+        * WebKitTestRunner/TestInvocation.h:
+        (WTR::TestInvocation::setDumpJSConsoleLogInStdErr):
+
+
 2016-12-14  Enrique Ocaña González  <eocanha@igalia.com>
 
         REGRESSION(r207879-207891): [GStreamer] Introduced many layout test failures and crashes, bots exiting early
index 304232b..44aa2bf 100644 (file)
@@ -57,12 +57,11 @@ void dump();
 void displayWebView();
 
 struct TestCommand {
-    TestCommand() : shouldDumpPixels(false), timeout(30000) { }
-
     std::string pathOrURL;
-    bool shouldDumpPixels;
+    bool shouldDumpPixels { false };
     std::string expectedPixelHash;
-    int timeout; // in ms
+    int timeout { 30000 }; // in ms
+    bool dumpJSConsoleLogInStdErr { false };
 };
 
 TestCommand parseInputLine(const std::string&);
index 3cb97eb..5f64894 100644 (file)
@@ -1,3 +1,28 @@
+/*
+ * Copyright (C) 2016 Apple Inc. All Rights Reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
 #include "config.h"
 #include "DumpRenderTree.h"
 
@@ -70,14 +95,16 @@ TestCommand parseInputLine(const std::string& inputLine)
     result.pathOrURL = arg;
     while (tokenizer.hasNext()) {
         arg = tokenizer.next();
-        if (arg == std::string("--timeout")) {
+        if (arg == "--timeout") {
             std::string timeoutToken = tokenizer.next();
             result.timeout = atoi(timeoutToken.c_str());
-        } else if (arg == std::string("-p") || arg == std::string("--pixel-test")) {
+        } else if (arg == "-p" || arg == "--pixel-test") {
             result.shouldDumpPixels = true;
             if (tokenizer.hasNext())
                 result.expectedPixelHash = tokenizer.next();
-        } else
+        } else if (arg == "--dump-jsconsolelog-in-stderr")
+            result.dumpJSConsoleLogInStdErr = true;
+        else
             die(inputLine);
     }
 
index a8d49c6..1d3a5c2 100644 (file)
@@ -367,9 +367,12 @@ public:
     double timeout() { return m_timeout; }
 
     unsigned imageCountInGeneralPasteboard() const;
-    
+
     void callUIScriptCallback(unsigned callbackID, JSStringRef result);
 
+    void setDumpJSConsoleLogInStdErr(bool inStdErr) { m_dumpJSConsoleLogInStdErr = inStdErr; }
+    bool dumpJSConsoleLogInStdErr() const { return m_dumpJSConsoleLogInStdErr; }
+
 private:
     TestRunner(const std::string& testURL, const std::string& expectedPixelHash);
 
@@ -434,6 +437,7 @@ private:
     bool m_areLegacyWebNotificationPermissionRequestsIgnored;
     bool m_customFullScreenBehavior;
     bool m_hasPendingWebNotificationClick;
+    bool m_dumpJSConsoleLogInStdErr { false };
 
     double m_databaseDefaultQuota;
     double m_databaseMaxQuota;
index b0956f9..846867b 100644 (file)
@@ -2001,6 +2001,7 @@ static void runTest(const string& inputLine)
     gTestRunner = TestRunner::create(testURL, command.expectedPixelHash);
     gTestRunner->setAllowedHosts(allowedHosts);
     gTestRunner->setCustomTimeout(command.timeout);
+    gTestRunner->setDumpJSConsoleLogInStdErr(command.dumpJSConsoleLogInStdErr);
     topLoadingFrame = nil;
 #if !PLATFORM(IOS)
     ASSERT(!draggingInfo); // the previous test should have called eventSender.mouseUp to drop!
index 18a58ce..f86f522 100644 (file)
@@ -82,10 +82,11 @@ DumpRenderTreeDraggingInfo *draggingInfo = nil;
     if (range.location != NSNotFound)
         message = [[message substringToIndex:range.location] stringByAppendingString:[[message substringFromIndex:NSMaxRange(range)] lastPathComponent]];
 
-    printf ("CONSOLE MESSAGE: ");
+    auto out = gTestRunner->dumpJSConsoleLogInStdErr() ? stderr : stdout;
+    fprintf(out, "CONSOLE MESSAGE: ");
     if ([lineNumber intValue])
-        printf ("line %d: ", [lineNumber intValue]);
-    printf ("%s\n", [message UTF8String]);
+        fprintf(out, "line %d: ", [lineNumber intValue]);
+    fprintf(out, "%s\n", [message UTF8String]);
 }
 
 - (void)modalWindowWillClose:(NSNotification *)notification
index 493d8f3..d454b11 100644 (file)
@@ -1121,6 +1121,8 @@ static void runTest(const string& inputLine)
 
     ::gTestRunner = TestRunner::create(testURL.data(), command.expectedPixelHash);
     ::gTestRunner->setCustomTimeout(command.timeout);
+    ::gTestRunner->setDumpJSConsoleLogInStdErr(command.dumpJSConsoleLogInStdErr);
+
     topLoadingFrame = nullptr;
     done = false;
 
index ddd8ead..e60d5e8 100644 (file)
@@ -507,10 +507,11 @@ HRESULT UIDelegate::webViewAddMessageToConsole(_In_opt_ IWebView* /*sender*/, _I
             newMessage = newMessage.substr(0, fileProtocol) + lastPathComponent(newMessage.substr(fileProtocol + fileURL.size()));
     }
 
-    printf("CONSOLE MESSAGE: ");
+    auto out = gTestRunner->dumpJSConsoleLogInStdErr() ? stderr : stdout;
+    fprintf(out, "CONSOLE MESSAGE: ");
     if (lineNumber)
-        printf("line %d: ", lineNumber);
-    printf("%s\n", toUTF8(newMessage).c_str());
+        fprintf(out, "line %d: ", lineNumber);
+    fprintf(out, "%s\n", toUTF8(newMessage).c_str());
     return S_OK;
 }
 
index 0e5a303..9e412f4 100644 (file)
@@ -131,13 +131,17 @@ class Manager(object):
     def _test_input_for_file(self, test_file):
         return TestInput(test_file,
             self._options.slow_time_out_ms if self._test_is_slow(test_file) else self._options.time_out_ms,
-            self._is_http_test(test_file))
+            self._is_http_test(test_file),
+            should_dump_jsconsolelog_in_stderr=self._test_should_dump_jsconsolelog_in_stderr(test_file))
 
     def _test_is_slow(self, test_file):
         if self._expectations.model().has_modifier(test_file, test_expectations.SLOW):
             return True
         return "slow" in self._tests_options.get(test_file, [])
 
+    def _test_should_dump_jsconsolelog_in_stderr(self, test_file):
+        return self._expectations.model().has_modifier(test_file, test_expectations.DUMPJSCONSOLELOGINSTDERR)
+
     def needs_servers(self, test_names):
         return any(self._is_http_test(test_name) for test_name in test_names) and self._options.http
 
index e858352..5028ae7 100644 (file)
@@ -58,6 +58,7 @@ class SingleTestRunner(object):
         self._worker_name = worker_name
         self._test_name = test_input.test_name
         self._should_run_pixel_test = test_input.should_run_pixel_test
+        self._should_dump_jsconsolelog_in_stderr = test_input.should_dump_jsconsolelog_in_stderr
         self._reference_files = test_input.reference_files
         self._stop_when_done = stop_when_done
         self._timeout = test_input.timeout
@@ -89,7 +90,7 @@ class SingleTestRunner(object):
         image_hash = None
         if self._should_fetch_expected_checksum():
             image_hash = self._port.expected_checksum(self._test_name)
-        return DriverInput(self._test_name, self._timeout, image_hash, self._should_run_pixel_test)
+        return DriverInput(self._test_name, self._timeout, image_hash, self._should_run_pixel_test, self._should_dump_jsconsolelog_in_stderr)
 
     def run(self):
         if self._reference_files:
index bb558c9..95e9c69 100644 (file)
@@ -43,7 +43,7 @@ _log = logging.getLogger(__name__)
 # FIXME: range() starts with 0 which makes if expectation checks harder
 # as PASS is 0.
 (PASS, FAIL, TEXT, IMAGE, IMAGE_PLUS_TEXT, AUDIO, TIMEOUT, CRASH, SKIP, WONTFIX,
- SLOW, REBASELINE, MISSING, FLAKY, NOW, NONE) = range(16)
+ SLOW, DUMPJSCONSOLELOGINSTDERR, REBASELINE, MISSING, FLAKY, NOW, NONE) = range(17)
 
 # FIXME: Perhas these two routines should be part of the Port instead?
 BASELINE_SUFFIX_LIST = ('png', 'wav', 'txt')
@@ -71,6 +71,7 @@ class TestExpectationParser(object):
     PASS_EXPECTATION = 'pass'
     SKIP_MODIFIER = 'skip'
     SLOW_MODIFIER = 'slow'
+    DUMPJSCONSOLELOGINSTDERR_MODIFIER = 'dumpjsconsoleloginstderr'
     WONTFIX_MODIFIER = 'wontfix'
 
     TIMEOUT_EXPECTATION = 'timeout'
@@ -319,7 +320,7 @@ class TestExpectationParser(object):
             elif state == 'configuration':
                 modifiers.append(cls._configuration_tokens.get(token, token))
             elif state == 'expectations':
-                if token in ('Rebaseline', 'Skip', 'Slow', 'WontFix'):
+                if token in ('Rebaseline', 'Skip', 'Slow', 'WontFix', 'DumpJSConsoleLogInStdErr'):
                     modifiers.append(token.upper())
                 elif token not in cls._expectation_tokens:
                     warnings.append('Unrecognized expectation "%s"' % token)
@@ -345,7 +346,8 @@ class TestExpectationParser(object):
             # FIXME: This is really a semantic warning and shouldn't be here. Remove when we drop the old syntax.
             warnings.append('A test marked Skip must not have other expectations.')
         elif not expectations:
-            if 'SKIP' not in modifiers and 'REBASELINE' not in modifiers and 'SLOW' not in modifiers:
+            # FIXME: We can probably simplify this adding 'SKIP' if modifiers is empty
+            if 'SKIP' not in modifiers and 'REBASELINE' not in modifiers and 'SLOW' not in modifiers and 'DUMPJSCONSOLELOGINSTDERR' not in modifiers:
                 modifiers.append('SKIP')
             expectations = ['PASS']
 
@@ -467,7 +469,7 @@ class TestExpectationLine(object):
             elif modifier.startswith('BUG'):
                 # FIXME: we should preserve case once we can drop the old syntax.
                 bugs.append('Bug(' + modifier[3:].lower() + ')')
-            elif modifier in ('SLOW', 'SKIP', 'REBASELINE', 'WONTFIX'):
+            elif modifier in ('SLOW', 'SKIP', 'REBASELINE', 'WONTFIX', 'DUMPJSCONSOLELOGINSTDERR'):
                 new_expectations.append(TestExpectationParser._inverted_expectation_tokens.get(modifier))
             else:
                 new_modifiers.append(TestExpectationParser._inverted_configuration_tokens.get(modifier, modifier))
@@ -786,6 +788,7 @@ class TestExpectations(object):
     MODIFIERS = {TestExpectationParser.SKIP_MODIFIER: SKIP,
                  TestExpectationParser.WONTFIX_MODIFIER: WONTFIX,
                  TestExpectationParser.SLOW_MODIFIER: SLOW,
+                 TestExpectationParser.DUMPJSCONSOLELOGINSTDERR_MODIFIER: DUMPJSCONSOLELOGINSTDERR,
                  TestExpectationParser.REBASELINE_MODIFIER: REBASELINE,
                  'none': NONE}
 
index 33e4458..e4a69e1 100644 (file)
@@ -31,7 +31,7 @@
 class TestInput(object):
     """Groups information about a test for easy passing of data."""
 
-    def __init__(self, test_name, timeout=None, needs_servers=None, reference_files=None, should_run_pixel_tests=None):
+    def __init__(self, test_name, timeout=None, needs_servers=None, reference_files=None, should_run_pixel_tests=None, should_dump_jsconsolelog_in_stderr=None):
         # TestInput objects are normally constructed by the manager and passed
         # to the workers, but these some fields are set lazily in the workers where possible
         # because they require us to look at the filesystem and we want to be able to do that in parallel.
@@ -40,6 +40,7 @@ class TestInput(object):
         self.needs_servers = needs_servers
         self.reference_files = reference_files
         self.should_run_pixel_tests = should_run_pixel_tests
+        self.should_dump_jsconsolelog_in_stderr = should_dump_jsconsolelog_in_stderr
 
     def __repr__(self):
-        return "TestInput('%s', timeout=%s, needs_servers=%s, reference_files=%s, should_run_pixel_tests=%s)" % (self.test_name, self.timeout, self.needs_servers, self.reference_files, self.should_run_pixel_tests)
+        return "TestInput('%s', timeout=%s, needs_servers=%s, reference_files=%s, should_run_pixel_tests=%s, should_dump_jsconsolelog_in_stderr=%s)" % (self.test_name, self.timeout, self.needs_servers, self.reference_files, self.should_run_pixel_tests, self.should_dump_jsconsolelog_in_stderr)
index 5ce58cc..e40c306 100644 (file)
@@ -28,7 +28,6 @@
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 import base64
-import copy
 import logging
 import re
 import shlex
@@ -38,22 +37,22 @@ import os
 
 from webkitpy.common.system import path
 from webkitpy.common.system.profiler import ProfilerFactory
-from webkitpy.layout_tests.servers.web_platform_test_server import WebPlatformTestServer
 
 
 _log = logging.getLogger(__name__)
 
 
 class DriverInput(object):
-    def __init__(self, test_name, timeout, image_hash, should_run_pixel_test, args=None):
+    def __init__(self, test_name, timeout, image_hash, should_run_pixel_test, should_dump_jsconsolelog_in_stderr=None, args=None):
         self.test_name = test_name
         self.timeout = timeout  # in ms
         self.image_hash = image_hash
         self.should_run_pixel_test = should_run_pixel_test
+        self.should_dump_jsconsolelog_in_stderr = should_dump_jsconsolelog_in_stderr
         self.args = args or []
 
     def __repr__(self):
-        return "DriverInput(test_name='{}', timeout={}, image_hash={}, should_run_pixel_test={}'".format(self.test_name, self.timeout, self.image_hash, self.should_run_pixel_test)
+        return "DriverInput(test_name='{}', timeout={}, image_hash={}, should_run_pixel_test={}, should_dump_jsconsolelog_in_stderr={}'".format(self.test_name, self.timeout, self.image_hash, self.should_run_pixel_test, self.should_dump_jsconsolelog_in_stderr)
 
 
 class DriverOutput(object):
@@ -487,6 +486,8 @@ class Driver(object):
             command += "'--timeout'%s" % driver_input.timeout
         if driver_input.should_run_pixel_test:
             command += "'--pixel-test"
+        if driver_input.should_dump_jsconsolelog_in_stderr:
+            command += "'--dump-jsconsolelog-in-stderr"
         if driver_input.image_hash:
             command += "'" + driver_input.image_hash
         return command + "\n"
index 6d98911..bdc165f 100644 (file)
@@ -155,6 +155,9 @@ void InjectedBundle::didReceiveMessageToPage(WKBundlePageRef page, WKStringRef m
         WKRetainPtr<WKStringRef> timeoutKey(AdoptWK, WKStringCreateWithUTF8CString("Timeout"));
         m_timeout = (int)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, timeoutKey.get())));
 
+        WKRetainPtr<WKStringRef> dumpJSConsoleLogInStdErrKey(AdoptWK, WKStringCreateWithUTF8CString("DumpJSConsoleLogInStdErr"));
+        m_dumpJSConsoleLogInStdErr = WKBooleanGetValue(static_cast<WKBooleanRef>(WKDictionaryGetItemForKey(messageBodyDictionary, dumpJSConsoleLogInStdErrKey.get())));
+
         WKRetainPtr<WKStringRef> ackMessageName(AdoptWK, WKStringCreateWithUTF8CString("Ack"));
         WKRetainPtr<WKStringRef> ackMessageBody(AdoptWK, WKStringCreateWithUTF8CString("BeginTest"));
         WKBundlePagePostMessage(page, ackMessageName.get(), ackMessageBody.get());
@@ -411,6 +414,17 @@ void InjectedBundle::dumpBackForwardListsForAllPages(StringBuilder& stringBuilde
         m_pages[i]->dumpBackForwardList(stringBuilder);
 }
 
+void InjectedBundle::dumpToStdErr(const String& output)
+{
+    if (m_state != Testing)
+        return;
+    if (output.isEmpty())
+        return;
+    WKRetainPtr<WKStringRef> messageName(AdoptWK, WKStringCreateWithUTF8CString("DumpToStdErr"));
+    WKRetainPtr<WKStringRef> messageBody(AdoptWK, WKStringCreateWithUTF8CString(output.utf8().data()));
+    WKBundlePagePostMessage(page()->page(), messageName.get(), messageBody.get());
+}
+
 void InjectedBundle::outputText(const String& output)
 {
     if (m_state != Testing)
index e4a4045..2edd5e7 100644 (file)
@@ -81,8 +81,10 @@ public:
 
     bool shouldDumpPixels() const { return m_dumpPixels; }
     bool useWaitToDumpWatchdogTimer() const { return m_useWaitToDumpWatchdogTimer; }
-    
+    bool dumpJSConsoleLogInStdErr() const { return m_dumpJSConsoleLogInStdErr; };
+
     void outputText(const String&);
+    void dumpToStdErr(const String&);
     void postNewBeforeUnloadReturnValue(bool);
     void postAddChromeInputField();
     void postRemoveChromeInputField();
@@ -180,6 +182,7 @@ private:
     bool m_useWorkQueue;
     int m_timeout;
     bool m_pixelResultIsPending { false };
+    bool m_dumpJSConsoleLogInStdErr { false };
 
     WKRetainPtr<WKDataRef> m_audioResult;
     WKRetainPtr<WKImageRef> m_pixelResult;
index dd79366..d1b6cb1 100644 (file)
@@ -1461,7 +1461,11 @@ void InjectedBundlePage::willAddMessageToConsole(WKStringRef message, uint32_t l
     }
     stringBuilder.append(messageString);
     stringBuilder.append('\n');
-    injectedBundle.outputText(stringBuilder.toString());
+
+    if (injectedBundle.dumpJSConsoleLogInStdErr())
+        injectedBundle.dumpToStdErr(stringBuilder.toString());
+    else
+        injectedBundle.outputText(stringBuilder.toString());
 }
 
 void InjectedBundlePage::willSetStatusbarText(WKStringRef statusbarText)
index d0bc249..fb58b57 100644 (file)
@@ -1022,12 +1022,11 @@ void TestController::configureViewForTest(const TestInvocation& test)
 }
 
 struct TestCommand {
-    TestCommand() : shouldDumpPixels(false), timeout(0) { }
-
     std::string pathOrURL;
-    bool shouldDumpPixels;
+    bool shouldDumpPixels { false };
     std::string expectedPixelHash;
-    int timeout;
+    int timeout { 0 };
+    bool dumpJSConsoleLogInStdErr { false };
 };
 
 class CommandTokenizer {
@@ -1100,7 +1099,9 @@ TestCommand parseInputLine(const std::string& inputLine)
             result.shouldDumpPixels = true;
             if (tokenizer.hasNext())
                 result.expectedPixelHash = tokenizer.next();
-        } else
+        } else if (arg == std::string("--dump-jsconsolelog-in-stderr"))
+            result.dumpJSConsoleLogInStdErr = true;
+        else
             die(inputLine);
     }
     return result;
@@ -1122,6 +1123,7 @@ bool TestController::runTest(const char* inputLine)
         m_currentInvocation->setIsPixelTest(command.expectedPixelHash);
     if (command.timeout > 0)
         m_currentInvocation->setCustomTimeout(command.timeout);
+    m_currentInvocation->setDumpJSConsoleLogInStdErr(command.dumpJSConsoleLogInStdErr);
 
     platformWillRunTest(*m_currentInvocation);
 
index 5ed7de5..684f9ed 100644 (file)
@@ -159,6 +159,10 @@ void TestInvocation::invoke()
     WKRetainPtr<WKUInt64Ref> timeoutValue = adoptWK(WKUInt64Create(m_timeout));
     WKDictionarySetItem(beginTestMessageBody.get(), timeoutKey.get(), timeoutValue.get());
 
+    WKRetainPtr<WKStringRef> dumpJSConsoleLogInStdErrKey = adoptWK(WKStringCreateWithUTF8CString("DumpJSConsoleLogInStdErr"));
+    WKRetainPtr<WKBooleanRef> dumpJSConsoleLogInStdErrValue = adoptWK(WKBooleanCreate(m_dumpJSConsoleLogInStdErr));
+    WKDictionarySetItem(beginTestMessageBody.get(), dumpJSConsoleLogInStdErrKey.get(), dumpJSConsoleLogInStdErrValue.get());
+
     WKPagePostMessageToInjectedBundle(TestController::singleton().mainWebView()->page(), messageName.get(), beginTestMessageBody.get());
 
     bool shouldOpenExternalURLs = false;
@@ -371,6 +375,13 @@ void TestInvocation::didReceiveMessageFromInjectedBundle(WKStringRef messageName
         return;
     }
 
+    if (WKStringIsEqualToUTF8CString(messageName, "DumpToStdErr")) {
+        ASSERT(WKGetTypeID(messageBody) == WKStringGetTypeID());
+        WKStringRef textOutput = static_cast<WKStringRef>(messageBody);
+        fprintf(stderr, "%s", toWTFString(textOutput).utf8().data());
+        return;
+    }
+
     if (WKStringIsEqualToUTF8CString(messageName, "BeforeUnloadReturnValue")) {
         ASSERT(WKGetTypeID(messageBody) == WKBooleanGetTypeID());
         WKBooleanRef beforeUnloadReturnValue = static_cast<WKBooleanRef>(messageBody);
index 1b6c5fe..df5885c 100644 (file)
@@ -51,6 +51,7 @@ public:
 
     // Milliseconds
     void setCustomTimeout(int duration) { m_timeout = duration; }
+    void setDumpJSConsoleLogInStdErr(bool value) { m_dumpJSConsoleLogInStdErr = value; }
 
     // Seconds
     double shortTimeout() const;
@@ -102,6 +103,7 @@ private:
     std::string m_expectedPixelHash;
 
     int m_timeout { 0 };
+    bool m_dumpJSConsoleLogInStdErr { false };
 
     // Invocation state
     bool m_gotInitialResponse { false };