webkitpy: Implement device type specific expected results (Part 1)
authorjbedard@apple.com <jbedard@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 15 Jan 2019 17:46:31 +0000 (17:46 +0000)
committerjbedard@apple.com <jbedard@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 15 Jan 2019 17:46:31 +0000 (17:46 +0000)
https://bugs.webkit.org/show_bug.cgi?id=192162

Reviewed by Lucas Forschler.

Device type specific expected results covers two related ideas. The first (covered by this change)
is where to search for -expected.* files. The second is which tests to run based on the
TestExpectations.txt file.

After this change, the baseline search path will be determined by the type of device running a
test. Device type will be passed into any function which picks a file based on the baseline search
path.

* Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
(Worker._update_test_input): Pass device_type to self._port.reference_files(...).
* Scripts/webkitpy/layout_tests/controllers/manager.py:
(Manager.run): Print baseline search path for each round of devices.
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner.__init__): Pass device_type into expected_*(...).
(SingleTestRunner._expected_driver_output): Ditto.
(SingleTestRunner._driver_input): Ditto.
* Scripts/webkitpy/layout_tests/views/printing.py:
(Printer.print_config): Move printing of baseline search path to a separate function.
(Printer.print_baseline_search_path): Moved from Printer.print_config.
* Scripts/webkitpy/layout_tests/views/printing_unittest.py:
(Testprinter.test_print_config):
(Testprinter.test_print_baseline_search_path):
* Scripts/webkitpy/port/base.py:
(Port.baseline_search_path): Pass device_type through.
(Port._expected_baselines_for_suffixes): Ditto.
(Port.expected_baselines): Ditto.
(Port.expected_filename): Ditto.
(Port.expected_checksum): Ditto.
(Port.expected_image): Ditto.
(Port.expected_audio): Ditto.
(Port.expected_text): Ditto.
(Port.reference_files): Ditto.
(Port.tests): Ditto.
(Port._expanded_paths): Ditto.
(Port.skipped_layout_tests): Ditto.
(Port.expectations_dict): Ditto.
(Port._port_specific_expectations_files): Ditto.
(Port.expectations_files): Ditto.
(Port.test_search_path): Ditto.
(Port._tests_for_other_platforms): Ditto.
* Scripts/webkitpy/port/base_unittest.py:
(PortTest.test_additional_platform_directory): Accept, but do not use, device_type.
(PortTest.test_nonexistant_expectations): Ditto.
(test_ref_tests_platform_directory): Ditto.
* Scripts/webkitpy/port/darwin.py:
(DarwinPort._port_specific_expectations_files): Accept device_type.
* Scripts/webkitpy/port/driver.py:
(DriverProxy):
(DriverProxy.host): Expose _target_host for device_type information.
* Scripts/webkitpy/port/gtk.py:
(GtkPort._port_specific_expectations_files): Accept, but do not use, device_type.
* Scripts/webkitpy/port/ios.py:
(IOSPort. default_baseline_search_path): Remove memoized, since it does not accept **kwargs.
* Scripts/webkitpy/port/mac.py:
(MacPort. default_baseline_search_path): Ditto.
* Scripts/webkitpy/port/port_testcase.py:
(TestWebKitPort._tests_for_other_platforms): Accept, but do not use, device_type.
* Scripts/webkitpy/port/test.py:
* Scripts/webkitpy/port/watch.py:
(WatchPort. default_baseline_search_path): Remove memoized, since it does not accept **kwargs.
* Scripts/webkitpy/port/win.py:
(WinPort.test_search_path): Accept, but do not use, device_type.
* Scripts/webkitpy/port/wpe.py:
(WPEPort._port_specific_expectations_files): Ditto.
* Scripts/webkitpy/tool/servers/rebaselineserver.py:
(get_test_baselines.AllPlatformsPort.baseline_search_path): Ditto.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@239989 268f45cc-cd09-0410-ab3c-d52691b4dbfc

19 files changed:
Tools/ChangeLog
Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py
Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py
Tools/Scripts/webkitpy/layout_tests/views/printing.py
Tools/Scripts/webkitpy/layout_tests/views/printing_unittest.py
Tools/Scripts/webkitpy/port/base.py
Tools/Scripts/webkitpy/port/base_unittest.py
Tools/Scripts/webkitpy/port/darwin.py
Tools/Scripts/webkitpy/port/driver.py
Tools/Scripts/webkitpy/port/gtk.py
Tools/Scripts/webkitpy/port/ios.py
Tools/Scripts/webkitpy/port/mac.py
Tools/Scripts/webkitpy/port/port_testcase.py
Tools/Scripts/webkitpy/port/test.py
Tools/Scripts/webkitpy/port/watch.py
Tools/Scripts/webkitpy/port/win.py
Tools/Scripts/webkitpy/port/wpe.py
Tools/Scripts/webkitpy/tool/servers/rebaselineserver.py

index 9a26240..bf7db9e 100644 (file)
@@ -1,3 +1,77 @@
+2019-01-15  Jonathan Bedard  <jbedard@apple.com>
+
+        webkitpy: Implement device type specific expected results (Part 1)
+        https://bugs.webkit.org/show_bug.cgi?id=192162
+
+        Reviewed by Lucas Forschler.
+
+        Device type specific expected results covers two related ideas. The first (covered by this change)
+        is where to search for -expected.* files. The second is which tests to run based on the
+        TestExpectations.txt file.
+
+        After this change, the baseline search path will be determined by the type of device running a
+        test. Device type will be passed into any function which picks a file based on the baseline search
+        path.
+
+        * Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
+        (Worker._update_test_input): Pass device_type to self._port.reference_files(...).
+        * Scripts/webkitpy/layout_tests/controllers/manager.py:
+        (Manager.run): Print baseline search path for each round of devices.
+        * Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
+        (SingleTestRunner.__init__): Pass device_type into expected_*(...).
+        (SingleTestRunner._expected_driver_output): Ditto.
+        (SingleTestRunner._driver_input): Ditto.
+        * Scripts/webkitpy/layout_tests/views/printing.py:
+        (Printer.print_config): Move printing of baseline search path to a separate function.
+        (Printer.print_baseline_search_path): Moved from Printer.print_config.
+        * Scripts/webkitpy/layout_tests/views/printing_unittest.py:
+        (Testprinter.test_print_config):
+        (Testprinter.test_print_baseline_search_path):
+        * Scripts/webkitpy/port/base.py:
+        (Port.baseline_search_path): Pass device_type through.
+        (Port._expected_baselines_for_suffixes): Ditto.
+        (Port.expected_baselines): Ditto.
+        (Port.expected_filename): Ditto.
+        (Port.expected_checksum): Ditto.
+        (Port.expected_image): Ditto.
+        (Port.expected_audio): Ditto.
+        (Port.expected_text): Ditto.
+        (Port.reference_files): Ditto.
+        (Port.tests): Ditto.
+        (Port._expanded_paths): Ditto.
+        (Port.skipped_layout_tests): Ditto.
+        (Port.expectations_dict): Ditto.
+        (Port._port_specific_expectations_files): Ditto.
+        (Port.expectations_files): Ditto.
+        (Port.test_search_path): Ditto.
+        (Port._tests_for_other_platforms): Ditto.
+        * Scripts/webkitpy/port/base_unittest.py:
+        (PortTest.test_additional_platform_directory): Accept, but do not use, device_type.
+        (PortTest.test_nonexistant_expectations): Ditto.
+        (test_ref_tests_platform_directory): Ditto.
+        * Scripts/webkitpy/port/darwin.py:
+        (DarwinPort._port_specific_expectations_files): Accept device_type.
+        * Scripts/webkitpy/port/driver.py:
+        (DriverProxy):
+        (DriverProxy.host): Expose _target_host for device_type information.
+        * Scripts/webkitpy/port/gtk.py:
+        (GtkPort._port_specific_expectations_files): Accept, but do not use, device_type.
+        * Scripts/webkitpy/port/ios.py:
+        (IOSPort. default_baseline_search_path): Remove memoized, since it does not accept **kwargs.
+        * Scripts/webkitpy/port/mac.py:
+        (MacPort. default_baseline_search_path): Ditto.
+        * Scripts/webkitpy/port/port_testcase.py:
+        (TestWebKitPort._tests_for_other_platforms): Accept, but do not use, device_type.
+        * Scripts/webkitpy/port/test.py:
+        * Scripts/webkitpy/port/watch.py:
+        (WatchPort. default_baseline_search_path): Remove memoized, since it does not accept **kwargs.
+        * Scripts/webkitpy/port/win.py:
+        (WinPort.test_search_path): Accept, but do not use, device_type.
+        * Scripts/webkitpy/port/wpe.py:
+        (WPEPort._port_specific_expectations_files): Ditto.
+        * Scripts/webkitpy/tool/servers/rebaselineserver.py:
+        (get_test_baselines.AllPlatformsPort.baseline_search_path): Ditto.
+
 2019-01-15  Zalan Bujtas  <zalan@apple.com>
 
         [LFC] Use the containing block's padding box to position out-of-flow elements.
index 884d78d..5428426 100644 (file)
@@ -294,7 +294,7 @@ class Worker(object):
     def _update_test_input(self, test_input):
         if test_input.reference_files is None:
             # Lazy initialization.
-            test_input.reference_files = self._port.reference_files(test_input.test_name)
+            test_input.reference_files = self._port.reference_files(test_input.test_name, device_type=self._port.target_host(self._worker_number).device_type)
         if test_input.reference_files:
             test_input.should_run_pixel_test = True
         else:
index 70842e0..96d1475 100644 (file)
@@ -274,6 +274,8 @@ class Manager(object):
                 else:
                     index += 1
 
+            self._printer.print_baseline_search_path(device_type=device_type)
+
             _log.info('Running {}{}'.format(pluralize(len(tests), 'test'), ' for {}'.format(str(device_type)) if device_type else ''))
             _log.info('')
             if not self._set_up_run(tests, device_type):
index c8a119c..718a089 100644 (file)
@@ -69,15 +69,15 @@ class SingleTestRunner(object):
             # 'foo-expected.txt', we should warn users. One test file must be used exclusively
             # in either layout tests or reftests, but not in both.
             for suffix in ('.txt', '.png', '.wav'):
-                expected_filename = self._port.expected_filename(self._test_name, suffix)
+                expected_filename = self._port.expected_filename(self._test_name, suffix, device_type=self._driver.host.device_type)
                 if self._filesystem.exists(expected_filename):
                     _log.error('%s is a reftest, but has an unused expectation file. Please remove %s.', self._test_name, expected_filename)
 
     def _expected_driver_output(self):
-        return DriverOutput(self._port.expected_text(self._test_name),
-                                 self._port.expected_image(self._test_name),
-                                 self._port.expected_checksum(self._test_name),
-                                 self._port.expected_audio(self._test_name))
+        return DriverOutput(self._port.expected_text(self._test_name, device_type=self._driver.host.device_type),
+                                 self._port.expected_image(self._test_name, device_type=self._driver.host.device_type),
+                                 self._port.expected_checksum(self._test_name, device_type=self._driver.host.device_type),
+                                 self._port.expected_audio(self._test_name, device_type=self._driver.host.device_type))
 
     def _should_fetch_expected_checksum(self):
         return self._should_run_pixel_test and not (self._options.new_baseline or self._options.reset_results)
@@ -89,7 +89,7 @@ class SingleTestRunner(object):
         # previous run will be copied into the baseline."""
         image_hash = None
         if self._should_fetch_expected_checksum():
-            image_hash = self._port.expected_checksum(self._test_name)
+            image_hash = self._port.expected_checksum(self._test_name, device_type=self._driver.host.device_type)
         return DriverInput(self._test_name, self._timeout, image_hash, self._should_run_pixel_test, self._should_dump_jsconsolelog_in_stderr)
 
     def run(self):
index 4e64c7d..a6aff60 100644 (file)
@@ -81,16 +81,6 @@ class Printer(object):
         if self._options.new_baseline:
             self._print_default("Placing new baselines in %s" % self._port.baseline_path())
 
-        fs = self._port.host.filesystem
-        full_baseline_search_path = self._port.baseline_search_path()
-        normalize_baseline = lambda baseline_search_path: [fs.relpath(x, self._port.layout_tests_dir()).replace("../", "") for x in baseline_search_path]
-
-        self._print_default('Verbose baseline search path: {} -> generic'.format(' -> '.join(normalize_baseline(full_baseline_search_path))))
-
-        self._print_default('')
-        self._print_default('Baseline search path: {} -> generic'.format(' -> '.join(normalize_baseline([path for path in full_baseline_search_path if fs.exists(path)]))))
-        self._print_default('')
-
         self._print_default("Using %s build" % self._options.configuration)
         if self._options.pixel_tests:
             self._print_default("Pixel tests enabled")
@@ -103,6 +93,20 @@ class Printer(object):
         self._print_default('Command line: ' + ' '.join(self._port.driver_cmd_line_for_logging()))
         self._print_default('')
 
+    def print_baseline_search_path(self, device_type=None):
+        fs = self._port.host.filesystem
+        full_baseline_search_path = self._port.baseline_search_path(device_type=device_type)
+        normalize_baseline = lambda baseline_search_path: [
+            fs.relpath(x, self._port.layout_tests_dir()).replace("../", "") for x in baseline_search_path]
+
+        self._print_default('Verbose baseline search path: {} -> generic'.format(
+            ' -> '.join(normalize_baseline(full_baseline_search_path))))
+
+        self._print_default('')
+        self._print_default('Baseline search path: {} -> generic'.format(
+            ' -> '.join(normalize_baseline([path for path in full_baseline_search_path if fs.exists(path)]))))
+        self._print_default('')
+
     def print_found(self, num_all_test_files, num_to_run, repeat_each, iterations):
         found_str = 'Found %s; running %d' % (grammar.pluralize(num_all_test_files, "test"), num_to_run)
         if repeat_each * iterations > 1:
index 9af5233..f012556 100644 (file)
@@ -111,16 +111,21 @@ class  Testprinter(unittest.TestCase):
         self.assertIn("Using port 'test-mac-leopard'", err.getvalue())
         self.assertIn('Test configuration: <leopard, x86, release>', err.getvalue())
         self.assertIn('Placing test results in /tmp', err.getvalue())
-        self.assertIn('Verbose baseline search path: platform/test-mac-leopard -> platform/test-mac-snowleopard -> generic', err.getvalue())
-        self.assertIn('Baseline search path: platform/test-mac-leopard -> generic', err.getvalue())
         self.assertIn('Using Release build', err.getvalue())
         self.assertIn('Pixel tests enabled', err.getvalue())
         self.assertIn('Command line:', err.getvalue())
         self.assertIn('Regular timeout: ', err.getvalue())
 
+    def test_print_baseline_search_path(self):
+        printer, err = self.get_printer()
+        printer.print_baseline_search_path()
+
+        self.assertIn('Verbose baseline search path: platform/test-mac-leopard -> platform/test-mac-snowleopard -> generic', err.getvalue())
+        self.assertIn('Baseline search path: platform/test-mac-leopard -> generic', err.getvalue())
+
         self.reset(err)
         printer._options.quiet = True
-        printer.print_config('/tmp')
+        printer.print_baseline_search_path()
         self.assertNotIn('Baseline search path: platform/test-mac-leopard -> platform/test-mac-snowleopard -> generic', err.getvalue())
 
     def test_print_one_line_summary(self):
index 20e8c76..a68f05b 100644 (file)
@@ -208,8 +208,8 @@ class Port(object):
         baseline_search_paths = self.baseline_search_path()
         return baseline_search_paths[0]
 
-    def baseline_search_path(self):
-        return self.get_option('additional_platform_directory', []) + self._compare_baseline() + self.default_baseline_search_path()
+    def baseline_search_path(self, device_type=None):
+        return self.get_option('additional_platform_directory', []) + self._compare_baseline() + self.default_baseline_search_path(device_type=device_type)
 
     def default_baseline_search_path(self, device_type=None):
         """Return a list of absolute paths to directories to search under for
@@ -408,8 +408,8 @@ class Port(object):
         """Returns a tuple of all of the non-reftest baseline extensions we use. The extensions include the leading '.'."""
         return ('.wav', '.webarchive', '.txt', '.png')
 
-    def _expected_baselines_for_suffixes(self, test_name, suffixes, all_baselines=False):
-        baseline_search_path = self.baseline_search_path() + [self.layout_tests_dir()]
+    def _expected_baselines_for_suffixes(self, test_name, suffixes, all_baselines=False, device_type=None):
+        baseline_search_path = self.baseline_search_path(device_type=device_type) + [self.layout_tests_dir()]
 
         baselines = []
         for platform_dir in baseline_search_path:
@@ -428,7 +428,7 @@ class Port(object):
             baselines.append((None, self._filesystem.splitext(test_name)[0] + '-expected' + suffix))
         return baselines
 
-    def expected_baselines(self, test_name, suffix, all_baselines=False):
+    def expected_baselines(self, test_name, suffix, all_baselines=False, device_type=None):
         """Given a test name, finds where the baseline results are located.
 
         Args:
@@ -455,9 +455,9 @@ class Port(object):
         conjunction with the other baseline and filename routines that are
         platform specific.
         """
-        return self._expected_baselines_for_suffixes(test_name, [suffix], all_baselines=all_baselines)
+        return self._expected_baselines_for_suffixes(test_name, [suffix], all_baselines=all_baselines, device_type=device_type)
 
-    def expected_filename(self, test_name, suffix, return_default=True):
+    def expected_filename(self, test_name, suffix, return_default=True, device_type=None):
         """Given a test name, returns an absolute path to its expected results.
 
         If no expected results are found in any of the searched directories,
@@ -477,14 +477,14 @@ class Port(object):
         This routine is generic but is implemented here to live alongside
         the other baseline and filename manipulation routines.
         """
-        platform_dir, baseline_filename = self.expected_baselines(test_name, suffix)[0]
+        platform_dir, baseline_filename = self.expected_baselines(test_name, suffix, device_type=device_type)[0]
         if platform_dir or return_default:
             return self._filesystem.join(platform_dir or self.layout_tests_dir(), baseline_filename)
         return None
 
-    def expected_checksum(self, test_name):
+    def expected_checksum(self, test_name, device_type=None):
         """Returns the checksum of the image we expect the test to produce, or None if it is a text-only test."""
-        png_path = self.expected_filename(test_name, '.png')
+        png_path = self.expected_filename(test_name, '.png', device_type=device_type)
 
         if self._filesystem.exists(png_path):
             with self._filesystem.open_binary_file_for_reading(png_path) as filehandle:
@@ -492,29 +492,29 @@ class Port(object):
 
         return None
 
-    def expected_image(self, test_name):
+    def expected_image(self, test_name, device_type=None):
         """Returns the image we expect the test to produce."""
-        baseline_path = self.expected_filename(test_name, '.png')
+        baseline_path = self.expected_filename(test_name, '.png', device_type=device_type)
         if not self._filesystem.exists(baseline_path):
             return None
         return self._filesystem.read_binary_file(baseline_path)
 
-    def expected_audio(self, test_name):
-        baseline_path = self.expected_filename(test_name, '.wav')
+    def expected_audio(self, test_name, device_type=None):
+        baseline_path = self.expected_filename(test_name, '.wav', device_type=device_type)
         if not self._filesystem.exists(baseline_path):
             return None
         return self._filesystem.read_binary_file(baseline_path)
 
-    def expected_text(self, test_name):
+    def expected_text(self, test_name, device_type=None):
         """Returns the text output we expect the test to produce, or None
         if we don't expect there to be any text output.
         End-of-line characters are normalized to '\n'."""
         # FIXME: DRT output is actually utf-8, but since we don't decode the
         # output from DRT (instead treating it as a binary string), we read the
         # baselines as a binary string, too.
-        baseline_path = self.expected_filename(test_name, '.txt')
+        baseline_path = self.expected_filename(test_name, '.txt', device_type=device_type)
         if not self._filesystem.exists(baseline_path):
-            baseline_path = self.expected_filename(test_name, '.webarchive')
+            baseline_path = self.expected_filename(test_name, '.webarchive', device_type=device_type)
             if not self._filesystem.exists(baseline_path):
                 return None
         text = self._filesystem.read_binary_file(baseline_path)
@@ -543,7 +543,7 @@ class Port(object):
             parsed_list.setdefault(filesystem.join(test_dirpath, test_file), []).append((expectation_type, filesystem.join(test_dirpath, ref_file)))
         return parsed_list
 
-    def reference_files(self, test_name):
+    def reference_files(self, test_name, device_type=None):
         """Return a list of expectation (== or !=) and filename pairs"""
 
         if self.get_option('treat_ref_tests_as_pixel_tests'):
@@ -558,7 +558,7 @@ class Port(object):
         for part1 in ['', '-mismatch']:
             for part2 in self._supported_reference_extensions:
                 suffixes.append(part1 + part2)
-        for platform_dir, baseline_filename in self._expected_baselines_for_suffixes(test_name, suffixes):
+        for platform_dir, baseline_filename in self._expected_baselines_for_suffixes(test_name, suffixes, device_type=device_type):
             if not platform_dir:
                 continue
             result.append((
@@ -579,12 +579,12 @@ class Port(object):
 
         return [self.host.filesystem.relpath(test, self.layout_tests_dir()) for test in self._filesystem.glob(re.sub('-expected.*', '.*', self._filesystem.join(self.layout_tests_dir(), path))) if self._filesystem.isfile(test)]
 
-    def tests(self, paths):
+    def tests(self, paths, device_type=None):
         """Return the list of tests found. Both generic and platform-specific tests matching paths should be returned."""
-        expanded_paths = self._expanded_paths(paths)
+        expanded_paths = self._expanded_paths(paths, device_type=device_type)
         return self._real_tests(expanded_paths)
 
-    def _expanded_paths(self, paths):
+    def _expanded_paths(self, paths, device_type=None):
         expanded_paths = []
         fs = self._filesystem
         all_platform_dirs = [path for path in fs.glob(fs.join(self.layout_tests_dir(), 'platform', '*')) if fs.isdir(path)]
@@ -592,7 +592,7 @@ class Port(object):
             expanded_paths.append(path)
             if self.test_isdir(path) and not path.startswith('platform') and not fs.isabs(path):
                 for platform_dir in all_platform_dirs:
-                    if fs.isdir(fs.join(platform_dir, path)) and platform_dir in self.baseline_search_path():
+                    if fs.isdir(fs.join(platform_dir, path)) and platform_dir in self.baseline_search_path(device_type=device_type):
                         expanded_paths.append(self.relative_test_filename(fs.join(platform_dir, path)))
 
         return expanded_paths
@@ -753,9 +753,9 @@ class Port(object):
     def perf_tests_dir(self):
         return self._webkit_finder.perf_tests_dir()
 
-    def skipped_layout_tests(self, test_list):
+    def skipped_layout_tests(self, test_list, device_type=None):
         """Returns tests skipped outside of the TestExpectations files."""
-        return set(self._tests_for_other_platforms()).union(self._skipped_tests_for_unsupported_features(test_list))
+        return set(self._tests_for_other_platforms(device_type=device_type)).union(self._skipped_tests_for_unsupported_features(test_list))
 
     @memoized
     def skipped_perf_tests(self):
@@ -1159,7 +1159,7 @@ class Port(object):
     def warn_if_bug_missing_in_test_expectations(self):
         return False
 
-    def expectations_dict(self):
+    def expectations_dict(self, device_type=None):
         """Returns an OrderedDict of name -> expectations strings.
         The names are expected to be (but not required to be) paths in the filesystem.
         If the name is a path, the file can be considered updatable for things like rebaselining,
@@ -1170,7 +1170,7 @@ class Port(object):
         # FIXME: rename this to test_expectations() once all the callers are updated to know about the ordered dict.
         expectations = OrderedDict()
 
-        for path in self.expectations_files():
+        for path in self.expectations_files(device_type=device_type):
             if self._filesystem.exists(path):
                 expectations[path] = self._filesystem.read_text_file(path)
 
@@ -1183,7 +1183,7 @@ class Port(object):
                 _log.warning("additional_expectations path '%s' does not exist" % path)
         return expectations
 
-    def _port_specific_expectations_files(self):
+    def _port_specific_expectations_files(self, **kwargs):
         # Unlike baseline_search_path, we only want to search [WK2-PORT, PORT-VERSION, PORT] and any directories
         # included via --additional-platform-directory, not the full casade.
         search_paths = [self.port_name]
@@ -1201,8 +1201,8 @@ class Port(object):
 
         return [self._filesystem.join(self._webkit_baseline_path(d), 'TestExpectations') for d in search_paths]
 
-    def expectations_files(self):
-        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files()
+    def expectations_files(self, device_type=None):
+        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files(device_type=device_type)
 
     def repository_paths(self):
         """Returns a list of (repository_name, repository_path) tuples of its depending code base.
@@ -1534,10 +1534,10 @@ class Port(object):
     def _build_driver_flags(self):
         return []
 
-    def test_search_path(self):
-        return self.baseline_search_path()
+    def test_search_path(self, device_type=None):
+        return self.baseline_search_path(device_type=device_type)
 
-    def _tests_for_other_platforms(self):
+    def _tests_for_other_platforms(self, device_type=None):
         # By default we will skip any directory under LayoutTests/platform
         # that isn't in our baseline search path (this mirrors what
         # old-run-webkit-tests does in findTestsToRun()).
@@ -1545,7 +1545,7 @@ class Port(object):
         entries = self._filesystem.glob(self._webkit_baseline_path('*'))
         dirs_to_skip = []
         for entry in entries:
-            if self._filesystem.isdir(entry) and entry not in self.test_search_path():
+            if self._filesystem.isdir(entry) and entry not in self.test_search_path(device_type=device_type):
                 basename = self._filesystem.basename(entry)
                 dirs_to_skip.append('platform/%s' % basename)
         return dirs_to_skip
index 73c24b7..e7409cc 100644 (file)
@@ -161,7 +161,7 @@ class PortTest(unittest.TestCase):
 
     def test_additional_platform_directory(self):
         port = self.make_port(port_name='foo')
-        port.default_baseline_search_path = lambda: ['LayoutTests/platform/foo']
+        port.default_baseline_search_path = lambda **kwargs: ['LayoutTests/platform/foo']
         test_file = 'fast/test.html'
 
         # No additional platform directory
@@ -187,7 +187,7 @@ class PortTest(unittest.TestCase):
 
     def test_nonexistant_expectations(self):
         port = self.make_port(port_name='foo')
-        port.expectations_files = lambda: ['/mock-checkout/LayoutTests/platform/exists/TestExpectations', '/mock-checkout/LayoutTests/platform/nonexistant/TestExpectations']
+        port.expectations_files = lambda **kwargs: ['/mock-checkout/LayoutTests/platform/exists/TestExpectations', '/mock-checkout/LayoutTests/platform/nonexistant/TestExpectations']
         port._filesystem.write_text_file('/mock-checkout/LayoutTests/platform/exists/TestExpectations', '')
         self.assertEqual('\n'.join(port.expectations_dict().keys()), '/mock-checkout/LayoutTests/platform/exists/TestExpectations')
 
@@ -413,7 +413,7 @@ class PortTest(unittest.TestCase):
 
     def test_ref_tests_platform_directory(self):
         port = self.make_port(port_name='foo')
-        port.default_baseline_search_path = lambda: ['/mock-checkout/LayoutTests/platform/foo']
+        port.default_baseline_search_path = lambda **kwargs: ['/mock-checkout/LayoutTests/platform/foo']
         port._filesystem.write_text_file('/mock-checkout/LayoutTests/fast/ref-expected.html', 'foo')
 
         # No platform directory
index 7833f2e..9c9ae99 100644 (file)
@@ -53,8 +53,8 @@ class DarwinPort(ApplePort):
             return 350 * 1000
         return super(DarwinPort, self).default_timeout_ms()
 
-    def _port_specific_expectations_files(self):
-        return list(reversed([self._filesystem.join(self._webkit_baseline_path(p), 'TestExpectations') for p in self.baseline_search_path()]))
+    def _port_specific_expectations_files(self, device_type=None):
+        return list(reversed([self._filesystem.join(self._webkit_baseline_path(p), 'TestExpectations') for p in self.baseline_search_path(device_type=device_type)]))
 
     def check_for_leaks(self, process_name, process_pid):
         if not self.get_option('leaks'):
index a803adb..11e8015 100644 (file)
@@ -719,6 +719,10 @@ class DriverProxy(object):
     def _make_driver(self, pixel_tests):
         return self._driver_instance_constructor(self._port, self._worker_number, pixel_tests, self._no_timeout)
 
+    @property
+    def host(self):
+        return self._driver._target_host
+
     # FIXME: this should be a @classmethod (or implemented on Port instead).
     def is_http_test(self, test_name):
         return self._driver.is_http_test(test_name)
index 29d00a5..a9dc525 100644 (file)
@@ -213,7 +213,7 @@ class GtkPort(Port):
     def default_baseline_search_path(self, **kwargs):
         return map(self._webkit_baseline_path, self._search_paths())
 
-    def _port_specific_expectations_files(self):
+    def _port_specific_expectations_files(self, **kwargs):
         return [self._filesystem.join(self._webkit_baseline_path(p), 'TestExpectations') for p in reversed(self._search_paths())]
 
     def print_leaks_summary(self):
index 6697e20..54b8130 100644 (file)
@@ -52,7 +52,6 @@ class IOSPort(DevicePort):
             return None
         return VersionNameMap.map(self.host.platform).to_name(self._os_version, platform=IOSPort.port_name)
 
-    @memoized
     def default_baseline_search_path(self, device_type=None):
         wk_string = 'wk1'
         if self.get_option('webkit_test_runner'):
index 4b3acbe..bcf8a12 100644 (file)
@@ -69,7 +69,6 @@ class MacPort(DarwinPort):
     def _build_driver_flags(self):
         return ['ARCHS=i386'] if self.architecture() == 'x86' else []
 
-    @memoized
     def default_baseline_search_path(self, **kwargs):
         versions_to_fallback = []
         version_name_map = VersionNameMap.map(self.host.platform)
index 9cc3d7a..9d86c9c 100644 (file)
@@ -70,7 +70,7 @@ class TestWebKitPort(Port):
     def _symbols_string(self):
         return self.symbols_string
 
-    def _tests_for_other_platforms(self):
+    def _tests_for_other_platforms(self, **kwargs):
         return ["media", ]
 
     def _tests_for_disabled_features(self):
index 686b6a1..339b53c 100644 (file)
@@ -407,7 +407,7 @@ class TestPort(Port):
         # the mock_drt Driver. We return something, but make sure it's useless.
         return 'MOCK _path_to_driver'
 
-    def baseline_search_path(self):
+    def baseline_search_path(self, **kwargs):
         search_paths = {
             'test-mac-snowleopard': ['test-mac-snowleopard'],
             'test-mac-leopard': ['test-mac-leopard', 'test-mac-snowleopard'],
index a57c2b0..9183e4a 100644 (file)
@@ -58,7 +58,6 @@ class WatchPort(DevicePort):
     def test_expectations_file_position(self):
         return 4
 
-    @memoized
     def default_baseline_search_path(self, **kwargs):
         versions_to_fallback = []
         if self.device_version() == self.CURRENT_VERSION:
index 693c541..e00c2a4 100644 (file)
@@ -217,7 +217,7 @@ class WinPort(ApplePort):
     def path_to_api_test_binaries(self):
         return {binary.split('.')[0]: self._build_path(binary) for binary in self.API_TEST_BINARY_NAMES}
 
-    def test_search_path(self):
+    def test_search_path(self, **kwargs):
         test_fallback_names = [path for path in self.baseline_search_path() if not path.startswith(self._webkit_baseline_path('mac'))]
         return map(self._webkit_baseline_path, test_fallback_names)
 
index 2d16955..18a9a5a 100644 (file)
@@ -105,7 +105,7 @@ class WPEPort(Port):
     def default_baseline_search_path(self, **kwargs):
         return map(self._webkit_baseline_path, self._search_paths())
 
-    def _port_specific_expectations_files(self):
+    def _port_specific_expectations_files(self, **kwargs):
         return map(lambda x: self._filesystem.join(self._webkit_baseline_path(x), 'TestExpectations'), reversed(self._search_paths()))
 
     def test_expectations_file_position(self):
index d8c62fb..ba9146c 100644 (file)
@@ -168,7 +168,7 @@ def get_test_baselines(test_file, test_config):
             super(AllPlatformsPort, self).__init__(host, 'mac')
             self._platforms_by_directory = dict([(self._webkit_baseline_path(p), p) for p in test_config.platforms])
 
-        def baseline_search_path(self):
+        def baseline_search_path(self, **kwargs):
             return self._platforms_by_directory.keys()
 
         def platform_from_directory(self, directory):