Web Inspector: chromium: I'd like to add a script for running perf tests for WebInspe...
authorloislo@chromium.org <loislo@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 1 Dec 2011 07:16:28 +0000 (07:16 +0000)
committerloislo@chromium.org <loislo@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 1 Dec 2011 07:16:28 +0000 (07:16 +0000)
https://bugs.webkit.org/show_bug.cgi?id=73079

The idea is to have performance tests for WebInspector.
I was suggested to put these tests into PerformanceTests/inspector.
They produce output in a format that is suitable for chromium perf bot drawing scripts.

I'd like to reuse code for scanning folders for test files.
Because of the fact that original test_files.py has common and layout specific parts
I decided to extract the generic part and put it into webkitpy/common
I think the two packages with name test_files look strange but I have no idea about a better name.

Reviewed by Dirk Pranke.

* Scripts/run-inspector-perf-tests.py: Added.
* Scripts/webkitpy/common/find_files.py: Renamed from Tools/Scripts/webkitpy/layout_tests/port/test_files.py.
(find):
(_normalize):
* Scripts/webkitpy/common/find_files_unittest.py: Added.
(MockWinFileSystem.join):
(MockWinFileSystem):
(MockWinFileSystem.normpath):
(TestWinNormalize.assert_filesystem_normalizes):
(TestWinNormalize.test_mocked_win):
(TestWinNormalize):
(TestWinNormalize.test_win):
* Scripts/webkitpy/layout_tests/port/base.py:
(Port.tests):
(Port.find_test_files):
(Port._driver_class):
(is_reference_html_file):
(_has_supported_extension):
* Scripts/webkitpy/layout_tests/port/base_unittest.py:
(PortTest.test_find_no_paths_specified):
(PortTest.test_find_one_test):
(PortTest.test_find_glob):
(PortTest.test_find_with_skipped_directories):
(PortTest.test_find_with_skipped_directories_2):
(PortTest):
(PortTest.test_is_test_file):
* Scripts/webkitpy/layout_tests/port/chromium_gpu.py:
(_tests):
* Scripts/webkitpy/layout_tests/port/test.py:
* Scripts/webkitpy/layout_tests/port/test_files_unittest.py: Removed.
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
* Scripts/webkitpy/performance_tests/__init__.py: Added.
* Scripts/webkitpy/performance_tests/perftestsrunner.py: Added.
(PerfTestsRunner.__init__):
(PerfTestsRunner._parse_args):
(PerfTestsRunner._collect_tests._is_test_file):
(PerfTestsRunner._collect_tests):
(PerfTestsRunner.run):
(PerfTestsRunner._run_tests_set):
* Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py: Added.
(MainTest.TestPort):
(MainTest.TestPort.create_driver):
(MainTest.TestDriver.run_test):
(MainTest.TestDriver):
(MainTest.TestDriver.stop):
(MainTest.create_runner):
(MainTest.run_test):
(MainTest.test_run_passing_test):
(MainTest.test_run_silent_test):
(MainTest.test_run_failed_test):
(MainTest.test_run_tonguey_test):
(MainTest.test_run_timeout_test):
(MainTest.test_run_crash_test):
(MainTest.test_run_test_set):
(MainTest.test_collect_tests):
(MainTest):
(MainTest.test_parse_args):
* Scripts/webkitpy/tool/commands/rebaseline.py:
(OptimizeBaselines.execute):
(AnalyzeBaselines.execute):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@101618 268f45cc-cd09-0410-ab3c-d52691b4dbfc

13 files changed:
Tools/ChangeLog
Tools/Scripts/run-inspector-perf-tests.py [new file with mode: 0755]
Tools/Scripts/webkitpy/common/find_files.py [moved from Tools/Scripts/webkitpy/layout_tests/port/test_files.py with 59% similarity]
Tools/Scripts/webkitpy/common/find_files_unittest.py [moved from Tools/Scripts/webkitpy/layout_tests/port/test_files_unittest.py with 60% similarity]
Tools/Scripts/webkitpy/layout_tests/port/base.py
Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
Tools/Scripts/webkitpy/layout_tests/port/chromium_gpu.py
Tools/Scripts/webkitpy/layout_tests/port/test.py
Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py
Tools/Scripts/webkitpy/performance_tests/__init__.py [new file with mode: 0644]
Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py [new file with mode: 0644]
Tools/Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py [new file with mode: 0755]
Tools/Scripts/webkitpy/tool/commands/rebaseline.py

index ab84cba6d2358b8dbc726fcf82fd8a68501d8177..c5debcfab7550304610318e5b820893b9c67794a 100644 (file)
@@ -1,3 +1,80 @@
+2011-11-30  Ilya Tikhonovsky  <loislo@chromium.org>
+
+        Web Inspector: chromium: I'd like to add a script for running perf tests for WebInspector.
+        https://bugs.webkit.org/show_bug.cgi?id=73079
+
+        The idea is to have performance tests for WebInspector.
+        I was suggested to put these tests into PerformanceTests/inspector.
+        They produce output in a format that is suitable for chromium perf bot drawing scripts.
+
+        I'd like to reuse code for scanning folders for test files.
+        Because of the fact that original test_files.py has common and layout specific parts
+        I decided to extract the generic part and put it into webkitpy/common
+        I think the two packages with name test_files look strange but I have no idea about a better name.
+
+        Reviewed by Dirk Pranke.
+
+        * Scripts/run-inspector-perf-tests.py: Added.
+        * Scripts/webkitpy/common/find_files.py: Renamed from Tools/Scripts/webkitpy/layout_tests/port/test_files.py.
+        (find):
+        (_normalize):
+        * Scripts/webkitpy/common/find_files_unittest.py: Added.
+        (MockWinFileSystem.join):
+        (MockWinFileSystem):
+        (MockWinFileSystem.normpath):
+        (TestWinNormalize.assert_filesystem_normalizes):
+        (TestWinNormalize.test_mocked_win):
+        (TestWinNormalize):
+        (TestWinNormalize.test_win):
+        * Scripts/webkitpy/layout_tests/port/base.py:
+        (Port.tests):
+        (Port.find_test_files):
+        (Port._driver_class):
+        (is_reference_html_file):
+        (_has_supported_extension):
+        * Scripts/webkitpy/layout_tests/port/base_unittest.py:
+        (PortTest.test_find_no_paths_specified):
+        (PortTest.test_find_one_test):
+        (PortTest.test_find_glob):
+        (PortTest.test_find_with_skipped_directories):
+        (PortTest.test_find_with_skipped_directories_2):
+        (PortTest):
+        (PortTest.test_is_test_file):
+        * Scripts/webkitpy/layout_tests/port/chromium_gpu.py:
+        (_tests):
+        * Scripts/webkitpy/layout_tests/port/test.py:
+        * Scripts/webkitpy/layout_tests/port/test_files_unittest.py: Removed.
+        * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
+        * Scripts/webkitpy/performance_tests/__init__.py: Added.
+        * Scripts/webkitpy/performance_tests/perftestsrunner.py: Added.
+        (PerfTestsRunner.__init__):
+        (PerfTestsRunner._parse_args):
+        (PerfTestsRunner._collect_tests._is_test_file):
+        (PerfTestsRunner._collect_tests):
+        (PerfTestsRunner.run):
+        (PerfTestsRunner._run_tests_set):
+        * Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py: Added.
+        (MainTest.TestPort):
+        (MainTest.TestPort.create_driver):
+        (MainTest.TestDriver.run_test):
+        (MainTest.TestDriver):
+        (MainTest.TestDriver.stop):
+        (MainTest.create_runner):
+        (MainTest.run_test):
+        (MainTest.test_run_passing_test):
+        (MainTest.test_run_silent_test):
+        (MainTest.test_run_failed_test):
+        (MainTest.test_run_tonguey_test):
+        (MainTest.test_run_timeout_test):
+        (MainTest.test_run_crash_test):
+        (MainTest.test_run_test_set):
+        (MainTest.test_collect_tests):
+        (MainTest):
+        (MainTest.test_parse_args):
+        * Scripts/webkitpy/tool/commands/rebaseline.py:
+        (OptimizeBaselines.execute):
+        (AnalyzeBaselines.execute):
+
 2011-11-30  nduca@chromium.org  <nduca@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
 
         [chromium] Enable threaded compositing via CCThreadProxy::hasThread only
diff --git a/Tools/Scripts/run-inspector-perf-tests.py b/Tools/Scripts/run-inspector-perf-tests.py
new file mode 100755 (executable)
index 0000000..0a33c54
--- /dev/null
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# Copyright (C) 2011 Google Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Run Inspector's perf tests in perf mode."""
+
+import logging
+import sys
+
+from webkitpy.performance_tests.perftestsrunner import PerfTestsRunner
+
+_log = logging.getLogger(__name__)
+
+if '__main__' == __name__:
+    logging.basicConfig(level=logging.INFO, format="%(message)s")
+    sys.exit(PerfTestsRunner('inspector').run())
similarity index 59%
rename from Tools/Scripts/webkitpy/layout_tests/port/test_files.py
rename to Tools/Scripts/webkitpy/common/find_files.py
index b8b84138bc842bb45e202606e934a75fd1b47837..665bcccd2144104f3b802447df564e3ab00c4b5a 100644 (file)
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# Copyright (C) 2010 Google Inc. All rights reserved.
+# Copyright (C) 2011 Google Inc. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted provided that the following conditions are
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-"""This module is used to find all of the layout test files used by
-run-webkit-tests. It exposes one public function - find() -
-which takes an optional list of paths. If a list is passed in, the returned
-list of test files is constrained to those found under the paths passed in,
-i.e. calling find(["LayoutTests/fast"]) will only return files
-under that directory."""
+"""This module is used to find files used by run-webkit-tests and
+perftestrunner. It exposes one public function - find() - which takes
+an optional list of paths, optional set of skipped directories and optional
+filter callback.
+
+If a list is passed in, the returned list of files is constrained to those
+found under the paths passed in. i.e. calling find(["LayoutTests/fast"])
+will only return files under that directory.
+
+If a set of skipped directories is passed in, the function will filter out
+the files lying in these directories i.e. find(["LayoutTests"], set(["fast"]))
+will return everything except files in fast subfolder.
+
+If a callback is passed in, it will be called for the each file and the file
+will be included into the result if the callback returns True.
+The callback has to take three arguments: filesystem, dirname and filename."""
 
 import time
 
@@ -42,32 +52,27 @@ from webkitpy.common.system import logutils
 _log = logutils.get_logger(__file__)
 
 
-# When collecting test cases, we include any file with these extensions.
-_supported_file_extensions = set(['.html', '.shtml', '.xml', '.xhtml', '.pl',
-                                  '.htm', '.php', '.svg', '.mht'])
-# When collecting test cases, skip these directories
-_skipped_directories = set(['.svn', '_svn', 'resources', 'script-tests'])
-
-
-def find(port, paths=None):
+def find(filesystem, base_dir, paths=None, skipped_directories=None, file_filter=None):
     """Finds the set of tests under a given list of sub-paths.
 
     Args:
-      paths: a list of path expressions relative to port.layout_tests_dir()
+      paths: a list of path expressions relative to base_dir
           to search. Glob patterns are ok, as are path expressions with
           forward slashes on Windows. If paths is empty, we look at
-          everything under the layout_tests_dir().
+          everything under the base_dir.
     """
+
+    global _skipped_directories
     paths = paths or ['*']
-    filesystem = port._filesystem
-    return normalized_find(filesystem, normalize(filesystem, port.layout_tests_dir(), paths))
+    skipped_directories = skipped_directories or set(['.svn', '_svn'])
+    return _normalized_find(filesystem, _normalize(filesystem, base_dir, paths), skipped_directories, file_filter)
 
 
-def normalize(filesystem, base_dir, paths):
+def _normalize(filesystem, base_dir, paths):
     return [filesystem.normpath(filesystem.join(base_dir, path)) for path in paths]
 
 
-def normalized_find(filesystem, paths):
+def _normalized_find(filesystem, paths, skipped_directories, file_filter):
     """Finds the set of tests under the list of paths.
 
     Args:
@@ -86,26 +91,12 @@ def normalized_find(filesystem, paths):
             paths_to_walk.add(path)
 
     # FIXME: I'm not sure there's much point in this being a set. A list would probably be faster.
-    test_files = set()
+    all_files = set()
     for path in paths_to_walk:
-        files = filesystem.files_under(path, _skipped_directories, _is_test_file)
-        test_files.update(set(files))
+        files = filesystem.files_under(path, skipped_directories, file_filter)
+        all_files.update(set(files))
 
     gather_time = time.time() - gather_start_time
     _log.debug("Test gathering took %f seconds" % gather_time)
 
-    return test_files
-
-
-def _has_supported_extension(filesystem, filename):
-    """Return true if filename is one of the file extensions we want to run a test on."""
-    extension = filesystem.splitext(filename)[1]
-    return extension in _supported_file_extensions
-
-
-def is_reference_html_file(filename):
-    return filename.endswith('-expected.html') or filename.endswith('-expected-mismatch.html')
-
-
-def _is_test_file(filesystem, dirname, filename):
-    return _has_supported_extension(filesystem, filename) and not is_reference_html_file(filename)
+    return all_files
similarity index 60%
rename from Tools/Scripts/webkitpy/layout_tests/port/test_files_unittest.py
rename to Tools/Scripts/webkitpy/common/find_files_unittest.py
index a29ba49ef8b52c3e42320985bff66561f6d9870c..b9068df48a1cc44ef96e5fa1c32fdf637e1c099a 100644 (file)
@@ -1,4 +1,4 @@
-# Copyright (C) 2010 Google Inc. All rights reserved.
+# Copyright (C) 2011 Google Inc. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted provided that the following conditions are
 import sys
 import unittest
 
-from webkitpy.layout_tests.port import test
-import test_files
-
-class TestFilesTest(unittest.TestCase):
-    def test_find_no_paths_specified(self):
-        port = test.TestPort()
-        layout_tests_dir = port.layout_tests_dir()
-        tests = test_files.find(port, [])
-        self.assertNotEqual(len(tests), 0)
-
-    def test_find_one_test(self):
-        port = test.TestPort()
-        tests = test_files.find(port, ['failures/expected/image.html'])
-        self.assertEqual(len(tests), 1)
-
-    def test_find_glob(self):
-        port = test.TestPort()
-        tests = test_files.find(port, ['failures/expected/im*'])
-        self.assertEqual(len(tests), 2)
-
-    def test_find_with_skipped_directories(self):
-        port = test.TestPort()
-        tests = port.tests('userscripts')
-        self.assertTrue('userscripts/resources/iframe.html' not in tests)
-
-    def test_find_with_skipped_directories_2(self):
-        port = test.TestPort()
-        tests = test_files.find(port, ['userscripts/resources'])
-        self.assertEqual(tests, set([]))
-
-    def test_is_test_file(self):
-        port = test.TestPort()
-        fs = port._filesystem
-        self.assertTrue(test_files._is_test_file(fs, '', 'foo.html'))
-        self.assertTrue(test_files._is_test_file(fs, '', 'foo.shtml'))
-        self.assertFalse(test_files._is_test_file(fs, '', 'foo.png'))
-        self.assertFalse(test_files._is_test_file(fs, '', 'foo-expected.html'))
-        self.assertFalse(test_files._is_test_file(fs, '', 'foo-expected-mismatch.html'))
+import find_files
 
 
 class MockWinFileSystem(object):
@@ -79,7 +42,7 @@ class MockWinFileSystem(object):
 
 class TestWinNormalize(unittest.TestCase):
     def assert_filesystem_normalizes(self, filesystem):
-        self.assertEquals(test_files.normalize(filesystem, "c:\\foo",
+        self.assertEquals(find_files._normalize(filesystem, "c:\\foo",
             ['fast/html', 'fast/canvas/*', 'compositing/foo.html']),
             ['c:\\foo\\fast\html', 'c:\\foo\\fast\canvas\*', 'c:\\foo\compositing\\foo.html'])
 
index 4c1bbe8e08708231f04a92b2230b8c083b7ec349..f2b0aa1a6714667736df045ee9ad7c47b1ebb4d3 100755 (executable)
@@ -46,6 +46,7 @@ try:
 except ImportError:
     multiprocessing = None
 
+from webkitpy.common import find_files
 from webkitpy.common.system import logutils
 from webkitpy.common.system import path
 from webkitpy.common.system.executive import ScriptError
@@ -54,7 +55,6 @@ from webkitpy.layout_tests.models.test_configuration import TestConfiguration
 from webkitpy.layout_tests.port import config as port_config
 from webkitpy.layout_tests.port import driver
 from webkitpy.layout_tests.port import http_lock
-from webkitpy.layout_tests.port import test_files
 from webkitpy.layout_tests.servers import apache_http_server
 from webkitpy.layout_tests.servers import http_server
 from webkitpy.layout_tests.servers import websocket_server
@@ -461,8 +461,13 @@ class Port(object):
 
     def tests(self, paths):
         """Return the list of tests found."""
-        # FIXME: Should test_files.find() return normalized, relative test names?
-        return set([self.relative_test_filename(f) for f in test_files.find(self, paths)])
+        # FIXME: Should port.find_test_files() return normalized, relative test names?
+        return set([self.relative_test_filename(f) for f in self.find_test_files(paths)])
+
+    def find_test_files(self, paths):
+        # When collecting test cases, skip these directories
+        skipped_directories = set(['.svn', '_svn', 'resources', 'script-tests'])
+        return find_files.find(self.filesystem, self.layout_tests_dir(), paths, skipped_directories, _is_test_file)
 
     def test_dirs(self):
         """Returns the list of top-level test directories."""
@@ -994,3 +999,22 @@ class Port(object):
     def _driver_class(self):
         """Returns the port's driver implementation."""
         raise NotImplementedError('Port._driver_class')
+
+
+# When collecting test cases, we include any file with these extensions.
+_supported_file_extensions = set(['.html', '.shtml', '.xml', '.xhtml', '.pl',
+                                 '.htm', '.php', '.svg', '.mht'])
+
+
+def is_reference_html_file(filename):
+    return filename.endswith('-expected.html') or filename.endswith('-expected-mismatch.html')
+
+
+def _has_supported_extension(filesystem, filename):
+    """Return true if filename is one of the file extensions we want to run a test on."""
+    extension = filesystem.splitext(filename)[1]
+    return extension in _supported_file_extensions
+
+
+def _is_test_file(filesystem, dirname, filename):
+    return _has_supported_extension(filesystem, filename) and not is_reference_html_file(filename)
index d705afc4b6be21afadff525a496e505299e5962b..e9e072199b17bdd435283c415002333a54034bdc 100644 (file)
@@ -42,6 +42,8 @@ from webkitpy.common.system.executive_mock import MockExecutive
 from webkitpy.common.host_mock import MockHost
 
 from webkitpy.layout_tests.port import Port, Driver, DriverOutput
+from webkitpy.layout_tests.port.base import _is_test_file
+from webkitpy.layout_tests.port.test import TestPort
 
 import config
 import config_mock
@@ -274,6 +276,40 @@ class PortTest(unittest.TestCase):
         port._filesystem = MockFileSystem({'/mock-results/test_expectations.txt': ''})
         self.assertTrue(port.uses_test_expectations_file())
 
+    def test_find_no_paths_specified(self):
+        port = TestPort()
+        layout_tests_dir = port.layout_tests_dir()
+        tests = port.find_test_files([])
+        self.assertNotEqual(len(tests), 0)
+
+    def test_find_one_test(self):
+        port = TestPort()
+        tests = port.find_test_files(['failures/expected/image.html'])
+        self.assertEqual(len(tests), 1)
+
+    def test_find_glob(self):
+        port = TestPort()
+        tests = port.find_test_files(['failures/expected/im*'])
+        self.assertEqual(len(tests), 2)
+
+    def test_find_with_skipped_directories(self):
+        port = TestPort()
+        tests = port.find_test_files('userscripts')
+        self.assertTrue('userscripts/resources/iframe.html' not in tests)
+
+    def test_find_with_skipped_directories_2(self):
+        port = TestPort()
+        tests = port.find_test_files(['userscripts/resources'])
+        self.assertEqual(tests, set([]))
+
+    def test_is_test_file(self):
+        filesystem = MockFileSystem()
+        self.assertTrue(_is_test_file(filesystem, '', 'foo.html'))
+        self.assertTrue(_is_test_file(filesystem, '', 'foo.shtml'))
+        self.assertFalse(_is_test_file(filesystem, '', 'foo.png'))
+        self.assertFalse(_is_test_file(filesystem, '', 'foo-expected.html'))
+        self.assertFalse(_is_test_file(filesystem, '', 'foo-expected-mismatch.html'))
+
 
 class VirtualTest(unittest.TestCase):
     """Tests that various methods expected to be virtual are."""
index e86b8b753ed7a1b0c189f9b544617b7d9da6759c..266103341e4d374477e36c08a1f012d2813722e9 100644 (file)
@@ -30,8 +30,6 @@ import chromium_linux
 import chromium_mac
 import chromium_win
 
-from webkitpy.layout_tests.port import test_files
-
 
 def get(host, platform=None, port_name='chromium-gpu', **kwargs):
     """Some tests have slightly different results when run while using
@@ -94,13 +92,13 @@ def _tests(port, paths):
 
         if not paths:
             # FIXME: This is a hack until we can turn of the webkit_gpu
-            # tests on the bots. If paths is empty, test_files.find()
+            # tests on the bots. If paths is empty, port.find_test_files()
             # finds *everything*. However, we have to return something,
             # or NRWT thinks there's something wrong. So, we return a single
             # short directory. See https://bugs.webkit.org/show_bug.cgi?id=72498.
             paths = ['fast/html']
 
-    return set([port.relative_test_filename(f) for f in test_files.find(port, paths)])
+    return set([port.relative_test_filename(f) for f in port.find_test_files(paths)])
 
 
 class ChromiumGpuLinuxPort(chromium_linux.ChromiumLinuxPort):
index a4fc9244c3a5d9c524595c238b017bf5ead1e1e5..1ef449a1eb2b120945732b88cdb53cfb2c1f18bb 100644 (file)
@@ -257,7 +257,7 @@ WONTFIX SKIP : failures/expected/exception.html = CRASH
 
     # FIXME: This test was only being ignored because of missing a leading '/'.
     # Fixing the typo causes several tests to assert, so disabling the test entirely.
-    # Add in a file should be ignored by test_files.find().
+    # Add in a file should be ignored by port.find_test_files().
     #files[LAYOUT_TEST_DIR + '/userscripts/resources/iframe.html'] = 'iframe'
 
     fs = MockFileSystem(files, dirs=set(['/mock-checkout']))  # Make sure at least the checkout_root exists as a directory.
index 5d37cfe38a6f07a8d5822b527bb05ce1990c17f1..f6adcfc791e893b019357cf7f13c3bce91bc1b60 100755 (executable)
@@ -66,7 +66,7 @@ from webkitpy.common.host_mock import MockHost
 from webkitpy.layout_tests import port
 from webkitpy.layout_tests import run_webkit_tests
 from webkitpy.layout_tests.port.test import TestPort, TestDriver, unit_test_filesystem
-from webkitpy.layout_tests.port.test_files import is_reference_html_file
+from webkitpy.layout_tests.port.base import is_reference_html_file
 from webkitpy.python24.versioning import compare_version
 from webkitpy.test.skip import skip_if
 
diff --git a/Tools/Scripts/webkitpy/performance_tests/__init__.py b/Tools/Scripts/webkitpy/performance_tests/__init__.py
new file mode 100644 (file)
index 0000000..ef65bee
--- /dev/null
@@ -0,0 +1 @@
+# Required for Python to search this directory for module files
diff --git a/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py b/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py
new file mode 100644 (file)
index 0000000..e8978ce
--- /dev/null
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+# Copyright (C) 2011 Google Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Run Inspector's perf tests in perf mode."""
+
+import logging
+import optparse
+import re
+import sys
+
+from webkitpy.common import find_files
+from webkitpy.common.host import Host
+from webkitpy.layout_tests.port.driver import DriverInput
+from webkitpy.layout_tests.views import printing
+
+_log = logging.getLogger(__name__)
+
+
+class PerfTestsRunner(object):
+    _perf_tests_base_dir = 'PerformanceTests'
+    _result_regex = re.compile('^RESULT .*$')
+
+    def __init__(self, perf_tests_dir, regular_output=sys.stderr, buildbot_output=sys.stdout, args=None):
+        self._perf_tests_dir = perf_tests_dir
+        self._buildbot_output = buildbot_output
+        self._options, self._args = self._parse_args(args)
+        self._host = Host()
+        self._host._initialize_scm()
+        self._port = self._host.port_factory.get(self._options.platform, self._options)
+        self._printer = printing.Printer(self._port, self._options, regular_output, buildbot_output, configure_logging=False)
+        self._webkit_base_dir_len = len(self._port.config.webkit_base_dir())
+
+    def _parse_args(self, args=None):
+        print_options = printing.print_options()
+
+        perf_option_list = [
+            optparse.make_option('--debug', action='store_const', const='Debug', dest="configuration",
+                                 help='Set the configuration to Debug'),
+            optparse.make_option('--release', action='store_const', const='Release', dest="configuration",
+                                 help='Set the configuration to Release'),
+            optparse.make_option("--platform",
+                                 help="Specify port/platform being tested (i.e. chromium-mac)"),
+            optparse.make_option("--build-directory",
+                                 help="Path to the directory under which build files are kept (should not include configuration)"),
+            optparse.make_option("--time-out-ms", default=30000,
+                                 help="Set the timeout for each test"),
+            ]
+
+        option_list = (perf_option_list + print_options)
+        return optparse.OptionParser(option_list=option_list).parse_args(args)
+
+    def _collect_tests(self, webkit_base, filesystem=None):
+        """Return the list of tests found."""
+
+        def _is_test_file(filesystem, dirname, filename):
+            return filename.endswith('.html')
+
+        filesystem = filesystem or self._host.filesystem
+        base_dir = filesystem.join(webkit_base, self._perf_tests_base_dir, self._perf_tests_dir)
+        return find_files.find(filesystem, base_dir, paths=self._args, file_filter=_is_test_file)
+
+    def run(self):
+        if self._options.help_printing:
+            self._printer.help_printing()
+            self._printer.cleanup()
+            return 0
+
+        if not self._port.check_build(needs_http=False):
+            _log.error("Build not up to date for %s" % self._port._path_to_driver())
+            return -1
+
+        # We wrap any parts of the run that are slow or likely to raise exceptions
+        # in a try/finally to ensure that we clean up the logging configuration.
+        unexpected = -1
+        try:
+            tests = self._collect_tests(self._port.webkit_base())
+            unexpected = self._run_tests_set(tests, self._port)
+        finally:
+            self._printer.cleanup()
+
+        return unexpected
+
+    def _run_tests_set(self, tests, port):
+        result_count = len(tests)
+        expected = 0
+        unexpected = 0
+        self._printer.print_one_line_summary(result_count, 0, 0)
+        driver_need_restart = False
+        driver = None
+
+        for test in tests:
+            if driver_need_restart:
+                _log.debug("%s killing driver" % test)
+                driver.stop()
+                driver = None
+            if not driver:
+                driver = port.create_driver(worker_number=1)
+
+            test_failed, driver_need_restart = self._run_single_test(test, driver)
+            if test_failed:
+                unexpected = unexpected + 1
+            else:
+                expected = expected + 1
+
+            self._printer.print_one_line_summary(result_count, expected, unexpected)
+
+        if driver:
+            driver.stop()
+
+        return unexpected
+
+    def _run_single_test(self, test, driver):
+        test_failed = False
+        driver_need_restart = False
+        output = driver.run_test(DriverInput(test, self._options.time_out_ms, None, False))
+
+        if output.text == None:
+            test_failed = True
+        elif output.timeout:
+            self._printer.write('timeout: %s' % test[self._webkit_base_dir_len + 1:])
+            test_failed = True
+            driver_need_restart = True
+        elif output.crash:
+            self._printer.write('crash: %s' % test[self._webkit_base_dir_len + 1:])
+            driver_need_restart = True
+            test_failed = True
+        else:
+            got_a_result = False
+            for line in re.split('\n', output.text):
+                if self._result_regex.match(line):
+                    self._buildbot_output.write("%s\n" % line)
+                    got_a_result = True
+                elif not len(line) == 0:
+                    test_failed = True
+                    self._printer.write("%s" % line)
+            test_failed = test_failed or not got_a_result
+
+        if len(output.error):
+            self._printer.write('error:\n%s' % output.error)
+            test_failed = True
+
+        return test_failed, driver_need_restart
diff --git a/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py b/Tools/Scripts/webkitpy/performance_tests/perftestsrunner_unittest.py
new file mode 100755 (executable)
index 0000000..782f5e7
--- /dev/null
@@ -0,0 +1,143 @@
+#!/usr/bin/python
+# Copyright (C) 2011 Google Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Unit tests for run_perf_tests."""
+
+import unittest
+
+from webkitpy.common import array_stream
+from webkitpy.common.system.filesystem_mock import MockFileSystem
+from webkitpy.layout_tests.port.driver import DriverInput, DriverOutput
+from webkitpy.layout_tests.views import printing
+from webkitpy.performance_tests.perftestsrunner import PerfTestsRunner
+
+
+class MainTest(unittest.TestCase):
+    class TestPort:
+        def create_driver(self, worker_number=None):
+            return MainTest.TestDriver()
+
+    class TestDriver:
+        def run_test(self, driver_input):
+            text = ''
+            timeout = False
+            crash = False
+            if driver_input.test_name == 'pass.html':
+                text = 'RESULT group_name: test_name= 42 ms'
+            elif driver_input.test_name == 'timeout.html':
+                timeout = True
+            elif driver_input.test_name == 'failed.html':
+                text = None
+            elif driver_input.test_name == 'tonguey.html':
+                text = 'we are not expecting an output from perf tests but RESULT blablabla'
+            elif driver_input.test_name == 'crash.html':
+                crash = True
+            return DriverOutput(text, '', '', '', crash=crash, timeout=timeout)
+
+        def stop(self):
+            """do nothing"""
+
+    def create_runner(self, buildbot_output=None):
+        buildbot_output = buildbot_output or array_stream.ArrayStream()
+        regular_output = array_stream.ArrayStream()
+        return PerfTestsRunner('', regular_output, buildbot_output, args=[])
+
+    def run_test(self, test_name):
+        runner = self.create_runner()
+        driver = MainTest.TestDriver()
+        return runner._run_single_test(test_name, driver)
+
+    def test_run_passing_test(self):
+        test_failed, driver_need_restart = self.run_test('pass.html')
+        self.assertFalse(test_failed)
+        self.assertFalse(driver_need_restart)
+
+    def test_run_silent_test(self):
+        test_failed, driver_need_restart = self.run_test('silent.html')
+        self.assertTrue(test_failed)
+        self.assertFalse(driver_need_restart)
+
+    def test_run_failed_test(self):
+        test_failed, driver_need_restart = self.run_test('failed.html')
+        self.assertTrue(test_failed)
+        self.assertFalse(driver_need_restart)
+
+    def test_run_tonguey_test(self):
+        test_failed, driver_need_restart = self.run_test('tonguey.html')
+        self.assertTrue(test_failed)
+        self.assertFalse(driver_need_restart)
+
+    def test_run_timeout_test(self):
+        test_failed, driver_need_restart = self.run_test('timeout.html')
+        self.assertTrue(test_failed)
+        self.assertTrue(driver_need_restart)
+
+    def test_run_crash_test(self):
+        test_failed, driver_need_restart = self.run_test('crash.html')
+        self.assertTrue(test_failed)
+        self.assertTrue(driver_need_restart)
+
+    def test_run_test_set(self):
+        buildbot_output = array_stream.ArrayStream()
+        runner = self.create_runner(buildbot_output)
+        port = MainTest.TestPort()
+        tests = ['pass.html', 'silent.html', 'failed.html', 'tonguey.html', 'timeout.html', 'crash.html']
+        unexpected_result_count = runner._run_tests_set(tests, port)
+        self.assertEqual(unexpected_result_count, len(tests) - 1)
+        self.assertEqual(len(buildbot_output.get()), 1)
+        self.assertEqual(buildbot_output.get()[0], 'RESULT group_name: test_name= 42 ms\n')
+
+    def test_collect_tests(self):
+        runner = self.create_runner()
+        webkit_base = '/test.checkout'
+        filesystem = MockFileSystem()
+        filename = filesystem.join(webkit_base, 'PerformanceTests', 'a_file.html')
+        filesystem.files[filename] = 'a content'
+        tests = runner._collect_tests(webkit_base, filesystem)
+        self.assertEqual(len(tests), 1)
+
+    def test_parse_args(self):
+        runner = self.create_runner()
+        options, args = runner._parse_args([
+                '--verbose',
+                '--build-directory=folder42',
+                '--platform=platform42',
+                '--time-out-ms=42',
+                '--debug', 'an_arg'])
+        self.assertEqual(options.verbose, True)
+        self.assertEqual(options.help_printing, None)
+        self.assertEqual(options.build_directory, 'folder42')
+        self.assertEqual(options.platform, 'platform42')
+        self.assertEqual(options.time_out_ms, '42')
+        self.assertEqual(options.configuration, 'Debug')
+        self.assertEqual(options.print_options, None)
+
+
+if __name__ == '__main__':
+    unittest.main()
index 9ea39a3b2c4f06efc7f702c43590552543ea9b4b..5db33335fba73fa854449538cce0caffd7c703ec 100644 (file)
@@ -41,7 +41,6 @@ from webkitpy.layout_tests.controllers.test_result_writer import TestResultWrite
 from webkitpy.layout_tests.models import test_failures
 from webkitpy.layout_tests.models.test_expectations import TestExpectations
 from webkitpy.layout_tests.port import builders
-from webkitpy.layout_tests.port import test_files
 from webkitpy.tool.grammar import pluralize
 from webkitpy.tool.multicommandtool import AbstractDeclarativeCommand
 
@@ -119,7 +118,7 @@ class OptimizeBaselines(AbstractDeclarativeCommand):
         self._baseline_optimizer = BaselineOptimizer(tool)
         self._port = tool.port_factory.get("chromium-win-win7")  # FIXME: This should be selectable.
 
-        for test_name in map(self._to_test_name, test_files.find(self._port, args)):
+        for test_name in map(self._to_test_name, self._port.find_test_files(args)):
             print "Optimizing %s." % test_name
             self._optimize_baseline(test_name)
 
@@ -149,7 +148,7 @@ class AnalyzeBaselines(AbstractDeclarativeCommand):
         self._baseline_optimizer = BaselineOptimizer(tool)
         self._port = tool.port_factory.get("chromium-win-win7")  # FIXME: This should be selectable.
 
-        for test_name in map(self._to_test_name, test_files.find(self._port, args)):
+        for test_name in map(self._to_test_name, self._port.find_test_files(args)):
             self._analyze_baseline(test_name)