2010-08-25 Dirk Pranke <dpranke@chromium.org>
authordpranke@chromium.org <dpranke@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 25 Aug 2010 21:45:26 +0000 (21:45 +0000)
committerdpranke@chromium.org <dpranke@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 25 Aug 2010 21:45:26 +0000 (21:45 +0000)
        Reviewed by Ojan Vafai.

        new-run-webkit-tests: add more unit tests

        This change adds more unit tests for the image_diff, test_failures,
        and test_files modules, bringing them to 100% coverage (and adds a
        couple tests to run_webkit_tests while we're at it).

        This test also turns on the "dryrun" port for testing when run on
        the mac, increasing coverage in a few other places.

        https://bugs.webkit.org/show_bug.cgi?id=44578

        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum.html: Copied from WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/image_checksum.html.
        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_check.html: Copied from WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_image.html.
        * Scripts/webkitpy/layout_tests/data/failures/expected/timeout.html: Added.
        * Scripts/webkitpy/layout_tests/data/platform/test/test_expectations.txt:
        * Scripts/webkitpy/layout_tests/data/resources/README.txt: Added.
        * Scripts/webkitpy/layout_tests/layout_package/test_failures_unittest.py:
        * Scripts/webkitpy/layout_tests/port/test.py:
        * Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py:

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@66045 268f45cc-cd09-0410-ab3c-d52691b4dbfc

14 files changed:
WebKitTools/ChangeLog
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.checksum [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.png [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.txt [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum.html [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.png [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.txt [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check.html [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/timeout.html [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/data/platform/test/test_expectations.txt
WebKitTools/Scripts/webkitpy/layout_tests/data/resources/README.txt [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/layout_tests/layout_package/test_failures_unittest.py
WebKitTools/Scripts/webkitpy/layout_tests/port/test.py
WebKitTools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py

index c27ea1191ef40563e117a9675f9e59d605121a8a..6b8fc2819993dea216ad7e456403c841a90d33ac 100644 (file)
@@ -1,3 +1,32 @@
+2010-08-25  Dirk Pranke  <dpranke@chromium.org>
+
+        Reviewed by Ojan Vafai.
+
+        new-run-webkit-tests: add more unit tests
+
+        This change adds more unit tests for the image_diff, test_failures,
+        and test_files modules, bringing them to 100% coverage (and adds a
+        couple tests to run_webkit_tests while we're at it).
+
+        This test also turns on the "dryrun" port for testing when run on
+        the mac, increasing coverage in a few other places.
+
+        https://bugs.webkit.org/show_bug.cgi?id=44578
+
+        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.checksum: Added.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.png: Added.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.txt: Added.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/checksum.html: Copied from WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/image_checksum.html.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.png: Added.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.txt: Added.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_check.html: Copied from WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_image.html.
+        * Scripts/webkitpy/layout_tests/data/failures/expected/timeout.html: Added.
+        * Scripts/webkitpy/layout_tests/data/platform/test/test_expectations.txt:
+        * Scripts/webkitpy/layout_tests/data/resources/README.txt: Added.
+        * Scripts/webkitpy/layout_tests/layout_package/test_failures_unittest.py:
+        * Scripts/webkitpy/layout_tests/port/test.py:
+        * Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py:
+
 2010-08-25  Dirk Pranke  <dpranke@chromium.org>
 
         Reviewed by Ojan Vafai.
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.checksum b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.checksum
new file mode 100644 (file)
index 0000000..5890112
--- /dev/null
@@ -0,0 +1 @@
+checksum-checksum
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.png b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.png
new file mode 100644 (file)
index 0000000..83a5de3
--- /dev/null
@@ -0,0 +1 @@
+checksum-png
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.txt b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum-expected.txt
new file mode 100644 (file)
index 0000000..5628d69
--- /dev/null
@@ -0,0 +1 @@
+checksum-txt
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum.html b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/checksum.html
new file mode 100644 (file)
index 0000000..2b78d31
--- /dev/null
@@ -0,0 +1 @@
+image_checksum
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.png b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.png
new file mode 100644 (file)
index 0000000..e45c7af
--- /dev/null
@@ -0,0 +1 @@
+missing_check-png
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.txt b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check-expected.txt
new file mode 100644 (file)
index 0000000..0ea9227
--- /dev/null
@@ -0,0 +1 @@
+missing_check-txt
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check.html b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/missing_check.html
new file mode 100644 (file)
index 0000000..0af8000
--- /dev/null
@@ -0,0 +1 @@
+missing_image
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/timeout.html b/WebKitTools/Scripts/webkitpy/layout_tests/data/failures/expected/timeout.html
new file mode 100644 (file)
index 0000000..790851a
--- /dev/null
@@ -0,0 +1 @@
+timeout
index 68d564116703a250702a562c9f6e8da5d01493da..6e66caab3fb6dd086cd2f4ddf223149c17209fa5 100644 (file)
@@ -1,5 +1,10 @@
-WONTFIX : failures/expected/text.html = TEXT
-WONTFIX SKIP : failures/expected/crash.html = CRASH
+WONTFIX : failures/expected/checksum.html = IMAGE
+WONTFIX : failures/expected/crash.html = CRASH
+// This one actually passes because the checksums will match.
+WONTFIX : failures/expected/image.html = PASS
 WONTFIX : failures/expected/image_checksum.html = IMAGE
-WONTFIX : failures/expected/missing_text.html = MISSING PASS
+WONTFIX : failures/expected/missing_check.html = MISSING PASS
 WONTFIX : failures/expected/missing_image.html = MISSING PASS
+WONTFIX : failures/expected/missing_text.html = MISSING PASS
+WONTFIX : failures/expected/text.html = TEXT
+WONTFIX : failures/expected/timeout.html = TIMEOUT
diff --git a/WebKitTools/Scripts/webkitpy/layout_tests/data/resources/README.txt b/WebKitTools/Scripts/webkitpy/layout_tests/data/resources/README.txt
new file mode 100644 (file)
index 0000000..b806b06
--- /dev/null
@@ -0,0 +1,2 @@
+This directory exists solely to make sure that when we gather the lists of
+tests, we skip over directories named 'resources'.
index 65c4a0b2281a11294df7241e98315ae26e04edd7..92fe2760c06a4569417ba83d10853cdfcc2a9ec4 100644 (file)
@@ -31,7 +31,6 @@
 from webkitpy.layout_tests.layout_package.test_failures import *
 import unittest
 
-
 class Test(unittest.TestCase):
     def assertResultHtml(self, failure_obj):
         self.assertNotEqual(failure_obj.result_html_output('foo'), None)
index 63c6896ea7c20446bd928d7065616db0ed09c78b..4fe3ec16cf515e17ac78577c10ff3af405f1ab04 100644 (file)
@@ -43,9 +43,6 @@ class TestPort(base.Port):
     def __init__(self, port_name=None, options=None):
         base.Port.__init__(self, port_name, options)
 
-    def base_platforms(self):
-        return ('test',)
-
     def baseline_path(self):
         return os.path.join(self.layout_tests_dir(), 'platform',
                             self.name() + self.version())
index 922d91366c7ea1d126239173c91f434a3604996c..3a3b14e5cbe54d93d2c7d7c2343bf26d694334b3 100644 (file)
@@ -52,7 +52,8 @@ def passing_run(args, port_obj=None, record_results=False,
                 tests_included=False):
     args.extend(['--print', 'nothing'])
     if not tests_included:
-        args.extend(['passes', 'failures/expected'])
+        # We use the glob to test that globbing works.
+        args.extend(['passes', 'failures/expected/*'])
     if not record_results:
         args.append('--no-record-results')
     options, args = run_webkit_tests.parse_args(args)
@@ -61,8 +62,11 @@ def passing_run(args, port_obj=None, record_results=False,
     res = run_webkit_tests.run(port_obj, options, args)
     return res == 0
 
-def logging_run(args):
+
+def logging_run(args, tests_included=False):
     args.extend(['--no-record-results'])
+    if not tests_included:
+        args.extend(['passes', 'failures/expected/*'])
     options, args = run_webkit_tests.parse_args(args)
     port_obj = port.get(options.platform, options)
     buildbot_output = array_stream.ArrayStream()
@@ -80,6 +84,11 @@ class MainTest(unittest.TestCase):
         self.assertTrue(passing_run(['--platform', 'test',
                                      'passes/text.html'], tests_included=True))
 
+    def test_unexpected_failures(self):
+        # Run tests including the unexpected failures.
+        self.assertFalse(passing_run(['--platform', 'test'],
+                         tests_included=True))
+
     def test_one_child_process(self):
         (res, buildbot_output, regular_output) = logging_run(
              ['--platform', 'test', '--print', 'config', '--child-processes',
@@ -101,6 +110,13 @@ class MainTest(unittest.TestCase):
         self.assertEqual(regular_output.get(), ['\n\n'])
         self.assertEqual(buildbot_output.get(), [])
 
+    def test_no_tests_found(self):
+        self.assertRaises(SystemExit, logging_run,
+                          ['--platform', 'test', 'resources'],
+                          tests_included=True)
+        self.assertRaises(SystemExit, logging_run,
+                          ['--platform', 'test', 'foo'],
+                          tests_included=True)
 
 def _mocked_open(original_open, file_list):
     def _wrapper(name, mode, encoding):
@@ -184,13 +200,14 @@ class TestRunnerTest(unittest.TestCase):
 
 
 class DryrunTest(unittest.TestCase):
-    def test_basics(self):
-        # FIXME: it's hard to know which platforms are safe to test; the
-        # chromium platforms require a chromium checkout, and the mac platform
-        # requires fcntl, so it can't be tested on win32, etc. There is
-        # probably a better way of handling this.
-        if sys.platform != "mac":
+    # FIXME: it's hard to know which platforms are safe to test; the
+    # chromium platforms require a chromium checkout, and the mac platform
+    # requires fcntl, so it can't be tested on win32, etc. There is
+    # probably a better way of handling this.
+    def test_darwin(self):
+        if sys.platform != "darwin":
             return
+
         self.assertTrue(passing_run(['--platform', 'dryrun',
                                      'fast/html']))
         self.assertTrue(passing_run(['--platform', 'dryrun-mac',