Teach buildbot to figure out how many webkitpy/webkitperl tests failed
authoraroben@apple.com <aroben@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 8 Jul 2011 19:05:17 +0000 (19:05 +0000)
committeraroben@apple.com <aroben@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 8 Jul 2011 19:05:17 +0000 (19:05 +0000)
Fixes <http://webkit.org/b/64192> It's hard to tell how many test-webkitpy/test-webkitperl
tests failed when looking at build.webkit.org

Reviewed by Eric Seidel.

* BuildSlaveSupport/build.webkit.org-config/master.cfg:
(TestWithFailureCount): New class that represents a test build step which has an associated
failure count. Eventually we should move more of our test classes to deriving from this.
(TestWithFailureCount.countFailures): Method for subclasses to override to say how many
failures occurred.

(TestWithFailureCount.commandComplete):
(TestWithFailureCount.evaluateCommand):
(TestWithFailureCount.getText):
(TestWithFailureCount.getText2):
These were all based on RunGtkAPITests.

(RunPythonTests): Changed to inherit from TestWithFailureCount.
(RunPythonTests.countFailures): Parses the test-webkitpy output looking for the count of
failures.
(RunPerlTests): Changed to inherit from TestWithFailureCount.
(RunPerlTests.countFailures): Parses the test-webkitperl output looking for the count of
failures.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@90650 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg
Tools/ChangeLog

index 12f2fbb..00dd7b8 100644 (file)
@@ -331,18 +331,72 @@ class RunUnitTests(shell.Test):
             self.setCommand(self.command + ['--no-build'])
         return shell.Test.start(self)
 
-class RunPythonTests(shell.Test):
+
+class TestWithFailureCount(shell.Test):
+    failedTestsFormatString = "%d tests failed"
+
+    def countFailures(self, cmd):
+        return 0
+
+    def commandComplete(self, cmd):
+        shell.Test.commandComplete(self, cmd)
+        self.failedTestCount = self.countFailures(cmd)
+
+    def evaluateCommand(self, cmd):
+        if self.failedTestCount:
+            return FAILURE
+
+        if cmd.rc != 0:
+            return FAILURE
+
+        return SUCCESS
+
+    def getText(self, cmd, results):
+        return self.getText2(cmd, results)
+
+    def getText2(self, cmd, results):
+        if results != SUCCESS and self.failedTestCount:
+            return [self.failedTestsFormatString % self.failedTestCount]
+
+        return [self.name]
+
+
+class RunPythonTests(TestWithFailureCount):
     name = "webkitpy-test"
     description = ["python-tests running"]
     descriptionDone = ["python-tests"]
     command = ["python", "./Tools/Scripts/test-webkitpy"]
+    failedTestsFormatString = "%d python tests failed"
 
+    def countFailures(self, cmd):
+        logText = cmd.logs['stdio'].getText()
+        # We're looking for the line that looks like this: FAILED (failures=2, errors=1)
+        regex = re.compile(r'^FAILED \((?P<counts>[^)]+)\)')
+        for line in logText.splitlines():
+            match = regex.match(line)
+            if not match:
+                continue
+            return sum(int(component.split('=')[1]) for component in match.group('counts').split(', '))
+        return 0
 
-class RunPerlTests(shell.Test):
+
+class RunPerlTests(TestWithFailureCount):
     name = "webkitperl-test"
     description = ["perl-tests running"]
     descriptionDone = ["perl-tests"]
     command = ["perl", "./Tools/Scripts/test-webkitperl"]
+    failedTestsFormatString = "%d perl tests failed"
+
+    def countFailures(self, cmd):
+        logText = cmd.logs['stdio'].getText()
+        # We're looking for the line that looks like this: Failed 2/19 test programs. 5/363 subtests failed.
+        regex = re.compile(r'^Failed \d+/\d+ test programs\. (?P<count>\d+)/\d+ subtests failed\.')
+        for line in logText.splitlines():
+            match = regex.match(line)
+            if not match:
+                continue
+            return int(match.group('count'))
+        return 0
 
 
 class RunBindingsTests(shell.Test):
index 4e48815..9da5cc7 100644 (file)
@@ -1,5 +1,33 @@
 2011-07-08  Adam Roben  <aroben@apple.com>
 
+        Teach buildbot to figure out how many webkitpy/webkitperl tests failed
+
+        Fixes <http://webkit.org/b/64192> It's hard to tell how many test-webkitpy/test-webkitperl
+        tests failed when looking at build.webkit.org
+
+        Reviewed by Eric Seidel.
+
+        * BuildSlaveSupport/build.webkit.org-config/master.cfg:
+        (TestWithFailureCount): New class that represents a test build step which has an associated
+        failure count. Eventually we should move more of our test classes to deriving from this.
+        (TestWithFailureCount.countFailures): Method for subclasses to override to say how many
+        failures occurred.
+
+        (TestWithFailureCount.commandComplete):
+        (TestWithFailureCount.evaluateCommand):
+        (TestWithFailureCount.getText):
+        (TestWithFailureCount.getText2):
+        These were all based on RunGtkAPITests.
+
+        (RunPythonTests): Changed to inherit from TestWithFailureCount.
+        (RunPythonTests.countFailures): Parses the test-webkitpy output looking for the count of
+        failures.
+        (RunPerlTests): Changed to inherit from TestWithFailureCount.
+        (RunPerlTests.countFailures): Parses the test-webkitperl output looking for the count of
+        failures.
+
+2011-07-08  Adam Roben  <aroben@apple.com>
+
         Ensure $CHANGE_LOG_EMAIL_ADDRESS is set when testing webkitpy's commit-log-editor integration
 
         Fixes <http://webkit.org/b/64180> REGRESSION (r90564): test-webkitpy failing on multiple