Completely ignore too-many-failures builds in TestFailures in most circumstances
authoraroben@apple.com <aroben@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 7 Jul 2011 17:18:09 +0000 (17:18 +0000)
committeraroben@apple.com <aroben@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 7 Jul 2011 17:18:09 +0000 (17:18 +0000)
Because a semi-arbitrary subset of tests are run in a too-many-failures build, we can't
really use them to perform regression analysis. The only time we want to pay attention to
too-many-failures builds is when we're trying to explain when the current bout of
too-many-failures started.

Fixes <http://webkit.org/b/64106> TestFailures page sometimes claims a test started failing
in a build that didn't even run it (because it exited early due to too many failues)

Reviewed by David Kilzer.

* BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:
(LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Removed old, broken
too-many-failures handling that would cause us to blame builds that didn't even run a given
test for breaking it. Instead, skip over all too-many-failures builds unless the most recent
build was itself a too-many-failures build.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@90571 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js
Tools/ChangeLog

index a048088..cbcadbd 100644 (file)
@@ -118,6 +118,20 @@ LayoutTestHistoryAnalyzer.prototype = {
 
         var self = this;
         self._loader.start(nextBuildName, function(tests, tooManyFailures) {
+            if (tooManyFailures) {
+                var firstBuildName = Object.keys(self._history)[0];
+                // If the first (i.e., current or most recent) build exited early due to too many
+                // failures, we want to process other too-many-failures builds normally to try to
+                // figure out when the too-many-failures started occurring. If the first/current
+                // build did not exit due to too many failures, then too-many-failures builds will
+                // only confuse our analysis (since they run a semi-arbitrary subset of tests), so
+                // we should just skip them entirely.
+                if (firstBuildName && !self._history[firstBuildName].tooManyFailures) {
+                    callback(true);
+                    return;
+                }
+            }
+
             ++self._testRunsSinceLastInterestingChange;
 
             var historyItem = {
@@ -150,16 +164,6 @@ LayoutTestHistoryAnalyzer.prototype = {
                 historyItem.tests[testName] = tests[testName];
             }
 
-            if (tooManyFailures && previousHistoryItem) {
-                // Not all tests were run due to too many failures. Just assume that all the tests
-                // that failed in the last build would still have failed in this build had they been
-                // run.
-                for (var testName in previousHistoryItem.tests) {
-                    historyItem.tests[testName] = previousHistoryItem.tests[testName];
-                    delete previousHistoryItem.tests[testName];
-                }
-            }
-
             var previousUnexplainedFailuresCount = previousBuildName ? Object.keys(self._history[previousBuildName].tests).length : 0;
             var unexplainedFailuresCount = Object.keys(self._history[nextBuildName].tests).length;
 
index 68e79d2..a98aac9 100644 (file)
@@ -1,5 +1,25 @@
 2011-07-07  Adam Roben  <aroben@apple.com>
 
+        Completely ignore too-many-failures builds in TestFailures in most circumstances
+
+        Because a semi-arbitrary subset of tests are run in a too-many-failures build, we can't
+        really use them to perform regression analysis. The only time we want to pay attention to
+        too-many-failures builds is when we're trying to explain when the current bout of
+        too-many-failures started.
+
+        Fixes <http://webkit.org/b/64106> TestFailures page sometimes claims a test started failing
+        in a build that didn't even run it (because it exited early due to too many failues)
+
+        Reviewed by David Kilzer.
+
+        * BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:
+        (LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Removed old, broken
+        too-many-failures handling that would cause us to blame builds that didn't even run a given
+        test for breaking it. Instead, skip over all too-many-failures builds unless the most recent
+        build was itself a too-many-failures build.
+
+2011-07-07  Adam Roben  <aroben@apple.com>
+
         Teach webkitpy's Checkout class to use commit-log-editor to create commit messages
 
         Fixes <http://webkit.org/b/26755> webkit-patch's commit messages are less readable than