[EWS] Do not retry layout-tests build if the flaky test failures are also present...
authoraakash_jain@apple.com <aakash_jain@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sat, 30 Nov 2019 08:15:36 +0000 (08:15 +0000)
committeraakash_jain@apple.com <aakash_jain@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sat, 30 Nov 2019 08:15:36 +0000 (08:15 +0000)
https://bugs.webkit.org/show_bug.cgi?id=204704

Reviewed by Alexey Proskuryakov.

* BuildSlaveSupport/ews-build/steps.py:
(AnalyzeLayoutTestsResults.start):
* BuildSlaveSupport/ews-build/steps_unittest.py:
(TestAnalyzeLayoutTestsResults.test_flaky_and_inconsistent_failures_with_clean_tree_failures): Changed expected outcome to SUCCESS.
(TestAnalyzeLayoutTestsResults.test_flaky_and_consistent_failures_with_clean_tree_failures): Ditto.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@252953 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/BuildSlaveSupport/ews-build/steps.py
Tools/BuildSlaveSupport/ews-build/steps_unittest.py
Tools/ChangeLog

index 3ef81fd..c8574ce 100644 (file)
@@ -1452,8 +1452,12 @@ class AnalyzeLayoutTestsResults(buildstep.BuildStep):
                 if failures_introduced_by_patch:
                     return self.report_failure(failures_introduced_by_patch)
 
-            # At this point we know that at least one test flaked, but no consistent failures
-            # were introduced. This is a bit of a grey-zone.
+            new_failing_or_flaky_tests = first_results_failing_tests.union(second_results_failing_tests) - clean_tree_results_failing_tests
+            if not new_failing_or_flaky_tests:
+                return self.report_pre_existing_failures(clean_tree_results_failing_tests)
+
+            # At this point we know that at least one test flaked, and those flaky tests passed on clean tree,
+            # and no consistent failures were introduced. This is a bit of a grey-zone. So, we retry the build.
             return self.retry_build()
 
         if clean_tree_results_did_exceed_test_failure_limit:
index dcb002e..53dfa5b 100644 (file)
@@ -1647,7 +1647,7 @@ class TestAnalyzeLayoutTestsResults(BuildStepMixinAdditions, unittest.TestCase):
         self.setProperty('first_run_failures', ['test1', 'test2'])
         self.setProperty('second_run_failures', ['test3'])
         self.setProperty('clean_tree_run_failures', ['test1', 'test2', 'test3'])
-        self.expectOutcome(result=RETRY, state_string='Unable to confirm if test failures are introduced by patch, retrying build (retry)')
+        self.expectOutcome(result=SUCCESS, state_string='Passed layout tests')
         return self.runStep()
 
     def test_flaky_and_consistent_failures_with_clean_tree_failures(self):
@@ -1655,7 +1655,7 @@ class TestAnalyzeLayoutTestsResults(BuildStepMixinAdditions, unittest.TestCase):
         self.setProperty('first_run_failures', ['test1', 'test2'])
         self.setProperty('second_run_failures', ['test1'])
         self.setProperty('clean_tree_run_failures', ['test1', 'test2'])
-        self.expectOutcome(result=RETRY, state_string='Unable to confirm if test failures are introduced by patch, retrying build (retry)')
+        self.expectOutcome(result=SUCCESS, state_string='Passed layout tests')
         return self.runStep()
 
     def test_mildly_flaky_patch_with_some_tree_redness_and_flakiness(self):
index a26ed49..6a1ab8b 100644 (file)
@@ -1,3 +1,16 @@
+2019-11-30  Aakash Jain  <aakash_jain@apple.com>
+
+        [EWS] Do not retry layout-tests build if the flaky test failures are also present in clean tree run
+        https://bugs.webkit.org/show_bug.cgi?id=204704
+
+        Reviewed by Alexey Proskuryakov.
+
+        * BuildSlaveSupport/ews-build/steps.py:
+        (AnalyzeLayoutTestsResults.start):
+        * BuildSlaveSupport/ews-build/steps_unittest.py:
+        (TestAnalyzeLayoutTestsResults.test_flaky_and_inconsistent_failures_with_clean_tree_failures): Changed expected outcome to SUCCESS.
+        (TestAnalyzeLayoutTestsResults.test_flaky_and_consistent_failures_with_clean_tree_failures): Ditto.
+
 2019-11-29  Zalan Bujtas  <zalan@apple.com>
 
         [LFC][IFC] Add support for hyphenate-limit-before and hyphenate-limit-after