run-webkit-tests prints confusing messages when test expectations list results that...
authorsimon.fraser@apple.com <simon.fraser@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 6 Sep 2018 23:25:39 +0000 (23:25 +0000)
committersimon.fraser@apple.com <simon.fraser@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 6 Sep 2018 23:25:39 +0000 (23:25 +0000)
commit16e5a3b05ad7a7d523199207f844c53b3ee9c2b3
treeefd3de6617896a0bca0c1950a330e80f8420b072
parentd8dd14ea1307513987a1a8c0fc36f483057d11b8
run-webkit-tests prints confusing messages when test expectations list results that are not compatible with the run options
https://bugs.webkit.org/show_bug.cgi?id=189219

Reviewed by Jon Lee.

If you call run-webkit-tests without --pixel-tests, and a non-ref test is marked as ImageOnlyFailure,
it will be reported as unexpectedly passing. This is more confusing if you run without --world-leaks, yet
tests are marked as [ Leak ] in TestExpectations.

Fix by filtering out expectations that don't apply given the run options. So without --pixel-tests,
a non-ref ImageOnlyFailure becomes a Pass, and without --world-leaks, a Leak becomes a Pass.

Add various unit tests to test the various combinations.

* Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
(LayoutTestRunner._update_summary_with_result):
(LayoutTestRunner._annotate_results_with_additional_failures):
* Scripts/webkitpy/layout_tests/controllers/layout_test_runner_unittest.py:
(LayoutTestRunnerTests.test_update_summary_with_result):
* Scripts/webkitpy/layout_tests/models/test_expectations.py:
(TestExpectationsModel.get_expectations_or_pass):
(TestExpectationsModel):
(TestExpectationsModel.expectations_to_string):
(TestExpectationsModel.get_expectations_string):
(TestExpectations.filtered_expectations_for_test):
(TestExpectations):
(TestExpectations.matches_an_expected_result):
* Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py:
* Scripts/webkitpy/layout_tests/models/test_run_results.py:
(summarize_results):
* Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py:
(summarized_results):
(SummarizedResultsTest.setUp):
(SummarizedResultsTest.test_summarized_results_include_passes):
(SummarizedResultsTest):
(SummarizedResultsTest.test_summarized_results_world_leaks_disabled):
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
* Scripts/webkitpy/layout_tests/views/buildbot_results_unittest.py:
(BuildBotPrinterTests.test_print_unexpected_results):
(BuildBotPrinterTests.test_print_unexpected_results_with_options):
(BuildBotPrinterTests.test_print_results):
* Scripts/webkitpy/port/test.py:
(unit_test_list):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@235764 268f45cc-cd09-0410-ab3c-d52691b4dbfc
Tools/ChangeLog
Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py
Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner_unittest.py
Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py
Tools/Scripts/webkitpy/layout_tests/models/test_expectations_unittest.py
Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py
Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py
Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py
Tools/Scripts/webkitpy/layout_tests/views/buildbot_results_unittest.py
Tools/Scripts/webkitpy/port/test.py