Unexpected reftest passes are only reported when pixel testing is enabled in results...
authorzandobersek@gmail.com <zandobersek@gmail.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 25 Sep 2012 06:17:58 +0000 (06:17 +0000)
committerzandobersek@gmail.com <zandobersek@gmail.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 25 Sep 2012 06:17:58 +0000 (06:17 +0000)
commit45a985e37827cd3724d81d06f1e9ff77b1647108
treeb282c4af0cc576e2f9c6a620a7c9e99fb8e3cc4f
parentab005855851059474537bfe9a89e0577f109905a
Unexpected reftest passes are only reported when pixel testing is enabled in results.html as well
https://bugs.webkit.org/show_bug.cgi?id=97426

Reviewed by Ojan Vafai.

Tools:

The 'is_reftest' member of the TestResult class has been replaced by the reftest_type
member which is a set containing either '!=', '==', both or none if the test represented
by the object is not a reftest.

When summarizing results, the test dictionary is updated to contain the 'reftest_type' key
if the test is a reftest. The value of this key is a list of all the reftest types of this test.

The test failure interpretation method has been refactored, it now reports only
missing results of various formats and image diff percent in case of image or
reftest mismatch.

Unit tests have been modified accordingly.

* Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py:
(LayoutTestRunner._update_summary_with_result):
* Scripts/webkitpy/layout_tests/controllers/layout_test_runner_unittest.py:
(LayoutTestRunnerTests.test_update_summary_with_result):
* Scripts/webkitpy/layout_tests/controllers/manager.py:
(interpret_test_failures):
(summarize_results):
* Scripts/webkitpy/layout_tests/controllers/manager_unittest.py:
(ResultSummaryTest.test_interpret_test_failures):
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner.run):
(SingleTestRunner._run_reftest):
* Scripts/webkitpy/layout_tests/models/test_results.py:
(TestResult.__init__):
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
(EndToEndTest.test_reftest_with_two_notrefs):

LayoutTests:

Report unexpectedly passing reftests as well. Test cases included.

* fast/harness/resources/results-test.js:
* fast/harness/results-expected.txt:
* fast/harness/results.html:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@129459 268f45cc-cd09-0410-ab3c-d52691b4dbfc
12 files changed:
LayoutTests/ChangeLog
LayoutTests/fast/harness/resources/results-test.js
LayoutTests/fast/harness/results-expected.txt
LayoutTests/fast/harness/results.html
Tools/ChangeLog
Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py
Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner_unittest.py
Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py
Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py
Tools/Scripts/webkitpy/layout_tests/models/test_results.py
Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py