Simplify and reformat the output of performance tests inside test runners
authorrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 18 Nov 2013 23:40:36 +0000 (23:40 +0000)
committerrniwa@webkit.org <rniwa@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 18 Nov 2013 23:40:36 +0000 (23:40 +0000)
commit90fde7ac1ff4f3c104fc0aa9a6d95306db084fad
tree9d8b48f6a633a07641e4c7717d78fb83e15434c4
parent2fefcf16304e9e8fcb3a550dd63176a9574876cd
Simplify and reformat the output of performance tests inside test runners
https://bugs.webkit.org/show_bug.cgi?id=124496

Reviewed by Antti Koivisto.

PerformanceTests:

As a preparation to support subtests for Dromaeo and DoYouEvenBench, simplify the output performance tests generate.
Also modernize the output to better support "metric" concept we introduced a while ago.

New output on Dromaeo/dom-attr looks like this:
-----------------------------------------------
Running 5 times
getAttribute -> [1105, 1108, 1134, 1137, 1154]
element.property -> [1634, 1655, 1685, 1696, 1723]
setAttribute -> [646.3536463536464, 651, 651, 656.3436563436563, 658]
element.property = value -> [934, 949, 963, 964, 974]
element.expando = value -> [419, 419.5804195804196, 421.57842157842157, 425.57442557442556, 429]
element.expando -> [501, 517, 519.4805194805194, 521.4785214785214, 525]

    1: 117.40644785571585 runs/s
    2: 118.84720469666297 runs/s
    3: 119.80547640905021 runs/s
    4: 120.51886194758805 runs/s
    5: 121.51924380569295 runs/s

:Time -> [117.40644785571585, 118.84720469666297, 119.80547640905021, 120.51886194758805, 121.51924380569295] runs/s
    mean: 119.619446942942 runs/s
    median: 119.80547640905021 runs/s
    stdev: 1.5769040458730506 runs/s
    min: 117.40644785571585 runs/s
    max: 121.51924380569295 runs/s
-----------------------------------------------

* Dromaeo/resources/dromaeorunner.js:
(DRT.progress): Use the new format for subtest reports.
* resources/runner.js:
(.): Declare verboseLogging, which is set to true outside of test runners.
(PerfTestRunner.logInfo): Use verboseLogging instead of directly checking window.testRunner.
(PerfTestRunner.logDetail): Added. Logs informative text with a label such as "mean: 123 s" with 4-space indentation.
(PerfTestRunner.logStatistics): Use logDetail.
(.start): Initialize verboseLogging. Also log "Running 20 times" as an informative log using logDetail.
(.ignoreWarmUpAndLog): Use logDetail for showing the progress. These logs were useless inside test runners anyway
because perftest didn't get to see any output until the test finished running.
(.finish): Call logStatistics with metric name as opposed to a label. Each metric name is now prefixed with ':' to be
distinguishable from subtests, making the new format forward compatible.

Tools:

As a preparation to support subtests for Dromaeo and DoYouEvenBench, simplify the output
performance tests generate. Instead of spitting out noise in PerfTestRunner (runner.js)
and ignoring it in PerfTest._filter_output (perftest.py), simply avoid generating it in
the first place.

Also modernize the output to adopt "metric" concept better and make it forward compatible
with subtests.

With this patch, performance tests written using runner.js only produces empty lines or
lines of the following format inside test runners (DumpRenderTree and WebKitTestRunner):
<subtest name> -> [<value 1>, <value 2>, ...]
:<metric name> -> [<value 1>, <value 2>, ...]

This greatly simplifies the parsing logic inside PerfTest._run_with_driver.

* Scripts/webkitpy/performance_tests/perftest.py:
(PerfTest): Removed a bunch of regular expressions that are no longer used.
(PerfTest._run_with_driver): Just parse the values and description and treat everything
else as errors.

* Scripts/webkitpy/performance_tests/perftest_unittest.py:
(TestPerfTest.test_parse_output): Removed the junk.
(TestPerfTest._assert_failed_on_line): Extracted from test_parse_output_with_failing_line,
which was removed in favor of the tests below.
(TestPerfTest.test_parse_output_with_running_five_times): Added.
(TestPerfTest.test_parse_output_with_detailed_info): Added.
(TestPerfTest.test_parse_output_with_statistics): Added.
(TestPerfTest.test_parse_output_with_description): Removed the junk.
(TestPerfTest.test_parse_output_with_subtests): Ditto.
(TestSingleProcessPerfTest.test_use_only_one_process): Ditto.

* Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py:
(EventTargetWrapperTestData): Ditto.
(SomeParserTestData): Ditto.
(MemoryTestData): Ditto.

LayoutTests:

Rebaseline the expected result now that the output has been simplified.

* fast/harness/perftests/runs-per-second-log-expected.txt:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@159465 268f45cc-cd09-0410-ab3c-d52691b4dbfc
LayoutTests/ChangeLog
LayoutTests/fast/harness/perftests/runs-per-second-log-expected.txt
PerformanceTests/ChangeLog
PerformanceTests/Dromaeo/resources/dromaeorunner.js
PerformanceTests/resources/runner.js
Tools/ChangeLog
Tools/Scripts/webkitpy/performance_tests/perftest.py
Tools/Scripts/webkitpy/performance_tests/perftest_unittest.py
Tools/Scripts/webkitpy/performance_tests/perftestsrunner_integrationtest.py