Add "-o/--output" option to startup.py and new_tab.py benchmark scripts to save the...
[WebKit-https.git] / PerformanceTests / ChangeLog
1 2018-12-17  Suresh Koppisetty  <skoppisetty@apple.com>
2
3         Add "-o/--output" option to startup.py and new_tab.py benchmark scripts to save the results in json format.
4         https://bugs.webkit.org/show_bug.cgi?id=192385
5
6         Reviewed by Ryosuke Niwa.
7
8         Sample json output for new tab benchmark script after running for 2 iterations and 2 groups. Values are in milliseconds.
9         {
10                 "NewTabBenchmark": {
11                         "metrics": {
12                                 "Time": {
13                                         "current": [
14                                                 [
15                                                         410.2939453125,
16                                                         307.81494140625
17                                                 ],
18                                                 [
19                                                         340.616943359375,
20                                                         265.94384765625
21                                                 ]
22                                         ]
23                                 }
24                         }
25                 }
26         }
27
28         Sample json output for startup time benchmark script after running for 2 iterations. Values are in milliseconds.
29         {
30                 "StartupBenchmark": {
31                         "metrics": {
32                                 "Time": {
33                                         "current": [
34                                                 [
35                                                         1415.2099609375,
36                                                         1439.552978515625
37                                                 ]
38                                         ]
39                                 }
40                         }
41                 }
42         }
43
44         * LaunchTime/launch_time.py:
45         * LaunchTime/new_tab.py:
46         (NewTabBenchmark.get_test_name):
47         * LaunchTime/startup.py:
48         (StartupBenchmark.get_test_name):
49
50 2018-12-17  Suresh Koppisetty  <skoppisetty@apple.com>
51
52         Import FeedbackServer only if "-f/--feedback-in-browser" option is enabled.
53         https://bugs.webkit.org/show_bug.cgi?id=192378
54
55         Reviewed by Ryosuke Niwa.
56
57         FeedbackServer currently depends on Tornado-5.1, which further adds
58         dependency of "singledispatch", "backports-abc" and "futures" python libraries.
59         Importing FeedbackServer only if "-f/--feedback-in-browser" option is enabled
60         will let us run the benchmark scripts without installing any new python libraries.
61
62         * LaunchTime/launch_time.py:
63         * LaunchTime/new_tab.py:
64
65 2018-12-13  Caio Lima  <ticaiolima@gmail.com>
66
67         [BigInt] Add ValueDiv into DFG
68         https://bugs.webkit.org/show_bug.cgi?id=186178
69
70         Reviewed by Yusuke Suzuki.
71
72         * BigIntBench/big-int-simple-div.js: Added.
73         * BigIntBench/value-div-type-propagation.js: Added.
74
75 2018-12-10  Caio Lima  <ticaiolima@gmail.com>
76
77         [BigInt] Add ValueMul into DFG
78         https://bugs.webkit.org/show_bug.cgi?id=186175
79
80         Reviewed by Yusuke Suzuki.
81
82         * BigIntBench/big-int-simple-mul.js: Added.
83         * BigIntBench/value-mul-type-propagation.js: Added.
84
85 2018-11-07  Caio Lima  <ticaiolima@gmail.com>
86
87         [BigInt] Add support to BigInt into ValueAdd
88         https://bugs.webkit.org/show_bug.cgi?id=186177
89
90         Reviewed by Keith Miller.
91
92         The idea of BigIntBench is to provide a set of microbenchmarks and
93         benchmarks to evaluate how fast BigInt computations are happening on
94         JSC implementation.
95
96         Now, we are adding microbenchmarks in this set,
97         but the plan is to move these tests to "JSTest/microbenchmarks" when
98         BigInt is enabled by default. After that, the focus of Bigint bench is
99         to provide a set of tests that represents real use cases of BigInt in
100         JS programs.
101
102         * BigIntBench/big-int-add-prediction-propagation.js: Added.
103         * BigIntBench/big-int-simple-add.js: Added.
104         * BigIntBench/big-int-simple-sub.js: Added.
105
106 2018-11-07  Tadeu Zagallo  <tzagallo@apple.com>
107
108         REGRESSION(r237547): Test failures on 32-bit JSC since the JIT was disabled
109         https://bugs.webkit.org/show_bug.cgi?id=191184
110
111         Reviewed by Saam Barati.
112
113         Skip test when the JIT is disabled, it takes too long to run on CLoop.
114
115         * JetStream/cdjs/main.js:
116
117 2018-11-05  Myles C. Maxfield  <mmaxfield@apple.com>
118
119         Cache glyph paths and share underline skipping code between all the ports
120         https://bugs.webkit.org/show_bug.cgi?id=191239
121
122         Reviewed by Alex Christensen.
123
124         Measures the performance of drawing a whole lot of underlines
125
126         * Layout/underline.html: Added.
127
128 2018-10-25  Saam Barati  <sbarati@apple.com>
129
130         Check in corresponding C code in JetStream 2
131         https://bugs.webkit.org/show_bug.cgi?id=190733
132
133         Rubber-stamped by Filip Pizlo.
134
135         * JetStream2/simple/float-mm.c: Added.
136         * JetStream2/wasm/HashSet.cpp: Added.
137         * JetStream2/wasm/gcc-loops.cpp: Added.
138         * JetStream2/wasm/quicksort.c: Added.
139         * JetStream2/wasm/TSF: Added.
140         * JetStream2/wasm/TSF/build.sh: Added.
141         * JetStream2/wasm/TSF/config.h: Added.
142         * JetStream2/wasm/TSF/gpc.h: Added.
143         * JetStream2/wasm/TSF/gpc_code_gen_util.c: Added.
144         * JetStream2/wasm/TSF/gpc_instruction.c: Added.
145         * JetStream2/wasm/TSF/gpc_instruction_dispatch.gen: Added.
146         * JetStream2/wasm/TSF/gpc_instruction_size.gen: Added.
147         * JetStream2/wasm/TSF/gpc_instruction_stack_effects.gen: Added.
148         * JetStream2/wasm/TSF/gpc_instruction_static_size.gen: Added.
149         * JetStream2/wasm/TSF/gpc_instruction_to_string.gen: Added.
150         * JetStream2/wasm/TSF/gpc_int_common.h: Added.
151         * JetStream2/wasm/TSF/gpc_intable.c: Added.
152         * JetStream2/wasm/TSF/gpc_internal.h: Added.
153         * JetStream2/wasm/TSF/gpc_interpreter.gen: Added.
154         * JetStream2/wasm/TSF/gpc_program.c: Added.
155         * JetStream2/wasm/TSF/gpc_proto.c: Added.
156         * JetStream2/wasm/TSF/gpc_stack_height.c: Added.
157         * JetStream2/wasm/TSF/gpc_threaded.c: Added.
158         * JetStream2/wasm/TSF/gpc_worklist.h: Added.
159         * JetStream2/wasm/TSF/tsf.h: Added.
160         * JetStream2/wasm/TSF/tsf_adaptive_reader.c: Added.
161         * JetStream2/wasm/TSF/tsf_asprintf.c: Added.
162         * JetStream2/wasm/TSF/tsf_atomics.h: Added.
163         * JetStream2/wasm/TSF/tsf_buf_reader.c: Added.
164         * JetStream2/wasm/TSF/tsf_buf_writer.c: Added.
165         * JetStream2/wasm/TSF/tsf_buffer.c: Added.
166         * JetStream2/wasm/TSF/tsf_build_defines.h: Added.
167         * JetStream2/wasm/TSF/tsf_config.h: Added.
168         * JetStream2/wasm/TSF/tsf_config_stub.h: Added.
169         * JetStream2/wasm/TSF/tsf_copier.c: Added.
170         * JetStream2/wasm/TSF/tsf_define_helpers.c: Added.
171         * JetStream2/wasm/TSF/tsf_define_helpers.h: Added.
172         * JetStream2/wasm/TSF/tsf_destructor.c: Added.
173         * JetStream2/wasm/TSF/tsf_error.c: Added.
174         * JetStream2/wasm/TSF/tsf_format.h: Added.
175         * JetStream2/wasm/TSF/tsf_fsdb.c: Added.
176         * JetStream2/wasm/TSF/tsf_fsdb_protocol.c: Added.
177         * JetStream2/wasm/TSF/tsf_fsdb_protocol.h: Added.
178         * JetStream2/wasm/TSF/tsf_generator.c: Added.
179         * JetStream2/wasm/TSF/tsf_gpc_code_gen.c: Added.
180         * JetStream2/wasm/TSF/tsf_indent.h: Added.
181         * JetStream2/wasm/TSF/tsf_internal.h: Added.
182         * JetStream2/wasm/TSF/tsf_internal_config.h: Added.
183         * JetStream2/wasm/TSF/tsf_internal_config_stub.h: Added.
184         * JetStream2/wasm/TSF/tsf_inttypes.h: Added.
185         * JetStream2/wasm/TSF/tsf_io.c: Added.
186         * JetStream2/wasm/TSF/tsf_io_utils.c: Added.
187         * JetStream2/wasm/TSF/tsf_ir.c: Added.
188         * JetStream2/wasm/TSF/tsf_ir.h: Added.
189         * JetStream2/wasm/TSF/tsf_ir_different.c: Added.
190         * JetStream2/wasm/TSF/tsf_ir_different.h: Added.
191         * JetStream2/wasm/TSF/tsf_ir_speed.c: Added.
192         * JetStream2/wasm/TSF/tsf_limits.c: Added.
193         * JetStream2/wasm/TSF/tsf_named_type.c: Added.
194         * JetStream2/wasm/TSF/tsf_native.c: Added.
195         * JetStream2/wasm/TSF/tsf_parser.c: Added.
196         * JetStream2/wasm/TSF/tsf_primitive.c: Added.
197         * JetStream2/wasm/TSF/tsf_ra_type_man.c: Added.
198         * JetStream2/wasm/TSF/tsf_reflect.c: Added.
199         * JetStream2/wasm/TSF/tsf_region.h: Added.
200         * JetStream2/wasm/TSF/tsf_serial_in_man.c: Added.
201         * JetStream2/wasm/TSF/tsf_serial_out_man.c: Added.
202         * JetStream2/wasm/TSF/tsf_serial_protocol.h: Added.
203         * JetStream2/wasm/TSF/tsf_sha1.c: Added.
204         * JetStream2/wasm/TSF/tsf_sha1.h: Added.
205         * JetStream2/wasm/TSF/tsf_sha1_writer.c: Added.
206         * JetStream2/wasm/TSF/tsf_sort.c: Added.
207         * JetStream2/wasm/TSF/tsf_st.c: Added.
208         * JetStream2/wasm/TSF/tsf_st.h: Added.
209         * JetStream2/wasm/TSF/tsf_st_typetable.c: Added.
210         * JetStream2/wasm/TSF/tsf_stream_file_input.c: Added.
211         * JetStream2/wasm/TSF/tsf_stream_file_output.c: Added.
212         * JetStream2/wasm/TSF/tsf_type.c: Added.
213         * JetStream2/wasm/TSF/tsf_type_in_map.c: Added.
214         * JetStream2/wasm/TSF/tsf_type_out_map.c: Added.
215         * JetStream2/wasm/TSF/tsf_type_table.c: Added.
216         * JetStream2/wasm/TSF/tsf_types.h: Added.
217         * JetStream2/wasm/TSF/tsf_util.h: Added.
218         * JetStream2/wasm/TSF/tsf_version.c: Added.
219         * JetStream2/wasm/TSF/tsf_zip_abstract.c: Added.
220         * JetStream2/wasm/TSF/tsf_zip_abstract.h: Added.
221         * JetStream2/wasm/TSF/tsf_zip_attr.c: Added.
222         * JetStream2/wasm/TSF/tsf_zip_reader.c: Added.
223         * JetStream2/wasm/TSF/tsf_zip_writer.c: Added.
224
225 2018-10-18  Saam Barati  <sbarati@apple.com>
226
227         Make JetStream 2
228         https://bugs.webkit.org/show_bug.cgi?id=187829
229
230         Rubber-stamped by Mark Lam.
231
232         This patch checks in the new JetStream 2 benchmark. JetStream 2 is
233         a new JavaScript and Web Assembly benchmark. JetStream 2's goal
234         is to measure the startup, worst case, and peak throughput performance
235         of the JavaScript engine. JetStream 2 incorporates these previous
236         benchmarks:
237         - JetStream
238         - ARES-6
239         - Kraken
240         - Web Tooling Benchmark
241         - WasmBench
242         - RexBench
243         
244         JetStream 2 also adds some new benchmarks:
245         - Two tests emphasizing web worker performance.
246         - One test emphasizing Promise, async iteration, and DataView performance.
247         - Two new code load tests.
248         - WSL: a test measuring all kinds of things, especially emphasizing exception performance.
249
250         * JetStream2: Added.
251
252 2018-10-02  Chris Dumez  <cdumez@apple.com>
253
254         Regression(r236613): Parser/html-parser.html performance test is failing
255         https://bugs.webkit.org/show_bug.cgi?id=190201
256
257         Unreviewed, apply the same fix as in r236699 to make sure the iframe is treated as
258         same origin, so that calls to document.open() / document.write() succeed.
259
260         * Parser/html-parser.html:
261
262 2018-10-01  Chris Dumez  <cdumez@apple.com>
263
264         Regression(r236613): Parser/HTML5-8266-ParseOnly.html performance tests no longer runs
265         https://bugs.webkit.org/show_bug.cgi?id=190174
266
267         Reviewed by Ryosuke Niwa.
268
269         Update iframe's sandbox to use 'allow-same-origin' so that the iframe is treated as same-origin and so that
270         the top frame can call document.open() / document.write() on the subframe. This test was also failing in
271         Firefox without this change.
272
273         * Parser/HTML5-8266-ParseOnly.html:
274
275 2018-08-27  Aditya Keerthi  <akeerthi@apple.com>
276
277         Consolidate ENABLE_INPUT_TYPE_COLOR and ENABLE_INPUT_TYPE_COLOR_POPOVER
278         https://bugs.webkit.org/show_bug.cgi?id=188931
279
280         Reviewed by Wenson Hsieh.
281
282         * StitchMarker/wtf/FeatureDefines.h: Removed ENABLE_INPUT_TYPE_COLOR_POPOVER.
283
284 2018-08-23  Don Olmstead  <don.olmstead@sony.com>
285
286         [CMake] Have checks are not getting set inside CMake properly
287         https://bugs.webkit.org/show_bug.cgi?id=188901
288
289         Reviewed by Michael Catanzaro.
290
291         Add notification that MallocBench is disabled.
292
293         * CMakeLists.txt:
294
295 2018-08-23  Don Olmstead  <don.olmstead@sony.com>
296
297         [CMake] Add HAVE_MALLOC_TRIM definition
298         https://bugs.webkit.org/show_bug.cgi?id=188897
299
300         Reviewed by Konstantin Tokarev.
301
302         MallocBench should only be built on Apple platforms and platforms that
303         have malloc_trim.
304
305         * CMakeLists.txt:
306
307 2018-08-13  Jon Lee  <jonlee@apple.com>
308
309         Update MotionMark version number
310         https://bugs.webkit.org/show_bug.cgi?id=188535
311         <rdar://problem/43254078>
312
313         Reviewed by Said Abou-Hallawa.
314
315         * MotionMark/resources/strings.js: To 1.1.
316
317 2018-08-13  Jon Lee  <jonlee@apple.com>
318
319         [MotionMark] Update Multiply test
320         https://bugs.webkit.org/show_bug.cgi?id=188532
321         <rdar://problem/43252151>
322
323         Reviewed by Said Abou-Hallawa.
324
325         Update Multiply test to have a larger upper limit of number of particles. Cycle through three
326         different ways of hiding an element. Add new Multiple suite that isolates those components for
327         debugging purposes.
328
329         * MotionMark/resources/debug-runner/tests.js:
330         * MotionMark/tests/dom/multiply.html: Added.
331         * MotionMark/tests/dom/resources/multiply.js: Added.
332         * MotionMark/tests/master/resources/multiply.js:
333
334 2018-08-13  Jon Lee  <jonlee@apple.com>
335
336         [MotionMark] Update Leaves test
337         https://bugs.webkit.org/show_bug.cgi?id=188530
338         <rdar://problem/43251862>
339
340         Reviewed by Said Abou-Hallawa.
341
342         Update Leaves test to include opacity and scale. Add new Leaves suite that isolate those components
343         for debugging purposes.
344
345         * MotionMark/resources/debug-runner/tests.js:
346         * MotionMark/tests/dom/leaves.html:
347         * MotionMark/tests/dom/resources/leaves.js:
348         * MotionMark/tests/master/resources/leaves.js:
349
350 2018-08-10  Ben Richards  <benton_richards@apple.com>
351
352         Add ability to ignore process prewarming for launch time benchmark
353         https://bugs.webkit.org/show_bug.cgi?id=188462
354
355         Reviewed by Ryosuke Niwa.
356
357         Added a flag to the new tab benchmark that will open a blank tab before the tab to be measured in order to ignore process prewarming.
358
359         * LaunchTime/launch_time.py:
360         (DefaultLaunchTimeHandler):
361         (DefaultLaunchTimeHandler.on_receive_stop_time): Deleted.
362         (DefaultLaunchTimeHandler.on_receive_stop_signal):
363         (DefaultLaunchTimeHandler.do_HEAD):
364         (DefaultLaunchTimeHandler.do_GET):
365         (DefaultLaunchTimeHandler.do_POST):
366         (LaunchTimeBenchmark):
367         (LaunchTimeBenchmark._standard_deviation): Fixed divide by zero bug when '-n' is set to 1
368         (LaunchTimeBenchmark.open_tab): Added option to open a blank tab
369         (LaunchTimeBenchmark.run):
370         * LaunchTime/new_tab.py:
371         (NewTabBenchmark.initialize):
372         (NewTabBenchmark.run_iteration):
373         (NewTabBenchmark.will_parse_arguments):
374         (NewTabBenchmark.did_parse_arguments):
375         (NewTabBenchmark.ResponseHandler.Handler.get_test_page):
376         (NewTabBenchmark.ResponseHandler.Handler.on_receive_stop_time): Deleted.
377         (NewTabBenchmark.ResponseHandler.Handler.on_receive_stop_signal):
378         (NewTabBenchmark):
379         * LaunchTime/startup.py:
380         (StartupBenchmark.ResponseHandler.Handler.get_test_page):
381         (StartupBenchmark.ResponseHandler.Handler.on_receive_stop_time): Deleted.
382         (StartupBenchmark.ResponseHandler.Handler.on_receive_stop_signal):
383         (StartupBenchmark):
384
385 2018-08-10  Truitt Savell  <tsavell@apple.com>
386
387         Unreviewed, rolling out r234750.
388
389         Caused 185 perf test failures.
390
391         Reverted changeset:
392
393         "Add ability to ignore process prewarming for launch time
394         benchmark"
395         https://bugs.webkit.org/show_bug.cgi?id=188462
396         https://trac.webkit.org/changeset/234750
397
398 2018-08-09  Ben Richards  <benton_richards@apple.com>
399
400         Add ability to ignore process prewarming for launch time benchmark
401         https://bugs.webkit.org/show_bug.cgi?id=188462
402
403         Reviewed by Ryosuke Niwa.
404
405         Added a flag to the new tab benchmark that will open a blank tab before the tab to be measured in order to ignore process prewarming.
406
407         * LaunchTime/launch_time.py:
408         (DefaultLaunchTimeHandler):
409         (DefaultLaunchTimeHandler.on_receive_stop_time): Deleted.
410         (DefaultLaunchTimeHandler.on_receive_stop_signal):
411         (DefaultLaunchTimeHandler.do_HEAD):
412         (DefaultLaunchTimeHandler.do_GET):
413         (DefaultLaunchTimeHandler.do_POST):
414         (LaunchTimeBenchmark):
415         (LaunchTimeBenchmark._standard_deviation): Fixed divide by zero bug when '-n' is set to 1
416         (LaunchTimeBenchmark.open_tab): Added option to open a blank tab
417         (LaunchTimeBenchmark.run):
418         * LaunchTime/new_tab.py:
419         (NewTabBenchmark.initialize):
420         (NewTabBenchmark.run_iteration):
421         (NewTabBenchmark.will_parse_arguments):
422         (NewTabBenchmark.did_parse_arguments):
423         (NewTabBenchmark.ResponseHandler.Handler.get_test_page):
424         (NewTabBenchmark.ResponseHandler.Handler.on_receive_stop_time): Deleted.
425         (NewTabBenchmark.ResponseHandler.Handler.on_receive_stop_signal):
426         (NewTabBenchmark):
427         * LaunchTime/startup.py:
428         (StartupBenchmark.ResponseHandler.Handler.get_test_page):
429         (StartupBenchmark.ResponseHandler.Handler.on_receive_stop_time): Deleted.
430         (StartupBenchmark.ResponseHandler.Handler.on_receive_stop_signal):
431         (StartupBenchmark):
432
433 2018-07-26  Andy VanWagoner  <andy@vanwagoner.family>
434
435         [INTL] Remove INTL sub-feature compile flags
436         https://bugs.webkit.org/show_bug.cgi?id=188081
437
438         Reviewed by Michael Catanzaro.
439
440         Removed old and unused i18n flag.
441
442         * StitchMarker/wtf/FeatureDefines.h:
443
444 2018-07-20  David Fenton  <david_fenton@apple.com>
445
446         REGRESSION (234006) ERROR: Blocked access to external URL http://code.jquery.com/jquery-1.9.1.min.js
447         https://bugs.webkit.org/show_bug.cgi?id=187858
448
449         Unreviewed test gardening, skipped test.
450
451         * Skipped: LaunchTime
452
453 2018-07-19  Ben Richards  <benton_richards@apple.com>
454
455         Add benchmark for WebKit process launch times
456         https://bugs.webkit.org/show_bug.cgi?id=186414
457
458         Reviewed by Ryosuke Niwa.
459
460         Added two benchmarks, one for measuring browser new tab launch time and one for browser startup time.
461
462         * LaunchTime/.gitignore: Added.
463         * LaunchTime/feedback_client.html: Added.
464         Displays benchmark progress in browser
465         * LaunchTime/feedback_server.py: Added.
466         (FeedbackServer): Sends data to feedback_client via websocket
467         (FeedbackServer.__init__):
468         (FeedbackServer._create_app):
469         (FeedbackServer._start_server):
470         (FeedbackServer._send_all_messages):
471         (FeedbackServer.start):
472         (FeedbackServer.stop):
473         (FeedbackServer.send_message): Send a message to the feedback_client
474         (FeedbackServer.wait_until_client_has_loaded): Wait until the feedback_client has opened a websocket connection to the feedback_server
475         (FeedbackServer.MainHandler): Handler factory to create handler that serves feedback_client.html
476         (FeedbackServer.MainHandler.Handler):
477         (FeedbackServer.MainHandler.Handler.get):
478         (FeedbackServer.WSHandler): Handler factory to create handler that sends data to feedback client
479         (FeedbackServer.WSHandler.Handler):
480         (FeedbackServer.WSHandler.Handler.open): On websocket connection opened
481         (FeedbackServer.WSHandler.Handler.on_close): On websocket connection closed
482         * LaunchTime/launch_time.py: Added.
483         (DefaultLaunchTimeHandler): Abstract HTTP request handler for launch time benchmarks
484         (DefaultLaunchTimeHandler.get_test_page): Default test page to be overridden by benchmarks
485         (DefaultLaunchTimeHandler.get_blank_page):
486         (DefaultLaunchTimeHandler.on_receive_stop_time):
487         (DefaultLaunchTimeHandler.do_HEAD):
488         (DefaultLaunchTimeHandler.do_GET):
489         (DefaultLaunchTimeHandler.do_POST):
490         (DefaultLaunchTimeHandler.log_message): Suppresses HTTP logs from SimpleHTTPRequestHandler
491         (LaunchTimeBenchmark): Abstract class which launch time benchmarks inherit from and override methods desired to customize
492         (LaunchTimeBenchmark.__init__):
493         (LaunchTimeBenchmark._parse_browser_bundle_path): Parser for bundle path option
494         (LaunchTimeBenchmark._parse_args):
495         (LaunchTimeBenchmark._run_server): Target function for main server thread
496         (LaunchTimeBenchmark._setup_servers):
497         (LaunchTimeBenchmark._clean_up):
498         (LaunchTimeBenchmark._exit_due_to_exception):
499         (LaunchTimeBenchmark._geometric_mean):
500         (LaunchTimeBenchmark._standard_deviation):
501         (LaunchTimeBenchmark._compute_results): Returns mean and std dev of list of results and pretty prints if should_print=True is specified
502         (LaunchTimeBenchmark._wait_times): Mimic numpy.linspace
503         (LaunchTimeBenchmark.open_tab): Open a browser tab with the html given by self.response_handler.get_test_page
504         (LaunchTimeBenchmark.launch_browser): Open a broser to either the feedback client (if option is set) or a blank page
505         (LaunchTimeBenchmark.quit_browser):
506         (LaunchTimeBenchmark.quit_browser.quit_app):
507         (LaunchTimeBenchmark.quit_browser.is_app_closed):
508         (LaunchTimeBenchmark.close_tab):
509         (LaunchTimeBenchmark.wait):
510         (LaunchTimeBenchmark.log): Print to console and send to feedback client if --feedback-in-browser flag is used
511         (LaunchTimeBenchmark.log_verbose): Only logs if --verbose flag is used
512         (LaunchTimeBenchmark.run):
513         (LaunchTimeBenchmark.group_init): Initialization done before each round of iterations
514         (LaunchTimeBenchmark.run_iteration):
515         (LaunchTimeBenchmark.initialize): Convenience method to be overriden by subclasses which is called at the end of __init__
516         (LaunchTimeBenchmark.will_parse_arguments): Called before argparse.parse_args to let subclasses add new command line arguments
517         (LaunchTimeBenchmark.did_parse_arguments): Called after argparse.parse_args to let subclass initialize based on command line arguments
518         * LaunchTime/new_tab.py: Added
519         (NewTabBenchmark):
520         (NewTabBenchmark._parse_wait_time): Parser for wait time option
521         (NewTabBenchmark.initialize):
522         (NewTabBenchmark.run_iteration):
523         (NewTabBenchmark.group_init):
524         (NewTabBenchmark.will_parse_arguments):
525         (NewTabBenchmark.did_parse_arguments):
526         (NewTabBenchmark.ResponseHandler):
527         (NewTabBenchmark.ResponseHandler.Handler):
528         (NewTabBenchmark.ResponseHandler.Handler.get_test_page):
529         * LaunchTime/startup.py: Added
530         (StartupBenchmark): This benchmark measures browser startup time and initial page load time
531         (StartupBenchmark.initialize):
532         (StartupBenchmark.run_iteration):
533         (StartupBenchmark.ResponseHandler):
534         (StartupBenchmark.ResponseHandler.Handler):
535         (StartupBenchmark.ResponseHandler.Handler.get_test_page):
536         * LaunchTime/thirdparty/__init__.py: Added.
537         (AutoinstallImportHook): Auto installs tornado package for feedback server
538         (AutoinstallImportHook.__init__):
539         (AutoinstallImportHook._ensure_autoinstalled_dir_is_in_sys_path):
540         (AutoinstallImportHook.find_module):
541         (AutoinstallImportHook._install_tornado):
542         (AutoinstallImportHook.greater_than_equal_to_version):
543         (AutoinstallImportHook._install):
544         (AutoinstallImportHook.get_latest_pypi_url):
545         (AutoinstallImportHook.install_binary):
546         (autoinstall_everything):
547         (get_os_info):
548
549 2018-06-25  Jon Lee  <jonlee@apple.com>
550
551         [MotionMark] Add support for version numbers
552         https://bugs.webkit.org/show_bug.cgi?id=186479
553
554         Reviewed by Said Abou-Hallawa.
555
556         Add support for displaying the version number as well as including it in the JSON results.
557
558         When loading the front page, script replaces any element with classname version with the
559         version number of the benchmark, which is stored in Strings.version.
560
561         The JSON structure for the results includes a new version property:
562             {
563               "version": "1.0",
564               "options": { ... },
565               "data": [ ... ]
566             }
567
568         When dragging a results file, the version listed will come from the JSON file. Older
569         results will not have had the version property, in which case it will default to "1.0".
570
571         * MotionMark/index.html: Update title to some other default. Script will update it.
572         Include the version number in the logo title.
573         * MotionMark/developer.html: Ditto.
574         * MotionMark/about.html: Ditto.
575
576         * MotionMark/resources/runner/motionmark.js:
577         (ResultsDashboard): Update constructor to include version. This is used when serializing
578         results out to JSON, and displaying the results panel in developer mode.
579         (ResultsDashboard._processData): When running the benchmark, include benchmark version string
580         in the results object.
581         (ResultsDashboard.version):
582         (window.benchmarkRunnerClient.willStartFirstIteration): When running the benchmark, pass the
583         benchmark version string to the dashboard, which holds the results.
584         (window.sectionsManager.setSectionVersion): Helper function to update the element in the
585         section with the class name version.
586         (window.benchmarkController.initialize): Populate all DOM elements with class name "version"
587         with the version string. Update the page title.
588         (window.benchmarkController.showResults): When showing results, update the version string
589         based on what is included in the JSON results, which would be the same as the benchmark version.
590         * MotionMark/resources/runner/motionmark.css: Include missing copyright. Wrap the SVG logo
591         in a div and include the version string.
592         * MotionMark/resources/strings.js: Add strings for the page title template, and the version.
593
594         * MotionMark/resources/debug-runner/motionmark.css:
595         * MotionMark/resources/debug-runner/motionmark.js:
596         (window.benchmarkRunnerClient.willStartFirstIteration): When running the benchmark, pass the
597         benchmark version string to the dashboard, which holds the results.
598         (window.benchmarkController.initialize): Populate all DOM elements with class name "version"
599         with the version string. Update the page title. When dragging in JSON results, look for
600         version to pass to the dashboard. If it doesn't exist, default to "1.0".
601         (window.benchmarkController.showResults): When showing results, update the version string
602         based on what is included in the JSON results, instead of the current benchmark version.
603         * MotionMark/resources/debug-runner/tests.js: Update page title template.
604
605 2018-06-08  Jon Lee  <jonlee@apple.com>
606
607         [MotionMark] Rename Suits test files
608         https://bugs.webkit.org/show_bug.cgi?id=186447
609
610         Reviewed by Said Abou-Hallawa.
611
612         * MotionMark/resources/runner/tests.js:
613         * MotionMark/tests/master/resources/suits.js: Renamed from PerformanceTests/MotionMark/tests/master/resources/svg-particles.js.
614         * MotionMark/tests/master/suits.html: Renamed from PerformanceTests/MotionMark/tests/master/svg-particles.html.
615         * MotionMark/tests/svg/suits.html:
616
617 2018-06-08  Jon Lee  <jonlee@apple.com>
618
619         Add sub-tests based on Suits
620         https://bugs.webkit.org/show_bug.cgi?id=186260
621
622         Reviewed by Said Abou-Hallawa.
623
624         Add a new developer Suits suite with sub-tests that isolate parts of each particle.
625         The sub-tests are:
626             - Particles using only clip paths
627             - Particles using only shapes
628             - Particles that have no gradients
629             - Particles that have no rotation
630             - Particles that do not move around at all (but all of the physics calculations are
631             still performed)
632
633         * MotionMark/resources/debug-runner/tests.js:
634         * MotionMark/tests/master/resources/svg-particles.js: Rename the stage, particle, and
635         benchmark to "Suits" from "SVG". Change SuitsParticle so that we guarantee 50% clip and
636         shape paths rather than relying on random chance. Wrap a check around creation of the
637         gradient element, and use a simple fill color when we aren't using a gradient.
638         * MotionMark/tests/svg/suits.html: Added.
639         * MotionMark/tests/svg/suits.js: Added.
640         Look for the query string and set the particle based on what is selected.
641
642 2018-06-06  Jon Lee  <jonlee@apple.com>
643
644         Allow the ramp controller to run tests that take less time than the initial ramp-up phase
645         https://bugs.webkit.org/show_bug.cgi?id=186257
646
647         Reviewed by Said Abou-Hallawa.
648
649         * MotionMark/tests/resources/main.js: Extend the desired end timestamp when ramp-up occurs.
650         Previously the controller assumed that the duration of the test was always longer than the
651         time it took to figure out the order of magnitude of particles to render. When that range
652         is determined, the end timestamp is then extended. If the duration is too short, the controller
653         would throw an exception. The fix is to continue extending the timestamp every time the order of
654         magnitude tiers up.
655
656         In general testing with really short durations isn't practical, but sometimes it is
657         desired for debugging.
658
659         To avoid confusion of the private variables, I renamed Benchmark._startTimestamp to
660         Benchmark._benchmarkStartTimestamp, to disambiguate from Controller._startTimestamp.
661         Benchmark._startTimestamp is set once and never changed, after the initial 100ms warm-up.
662         Controller._startTimestamp is initialized to 0 (which means "not initialized"), and then after
663         the warm up is complete, set in Controller.start() to the same wall clock timestamp, and never
664         changed afterwards.
665
666 2018-06-06  Jon Lee  <jonlee@apple.com>
667
668         Remove unneeded data processing in MotionMark
669         https://bugs.webkit.org/show_bug.cgi?id=186256
670
671         Reviewed by Said Abou-Hallawa.
672
673         Remove calculation of average frame rate based on multiple samples at a given complexity.
674         It is not used in the calculation of the score, and is just extra work to do at the end
675         of a test.
676
677         * MotionMark/developer.html: Remove options to show average data and regression based on
678         that data.
679         * MotionMark/resources/debug-runner/graph.js: Remove data. Update the time graph that
680         shows the relationship between the break point in the regression against the complexity
681         graph.
682         * MotionMark/resources/debug-runner/motionmark.css: Make the confidence intervals easier
683         to see.
684         * MotionMark/resources/runner/motionmark.js: Simplify score calculation now that we are
685         only calculating regressions on the raw data. We were calculating it based on the averages,
686         but that was just for analysis, and not used in the actual score. Get rid of the
687         |complexitySamples| local variable.
688         * MotionMark/resources/strings.js: Remove unneeded string constant.
689         * MotionMark/tests/resources/main.js:
690         (processSamples): Refactor so that more of it can be shared between Controller and
691         RampController. It also avoids processing the raw samples (controllerSamples) twice for
692         the ramp controller.
693         (_processComplexitySamples): Remove logic to calculate averages based on complexity.
694         (_processMarks): Split this out from the old processSamples.
695         (_processControllerSamples): Split this out from the old processSamples.
696
697 2018-06-06  Jon Lee  <jonlee@apple.com>
698
699         Strip out more characters when creating permalinks
700         https://bugs.webkit.org/show_bug.cgi?id=186259
701
702         Reviewed by Said Abou-Hallawa.
703
704         Tests and suites may have names that make permalinks to the tests less readable when
705         shared. Expand the set of characters that are stripped out of the names.
706
707         * MotionMark/resources/debug-runner/motionmark.js: Refactor to call
708         stripUnwantedCharactersForURL. For existing links, decode the suite and test names.
709         * MotionMark/resources/extensions.js:
710         (Utilities.stripUnwantedCharactersForURL): Rename from stripNonASCIICharacters.
711         Remove any non-alphanumeric character.
712
713 2018-06-04  Jon Lee  <jonlee@apple.com>
714
715         Remove unnecessary MotionMark controllers
716         https://bugs.webkit.org/show_bug.cgi?id=186255
717
718         Reviewed by Said Abou-Hallawa.
719
720         Get rid of the ramp30 and the fixed-with-a-step controllers, which aren't used for testing
721         or debugging.
722
723         * MotionMark/developer.html: Move the "ramp" option as first, since that is the default
724         controller for the benchmark.
725         * MotionMark/resources/debug-runner/motionmark.js:
726         * MotionMark/resources/runner/motionmark.js: Added missing copyright notice.
727         * MotionMark/tests/resources/main.js:
728
729 2018-05-25  Myles C. Maxfield  <mmaxfield@apple.com>
730
731         Improve the performance of Font::canRenderCombiningCharacterSequence()
732         https://bugs.webkit.org/show_bug.cgi?id=185933
733
734         Reviewed by Ryosuke Niwa.
735
736         * Layout/ComplexLongUnique.html: Added.
737
738 2018-05-25  Saam Barati  <sbarati@apple.com>
739
740         Have a memory test where we can validate JSCs mini memory mode
741         https://bugs.webkit.org/show_bug.cgi?id=185932
742
743         Reviewed by Mark Lam.
744
745         We add a directory here with the contents of the testmem benchmark.
746         To run it, use `Tools/Scripts/run-testmem`. To add new tests in the future,
747         you just need to add JS files to this directory.
748
749         * testmem: Added.
750         * testmem/air.js: Added.
751         * testmem/base64.js: Added.
752         * testmem/basic.js: Added.
753         * testmem/box2d.js: Added.
754         * testmem/crypto-md5.js: Added.
755         * testmem/date-format-tofte.js: Added.
756         * testmem/earley-boyer.js: Added.
757         * testmem/hash-map.js: Added.
758         * testmem/regex-dna.js: Added.
759         * testmem/splay.js: Added.
760         * testmem/tagcloud.js: Added.
761
762 2018-05-22  Ryan Haddad  <ryanhaddad@apple.com>
763
764         Unreviewed, rolling out r232052.
765
766         Breaks internal builds.
767
768         Reverted changeset:
769
770         "Use more C++17"
771         https://bugs.webkit.org/show_bug.cgi?id=185176
772         https://trac.webkit.org/changeset/232052
773
774 2018-05-21  Yusuke Suzuki  <utatane.tea@gmail.com>
775
776         Use more C++17
777         https://bugs.webkit.org/show_bug.cgi?id=185176
778
779         Reviewed by JF Bastien.
780
781         * MallocBench/MallocBench.xcodeproj/project.pbxproj:
782         * MediaTime/Configurations/Base.xcconfig:
783         * MediaTime/MediaTime.xcodeproj/project.pbxproj:
784
785 2018-04-30  Myles C. Maxfield  <mmaxfield@apple.com>
786
787         Improve the performance of FontCascadeDescription's effectiveFamilies
788         https://bugs.webkit.org/show_bug.cgi?id=184720
789
790         Reviewed by Simon Fraser.
791
792         This performance test calls SystemFontDatabase::systemFontCascadeList() around 2,000,000 times (before
793         this patch is applied), which is roughly equivalent to the page we found the performance problem on.
794         The calling pattern is roughly equivalent in this test.
795
796         * Layout/system-ui.html: Added.
797
798 2018-03-08  Antti Koivisto  <antti@apple.com>
799
800         Update StyleBench version number in page title to 0.3
801
802         * StyleBench/index.html:
803
804         There has been a bunch of changes, most notably the addition of attribute selector test step.
805
806 2018-02-16  Dean Jackson  <dino@apple.com>
807
808         Use OPENGL macros to be more clear about which OpenGL/ES WebGL uses on Cocoa
809         https://bugs.webkit.org/show_bug.cgi?id=182894
810
811         Reviewed by Tim Horton.
812
813         Rename OPENGL_ES_2 to OPENGL_ES.
814
815         * StitchMarker/wtf/Platform.h:
816
817 2018-02-09  Ryosuke Niwa  <rniwa@webkit.org>
818
819         Make run-perf-tests work with Speedometer 2.0 and re-enable the test
820         https://bugs.webkit.org/show_bug.cgi?id=182088
821
822         Reviewed by Antti Koivisto.
823
824         This patch updates the test harness to report new metrics of Speedometer 2.0.
825
826         Replaced the total time by the geometric mean of the time spent in each suite, and added scores
827         as a metric in addition to time. Also, updated the PerfTestRunner harness and the results template
828         to support "Score" metric with "pt" as its unit.
829
830         * Skipped: Unskipped the test.
831         * Speedometer/resources/benchmark-report.js:
832         (window.benchmarkClient.willStartFirstIteration..createTest): Added the unit as an optional argument
833         to report the score in addition to time. Also report the geomean of time instead of the total time.
834         (window.benchmarkClient.willStartFirstIteration): Report the geomean of time.
835         (window.benchmarkClient.didRunSuites): Record the entire measuredValues instead of just tests.
836         (window.benchmarkClient.didFinishLastIteration): Report the scores.
837
838 2018-02-09  Ross Kirsling  <ross.kirsling@sony.com>
839
840         Use REQUEST_ANIMATION_FRAME_DISPLAY_MONITOR on any non-Windows port.
841         https://bugs.webkit.org/show_bug.cgi?id=182623
842
843         Reviewed by Alex Christensen.
844
845         * StitchMarker/wtf/Platform.h:
846         Simplify #if.
847
848 2018-02-01  Antti Koivisto  <antti@apple.com>
849
850         StyleBench: Attribute selectors and other improvements
851         https://bugs.webkit.org/show_bug.cgi?id=182387
852
853         Reviewed by Joseph Pecoraro.
854
855         - Add some attributes to elements in all tests
856         - Add some attribute selectors to stylesheets in all tests
857         - Also add some * selectors to all stylesheets.
858         - Add attribute mutation step to all suites
859         - Make test steps do more mutations (25->100) and reduce the number of steps to keep testing time in check.
860           Too fast steps were running into timer resolution limits.
861
862         * StyleBench/resources/style-bench.js:
863         (defaultConfiguration):
864         (prototype.randomAttributeName):
865         (prototype.randomAttributeValue):
866         (prototype.randomAttributeSelector):
867         (prototype.makeCompoundSelector):
868         (prototype.makeElement):
869         (prototype.addClasses):
870         (prototype.removeClasses):
871         (prototype.mutateAttributes):
872         (prototype.async.runForever):
873         * StyleBench/resources/tests.js:
874         (makeSteps):
875         (makeSuite):
876
877 2018-02-01  Geoffrey Garen  <ggaren@apple.com>
878
879         Make MallocBench easier for non-WebKit engineers to run
880         https://bugs.webkit.org/show_bug.cgi?id=182415
881
882         Reviewed by Saam Barati.
883
884         * MallocBench/MallocBench.xcodeproj/project.pbxproj: Use c++14 so we
885         can make_unique.
886
887         Specify that we support all Darwin platforms so you can test them.
888
889         * MallocBench/run-malloc-benchmarks: Specify the path to MallocBench
890         and libmbmalloc explicitly, rather than computing them implicitly
891         using webkitdirs. Non-WebKit folks don't have the directory structure
892         required by webkitdirs.
893
894         Remove Linux-specific and cmake-specific behaviors because we only
895         needed them in the world of implicit path computation.
896
897 2018-01-31  Ryosuke Niwa  <rniwa@webkit.org>
898
899         Make run-perf-tests work with StyleBench and re-enable it
900         https://bugs.webkit.org/show_bug.cgi?id=182322
901
902         Reviewed by Antti Koivisto.
903
904         This patch updates the test harness to report new metrics of StyleBench.
905
906         Replaced the total time by the geometric mean of the time spent in each suite, and added scores
907         as a metric in addition to time. Also, updated the PerfTestRunner harness and the results template
908         to support "Score" metric with "pt" as its unit.
909
910         * Skipped: Unskipped the test.
911         * StyleBench/resources/benchmark-report.js:
912         (window.benchmarkClient.willStartFirstIteration..createTest): Added the unit as an optional argument
913         to report the score in addition to time. Also report the geomean of time instead of the total time.
914         (window.benchmarkClient.willStartFirstIteration): Report the geomean of time.
915         (window.benchmarkClient.didRunSuites): Record the entire measuredValues instead of just tests.
916         (window.benchmarkClient.didFinishLastIteration): Report the scores.
917         * resources/results-template.html: Added "Score" as a metric and "pt" as its unit.
918         * resources/runner.js:
919         (PerfTestRunner..finish): Ditto.
920
921 2018-01-30  Ryosuke Niwa  <rniwa@webkit.org>
922
923         StyleBench: Generate more mutation steps
924         https://bugs.webkit.org/show_bug.cgi?id=182291
925
926         Rubber-stamped by Antti Koivisto
927
928         Use zero-based index instead of one-based index.
929
930         * StyleBench/resources/tests.js:
931         (makeSteps):
932
933 2018-01-30  Antti Koivisto  <antti@apple.com>
934
935         StyleBench: Generate more mutation steps
936         https://bugs.webkit.org/show_bug.cgi?id=182291
937
938         Reviewed by Ryosuke Niwa.
939
940         The test should be doing multiple rounds of mutations in a single document to test what is supposed to test (like it did before r227756).
941         This also makes test results more stable and less dependent on possible first-time costs.
942
943         * StyleBench/resources/benchmark-runner.js:
944
945         Adjust correctionFactor to match the new steps count. With 1.5 Firefox is ~72rpm.
946
947         * StyleBench/resources/tests.js:
948         (makeSteps):
949
950         Generate 10 rounds of class/element mutation steps.
951         Give each generated step unique name so reporting works.
952         Make individual steps do fewer mutations.
953
954 2018-01-29  Antti Koivisto  <antti@apple.com>
955
956         StyleBench: Remove : from a test name
957         https://bugs.webkit.org/show_bug.cgi?id=182258
958
959         Reviewed by Ryosuke Niwa.
960
961         It breaks reporting.
962
963         * StyleBench/resources/style-bench.js:
964         (nthPseudoClassConfiguration):
965
966 2018-01-29  Ryosuke Niwa  <rniwa@webkit.org>
967
968         Temporarily skip StyleBench until webkit.org/b/182088 is fixed.
969
970         * Skipped:
971
972 2018-01-29  Ryosuke Niwa  <rniwa@webkit.org>
973
974         Make StyleBench compatible with run-benchmark and run-perf-tests
975         https://bugs.webkit.org/show_bug.cgi?id=182262
976
977         Reviewed by Antti Koivisto.
978
979         Copied resource files referenced from Speedometer directory since run-benchmark needs to be able
980         to checkout each benchmark separately.
981
982         Removed the code to create tests of the same name five times in makeSteps since this
983         won't be compatible with either run-benchmark or run-perf-tests.
984
985         * StyleBench/index.html: Removed the code to show warnings for local files since run-benchmark
986         doesn't use HTTP server in WebDriver mode.
987         * StyleBench/resources/benchmark-report.js: Copied from resources/benchmark-report.js.
988         * StyleBench/resources/benchmark-runner.js: Copied from resources/benchmark-runner.js.
989         (BenchmarkRunner.prototype._finalize): Use the correction factor of 8 instead of 5 in StyleBench.
990         * StyleBench/resources/gauge.png: Copied from resources/gauge.png.
991         * StyleBench/resources/gauge@2x.png: Copied from resources/gauge@2x.png.
992         * StyleBench/resources/main.css: Copied from resources/main.css.
993         * StyleBench/resources/main.js: Copied from resources/main.js.
994         * StyleBench/resources/tests.js:
995         (makeSteps): Only make each test once.
996
997 2018-01-26  Chris Nardi  <cnardi@chromium.org>
998
999         Addressing post-review comments after r226614
1000         https://bugs.webkit.org/show_bug.cgi?id=182151
1001
1002         Reviewed by Myles C. Maxfield.
1003
1004         * StitchMarker/wtf/text/StringImpl.h:
1005         (WTF::isSpaceOrNewline):
1006         * StitchMarker/wtf/text/TextBreakIterator.cpp:
1007         (WTF::numCodeUnitsInGraphemeClusters):
1008         * StitchMarker/wtf/text/TextBreakIterator.h:
1009
1010 2018-01-22  Antti Koivisto  <antti@apple.com>
1011
1012         StyleBench: Separate test for :nth pseudo classes
1013         https://bugs.webkit.org/show_bug.cgi?id=181932
1014
1015         Reviewed by Sam Weinig.
1016
1017         Optimization strategies for nth are different from other structural pseudo classes (:first-child etc). Move them to a separate test.
1018
1019         * StyleBench/resources/style-bench.js:
1020         (structuralPseudoClassConfiguration):
1021         (nthPseudoClassConfiguration):
1022
1023             Add a new configuration.
1024
1025         (predefinedConfigurations):
1026         (pseudoClassConfiguration): Deleted.
1027
1028 2018-01-17  Ryosuke Niwa  <rniwa@webkit.org>
1029
1030         Temporarily skip Speedometer until we fix the metrics.
1031
1032         * Skipped:
1033
1034 2018-01-16  Alex Christensen  <achristensen@webkit.org>
1035
1036         Fix Mac CMake build after r222900
1037
1038         * MallocBench/MallocBench/CMakeLists.txt:
1039         The mbmalloc target doesn't exist if we're not in developer mode, so only use it if we are.
1040
1041 2018-01-09  Ryosuke Niwa  <rniwa@webkit.org>
1042
1043         Speedometer 2.0: Update the About page.
1044         https://bugs.webkit.org/show_bug.cgi?id=175918
1045
1046         Reviewed by Keith Miller.
1047
1048         Updated the about page for Speedometer 2.0.
1049
1050         * Speedometer/index.html:
1051
1052 == Rolled over to ChangeLog-2018-01-01 ==