2 This is a script for automating the browser based benchmarks(e.g. Speedometer, JetStream)
8 ├── benchmark_builder.py
9 ├── benchmark_results.py
10 ├── benchmark_results_unittest.py
11 ├── benchmark_runner.py
14 │ ├── browser_driver.py
15 │ ├── browser_driver_factory.py
16 │ ├── osx_browser_driver.py
17 │ ├── osx_chrome_driver.py
18 │ ├── osx_firefox_driver.py
19 │ └── osx_safari_driver.py
24 │ │ ├── JetStream.patch
27 │ │ ├── Speedometer.patch
28 │ │ └── SunSpider.patch
30 │ ├── dromaeo-cssquery.plan
31 │ ├── dromaeo-dom.plan
32 │ ├── dromaeo-jslib.plan
37 │ ├── speedometer.plan
39 ├── generic_factory.py
40 ├── http_server_driver
43 │ │ └── twisted_http_server.py
44 │ ├── http_server_driver.py
45 │ ├── http_server_driver_factory.py
46 │ └── simple_http_server_driver.py
58 python path/to/run-benchmark --build-directory path/to/browser/directory --plan json_format_plan --platform target_platform --browser target_browser
60 * **path/to/browser/directory**: should be the folder containing the executable binary(e.g. /Application/ on OSX which contains Safari.app)
61 * **json_format_plan**: the benchmark plan you want to execute
63 ### How to create a plan
64 To create a plan, you may refer to Plans/jetstream.plan.
69 "local_copy": "../../../../PerformanceTests/JetStream",
70 "benchmark_patch": "data/patches/JetStream.patch",
71 "entry_point": "JetStream/JetStream-1.0.1/index.html",
73 "orientation": "portrait"
75 "output_file": "jetstream.result"
78 Plan is a json-formatted dictionary which contains following keys
79 * **timeout**: time limit for **EACH RUN** of the benchmark. This can avoid program getting stuck in the extreme circumstances. The time limit is suggested to be 1.5-2x the time spent in a normal run.
80 * **count**: the number of times you want to run benchmark
81 * **local_copy**: (**OPTIONAL**) Path of benchmark, a relative path to the root of this project ('benchmark_runner' directory)
82 * **remote_archive**: (**OPTIONAL**) URL of the remote (http/https) ZIP file that contains the benchmark.
83 * **benchmark_path**: (**OPTIONAL**) path of patch, a relative path to the root of this project ('benchmark_runner' directory)
84 * **entry_point**: the relative url you want browser to launch (a relative path to the benchmark directory)
85 * **config**: a dictionary that specifies the environment configurations for the test (e.g. orientation while the test is running)
86 * **output_file**: specify the output file, this can be overwritten by specifying '--output-file' while invoking run-benchmark script
88 ### How to import a benchmark
89 * Modify the benchmark html file, make sure the page has following functionalities:
90 * When you launch the page('entry_point' in benchmark), the test will start automatically
91 * Organizing the results
92 * 'POST' information to '/report', and 'GET' '/shutdown' when post succeeds. Example:
94 var xhr = new XMLHttpRequest();
95 xhr.open("POST", "/report");
96 xhr.setRequestHeader("Content-type", "application/json");
97 xhr.setRequestHeader("Content-length", results.length);
98 xhr.setRequestHeader("Connection", "close");
99 xhr.onreadystatechange = function() {
100 if(xhr.readyState == 4 && xhr.status == 200) {
101 var closeRequest = new XMLHttpRequest();
102 closeRequest.open("GET", '/shutdown');
108 * Create a patch file against original file
109 * Go to the directory contains the benchmark directory (e.g. for JetStream, you should go to PerformaceTest folder)
110 * Use ```git diff --relative HEAD > your.patch``` to create your patch
111 * (**Suggested**) move the patch to the 'Patches' directory under project directory
112 * Create a plan for the benchmark (refer to **"How to create a plan"** for more details)
113 * Do following instruction **ONLY IF NEEDED**. In most case, you do not have to.
114 * If you want to customize BrowserDriver for specific browser/platform, you need to extend browser_driver/browser_driver.py and register your module in browser_driver/browser_driversjson.
115 * If you want to customize HTTPServerDriver, you need to extend http_server_drirver/http_server_driver and register your module in http_server_driver/http_server_drivers.json.