1 2015-11-09 Ryosuke Niwa <rniwa@webkit.org>
3 Allow , in the builder name.
5 Rubber-stamped by Alexey Proskuryakov.
7 * public/api/report.php:
9 2015-08-10 Dana Burkart <dburkart@apple.com>
11 Fix flakiness dashboard stability and performance issues.
12 https://bugs.webkit.org/show_bug.cgi?id=147835
14 Reviewed by Ryosuke Niwa.
18 * public/include/json-shared.php:
19 * public/include/test-results.php:
21 2014-01-23 Ryosuke Niwa <rniwa@webkit.org>
23 Upstream changes to json-shared.php from the perf dashboard
24 https://bugs.webkit.org/show_bug.cgi?id=127544
26 Reviewed by Joseph Pecoraro.
28 json-shared.php originated from the perf dashboard.
29 Upstream changes made to the version in the perf. dashboard so that two apps could share the same code.
31 * public/include/json-shared.php:
36 (merge_additional_details):
38 2013-11-06 Ryosuke Niwa <rniwa@webkit.org>
40 New flakiness dashboard should generate JSON in the background process
41 https://bugs.webkit.org/show_bug.cgi?id=123797
43 Reviewed by Alexey Proskuryakov.
45 * config.json: Add defaultBuildWaitInterval used by process-builds.php.
46 * init-database.sql: Add is_process column to builds table so that we can track of "unprocessed" builds.
48 * public/admin/process-builds.php: Added.
49 (process_latest_five_builds): Update flakiness states and generate JSONs for the latest five builds.
50 We go backwards in the time so that we don't end up infinite looping over a single build that fails.
51 (main): Call processed_builds with an exponential back off.
53 * public/api/report.php:
54 (store_results): No need to return build or builder ids.
55 (main): Don't update flakiness states or generate JSONs.
57 * public/include/test-results.php:
58 (add_build): Take the slave name; it doesn't make any sense to create a build without it and later
59 updating it in store_test_results.
60 (store_test_results): Once new test results are added, explicitly mark this build as unprocessed.
62 2013-11-04 Ryosuke Niwa <rniwa@webkit.org>
64 Build fix. Update is_flaky when it's NULL.
66 * public/include/test-results.php:
68 2013-11-04 Ryosuke Niwa <rniwa@webkit.org>
70 JSON generation on new flakiness dashboard is still too slow
71 https://bugs.webkit.org/show_bug.cgi?id=123776
73 Reviewed by Mark Rowe.
75 * init-database.sql: Added a constraint that result is unique for each (build, test) pair.
77 * public/include/test-results.php:
78 (ResultsJSONGenerator::generate): Don't sort results by latest_revision_time since that's done
79 in the client side anyway.
80 (update_flakiness_for_build): Don't update is_flaky flag if it's already set to the right value.
83 2013-11-04 Ryosuke Niwa <rniwa@webkit.org>
85 Build fixes. In particular, we were missing the code to generate JSONs in report.php :(
87 * public/api/report.php:
88 * public/include/test-results.php:
90 2013-11-04 Ryosuke Niwa <rniwa@webkit.org>
92 Fix JSON generations on new flakiness dashboard
93 https://bugs.webkit.org/show_bug.cgi?id=123723
95 Reviewed by Andreas Kling.
97 Add is_flaky column on results table so that this column can be used to find flaky tests on a given builder
98 efficiently without having to through results for all tests in PHP. This column is updated in report.php
99 when a new build is added. Because is_flaky depends on the preceding and succeeding results, we must update
100 is_flaky flag on results for builds immediately before and after the new build as well.
102 To see why, suppose we had two consecutive results [PASS] [PASS]. If we were to insert [FAIL] result between
103 the two, those two results may also turn into flaky results if they were surrounded by [FAIL]. Similarly,
104 if we had [PASS] [FAIL] and the second result was marked flaky, inserting new [FAIL] must unmark it.
107 * init-database.sql: Added is_flaky column to results table with an index. Also added an index on
108 build_revisions.time as many queries filter results by this quantity. Also set the work_mem to 50MB avoid
109 disk thrashing while sorting results in various queries.
111 * public/api/failing-tests.php: Handle builder ids as well as names. Call generate() with failure types.
112 No longer generates *-failing.json since it's a subset of *-wrongexpectations.json to save time.
114 * public/api/report.php: Rewritten. Calls update_flakiness_after_inserting_build to update is_flaky flags
115 on the newly added results.
116 (store_results): Added.
119 * public/include/test-results.php:
121 (ResultsJSONWriter::add_results_for_test): Renamed from add_results_for_test_if_matches.
122 (ResultsJSONGenerator::generate): Takes the failure type. Instead of generating JSONs for all failure types
123 at once, generate one JSON for the specified type. We generate the list of test ids based on the failure type
124 and query results based on that. This dramatically cuts down the time spent in PHP.
125 (ResultsJSONGenerator::latest_build): Added.
126 (ResultsJSONGenerator::write_jsons): Takes single writer now.
127 (update_flakiness_for_build): Added.
128 (update_flakiness_after_inserting_build): Added.
131 (TestResultsView._populateBuilderPane): Emulate *-failing.json upon *-wrongexpectations.json.
132 (TestResultsView.fetchFailingTestsForBuilder): Ditto.
134 * public/main.css: Minor style tweaks.
135 (.testResults): Extend the border that wraps the test results as needed.
136 (.tooltip): Don't wrap text inside tooltips.
138 2013-10-26 Ryosuke Niwa <rniwa@webkit.org>
140 Make new bug link in flakiness dashboard configurable
141 https://bugs.webkit.org/show_bug.cgi?id=123386
143 Reviewed by Alexey Proskuryakov.
145 * config.json: Added Bugzilla as the default destination for new bugs.
146 * public/api/manifest.php: Include newBugLinks in the manifest.
148 (TestResultsView): Initialize _newBugLinks.
149 (TestResultsView.setNewBugLinks): Added.
150 (TestResultsView._populateTestPane):
151 (TestResultsView._createTestResultRow): Replaced the hard-coded Bugzilla URL by the code
152 to generate hyper-links based on _newBugLinks. Also added a nullity check while formatting
153 revision checks so that we don't blow up when some build doesn't contain all revision info.
154 (TestResultsView._populateBuilderPane):
156 2013-10-26 Ryosuke Niwa <rniwa@webkit.org>
158 New flakiness dashboard should support substring matching
159 https://bugs.webkit.org/show_bug.cgi?id=123393
161 Reviewed by Alexey Proskuryakov.
163 Addressed the use cases by
164 1. Always showing the candidate even when there is exactly one test matching the current value.
165 2. Adding all tests that match the current value upon the user pressing enter key.
168 (fetchManifest): Add all tests that match the current value. Confirm whether the user really
169 want to add all the tests when there are more than 15 tests to add.
170 * public/js/autocompleter.js:
171 (Autocompleter.prototype.filterCandidates): Extracted from _updateCandidates.
172 (Autocompleter.prototype._updateCandidates): Show the candidate window even when there is
173 exactly one test that matches the criteria so that the user can select this test.
175 2013-10-26 Ryosuke Niwa <rniwa@webkit.org>
177 New flakiness dashboard shouldn't treat tests with right expectations as failing
178 https://bugs.webkit.org/show_bug.cgi?id=123385
180 Reviewed by Alexey Proskuryakov.
182 We define failing tests to be tests with wrong expectations whose actual results are not PASS
183 since tests with TEXT, IMAGE, etc... failures do not turn the bots red as long as the expectation
184 of the same type is specified in TestExpectation files.
186 * public/include/test-results.php:
187 (FailingResultsJSONWriter): Inherit from WrongExpectationsResultsJSONWriter.
189 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
191 Build fix. The query results weren't sorted by the latest commit time,
192 yielding wrong set of tests to be listed in the builder pane.
194 * public/include/test-results.php:
196 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
198 Links in new flakiness dashboard could benefit from tooltips
199 https://bugs.webkit.org/show_bug.cgi?id=123381
201 Reviewed by Alexey Proskuryakov.
203 Add title content attribute on each hyperlink in the builder pane.
206 (TestResultsView._populateBuilderPane):
208 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
210 Commit the right change.
214 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
216 The new flakiness dashboard should sort builder and test names
217 https://bugs.webkit.org/show_bug.cgi?id=123363
219 Reviewed by Simon Fraser.
221 Sort the forgotten builders in the menu.
225 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
227 The new flakiness dashboard should sort builder and test names
228 https://bugs.webkit.org/show_bug.cgi?id=123363
230 Reviewed by Simon Fraser.
232 Sort builders and tests by their name in each pane.
235 (TestResultsView._populateTestPane): Lexicologically sort builders by their name.
236 (TestResultsView._sortObjectsByName): Added.
237 (TestResultsView._populateBuilderPane): Lexicologically sort builders by their name.
239 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
241 Merge revision columns in flakiness dashboard
242 https://bugs.webkit.org/show_bug.cgi?id=123360
244 Reviewed by Simon Fraser.
246 Merge revision numbers into one cell separated by /, and clearly label them as "latest revisions".
247 Also add colors for missing & audio failures so that they won't be an empty cell.
250 (TestResultsView._createTestResultHeader): Create a single th for the list of repository names.
251 (TestResultsView._createTestResultRow): Use single td for all revision numbers.
252 * public/js/build.js:
253 (.this.formattedRevision): Don't prefix revision numbers with '@' when we're not in the tooltip.
255 (.resultsTable .AUDIO a): Added.
256 (.resultsTable .MISSING a): Added.
258 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
260 One more build fix. Put each latest revision information in an actual td.
264 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
266 Yet another set of build fixes.
268 1. Manually strip / from revisions JSON if there is any.
269 2. Suppress warnings from ob_end_flush.
270 3. Allow "+" in actual and expected results as used in "TEXT+IMAGE".
272 * public/api/report.php:
273 * public/include/test-results.php:
275 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
277 Another build fix. Move .htaccess to public.
279 * .htaccess: Removed.
280 * public/.htaccess: Copied from .htaccess.
282 2013-10-25 Ryosuke Niwa <rniwa@webkit.org>
284 It turns out that the current instruction only works on Mountain Lion.
286 Also fixed some typos.
290 2013-10-24 Ryosuke Niwa <rniwa@webkit.org>
292 Add an instruction on how to setup test-results app
293 https://bugs.webkit.org/show_bug.cgi?id=123321
295 Rubber-stamped by Stephanie Lewis.
297 Added Install.md and updated config.json accordingly.
299 Also moved init-database.sql out of public/include since it doesn't need to be accessible via httpd.
303 * init-database.sql: Moved from public/include/init-database.sql.
305 2013-10-24 Ryosuke Niwa <rniwa@webkit.org>
307 Move everything except ChangeLog and config.json into public directory.
308 https://bugs.webkit.org/show_bug.cgi?id=123319
310 Rubber-stamped by Stephanie Lewis.
313 * admin/admin.css: Removed.
314 * admin/builders.php: Removed.
315 * admin/index.php: Removed.
316 * admin/repositories.php: Removed.
318 * api/failing-tests.php: Removed.
319 * api/manifest.php: Removed.
320 * api/report.php: Removed.
321 * api/results.php: Removed.
322 * common.css: Removed.
323 * config.json: Copied from include/config.json.
325 * include/admin-footer.php: Removed.
326 * include/admin-header.php: Removed.
327 * include/config.json: Removed.
328 * include/db.php: Removed.
329 * include/init-database.sql: Removed.
330 * include/json-shared.php: Removed.
331 * include/test-results.php: Removed.
332 * index.html: Removed.
334 * js/autocompleter.js: Removed.
335 * js/build.js: Removed.
336 * js/dom.js: Removed.
339 * public/admin: Copied from admin.
340 * public/api: Copied from api.
341 * public/common.css: Copied from common.css.
342 * public/include: Copied from include.
343 * public/include/config.json: Removed.
344 * public/include/db.php:
345 * public/index.html: Copied from index.html.
346 * public/js: Copied from js.
347 * public/main.css: Copied from main.css.
349 2013-10-24 Ryosuke Niwa <rniwa@webkit.org>
351 Showing lists of flaky tests for a builder takes too long
352 https://bugs.webkit.org/show_bug.cgi?id=123311
354 Reviewed by Sam Weinig.
356 Generate JSONs for tests failing, flaky, or with wrong expectation at the time a builder reports results
357 instead when the frontend requests to those those results since it takes multiple seconds or minutes to
358 generate those JSON files.
360 * api/failing-tests.php: Moved and renamed to manually generate all JSON files for a given builder.
363 * api/report.php: Manually flush and end the request (to avoid blocking run-webkit-tests on the other side
364 for minutes), then generate JSONs for tests that are failing, are flaky, and have wrong expectations.
366 * api/results.php: Merge format_result_rows here since it's not used anywhere else.
368 * include/config.json: Added the path to the data directory into which JSON files are generated.
371 (configPath): Takes a relative path value from config.json, and resolves it.
373 * include/json-shared.php:
374 (echo_success): Extracted from exit_with_success.
377 * include/test-results.php:
378 (add_builder): Extracted from add_build.
380 (ResultsJSONWriter): Extracted from api/failing-tests.php.
381 (ResultsJSONWriter.__construct):
382 (ResultsJSONWriter.start):
383 (ResultsJSONWriter.end):
384 (ResultsJSONWriter.add_results_for_test_if_matches):
385 (ResultsJSONWriter.pass_for_failure_type):
386 (FailingResultsJSONWriter): Extracted from index.html's TestResultsView._matchesFailureType.
387 (FailingResultsJSONWriter.__construct):
388 (FailingResultsJSONWriter.pass_for_failure_type):
389 (FlakyResultsJSONWriter): Ditto.
390 (FlakyResultsJSONWriter.__construct):
391 (FlakyResultsJSONWriter.pass_for_failure_type):
392 (WrongExpectationsResultsJSONWriter): Ditto.
393 (WrongExpectationsResultsJSONWriter.__construct):
394 (WrongExpectationsResultsJSONWriter.pass_for_failure_type):
395 (ResultsJSONGenerator): Ditto.
396 (ResultsJSONGenerator.__construct):
397 (ResultsJSONGenerator.generate):
398 (ResultsJSONGenerator.open_json_for_failure_type):
399 (ResultsJSONGenerator.write_jsons):
403 (TestResultsView.setBuilders):
404 (TestResultsView._createResultCell): Add a hyperlink to results.html in the tooltip.
405 (TestResultsView.fetchFailingTestsForBuilder): Fetch the generated JSON files.
407 2013-10-23 Ryosuke Niwa <rniwa@webkit.org>
409 Reverted erroneously committed changes from the previous commit.
413 2013-10-23 Ryosuke Niwa <rniwa@webkit.org>
415 Clicking on a test name on the new flakiness dashboard should add a new test pane
416 https://bugs.webkit.org/show_bug.cgi?id=123194
418 Reviewed by Simon Fraser.
420 Added a new click handler on each test name inside the builder test view add a new test pane.
421 Moved the old trac link to a right-arrow inline SVG icon.
423 Also tweaked the style so that the builder test view looks different from other test panes.
426 (TestResultsView._urlFromTest): Extracted from TestResultsView._linkifiedTestName.
427 (TestResultsView._populateBuilderPane): Fetch the test when its name is clicked.
428 Added a circled right arrow for the old trac link.
429 * main.css: Tweaked the style to move the rounded border around the builder test view to be
430 around the form controls to clearly differentiate it from a regular test pane.
432 2013-10-22 Ryosuke Niwa <rniwa@webkit.org>
434 New flakiness dashboard's test pane should show the latest WebKit revision for each builder
435 https://bugs.webkit.org/show_bug.cgi?id=123189
437 Reviewed by Simon Fraser.
440 (TestResultsView._populateTestPane): Call _createTestResultHeader and _createTestResultRow with
441 a list of repository information.
442 (TestResultsView._createTestResultHeader): Optionally creates headers for a list of repositories.
443 (TestResultsView._createTestResultRow): Add a hyperlinked revision information for each builder.
444 Also add a hyperlink to file a Bugzilla bug when there is no bug associated with the test already.
446 (Build.formattedRevision): Extracted from Build.formattedRevisions.
447 (Build.formattedRevisions):
449 2013-10-22 Ryosuke Niwa <rniwa@webkit.org>
451 Make tables on the new flakiness dashboard sortable
452 https://bugs.webkit.org/show_bug.cgi?id=123141
454 Reviewed by Simon Fraser.
456 Use jquery.tablesorter.js to make test and builder tables sortable. The jquery plugin is already used by run-perf-tests.
459 (TestResultsView._populateTestPane): Add tablesorter to the class name and wrap tr's inside a tbody so that tablesorter
461 (TestResultsView._populateBuilderPane): Ditto.
463 (.resultsTable thead): Use cursor: pointer to signify the fact it's clickable.
464 (.resultsTable th): Don't repeat arrows.
465 (.resultsTable th.headerSortUp): Inline SVG up arrow.
466 (.resultsTable th.headerSortDown): Inline SVG down arrow.
468 2013-10-21 Ryosuke Niwa <rniwa@webkit.org>
470 New flakiness dashboard should hyperlink test names, WebKit revisions, and bubbles
471 https://bugs.webkit.org/show_bug.cgi?id=123134
473 Reviewed by Stephanie Lewis.
475 Copied admin.css, admin-header.php, admin-footer.php, builders.php, repositories.php from WebKit Perf Monitor.
476 (Unfortunately WebKit Perf Monitor hasn't been committed into WebKit repository just yet.)
478 Updated various parts of index.html to linkify test names, build numbers, and bubbles (to results page).
480 * admin/admin.css: Added.
481 * admin/builders.php: Added.
482 * admin/index.php: Removed the duplicated code now that it uses admin-header.php.
483 * admin/repositories.php: Added.
484 * api/manifest.php: Use camelCase for blame_url and build_url to be consistent with other JSON properties.
485 Also exported testCategories from config so that we can linkify test names in the dashboard.
486 * include/admin-footer.php: Added.
487 * include/admin-header.php: Added.
488 * include/config.json: Added test categories. This avoids hard-coding the URL to trac in php/js.
489 * include/init-database.sql: Added name and build_url to builders table and category to tests.
490 * include/test-results.php: Assume the test category to be LayoutTest for now.
493 (TestResultsView): Initialize _builders, _slaves, _repositories, _testCategories as dictionaries as intended.
494 (TestResultsView.setTestCategories): Added.
495 (TestResultsView._createResultCell): Dynamically resolve URLs of results page and and build page.
496 (TestResultsView._populateTestPane): Linkify the test name. Unfortunately we don't have a test object anywhere.
497 We need to figure out a way to find the test object here eventually. For now, hard-coding "LayoutTest" works.
498 (TestResultsView._linkifiedTestName): Added.
499 (TestResultsView._createBuildsAndComputeSlownessOfResults): Takes builderId to set "builder" property on each
500 result object as it's used by Build class.
501 (TestResultsView._populateBuilderPane):
505 (Build.buildUrl): Support $builderName so that we don't have to keep repeating builder names in the database.
506 (Build.revision): Don't access [0] if revisions[repositoryId] was undefined.
508 2013-10-21 Ryosuke Niwa <rniwa@webkit.org>
510 New flakiness dashboard should align results by revision numbers
511 https://bugs.webkit.org/show_bug.cgi?id=123129
513 Reviewed by Tim Horton.
515 * include/config.json: Specify the default cache time to be 10 minutes.
516 * include/json-shared.php: Added Expires and Cache-Control headers. It's configurable via config.json.
518 (TestResultsView._createResultCell): Don't put '-' in a result cell without time.
519 (TestResultsView._populateTestPane): Create an array of build times ordered from the newest to the oldest.
520 Not that build.time() is the newest commit time among all repository for each result. e.g. if we had
521 both WebKit and Safari repository information in a given result, the newer of the two will be used.
522 (TestResultsView._createTestResultHeader): Added.
523 (TestResultsView._createBuildsAndComputeSlownessOfResults): Extracted from _createTestResultRow so
524 that _populateTestPane could use Build object before calling _createTestResultRow.
525 (TestResultsView._createTestResultRow): Takes buildTimes. Insert an empty cell for a build time if
526 the current row doesn't contain that a result for that build time.
527 (TestResultsView._populateBuilderPane):
528 * main.css: Tweaked the style so that bubbles are aligned vertically without '-'.
530 (.resultsTable thead th):
531 (.resultsTable .resultCell):
533 (.resultsTable span a):
535 2013-10-21 Ryosuke Niwa <rniwa@webkit.org>
537 New flakiness dashboard show test time, modifiers, and flaky tests
538 https://bugs.webkit.org/show_bug.cgi?id=123119
540 Reviewed by Tim Horton.
542 * api/failing-tests.php: Manually serialize each row in the results to avoid hitting the memory limit.
544 (Database::query): Added.
545 (Database::fetch_next_row): Added.
546 * include/init-database.sql: Added modifiers and time columns to results table.
547 * include/test-results.php:
548 (store_test_results): Update start_time and end_time to the union of the new interval and the existing interval.
549 (recursively_add_test_results): Handle empty $full_name to eliminate the loop over tests in store_test_results.
550 Also verify that each test name, expected and actual results conform to the specific format to prevent XSS.
551 Also use insert_row instead of select_or_insert_row to avoid issuing an unnecessary SQL query.
552 (format_result): Extracted from format_result_rows. Used in failing-tests.php.
554 (TestResultsView): Added _currentBuilderFailureType and _currentBuilderDays.
555 (TestResultsView._createResultCell): Show the test time and the expected result.
556 (TestResultsView._createTestResultRow): Compute the slowest run and also round time to tenth of second for time
557 less than 10s or second if it's more than 10s so that the test time will always be shown in two digits.
558 Also show the bug number and the latest expected result on the left columns after linkifying the bug numbers.
559 (TestResultsView._matchesFailureType): Added. Determines whether results is of a particular failure type.
560 (TestResultsView._populateBuilderPane):
561 (TestResultsView.fetchFailingTestsForBuilder): Store the failure type such as flaky, wrongtestexpectations.
562 (TestResultsView.updateLocationHash):
563 (TestResultsView.loadTestsFromLocationHash):
566 (element): appendChild if an item is a Node. Otherwise, e.g. integer, create a text node out of toString() call.
567 * main.css: Updated styles.
569 2013-10-18 Ryosuke Niwa <rniwa@webkit.org>
571 New flakiness dashboard should support showing the failing tests per builder
572 https://bugs.webkit.org/show_bug.cgi?id=123011
574 Reviewed by Timothy Hatcher.
576 Added the feature. Also did some refactoring to add this feature.
579 * api/failing-tests.php: Added.
580 * api/manifest.php: Removed the code to make maps by id. The work is now done in index.html.
582 * common.css: Added. Extracted from index.html.
583 * include/test-results.php: Extracted parse_revisions_array and format_result_rows from results.php.
586 (TestResultsView.setAvailableTests): Added.
587 (TestResultsView.showTooltip): Fixed the code to compute x and y coordinates of the tooltip to take
588 scrolled positions into account.
589 (TestResultsView._createTestResultRow): Extracted from _populateTestPane.
590 (TestResultsView.fetchTest): Added the code to show "Loading..." in the pane while loading the JSON.
591 (TestResultsView.fetchTests): Respect the doNotUpdateHash flag.
592 (TestResultsView._populateBuilderPane): Added.
593 (TestResultsView.fetchFailingTestsForBuilder): Added.
594 (TestResultsView.updateLocationHash): Serialize builder & builderDays.
595 (TestResultsView.locationHashChanged): Don't delete existing test panes since that's now done in
596 loadTestsFromLocationHash.
597 (TestResultsView.loadTestsFromLocationHash): Take care of both 'tests' and 'builder' components.
598 (fetchManifest): Setup the UI to select a builder.