2010-11-16 Mihai Parparita <mihaip@chromium.org>
authormihaip@chromium.org <mihaip@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 17 Nov 2010 02:37:28 +0000 (02:37 +0000)
committermihaip@chromium.org <mihaip@chromium.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 17 Nov 2010 02:37:28 +0000 (02:37 +0000)
        Reviewed by Tony Chang.

        Rebaseline server: display test results
        https://bugs.webkit.org/show_bug.cgi?id=49626

        Adds basic result display to the rebaseline server. On the Python side
        this involves:
        - Parsing the unexpected_results.json into a dictionary.
        - Serving it as JSON under /results.json.
        (the JSON -> dict -> JSON transform isn't strictly necessary right now,
        but I'll need to have access to the parsed results on the Python side
        for follow-up changes).

        On the web UI side this adds:
        - Markup for display image and text results (expected, actual, diff),
          and JS for populating it.
        - Markup for breaking down test results by failure type and directory,
          and JS for populating it.

        * Scripts/webkitpy/tool/commands/data/rebaselineserver/index.html:
        * Scripts/webkitpy/tool/commands/data/rebaselineserver/main.css:
        * Scripts/webkitpy/tool/commands/data/rebaselineserver/main.js:
        * Scripts/webkitpy/tool/commands/data/rebaselineserver/util.js: Added.
        * Scripts/webkitpy/tool/commands/rebaselineserver.py:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@72157 268f45cc-cd09-0410-ab3c-d52691b4dbfc

WebKitTools/ChangeLog
WebKitTools/Scripts/webkitpy/tool/commands/data/rebaselineserver/index.html
WebKitTools/Scripts/webkitpy/tool/commands/data/rebaselineserver/main.css
WebKitTools/Scripts/webkitpy/tool/commands/data/rebaselineserver/main.js
WebKitTools/Scripts/webkitpy/tool/commands/data/rebaselineserver/util.js [new file with mode: 0644]
WebKitTools/Scripts/webkitpy/tool/commands/rebaselineserver.py

index 0aed1425f54f96f162787c61a22141023fd26d81..aea9504b3589ace3f32036898bbdcb265a7c51fd 100644 (file)
@@ -1,3 +1,30 @@
+2010-11-16  Mihai Parparita  <mihaip@chromium.org>
+
+        Reviewed by Tony Chang.
+
+        Rebaseline server: display test results
+        https://bugs.webkit.org/show_bug.cgi?id=49626
+        
+        Adds basic result display to the rebaseline server. On the Python side
+        this involves:
+        - Parsing the unexpected_results.json into a dictionary.
+        - Serving it as JSON under /results.json.
+        (the JSON -> dict -> JSON transform isn't strictly necessary right now,
+        but I'll need to have access to the parsed results on the Python side
+        for follow-up changes).
+        
+        On the web UI side this adds:
+        - Markup for display image and text results (expected, actual, diff),
+          and JS for populating it.
+        - Markup for breaking down test results by failure type and directory,
+          and JS for populating it.
+
+        * Scripts/webkitpy/tool/commands/data/rebaselineserver/index.html:
+        * Scripts/webkitpy/tool/commands/data/rebaselineserver/main.css:
+        * Scripts/webkitpy/tool/commands/data/rebaselineserver/main.js:
+        * Scripts/webkitpy/tool/commands/data/rebaselineserver/util.js: Added.
+        * Scripts/webkitpy/tool/commands/rebaselineserver.py:
+
 2010-11-16  Dirk Pranke  <dpranke@chromium.org>
 
         Reviewed by Ojan Vafai.
index 5667cd2af274d7e02e92e3ba758e7b7ea786629a..142e16447c821460a6d8f722c9bdc12f875b404b 100644 (file)
@@ -1,11 +1,11 @@
 <!DOCTYPE html>
 <!--
   Copyright (c) 2010 Google Inc. All rights reserved.
-  
+
   Redistribution and use in source and binary forms, with or without
   modification, are permitted provided that the following conditions are
   met:
-  
+
      * Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
      * Redistributions in binary form must reproduce the above
@@ -15,7 +15,7 @@
      * Neither the name of Google Inc. nor the names of its
   contributors may be used to endorse or promote products derived from
   this software without specific prior written permission.
-  
+
   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 <head>
   <title>Layout Test Rebaseline Server</title>
   <link rel="stylesheet" href="/main.css" type="text/css">
+  <script src="/util.js"></script>
   <script src="/main.js"></script>
 </head>
 <body class="loading">
+
+<div id="header">
+  <div id="controls">
+    <!-- Add a dummy <select> node so that this lines up with the text on the left -->
+    <select style="visibility: hidden"></select>
     <a href="/quitquitquit">Exit</a>
+  </div>
+
+  <span id="selectors">
+    <label>
+      Failure type:
+      <select id="failure-type-selector"></select>
+    </label>
+
+    <label>
+      Directory:
+      <select id="directory-selector"></select>
+    </label>
+
+    <label>
+      Test:
+      <select id="test-selector"></select>
+    </label>
+  </span>
+
+  <a id="test-link">View test</a>
+
+  <span id="nav-buttons">
+    <button id="previous-test">&laquo;</button>
+    <span id="test-index"></span> of <span id="test-count"></span>
+    <button id="next-test">&raquo;</button>
+  </span>
+</div>
+
+<table id="test-output">
+  <thead id="labels">
+    <tr>
+      <th>Expected</th>
+      <th>Actual</th>
+      <th>Diff</th>
+    </tr>
+  </thead>
+  <tbody id="image-outputs" style="display: none">
+    <tr>
+      <td colspan="3"><h2>Image</h2></td>
+    </tr>
+    <tr>
+      <td><img id="expected-image"></td>
+      <td><img id="actual-image"></td>
+      <td><img id="diff-image"></td>
+    </tr>
+  </tbody>
+  <tbody id="text-outputs" style="display: none">
+    <tr>
+      <td colspan="3"><h2>Text</h2></td>
+    </tr>
+    <tr>
+      <td><pre id="expected-text"></pre></td>
+      <td><pre id="actual-text"></pre></td>
+      <td><pre id="diff-text"><pre></td>
+    </tr>
+  </tbody>
+</table>
+
 </body>
 </html>
index 35bd6a5ad1c9f9bb3b581963256815acea0e17b2..329f95f44f6f5e7c198957efed6b17f0a90c5082 100644 (file)
@@ -46,7 +46,7 @@ div {
 a, .link {
   color: #aaf;
   text-decoration: underline;
-  cursor: pointer;      
+  cursor: pointer;
 }
 
 .link.selected {
@@ -54,3 +54,91 @@ a, .link {
   font-weight: bold;
   text-decoration: none;
 }
+
+#header {
+  padding: .5em 1em;
+  background: #333;
+  color: #fff;
+  -webkit-box-shadow: 0 1px 5px rgba(0, 0, 0, 0.5);
+  margin-bottom: 1em;
+}
+
+#header label {
+  padding-right: 1em;
+  color: #ccc;
+}
+
+#test-link {
+  margin-right: 1em;
+}
+
+#header label span {
+  color: #fff;
+  font-weight: bold;
+}
+
+#nav-buttons {
+  white-space: nowrap;
+}
+
+#nav-buttons button {
+  background: #fff;
+  border: 0;
+  border-radius: 10px;
+}
+
+#nav-buttons button:active {
+  -webkit-box-shadow: 0 0 5px #33f inset;
+  background: #aaa;
+}
+
+#nav-buttons button[disabled] {
+  opacity: .5;
+}
+
+#controls {
+  float: right;
+}
+
+#test-output {
+  border-spacing: 0;
+  border-collapse: collapse;
+  margin: 0 auto;
+  width: 100%;
+}
+
+#test-output td,
+#test-output th {
+  padding: 0;
+  vertical-align: top;
+}
+
+#image-outputs img {
+  width: 800px;
+  height: 600px;
+  border: solid 1px #ddd;
+  -webkit-user-select: none;
+  -webkit-user-drag: none;
+}
+
+#image-outputs #actual-image {
+  margin: 0 1em;
+}
+
+#test-output #labels th {
+  text-align: center;
+  color: #666;
+}
+
+#text-outputs pre {
+  height: 600px;
+  width: 800px;
+  overflow: auto;
+}
+
+#test-output h2 {
+  border-bottom: solid 1px #ccc;
+  font-weight: bold;
+  margin: 0;
+  background: #eee;
+}
index 55f19a4f3e29709ce3f13dd2e9d8d1174de2f41a..5387cd9365f04b414266b7c8eb249f326db82d33 100644 (file)
  * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
+var ALL_DIRECTORY_PATH = '[all]';
+
+var results;
+var testsByFailureType = {};
+var testsByDirectory = {};
+var selectedTests = [];
+
 function main()
 {
+    $('failure-type-selector').addEventListener('change', selectFailureType);
+    $('directory-selector').addEventListener('change', selectDirectory);
+    $('test-selector').addEventListener('change', selectTest);
+    $('next-test').addEventListener('click', nextTest);
+    $('previous-test').addEventListener('click', previousTest);
+
+    document.addEventListener('keydown', function(event) {
+        if (event.altKey || event.ctrlKey || event.metaKey || event.shiftKey) {
+            return;
+        }
+
+        switch (event.keyIdentifier) {
+        case 'Left':
+            event.preventDefault();
+            previousTest();
+            break;
+        case 'Right':
+            event.preventDefault();
+            nextTest();
+            break;
+        }
+    });
+
+    loadText('/results.json', function(text) {
+        results = JSON.parse(text);
+        displayResults();
+    });
+}
+
+/**
+ * Groups test results by failure type.
+ */
+function displayResults()
+{
+    var failureTypeSelector = $('failure-type-selector');
+    var failureTypes = [];
+
+    for (var testName in results.tests) {
+        var test = results.tests[testName];
+        if (test.actual == 'PASS') {
+            continue;
+        }
+        var failureType = test.actual + ' (expected ' + test.expected + ')';
+        if (!(failureType in testsByFailureType)) {
+            testsByFailureType[failureType] = [];
+            failureTypes.push(failureType);
+        }
+        testsByFailureType[failureType].push(testName);
+    }
+
+    // Sort by number of failures
+    failureTypes.sort(function(a, b) {
+        return testsByFailureType[b].length - testsByFailureType[a].length;
+    });
+
+    for (var i = 0, failureType; failureType = failureTypes[i]; i++) {
+        var failureTypeOption = document.createElement('option');
+        failureTypeOption.value = failureType;
+        failureTypeOption.textContent = failureType + ' - ' + testsByFailureType[failureType].length + ' tests';
+        failureTypeSelector.appendChild(failureTypeOption);
+    }
+
+    selectFailureType();
+
     document.body.classList.remove('loading');
 }
 
+/**
+ * For a given failure type, gets all the tests and groups them by directory
+ * (populating the directory selector with them).
+ */
+function selectFailureType()
+{
+    var selectedFailureType = getSelectValue('failure-type-selector');
+    var tests = testsByFailureType[selectedFailureType];
+
+    testsByDirectory = {}
+    var displayDirectoryNamesByDirectory = {};
+    var directories = [];
+
+    // Include a special option for all tests
+    testsByDirectory[ALL_DIRECTORY_PATH] = tests;
+    displayDirectoryNamesByDirectory[ALL_DIRECTORY_PATH] = 'all';
+    directories.push(ALL_DIRECTORY_PATH);
+
+    // Roll up tests by ancestor directories
+    tests.forEach(function(test) {
+        var pathPieces = test.split('/');
+        var pathDirectories = pathPieces.slice(0, pathPieces.length -1);
+        var ancestorDirectory = '';
+
+        pathDirectories.forEach(function(pathDirectory, index) {
+            ancestorDirectory += pathDirectory + '/';
+            if (!(ancestorDirectory in testsByDirectory)) {
+                testsByDirectory[ancestorDirectory] = [];
+                var displayDirectoryName = new Array(index * 6).join('&nbsp;') + pathDirectory;
+                displayDirectoryNamesByDirectory[ancestorDirectory] = displayDirectoryName;
+                directories.push(ancestorDirectory);
+            }
+
+            testsByDirectory[ancestorDirectory].push(test);
+        });
+    });
+
+    directories.sort();
+
+    var directorySelector = $('directory-selector');
+    directorySelector.innerHTML = '';
+
+    directories.forEach(function(directory) {
+        var directoryOption = document.createElement('option');
+        directoryOption.value = directory;
+        directoryOption.innerHTML =
+            displayDirectoryNamesByDirectory[directory] + ' - ' +
+            testsByDirectory[directory].length + ' tests';
+        directorySelector.appendChild(directoryOption);
+    });
+
+    selectDirectory();
+}
+
+/**
+ * For a given failure type and directory and failure type, gets all the tests
+ * in that directory and populatest the test selector with them.
+ */
+function selectDirectory()
+{
+    var selectedDirectory = getSelectValue('directory-selector');
+    selectedTests = testsByDirectory[selectedDirectory];
+
+    selectedTests.sort();
+
+    var testSelector = $('test-selector');
+    testSelector.innerHTML = '';
+
+    selectedTests.forEach(function(testName) {
+        var testOption = document.createElement('option');
+        testOption.value = testName;
+        var testDisplayName = testName;
+        if (testName.lastIndexOf(selectedDirectory) == 0) {
+            testDisplayName = testName.substring(selectedDirectory.length);
+        }
+        testOption.innerHTML = '&nbsp;&nbsp;' + testDisplayName;
+        testSelector.appendChild(testOption);
+    });
+
+    selectTest();
+}
+
+function getSelectedTest()
+{
+    return getSelectValue('test-selector');
+}
+
+function selectTest()
+{
+    var selectedTest = getSelectedTest();
+
+    if (results.tests[selectedTest].actual.indexOf('IMAGE') != -1) {
+        $('image-outputs').style.display = '';
+        displayImageResults(selectedTest);
+    } else {
+        $('image-outputs').style.display = 'none';
+    }
+
+    if (results.tests[selectedTest].actual.indexOf('TEXT') != -1) {
+        $('text-outputs').style.display = '';
+        displayTextResults(selectedTest);
+    } else {
+        $('text-outputs').style.display = 'none';
+    }
+
+    updateState();
+}
+
+function updateState()
+{
+    var testName = getSelectedTest();
+    var testIndex = selectedTests.indexOf(testName);
+    var testCount = selectedTests.length
+    $('test-index').textContent = testIndex + 1;
+    $('test-count').textContent = testCount;
+
+    $('next-test').disabled = testIndex == testCount - 1;
+    $('previous-test').disabled = testIndex == 0;
+
+    $('test-link').href =
+        'http://trac.webkit.org/browser/trunk/LayoutTests/' + testName;
+}
+
+function getTestResultUrl(testName, mode)
+{
+    return '/test_result?test=' + testName + '&mode=' + mode;
+}
+
+function displayImageResults(testName)
+{
+    function displayImageResult(mode) {
+        $(mode).src = getTestResultUrl(testName, mode);
+    }
+
+    displayImageResult('expected-image');
+    displayImageResult('actual-image');
+    displayImageResult('diff-image');
+}
+
+function displayTextResults(testName)
+{
+    function loadTextResult(mode) {
+        loadText(getTestResultUrl(testName, mode), function(text) {
+            $(mode).textContent = text;
+        });
+    }
+
+    loadTextResult('expected-text');
+    loadTextResult('actual-text');
+    loadTextResult('diff-text');
+}
+
+function nextTest()
+{
+    var testSelector = $('test-selector');
+    var nextTestIndex = testSelector.selectedIndex + 1;
+    while (true) {
+        if (nextTestIndex == testSelector.options.length) {
+            return;
+        }
+        if (testSelector.options[nextTestIndex].disabled) {
+            nextTestIndex++;
+        } else {
+            testSelector.selectedIndex = nextTestIndex;
+            selectTest();
+            return;
+        }
+    }
+}
+
+function previousTest()
+{
+    var testSelector = $('test-selector');
+    var previousTestIndex = testSelector.selectedIndex - 1;
+    while (true) {
+        if (previousTestIndex == -1) {
+            return;
+        }
+        if (testSelector.options[previousTestIndex].disabled) {
+            previousTestIndex--;
+        } else {
+            testSelector.selectedIndex = previousTestIndex;
+            selectTest();
+            return
+        }
+    }
+}
+
 window.addEventListener('DOMContentLoaded', main);
diff --git a/WebKitTools/Scripts/webkitpy/tool/commands/data/rebaselineserver/util.js b/WebKitTools/Scripts/webkitpy/tool/commands/data/rebaselineserver/util.js
new file mode 100644 (file)
index 0000000..1c8782b
--- /dev/null
@@ -0,0 +1,57 @@
+/*
+ * Copyright (c) 2010 Google Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following disclaimer
+ * in the documentation and/or other materials provided with the
+ * distribution.
+ *     * Neither the name of Google Inc. nor the names of its
+ * contributors may be used to endorse or promote products derived from
+ * this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+var results;
+var testsByFailureType = {};
+var testsByDirectory = {};
+var selectedTests = [];
+
+function $(id)
+{
+    return document.getElementById(id);
+}
+
+function getSelectValue(id) 
+{
+    var select = $(id);
+    if (select.selectedIndex == -1) {
+        return null;
+    } else {
+        return select.options[select.selectedIndex].value;
+    }
+}
+
+function loadText(url, callback)
+{
+    var xhr = new XMLHttpRequest();
+    xhr.open('GET', url);
+    xhr.addEventListener('load', function() { callback(xhr.responseText); });
+    xhr.send();
+}
index 0a37677b309d16e6d551f0b049575c16fa6d23e2..3aa464aa1cf843d08aa48b00665b301fa5feeba1 100644 (file)
@@ -46,12 +46,13 @@ from optparse import make_option
 from wsgiref.handlers import format_date_time
 
 from webkitpy.tool.multicommandtool import AbstractDeclarativeCommand
-
+import webkitpy.thirdparty.simplejson as simplejson
 
 class RebaselineHTTPServer(BaseHTTPServer.HTTPServer):
-    def __init__(self, httpd_port, results_directory):
+    def __init__(self, httpd_port, results_directory, results_json):
         BaseHTTPServer.HTTPServer.__init__(self, ("", httpd_port), RebaselineHTTPRequestHandler)
         self.results_directory = results_directory
+        self.results_json = results_json
 
 
 class RebaselineHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
@@ -59,6 +60,7 @@ class RebaselineHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
         "index.html",
         "main.js",
         "main.css",
+        "util.js",
     ])
 
     STATIC_FILE_DIRECTORY = os.path.join(
@@ -111,6 +113,38 @@ class RebaselineHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
         # otherwise there's a deadlock
         threading.Thread(target=lambda: self.server.shutdown()).start()
 
+    def test_result(self):
+        test_name, _ = os.path.splitext(self.query['test'][0])
+        mode = self.query['mode'][0]
+        if mode == 'expected-image':
+            file_name = test_name + '-expected.png'
+        elif mode == 'actual-image':
+            file_name = test_name + '-actual.png'
+        if mode == 'expected-checksum':
+            file_name = test_name + '-expected.checksum'
+        elif mode == 'actual-checksum':
+            file_name = test_name + '-actual.checksum'
+        elif mode == 'diff-image':
+            file_name = test_name + '-diff.png'
+        if mode == 'expected-text':
+            file_name = test_name + '-expected.txt'
+        elif mode == 'actual-text':
+            file_name = test_name + '-actual.txt'
+        elif mode == 'diff-text':
+            file_name = test_name + '-diff.txt'
+
+        file_path = os.path.join(self.server.results_directory, file_name)
+
+        # Let results be cached for 60 seconds, so that they can be pre-fetched
+        # by the UI
+        self._serve_file(file_path, cacheable_seconds=60)
+
+    def results_json(self):
+        self.send_response(200)
+        self.send_header('Content-type', 'application/json')
+        self.end_headers()
+        simplejson.dump(self.server.results_json, self.wfile)
+
     def _serve_file(self, file_path, cacheable_seconds=0):
         if not os.path.exists(file_path):
             self.send_error(404, "File not found")
@@ -147,11 +181,19 @@ class RebaselineServer(AbstractDeclarativeCommand):
     def execute(self, options, args, tool):
         results_directory = args[0]
 
+        print 'Parsing unexpected_results.json...'
+        results_json_path = os.path.join(
+            results_directory, 'unexpected_results.json')
+        with codecs.open(results_json_path, "r") as results_json_file:
+            results_json_file = file(results_json_path)
+            results_json = simplejson.load(results_json_file)
+
         print "Starting server at http://localhost:%d/" % options.httpd_port
         print ("Use the 'Exit' link in the UI, http://localhost:%d/"
             "quitquitquit or Ctrl-C to stop") % options.httpd_port
 
         httpd = RebaselineHTTPServer(
             httpd_port=options.httpd_port,
-            results_directory=results_directory)
+            results_directory=results_directory,
+            results_json=results_json)
         httpd.serve_forever()