run-jsc-stress-tests should be better at telling you details about test failures
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 4 Feb 2016 21:14:54 +0000 (21:14 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 4 Feb 2016 21:14:54 +0000 (21:14 +0000)
https://bugs.webkit.org/show_bug.cgi?id=153845

Reviewed by Geoffrey Garen.

Here's an example of the contents of results/resultsByFamily, one of the new files that this
generates, for some local testing I'm doing.  Notice how it groups families of tests together.

You won't see this unless you cat results/resultsByFamily.

regress.yaml/Regress/radar-24289839.js:
    regress.yaml/Regress/radar-24289839.js.default: PASS
    regress.yaml/Regress/radar-24289839.js.ftl: PASS
    regress.yaml/Regress/radar-24289839.js.ftl-no-cjit: FAIL
    regress.yaml/Regress/radar-24289839.js.ftl-eager-no-cjit: FAIL

regress.yaml/Regress/radar-24290639.js:
    regress.yaml/Regress/radar-24290639.js.default: PASS
    regress.yaml/Regress/radar-24290639.js.ftl: PASS
    regress.yaml/Regress/radar-24290639.js.ftl-no-cjit: PASS
    regress.yaml/Regress/radar-24290639.js.ftl-eager-no-cjit: FAIL

regress.yaml/Regress/radar-24290670.js: FAILED

* Scripts/run-jsc-stress-tests:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@196143 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/ChangeLog
Tools/Scripts/run-jsc-stress-tests

index 4d61749..a432c5c 100644 (file)
@@ -1,3 +1,31 @@
+2016-02-03  Filip Pizlo  <fpizlo@apple.com>
+
+        run-jsc-stress-tests should be better at telling you details about test failures
+        https://bugs.webkit.org/show_bug.cgi?id=153845
+
+        Reviewed by Geoffrey Garen.
+
+        Here's an example of the contents of results/resultsByFamily, one of the new files that this
+        generates, for some local testing I'm doing.  Notice how it groups families of tests together.
+
+        You won't see this unless you cat results/resultsByFamily.
+
+        regress.yaml/Regress/radar-24289839.js:
+            regress.yaml/Regress/radar-24289839.js.default: PASS
+            regress.yaml/Regress/radar-24289839.js.ftl: PASS
+            regress.yaml/Regress/radar-24289839.js.ftl-no-cjit: FAIL
+            regress.yaml/Regress/radar-24289839.js.ftl-eager-no-cjit: FAIL
+        
+        regress.yaml/Regress/radar-24290639.js:
+            regress.yaml/Regress/radar-24290639.js.default: PASS
+            regress.yaml/Regress/radar-24290639.js.ftl: PASS
+            regress.yaml/Regress/radar-24290639.js.ftl-no-cjit: PASS
+            regress.yaml/Regress/radar-24290639.js.ftl-eager-no-cjit: FAIL
+        
+        regress.yaml/Regress/radar-24290670.js: FAILED
+        
+        * Scripts/run-jsc-stress-tests:
+
 2016-02-03  Carlos Garcia Campos  <cgarcia@igalia.com>
 
         [WTR] Crash in EventSendingController::contextClick() when context menu event is not handled
index c871a18..41db5d4 100755 (executable)
@@ -37,6 +37,12 @@ module URI
     @@schemes['SSH'] = SSH
 end
 
+class String
+    def scrub
+        encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
+    end
+end
+
 THIS_SCRIPT_PATH = Pathname.new(__FILE__).realpath
 SCRIPTS_PATH = THIS_SCRIPT_PATH.dirname
 WEBKIT_PATH = SCRIPTS_PATH.dirname.dirname
@@ -387,6 +393,7 @@ if !$testRunnerType
 end
 
 $numFailures = 0
+$numPasses = 0
 
 BASE_OPTIONS = ["--useFTLJIT=false", "--useFunctionDotArguments=true"]
 EAGER_OPTIONS = ["--thresholdForJITAfterWarmUp=10", "--thresholdForJITSoon=10", "--thresholdForOptimizeAfterWarmUp=20", "--thresholdForOptimizeAfterLongWarmUp=20", "--thresholdForOptimizeSoon=20", "--thresholdForFTLOptimizeAfterWarmUp=20", "--thresholdForFTLOptimizeSoon=20", "--maximumEvalCacheableSourceLength=150000"]
@@ -599,12 +606,13 @@ end
 $runCommandOptions = {}
 
 class Plan
-    attr_reader :directory, :arguments, :name, :outputHandler, :errorHandler
+    attr_reader :directory, :arguments, :family, :name, :outputHandler, :errorHandler
     attr_accessor :index
     
-    def initialize(directory, arguments, name, outputHandler, errorHandler)
+    def initialize(directory, arguments, family, name, outputHandler, errorHandler)
         @directory = directory
         @arguments = arguments
+        @family = family
         @name = name
         @outputHandler = outputHandler
         @errorHandler = errorHandler
@@ -689,7 +697,9 @@ def addRunCommand(kind, command, outputHandler, errorHandler)
     if $filter and name !~ $filter
         return
     end
-    plan = Plan.new($benchmarkDirectory, command, name, outputHandler, errorHandler)
+    plan = Plan.new(
+        $benchmarkDirectory, command, "#{$collectionName}/#{$benchmark}", name, outputHandler,
+        errorHandler)
     if $numChildProcesses > 1 and $runCommandOptions[:isSlow]
         $runlist.unshift plan
     else
@@ -1411,6 +1421,21 @@ def appendFailure(plan)
     $numFailures += 1
 end
 
+def appendPass(plan)
+    File.open($outputDir + "passed", "a") {
+        | outp |
+        outp.puts plan.name
+    }
+    $numPasses += 1
+end
+
+def appendResult(plan, didPass)
+    File.open($outputDir + "results", "a") {
+        | outp |
+        outp.puts "#{plan.name}: #{didPass ? 'PASS' : 'FAIL'}"
+    }
+end
+
 def prepareBundle
     raise if $bundle
 
@@ -1688,7 +1713,7 @@ def runAndMonitorTestRunnerCommand(*cmd)
            | inp |
            inp.each_line {
                | line |
-               line.chomp!
+               line = line.scrub.chomp
                if line =~ /^Running /
                    running[$~.post_match] = true
                elsif line =~ /^PASS: /
@@ -1774,20 +1799,82 @@ end
 def detectFailures
     raise if $bundle
 
+    failures = []
+    
     if $remote
         output = sshRead("cd #{$remoteDirectory}/#{$outputDir.basename}/.runner && find . -maxdepth 1 -name \"test_fail_*\"")
         output.split(/\n/).each {
             | line |
             next unless line =~ /test_fail_/
-            appendFailure($runlist[$~.post_match.to_i])
+            failures << $~.post_match.to_i
         }
     else
         Dir.foreach($runnerDir) {
             | filename |
             next unless filename =~ /test_fail_/
-            appendFailure($runlist[$~.post_match.to_i])
+            failures << $~.post_match.to_i
         }
     end
+
+    failureSet = {}
+
+    failures.each {
+        | failure | 
+        appendFailure($runlist[failure])
+        failureSet[failure] = true
+    }
+
+    familyMap = {}
+    $runlist.each_with_index {
+        | plan, index |
+        unless familyMap[plan.family]
+            familyMap[plan.family] = []
+        end
+        if failureSet[index]
+            appendResult(plan, false)
+            familyMap[plan.family] << {:result => "FAIL", :plan => plan};
+            next
+        else
+            appendResult(plan, true)
+            familyMap[plan.family] << {:result => "PASS", :plan => plan};
+        end
+        appendPass(plan)
+    }
+
+    File.open($outputDir + "resultsByFamily", "w") {
+        | outp |
+        first = true
+        familyMap.keys.sort.each {
+            | familyName |
+            if first
+                first = false
+            else
+                outp.puts
+            end
+            
+            outp.print "#{familyName}:"
+
+            numPassed = 0
+            familyMap[familyName].each {
+                | entry |
+                if entry[:result] == "PASS"
+                    numPassed += 1
+                end
+            }
+
+            if numPassed == familyMap[familyName].size
+                outp.puts " PASSED"
+            elsif numPassed == 0
+                outp.puts " FAILED"
+            else
+                outp.puts
+                familyMap[familyName].each {
+                    | entry |
+                    outp.puts "    #{entry[:plan].name}: #{entry[:result]}"
+                }
+            end
+        }
+    }
 end
 
 def compressBundle
@@ -1801,6 +1888,9 @@ def clean(file)
 end
 
 clean($outputDir + "failed")
+clean($outputDir + "passed")
+clean($outputDir + "results")
+clean($outputDir + "resultsByFamily")
 clean($outputDir + ".vm")
 clean($outputDir + ".helpers")
 clean($outputDir + ".runner")