API test harness should be Python
authorjbedard@apple.com <jbedard@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 25 Apr 2018 16:35:34 +0000 (16:35 +0000)
committerjbedard@apple.com <jbedard@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 25 Apr 2018 16:35:34 +0000 (16:35 +0000)
https://bugs.webkit.org/show_bug.cgi?id=181043
<rdar://problem/36164410>

Reviewed by David Kilzer.

Run API tests from a Python script rather than Perl. The Python API tests have
the same structure as the layout tests.

* Scripts/run-api-tests: Changed from Perl to Python.
* Scripts/webkitpy/api_tests: Added.
* Scripts/webkitpy/api_tests/__init__.py: Added.
* Scripts/webkitpy/api_tests/manager.py: Added.
(Manager):
(Manager.__init__):
(Manager._test_list_from_output): Construct a list of tests from the output of
TestWTF and TestWebKitAPI.
(Manager._find_test_subset): Given a list of all possible tests and command line
arguments, return a list of tests which match the provided arguments.
(Manager._collect_tests): Run TestWTF and TestWebKitAPI with --gtest_list_tests
to determine which tests will be run.
(Manager._print_test_result): Formatted print of a test with it's output.
(Manager._print_tests_result_with_status): Print all tests and output for tests
which ran with a specified exit status.
(Manager.run): Collect all tests, run them and then print results from the run.
* Scripts/webkitpy/api_tests/run_api_tests.py: Added.
(main): Parse arguments, construct a port, early exit for illegal ports and
then run tests.
(run): Setup logging and printing before using the Manager to run tests.
(parse_args): Use argparse to define all options used by run-api-tests.
* Scripts/webkitpy/api_tests/runner.py: Added.
(Runner):
(Runner.__init__):
(Runner._shard_tests): Split tests so that each suite runs together.
(Runner.command_for_port): Run a command on the device specified by the given
port object. Once <https://bugs.webkit.org/show_bug.cgi?id=175204> is completed,
this function would be entirely replaced by the server_process factory owned
by the port object.
(Runner.run): Run all shards in the message_pool.
(Runner.handle): Handle reports from child processes.
(Runner.result_map_by_status): Return a mapping of tests to output for tests
which exit with a specific status.
(_Worker):
(_Worker.__init__):
(_Worker._filter_noisy_output): Filter out objc warnings since these polite the log.
(_Worker._run_single_test): Runs a single test in a single child process.
(_Worker._run_shard_with_binary): Attempts to run a group of tests in the same
child process. If this technique fails, the remaining tests are run singly.
(_Worker.handle): Run the specified shard.
* Scripts/webkitpy/port/base.py:
(Port):
(Port.check_api_test_build): Check if TestWTF and TestWebKitAPI are built.
(Port.environment_for_api_tests): Return the environment needed to run
the TestWebKitAPI binary.
(Port. path_to_api_test_binaries): Return a list of the path to all binaries
used when running API tests.
(Port._build_api_tests): Build TestWTF and TestWebKitAPI if required.
* Scripts/webkitpy/port/mac.py:
(MacPort.environment_for_api_tests): Enable GuardMalloc for API tests.
* Scripts/webkitpy/port/server_process.py:
(ServerProcess.pop_all_buffered_stdout):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@230998 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/ChangeLog
Tools/Scripts/run-api-tests
Tools/Scripts/webkitpy/api_tests/__init__.py [new file with mode: 0644]
Tools/Scripts/webkitpy/api_tests/manager.py [new file with mode: 0644]
Tools/Scripts/webkitpy/api_tests/run_api_tests.py [new file with mode: 0644]
Tools/Scripts/webkitpy/api_tests/runner.py [new file with mode: 0644]
Tools/Scripts/webkitpy/port/base.py
Tools/Scripts/webkitpy/port/ios_simulator.py
Tools/Scripts/webkitpy/port/mac.py
Tools/Scripts/webkitpy/port/server_process.py

index 2ef2874..a40cf3c 100644 (file)
@@ -1,3 +1,67 @@
+2018-04-25  Jonathan Bedard  <jbedard@apple.com>
+
+        API test harness should be Python
+        https://bugs.webkit.org/show_bug.cgi?id=181043
+        <rdar://problem/36164410>
+
+        Reviewed by David Kilzer.
+
+        Run API tests from a Python script rather than Perl. The Python API tests have
+        the same structure as the layout tests.
+
+        * Scripts/run-api-tests: Changed from Perl to Python.
+        * Scripts/webkitpy/api_tests: Added.
+        * Scripts/webkitpy/api_tests/__init__.py: Added.
+        * Scripts/webkitpy/api_tests/manager.py: Added.
+        (Manager):
+        (Manager.__init__):
+        (Manager._test_list_from_output): Construct a list of tests from the output of
+        TestWTF and TestWebKitAPI.
+        (Manager._find_test_subset): Given a list of all possible tests and command line
+        arguments, return a list of tests which match the provided arguments.
+        (Manager._collect_tests): Run TestWTF and TestWebKitAPI with --gtest_list_tests
+        to determine which tests will be run.
+        (Manager._print_test_result): Formatted print of a test with it's output.
+        (Manager._print_tests_result_with_status): Print all tests and output for tests
+        which ran with a specified exit status.
+        (Manager.run): Collect all tests, run them and then print results from the run.
+        * Scripts/webkitpy/api_tests/run_api_tests.py: Added.
+        (main): Parse arguments, construct a port, early exit for illegal ports and
+        then run tests.
+        (run): Setup logging and printing before using the Manager to run tests.
+        (parse_args): Use argparse to define all options used by run-api-tests.
+        * Scripts/webkitpy/api_tests/runner.py: Added.
+        (Runner):
+        (Runner.__init__):
+        (Runner._shard_tests): Split tests so that each suite runs together.
+        (Runner.command_for_port): Run a command on the device specified by the given
+        port object. Once <https://bugs.webkit.org/show_bug.cgi?id=175204> is completed,
+        this function would be entirely replaced by the server_process factory owned
+        by the port object.
+        (Runner.run): Run all shards in the message_pool.
+        (Runner.handle): Handle reports from child processes.
+        (Runner.result_map_by_status): Return a mapping of tests to output for tests
+        which exit with a specific status.
+        (_Worker):
+        (_Worker.__init__):
+        (_Worker._filter_noisy_output): Filter out objc warnings since these polite the log.
+        (_Worker._run_single_test): Runs a single test in a single child process.
+        (_Worker._run_shard_with_binary): Attempts to run a group of tests in the same
+        child process. If this technique fails, the remaining tests are run singly.
+        (_Worker.handle): Run the specified shard.
+        * Scripts/webkitpy/port/base.py:
+        (Port):
+        (Port.check_api_test_build): Check if TestWTF and TestWebKitAPI are built.
+        (Port.environment_for_api_tests): Return the environment needed to run
+        the TestWebKitAPI binary.
+        (Port. path_to_api_test_binaries): Return a list of the path to all binaries
+        used when running API tests.
+        (Port._build_api_tests): Build TestWTF and TestWebKitAPI if required.
+        * Scripts/webkitpy/port/mac.py:
+        (MacPort.environment_for_api_tests): Enable GuardMalloc for API tests.
+        * Scripts/webkitpy/port/server_process.py: 
+        (ServerProcess.pop_all_buffered_stdout):
+
 2018-04-25  Carlos Garcia Campos  <cgarcia@igalia.com>
 
         Incorrect number of WebDriver test failures shown in the bots
index cf0b92c..6c524ec 100755 (executable)
-#!/usr/bin/env perl
+#!/usr/bin/env python
 
-# Copyright (C) 2010-2012, 2014-2015 Apple Inc. All rights reserved.
+# Copyright (C) 2018 Apple Inc. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted provided that the following conditions
 # are met:
-# 1. Redistributions of source code must retain the above copyright
+# 1.  Redistributions of source code must retain the above copyright
 #    notice, this list of conditions and the following disclaimer.
-# 2. Redistributions in binary form must reproduce the above copyright
+# 2.  Redistributions in binary form must reproduce the above copyright
 #    notice, this list of conditions and the following disclaimer in the
 #    documentation and/or other materials provided with the distribution.
 #
-# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
-# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
-# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
-# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
-# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
-# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
-# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
-# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
-# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
-# THE POSSIBILITY OF SUCH DAMAGE.
-
-use strict;
-use warnings;
-
-use File::Basename;
-use File::Spec;
-use FindBin;
-use Getopt::Long qw(:config pass_through);
-use IPC::Open3;
-use JSON::PP;
-use lib $FindBin::Bin;
-use sigtrap qw(die normal-signals);
-use webkitdirs;
-use VCSUtils;
-
-sub buildTestTool();
-sub dumpTestsBySuite(\@);
-sub listAllTests();
-sub runTest($$);
-sub runTestsBySuite(\@);
-sub prepareEnvironmentForRunningTestTool();
-sub archCommandLineArgumentsForRestrictedEnvironmentVariables();
-sub testToolPaths();
-sub writeJSONDataIfApplicable();
-
-# Defined in VCSUtils.
-sub possiblyColored($$);
-
-# Timeout for individual test, in sec
-my $timeout = 30;
-
-my $showHelp = 0;
-my $verbose = 0;
-my $showLeaks = 0;
-my $dumpTests = 0;
-my $disableTimeout = 0;
-my $build = 1;
-my $root;
-my $buildDefault = $build ? "build" : "do not build";
-my @testsFailed;
-my @testsTimedOut;
-my $wtfOnly = 0;
-my %testToToolMap;
-my %jsonData = ();
-my $jsonFileName;
-
-
-my $programName = basename($0);
-my $usage = <<EOF;
-Usage: $programName [options] [suite or test prefixes]
-  --help                Show this help message
-  -v|--verbose          Verbose output
-  -d|--dump-tests       Dump the names of testcases without running them
-  --[no-]build          Build (or do not build) unit tests prior to running (default: $buildDefault)
-  --json-output=        Create a file at the specified path, listing test failures and timeouts in JSON format.
-  --root=               Path to the pre-built root containing TestWebKitAPI
-  --show-leaks          Show leaks in the output
-  --no-timeout          Disable test timeouts
-  --wtf-only            Only build and run TestWTF
-
-Platform options:
-  --ios-simulator       Run tests in the iOS Simulator
-  --simulator           DEPRECATED alias of --ios-simulator
-
-@{[ sharedCommandLineOptionsUsage(indent => 2, switchWidth => 21) ]}
-Examples
-
-The following command will run a single test:
-    $programName WebKit.AboutBlank
-
-The following command will run all tests in suites that begin with 'WebKit':
-    $programName WebKit
-
-EOF
-
-my $getOptionsResult = GetOptions(
-    sharedCommandLineOptions(),
-    'help' => \$showHelp,
-    'verbose|v' => \$verbose,
-    'show-leaks' => \$showLeaks,
-    'no-timeout' => \$disableTimeout,
-    'json-output=s' => \$jsonFileName,
-    'dump|d' => \$dumpTests,
-    'build!' => \$build,
-    'root=s' => \$root,
-    'wtf-only' => \$wtfOnly,
-);
-
-if (!$getOptionsResult || $showHelp) {
-   print STDERR $usage;
-   exit 1;
-}
-
-setConfiguration();
-
-setConfigurationProductDir(Cwd::abs_path($root)) if (defined($root));
-
-if (defined($jsonFileName)) {
-    $jsonFileName = File::Spec->rel2abs($jsonFileName);
-}
-
-buildTestTool() if $build && !defined($root);
-setPathForRunningWebKitApp(\%ENV);
-
-my $simulatorDevice;
-if (willUseIOSSimulatorSDK()) {
-    $simulatorDevice = findOrCreateSimulatorForIOSDevice(SIMULATOR_DEVICE_SUFFIX_FOR_WEBKIT_DEVELOPMENT);
-    relaunchIOSSimulator($simulatorDevice);
-}
-my @testsToRun = listAllTests();
-
-@testsToRun = grep { my $test = $_; grep { $test =~ m/^\Q$_\E/ } @ARGV; } @testsToRun if @ARGV;
-
-if ($dumpTests) {
-    dumpTestsBySuite(@testsToRun);
-    exit 0;
-}
-
-END { shutDownIOSSimulatorDevice($simulatorDevice) if $simulatorDevice; }
-
-exit runTestsBySuite(@testsToRun);
-
-sub isSupportedPlatform()
-{
-    return isAppleCocoaWebKit() || isAppleWinWebKit();
-}
-
-sub dumpTestsBySuite(\@)
-{
-    my ($tests) = @_;
-    print "Dumping test cases\n";
-    print "------------------\n";
-    my $lastSuite = "";
-    for my $suiteAndTest (sort @$tests) {
-        my ($suite, $test) = split(/\./, $suiteAndTest);
-        if ($lastSuite ne $suite) {
-            $lastSuite = $suite;
-            print "$suite:\n";
-        }
-        print "   $test\n";
-    }
-    print "------------------\n";
-}
-
-sub runTestsBySuite(\@)
-{
-    my ($tests) = @_;
-    for my $suiteAndTest (sort @$tests) {
-        my ($suite, $test) = split(/\./, $suiteAndTest);
-        runTest($suite, $test);
-    }
-
-    if (@testsFailed) {
-        print "\nTests that failed:\n";
-        for my $test (@testsFailed) {
-            print "  $test\n";
-        }
-    }
-    if (@testsTimedOut) {
-        print "\nTests that timed out:\n";
-        for my $test (@testsTimedOut) {
-            print "  $test\n";
-        }
-    }
-
-    if (defined($jsonFileName)) {
-        $jsonData{'failures'} = \@testsFailed;
-        $jsonData{'timeouts'} = \@testsTimedOut;
-    }
-
-    writeJSONDataIfApplicable();
-
-    return @testsFailed > 0 || @testsTimedOut > 0;
-}
-
-sub runTest($$)
-{
-    my ($suite, $testName) = @_;
-    my $test = $suite . "." . $testName;
-
-    my $gtestArg = "--gtest_filter=" . $test;
-
-    my $result = 0;
-    my $timedOut = 0;
-
-    die "run-api-tests is not supported on this platform.\n" unless isSupportedPlatform();
-
-    local %ENV = %ENV;
-    prepareEnvironmentForRunningTestTool();
-
-    local *DEVNULL;
-    my ($childIn, $childOut, $childErr);
-    if ($verbose || $showLeaks) {
-        $childErr = 0;
-    } else {
-        open(DEVNULL, ">", File::Spec->devnull()) or die "Failed to open /dev/null";
-        $childErr = ">&DEVNULL";
-    }
-
-    my $pid;
-    my @commonArguments = ($testToToolMap{$test}, $gtestArg, @ARGV);
-    if (willUseIOSSimulatorSDK()) {
-        $pid = open3($childIn, $childOut, $childErr, qw(xcrun --sdk iphonesimulator simctl spawn), $simulatorDevice->{UDID}, @commonArguments) or die "Failed to run test: $test.";
-    } elsif (isAppleCocoaWebKit() && architecture()) {
-        $pid = open3($childIn, $childOut, $childErr, "arch", "-" . architecture(), archCommandLineArgumentsForRestrictedEnvironmentVariables(), @commonArguments) or die "Failed to run test: $test.";
-    } else {
-        $pid = open3($childIn, $childOut, $childErr, @commonArguments) or die "Failed to run test: $test.";
-    }
-
-    eval {
-        if ($disableTimeout) {
-            waitpid($pid, 0);    
-        } else {
-            local $SIG{ALRM} = sub { die "alarm\n" };
-            alarm $timeout;
-            waitpid($pid, 0);
-            alarm 0;
-        }
-        $result = $?;
-    };
-    if ($@) {
-        die unless $@ eq "alarm\n";
-        kill SIGTERM, $pid or kill SIGKILL, $pid;
-        $timedOut = 1;
-    }
-
-    my @testOutput = <$childOut>;
-    @testOutput = grep { !/^LEAK:/ } @testOutput unless $showLeaks;
-    map { s/\*\*PASS\*\*/possiblyColored("bold green", "PASS")/eg } @testOutput;
-    map { s/\*\*FAIL\*\*/possiblyColored("bold red", "FAIL")/eg } @testOutput;
-
-    if ($result) {
-        push @testsFailed, $test;
-        if (!$timedOut && index("@testOutput", $test) == -1) {
-            print STDOUT possiblyColored("bold red", "UNEXPECTEDLY EXITED"), " $test\n";
-        }
-    } elsif ($timedOut) {
-        push @testsTimedOut, $test;
-        print STDOUT possiblyColored("bold yellow", "TIMEOUT"), " $test\n";
-    }
-
-    print STDOUT @testOutput;
-
-    close($childIn);
-    close($childOut);
-    close($childErr) unless ($verbose || $showLeaks);
-    close(DEVNULL) unless ($verbose || $showLeaks);
-
-    if ($timedOut || $result) {
-        return $timedOut || $result;
-    }
-
-    return 0;
-}
-
-sub listAllTests()
-{
-    my @toolOutput;
-    my $timedOut;
-
-    die "run-api-tests is not supported on this platform.\n" unless isSupportedPlatform();
-
-    prepareEnvironmentForRunningTestTool();
-
-    local *DEVNULL;
-    my ($childIn, $childOut, $childErr);
-    if ($verbose) {
-        $childErr = ">&STDERR";
-    } else {
-        open(DEVNULL, ">", File::Spec->devnull()) or die "Failed to open /dev/null";
-        $childErr = ">&DEVNULL";
-    }
-
-    my @tests = ();
-    foreach (testToolPaths()) {
-        my $pid;
-        my $testTool = $_;
-        my @commonArguments = ($testTool, "--gtest_list_tests");
-        if (isIOSWebKit()) {
-            $pid = open3($childIn, $childOut, $childErr, qw(xcrun --sdk iphonesimulator simctl spawn), $simulatorDevice->{UDID}, @commonArguments) or die "Failed to build list of tests!";
-        } elsif (isAppleCocoaWebKit() && architecture()) {
-            $pid = open3($childIn, $childOut, $childErr, "arch", "-" . architecture(), archCommandLineArgumentsForRestrictedEnvironmentVariables(), @commonArguments) or die "Failed to build list of tests!";
-        } else {
-            $pid = open3($childIn, $childOut, $childErr, @commonArguments) or die "Failed to build list of tests!";
-        }
-
-        close($childIn);
-        @toolOutput = <$childOut>;
-        close($childOut);
-        close($childErr);
-
-        waitpid($pid, 0);
-        my $result = $?;
-
-        if ($result) {
-            print STDERR "Failed to build list of tests!--\n";
-            exit exitStatus($result);
-        }
-
-        my $suite;
-        for my $line (@toolOutput) {
-           $line =~ s/[\r\n]*$//;
-           if ($line =~ m/\.$/) {
-              $suite = $line; # "SuiteName."
-           } else {
-              $line =~ s/^\s*//; # "TestName"
-              my $fullName = $suite . $line; # "SuiteName.TestName";
-              push @tests, $fullName;
-              $testToToolMap{$fullName} = $testTool;
-            }
-        }
-    }
-
-    close(DEVNULL) unless ($verbose);
-
-    return @tests;
-}
-
-sub buildTestTool()
-{
-    my $originalCwd = getcwd();
-
-    chdirWebKit();
-
-    my $buildTestTool = "build-api-tests";
-    print STDERR "Running $buildTestTool\n";
-
-    local *DEVNULL;
-    my ($childIn, $childOut, $childErr);
-    if ($verbose) {
-        # When not quiet, let the child use our stdout/stderr.
-        $childOut = ">&STDOUT";
-        $childErr = ">&STDERR";
-    } else {
-        open(DEVNULL, ">", File::Spec->devnull()) or die "Failed to open /dev/null";
-        $childOut = ">&DEVNULL";
-        $childErr = ">&DEVNULL";
-    }
-
-    my @args = argumentsForConfiguration();
-    if ($wtfOnly) {
-        push @args, "--wtf-only";
-    }
-
-    my $pathToBuildTestTool = File::Spec->catfile("Tools", "Scripts", $buildTestTool);
-    my $buildProcess = open3($childIn, $childOut, $childErr, "perl", $pathToBuildTestTool, @args) or die "Failed to run " . $buildTestTool;
-
-    close($childIn);
-    close($childOut);
-    close($childErr);
-    close(DEVNULL) unless ($verbose);
-
-    waitpid($buildProcess, 0);
-    my $buildResult = $?;
-
-    if ($buildResult) {
-        print STDERR "Compiling TestWebKitAPI failed!\n";
-        exit exitStatus($buildResult);
-    }
-
-    chdir $originalCwd;
-}
-
-sub prepareEnvironmentForRunningTestTool()
-{
-    return unless isAppleCocoaWebKit();
-
-    if (willUseIOSSimulatorSDK()) {
-        my %simulatorENV;
-        {
-            local %ENV;
-            setupIOSWebKitEnvironment(productDir());
-            %simulatorENV = %ENV;
-        }
-        # Prefix the environment variables with SIMCTL_CHILD_ per `xcrun simctl help launch`.
-        foreach my $key (keys %simulatorENV) {
-            $ENV{"SIMCTL_CHILD_$key"} = $simulatorENV{$key};
-        }
-        return;
-    }
-    setupMacWebKitEnvironment(productDir());
-}
-
-sub testToolPaths()
-{
-    if (!isAppleWinWebKit()) {
-        my @toolPaths = ();
-        if (!$wtfOnly) {
-            push @toolPaths, File::Spec->catfile(productDir(), "TestWebKitAPI");
-        }
-        push @toolPaths, File::Spec->catfile(productDir(), "TestWTF");
-        return @toolPaths;
-    }
-
-    my $binDir = isWin64() ? "bin64" : "bin32";
-    my $pathWTF = File::Spec->catfile(productDir(), $binDir, "TestWTF");
-    my $pathWebCore = File::Spec->catfile(productDir(), $binDir, "TestWebCore");
-    my $pathWebKit = File::Spec->catfile(productDir(), $binDir, "TestWebKitLegacy");
-
-    my $suffix;
-    if (configuration() eq "Debug_All") {
-        $suffix = "_debug";
-    } else {
-        $suffix = "";
-    }
-    return ("$pathWTF$suffix.exe", "$pathWebCore$suffix.exe", "$pathWebKit$suffix.exe");
-}
-
-sub writeJSONDataIfApplicable()
-{
-    if (defined($jsonFileName)) {
-        open(my $fileHandler, ">", $jsonFileName) or die;
-        print $fileHandler "${\encode_json(\%jsonData)}\n";
-        close($fileHandler);
-    }
-}
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Wrapper around webkitpy/api_tests/run_api_tests.py"""
+from webkitpy.common import multiprocessing_bootstrap
+
+multiprocessing_bootstrap.run('webkitpy', 'api_tests', 'run_api_tests.py')
diff --git a/Tools/Scripts/webkitpy/api_tests/__init__.py b/Tools/Scripts/webkitpy/api_tests/__init__.py
new file mode 100644 (file)
index 0000000..b376bf2
--- /dev/null
@@ -0,0 +1,13 @@
+# Required for Python to search this directory for module files
+
+# Keep this file free of any code or import statements that could
+# cause either an error to occur or a log message to be logged.
+# This ensures that calling code can import initialization code from
+# webkitpy before any errors or log messages due to code in this file.
+# Initialization code can include things like version-checking code and
+# logging configuration code.
+#
+# We do not execute any version-checking code or logging configuration
+# code in this file so that callers can opt-in as they want.  This also
+# allows different callers to choose different initialization code,
+# as necessary.
diff --git a/Tools/Scripts/webkitpy/api_tests/manager.py b/Tools/Scripts/webkitpy/api_tests/manager.py
new file mode 100644 (file)
index 0000000..deed311
--- /dev/null
@@ -0,0 +1,191 @@
+# Copyright (C) 2018 Apple Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import logging
+import os
+
+from webkitpy.api_tests.runner import Runner
+from webkitpy.common.system.executive import ScriptError
+from webkitpy.xcode.device_type import DeviceType
+from webkitpy.xcode.simulated_device import DeviceRequest, SimulatedDeviceManager
+
+_log = logging.getLogger(__name__)
+
+
+class Manager(object):
+    """A class for managing running API and WTF tests
+    """
+
+    SUCCESS = 0
+    FAILED_BUILD_CHECK = 1
+    FAILED_COLLECT_TESTS = 2
+    FAILED_TESTS = 3
+
+    def __init__(self, port, options, stream):
+        self._port = port
+        self.host = port.host
+        self._options = options
+        self._stream = stream
+
+    @staticmethod
+    def _test_list_from_output(output, prefix=''):
+        result = []
+        current_test_suite = None
+        for line in output.split('\n'):
+            striped_line = line.lstrip().rstrip()
+            if not striped_line:
+                continue
+
+            if striped_line[-1] == '.':
+                current_test_suite = striped_line[:-1]
+            else:
+                striped_line = striped_line.lstrip()
+                if ' ' in striped_line:
+                    continue
+                val = '{}{}.{}'.format(prefix, current_test_suite, striped_line)
+                if val not in result:
+                    result.append(val)
+        return result
+
+    @staticmethod
+    def _find_test_subset(superset, arg_filter):
+        result = []
+        for arg in arg_filter:
+            split_arg = arg.split('.')
+            for test in superset:
+                # Might match <binary>.<suite>.<test> or just <suite>.<test>
+                split_test = test.split('.')
+                if len(split_arg) == 1:
+                    if test not in result and (arg == split_test[0] or arg == split_test[1]):
+                        result.append(test)
+                elif len(split_arg) == 2:
+                    if test not in result and (split_arg == split_test[0:2] or split_arg == split_test[1:3]):
+                        result.append(test)
+                else:
+                    if arg == test and test not in result:
+                        result.append(test)
+        return result
+
+    def _collect_tests(self, args):
+        available_tests = []
+        for binary in self._port.path_to_api_test_binaries():
+            stripped_name = os.path.splitext(os.path.basename(binary))[0]
+            try:
+                output = self.host.executive.run_command(
+                    Runner.command_for_port(self._port, [binary, '--gtest_list_tests']),
+                    env=self._port.environment_for_api_tests())
+                available_tests += Manager._test_list_from_output(output, '{}.'.format(stripped_name))
+            except ScriptError:
+                _log.error('Failed to list {} tests'.format(stripped_name))
+                raise
+
+        if len(args) == 0:
+            return sorted(available_tests)
+        return sorted(Manager._find_test_subset(available_tests, args))
+
+    @staticmethod
+    def _print_test_result(stream, test_name, output):
+        stream.writeln('    {}'.format(test_name))
+        has_output = False
+        for line in output.splitlines():
+            stream.writeln('        {}'.format(line))
+            has_output = True
+        if has_output:
+            stream.writeln('')
+        return not has_output
+
+    def _print_tests_result_with_status(self, status, runner):
+        mapping = runner.result_map_by_status(status)
+        if mapping:
+            self._stream.writeln(runner.NAME_FOR_STATUS[status])
+            self._stream.writeln('')
+            need_newline = False
+            for test, output in mapping.iteritems():
+                need_newline = Manager._print_test_result(self._stream, test, output)
+            if need_newline:
+                self._stream.writeln('')
+
+    def _initialize_devices(self):
+        if 'simulator' in self._port.port_name:
+            SimulatedDeviceManager.initialize_devices(DeviceRequest(DeviceType.from_string(self._port.DEFAULT_DEVICE_CLASS), allow_incomplete_match=True), self.host, simulator_ui=False)
+        elif 'device' in self._port.port_name:
+            raise RuntimeError('Running api tests on {} is not supported'.format(self._port.port_name))
+
+    def run(self, args):
+        self._stream.write_update('Checking build ...')
+        if not self._port.check_api_test_build():
+            _log.error('Build check failed')
+            return Manager.FAILED_BUILD_CHECK
+
+        self._initialize_devices()
+
+        self._stream.write_update('Collecting tests ...')
+        try:
+            test_names = self._collect_tests(args)
+        except ScriptError:
+            self._stream.writeln('Failed to collect tests')
+            return Manager.FAILED_COLLECT_TESTS
+        self._stream.write_update('Found {} tests'.format(len(test_names)))
+        if len(test_names) == 0:
+            self._stream.writeln('No tests found')
+            return Manager.FAILED_COLLECT_TESTS
+
+        if self._port.get_option('dump'):
+            for test in test_names:
+                self._stream.writeln(test)
+            return Manager.SUCCESS
+
+        try:
+            _log.info('Running tests')
+            runner = Runner(self._port, self._stream)
+            runner.run(test_names, int(self._options.child_processes) if self._options.child_processes else self._port.default_child_processes())
+        except KeyboardInterrupt:
+            # If we receive a KeyboardInterrupt, print results.
+            self._stream.writeln('')
+
+        successful = runner.result_map_by_status(runner.STATUS_PASSED)
+        _log.info('Ran {} tests of {} with {} successful'.format(len(runner.results), len(test_names), len(successful)))
+
+        self._stream.writeln('------------------------------')
+        if len(successful) + len(runner.result_map_by_status(runner.STATUS_DISABLED)) == len(test_names):
+            self._stream.writeln('All tests successfully passed!')
+            return Manager.SUCCESS
+
+        self._stream.writeln('Test suite failed')
+        self._stream.writeln('')
+
+        skipped = []
+        for test in test_names:
+            if test not in runner.results:
+                skipped.append(test)
+        if skipped:
+            self._stream.writeln('Skipped {} tests'.format(len(skipped)))
+            self._stream.writeln('')
+            if self._options.verbose:
+                for test in skipped:
+                    self._stream.writeln('    {}'.format(test))
+
+        self._print_tests_result_with_status(runner.STATUS_FAILED, runner)
+        self._print_tests_result_with_status(runner.STATUS_CRASHED, runner)
+        self._print_tests_result_with_status(runner.STATUS_TIMEOUT, runner)
+
+        return Manager.FAILED_TESTS
diff --git a/Tools/Scripts/webkitpy/api_tests/run_api_tests.py b/Tools/Scripts/webkitpy/api_tests/run_api_tests.py
new file mode 100644 (file)
index 0000000..7b42c6b
--- /dev/null
@@ -0,0 +1,134 @@
+# Copyright (C) 2018 Apple Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+from __future__ import print_function
+import logging
+import optparse
+import sys
+import traceback
+
+from webkitpy.api_tests.manager import Manager
+from webkitpy.common.host import Host
+from webkitpy.layout_tests.views.metered_stream import MeteredStream
+from webkitpy.port import configuration_options, platform_options
+
+EXCEPTIONAL_EXIT_STATUS = -1
+INTERRUPT_EXIT_STATUS = -2
+
+_log = logging.getLogger(__name__)
+
+
+def main(argv, stdout, stderr):
+    options, args = parse_args(argv)
+    host = Host()
+
+    try:
+        options.webkit_test_runner = True
+        port = host.port_factory.get(options.platform, options)
+    except NotImplementedError as e:
+        print(str(e), file=stderr)
+        return EXCEPTIONAL_EXIT_STATUS
+
+    # Some platforms do not support API tests
+    does_not_support_api_tests = ['ios-device']
+    if port.operating_system() in does_not_support_api_tests:
+        print('{} cannot run API tests'.format(port.operating_system()), file=stderr)
+        return EXCEPTIONAL_EXIT_STATUS
+
+    try:
+        return run(port, options, args, stderr)
+    except KeyboardInterrupt:
+        return INTERRUPT_EXIT_STATUS
+    except BaseException as e:
+        if isinstance(e, Exception):
+            print('\n%s raised: %s' % (e.__class__.__name__, str(e)), file=stderr)
+            traceback.print_exc(file=stderr)
+        return EXCEPTIONAL_EXIT_STATUS
+
+
+def run(port, options, args, logging_stream):
+    logger = logging.getLogger()
+    logger.setLevel(logging.DEBUG if options.verbose else logging.ERROR if options.quiet else logging.INFO)
+
+    try:
+        stream = MeteredStream(logging_stream, options.verbose, logger=logger, number_of_columns=port.host.platform.terminal_width())
+        manager = Manager(port, options, stream)
+
+        result = manager.run(args)
+        _log.debug("Testing completed, Exit status: %d" % result)
+        return result
+    finally:
+        stream.cleanup()
+
+
+def parse_args(args):
+    option_group_definitions = []
+
+    option_group_definitions.append(('Platform options', platform_options()))
+    option_group_definitions.append(('Configuration options', configuration_options()))
+    option_group_definitions.append(('Printing Options', [
+        optparse.make_option('-q', '--quiet', action='store_true', default=False,
+                             help='Run quietly (errors, warnings, and progress only)'),
+        optparse.make_option('-v', '--verbose', action='store_true', default=False,
+                             help='Enable verbose printing'),
+    ]))
+
+    option_group_definitions.append(('WebKit Options', [
+        optparse.make_option('-g', '--guard-malloc', action='store_true', default=False,
+                             help='Enable Guard Malloc (OS X only)'),
+        optparse.make_option('--root', action='store',
+                             help='Path to a directory containing the executables needed to run tests.'),
+    ]))
+
+    option_group_definitions.append(('Testing Options', [
+        optparse.make_option('-d', '--dump', action='store_true', default=False,
+                             help='Dump all test names without running them'),
+        optparse.make_option('--build', dest='build', action='store_true', default=True,
+                             help='Check to ensure the build is up-to-date (default).'),
+        optparse.make_option('--no-build', dest='build', action='store_false',
+                             help="Don't check to see if the build is up-to-date."),
+        optparse.make_option('--timeout', default=30,
+                             help='Number of seconds to wait before a test times out'),
+        optparse.make_option('--no-timeout', dest='timeout', action='store_false',
+                             help='Disable timeouts for all tests'),
+
+        # FIXME: Remove the default, API tests should be multiprocess
+        optparse.make_option('--child-processes', default=1,
+                             help='Number of processes to run in parallel.'),
+
+        # FIXME: Default should be false, API tests should not be forced to run singly
+        optparse.make_option('--run-singly', action='store_true', default=True,
+                             help='Run a separate process for each test'),
+    ]))
+
+    option_parser = optparse.OptionParser(usage='%prog [options] [<path>...]')
+
+    for group_name, group_options in option_group_definitions:
+        option_group = optparse.OptionGroup(option_parser, group_name)
+        option_group.add_options(group_options)
+        option_parser.add_option_group(option_group)
+
+    return option_parser.parse_args(args)
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:], sys.stdout, sys.stderr))
diff --git a/Tools/Scripts/webkitpy/api_tests/runner.py b/Tools/Scripts/webkitpy/api_tests/runner.py
new file mode 100644 (file)
index 0000000..5eedbc3
--- /dev/null
@@ -0,0 +1,240 @@
+# Copyright (C) 2018 Apple Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import os
+import time
+
+from webkitpy.common import message_pool
+from webkitpy.port.server_process import ServerProcess
+from webkitpy.xcode.simulated_device import SimulatedDeviceManager
+
+
+class Runner(object):
+    STATUS_PASSED = 0
+    STATUS_FAILED = 1
+    STATUS_CRASHED = 2
+    STATUS_TIMEOUT = 3
+    STATUS_DISABLED = 4
+
+    NAME_FOR_STATUS = [
+        'Passed',
+        'Failed',
+        'Crashed',
+        'Timeout',
+        'Disabled',
+    ]
+
+    def __init__(self, port, printer):
+        self.port = port
+        self.printer = printer
+        self.tests_run = 0
+        self.results = {}
+
+    @staticmethod
+    def _shard_tests(tests):
+        shards = {}
+        for test in tests:
+            shard_prefix = '.'.join(test.split('.')[:-1])
+            if shard_prefix not in shards:
+                shards[shard_prefix] = []
+            shards[shard_prefix].append(test)
+        return shards
+
+    # FIXME API tests should run as an app, we won't need this function <https://bugs.webkit.org/show_bug.cgi?id=175204>
+    @staticmethod
+    def command_for_port(port, args):
+        if 'simulator' in port.port_name:
+            assert SimulatedDeviceManager.INITIALIZED_DEVICES
+            return ['/usr/bin/xcrun', 'simctl', 'spawn', SimulatedDeviceManager.INITIALIZED_DEVICES[0].udid] + args
+        elif 'device' in port.port_name:
+            raise RuntimeError('Running api tests on {} is not supported'.format(port.port_name))
+        elif port.host.platform.is_win():
+            args[0] = os.path.splitext(args[0])[0] + '.exe'
+        return args
+
+    def run(self, tests, num_workers):
+        if not tests:
+            return
+
+        self.printer.write_update('Sharding tests ...')
+        shards = Runner._shard_tests(tests)
+
+        with message_pool.get(self, lambda caller: _Worker(caller, self.port, shards), min(num_workers, len(shards))) as pool:
+            pool.run(('test', shard) for shard, _ in shards.iteritems())
+
+    def handle(self, message_name, source, test_name=None, status=0, output=''):
+        if message_name == 'did_spawn_worker':
+            return
+        if message_name == 'ended_test':
+            update = '{} {} {}'.format(source, test_name, Runner.NAME_FOR_STATUS[status])
+
+            # Don't print test output if --quiet.
+            if status != Runner.STATUS_PASSED or (output and not self.port.get_option('quiet')):
+                self.printer.writeln(update)
+                for line in output.splitlines():
+                    self.printer.writeln('    {}'.format(line))
+            else:
+                self.printer.write_update(update)
+            self.tests_run += 1
+            self.results[test_name] = (status, output)
+
+    def result_map_by_status(self, status=None):
+        map = {}
+        for test_name, result in self.results.iteritems():
+            if result[0] == status:
+                map[test_name] = result[1]
+        return map
+
+
+class _Worker(object):
+    def __init__(self, caller, port, shard_map):
+        self._caller = caller
+        self._port = port
+        self.host = port.host
+        self._shard_map = shard_map
+
+        # ServerProcess doesn't allow for a timeout of 'None,' this uses a week instead of None.
+        self._timeout = int(self._port.get_option('timeout')) if self._port.get_option('timeout') else 60 * 24 * 7
+
+    @staticmethod
+    def _filter_noisy_output(output):
+        result = ''
+        for line in output.splitlines():
+            if line.lstrip().startswith('objc['):
+                continue
+            result += line + '\n'
+        return result
+
+    def _run_single_test(self, binary_name, test):
+        server_process = ServerProcess(
+            self._port, binary_name,
+            Runner.command_for_port(self._port, [self._port._build_path(binary_name), '--gtest_filter={}'.format(test)]),
+            env=self._port.environment_for_api_tests())
+
+        try:
+            deadline = time.time() + self._timeout
+            server_process.start()
+
+            if not test.split('.')[1].startswith('DISABLED_'):
+                stdout_line = server_process.read_stdout_line(deadline)
+            else:
+                stdout_line = None
+
+            if not stdout_line and server_process.timed_out:
+                status = Runner.STATUS_TIMEOUT
+            elif not stdout_line and server_process.has_crashed():
+                status = Runner.STATUS_CRASHED
+            elif not stdout_line:
+                status = Runner.STATUS_DISABLED
+            elif '**PASS**' in stdout_line:
+                status = Runner.STATUS_PASSED
+            else:
+                status = Runner.STATUS_FAILED
+
+        finally:
+            output_buffer = server_process.pop_all_buffered_stdout() + server_process.pop_all_buffered_stderr()
+            server_process.stop()
+
+        self._caller.post('ended_test', '{}.{}'.format(binary_name, test), status, self._filter_noisy_output(output_buffer))
+
+    def _run_shard_with_binary(self, binary_name, tests):
+        remaining_tests = list(tests)
+
+        # Try to run the shard in a single process.
+        while remaining_tests and not self._port.get_option('run_singly'):
+            starting_length = len(remaining_tests)
+            server_process = ServerProcess(
+                self._port, binary_name,
+                Runner.command_for_port(self._port, [self._port._build_path(binary_name), '--gtest_filter={}'.format(':'.join(remaining_tests))]),
+                env=self._port.environment_for_api_tests())
+
+            try:
+                deadline = time.time() + self._timeout
+                last_test = None
+                last_status = None
+                stdout_buffer = ''
+
+                server_process.start()
+                while remaining_tests:
+                    stdout = server_process.read_stdout_line(deadline)
+
+                    # If we've triggered a timeout, we don't know which test caused it. Break out and run singly.
+                    if stdout is None and server_process.timed_out:
+                        break
+
+                    if stdout is None and server_process.has_crashed():
+                        # It's possible we crashed before printing anything.
+                        if last_status == Runner.STATUS_PASSED:
+                            last_test = None
+                        else:
+                            last_status = Runner.STATUS_CRASHED
+                        break
+
+                    assert stdout is not None
+                    stdout_split = stdout.rstrip().split(' ')
+                    if len(stdout_split) != 2 or not (stdout_split[0].startswith('**') and stdout_split[0].endswith('**')):
+                        stdout_buffer += stdout
+                        continue
+                    if last_test is not None:
+                        remaining_tests.remove(last_test)
+                        self._caller.post('ended_test', '{}.{}'.format(binary_name, last_test), last_status, stdout_buffer)
+                        deadline = time.time() + self._timeout
+                        stdout_buffer = ''
+
+                    if '**PASS**' == stdout_split[0]:
+                        last_status = Runner.STATUS_PASSED
+                    else:
+                        last_status = Runner.STATUS_FAILED
+                    last_test = stdout_split[1]
+
+                # We assume that stderr is only relevant if there is a crash (meaning we triggered an assert)
+                if last_test:
+                    remaining_tests.remove(last_test)
+                    stdout_buffer += server_process.pop_all_buffered_stdout()
+                    stderr_buffer = server_process.pop_all_buffered_stderr() if last_status == Runner.STATUS_CRASHED else ''
+                    self._caller.post('ended_test', '{}.{}'.format(binary_name, last_test), last_status, self._filter_noisy_output(stdout_buffer + stderr_buffer))
+
+                if server_process.timed_out:
+                    break
+
+                # If we weren't able to determine the results for any tests, we need to run what remains singly.
+                if starting_length == len(remaining_tests):
+                    break
+            finally:
+                server_process.stop()
+
+        # Now, just try and run the rest of the tests singly.
+        for test in remaining_tests:
+            self._run_single_test(binary_name, test)
+
+    def handle(self, message_name, source, shard_name):
+        assert message_name == 'test'
+        self._caller.post('started_shard', shard_name)
+
+        binary_map = {}
+        for test in self._shard_map[shard_name]:
+            split_test_name = test.split('.')
+            if split_test_name[0] not in binary_map:
+                binary_map[split_test_name[0]] = []
+            binary_map[split_test_name[0]].append('.'.join(split_test_name[1:]))
+        for binary_name, test_list in binary_map.iteritems():
+            self._run_shard_with_binary(binary_name, test_list)
index d4a5362..f83fc42 100644 (file)
@@ -240,6 +240,27 @@ class Port(object):
                 return False
         return True
 
+    def check_api_test_build(self):
+        if not self._root_was_set and self.get_option('build') and not self._build_api_tests():
+            return False
+        if self.get_option('install') and not self._check_port_build():
+            return False
+
+        for binary in self.path_to_api_test_binaries():
+            if not self._filesystem.exists(binary):
+                _log.error('{} was not found at {}'.format(os.path.basename(binary), binary))
+                return False
+        return True
+
+    def environment_for_api_tests(self):
+        build_root_path = str(self._build_path())
+        return {
+            'DYLD_LIBRARY_PATH': build_root_path,
+            '__XPC_DYLD_LIBRARY_PATH': build_root_path,
+            'DYLD_FRAMEWORK_PATH': build_root_path,
+            '__XPC_DYLD_FRAMEWORK_PATH': build_root_path,
+        }
+
     def _check_driver(self):
         driver_path = self._path_to_driver()
         if not self._filesystem.exists(driver_path):
@@ -1336,6 +1357,17 @@ class Port(object):
         This is likely used only by diff_image()"""
         return self._build_path('ImageDiff')
 
+    def path_to_api_test_binaries(self):
+        binary_names = ['TestWTF']
+        if self.host.platform.is_win():
+            binary_names += ['TestWebCore', 'TestWebKitLegacy']
+        else:
+            binary_names += ['TestWebKitAPI']
+        binary_paths = [self._build_path(binary_name) for binary_name in binary_names]
+        if self.host.platform.is_win():
+            binary_paths = [os.path.splitext(binary_path)[0] + '.exe' for binary_path in binary_paths]
+        return binary_paths
+
     def _path_to_lighttpd(self):
         """Returns the path to the LigHTTPd binary.
 
@@ -1449,6 +1481,15 @@ class Port(object):
             return False
         return True
 
+    def _build_api_tests(self):
+        environment = self.host.copy_current_environment().to_dictionary()
+        try:
+            self._run_script('build-api-tests', args=self._build_driver_flags(), env=environment)
+        except ScriptError as e:
+            _log.error(e.message_with_output(output_limit=None))
+            return False
+        return True
+
     def _build_image_diff(self):
         environment = self.host.copy_current_environment()
         env = environment.to_dictionary()
index 4e2dd14..547e096 100644 (file)
@@ -108,6 +108,17 @@ class IOSSimulatorPort(IOSPort):
 
         SimulatedDeviceManager.tear_down(self.host)
 
+    def environment_for_api_tests(self):
+        no_prefix = super(IOSSimulatorPort, self).environment_for_api_tests()
+        result = {}
+        SIMCTL_ENV_PREFIX = 'SIMCTL_CHILD_'
+        for value in no_prefix:
+            if not value.startswith(SIMCTL_ENV_PREFIX):
+                result[SIMCTL_ENV_PREFIX + value] = no_prefix[value]
+            else:
+                result[value] = no_prefix[value]
+        return result
+
     def setup_environ_for_server(self, server_name=None):
         _log.debug("setup_environ_for_server")
         env = super(IOSSimulatorPort, self).setup_environ_for_server(server_name)
index b74bd01..710a810 100644 (file)
@@ -135,6 +135,13 @@ class MacPort(DarwinPort):
                 config_map[version_name.lower().replace(' ', '') + '+'] = version_names
         return config_map
 
+    def environment_for_api_tests(self):
+        result = super(MacPort, self).environment_for_api_tests()
+        if self.get_option('guard_malloc'):
+            result['DYLD_INSERT_LIBRARIES'] = '/usr/lib/libgmalloc.dylib'
+            result['__XPC_DYLD_INSERT_LIBRARIES'] = '/usr/lib/libgmalloc.dylib'
+        return result
+
     def setup_environ_for_server(self, server_name=None):
         env = super(MacPort, self).setup_environ_for_server(server_name)
         if server_name == self.driver_name():
index 118b7c0..dd29258 100644 (file)
@@ -186,6 +186,9 @@ class ServerProcess(object):
             return self._pop_error_bytes(index_after_newline)
         return None
 
+    def pop_all_buffered_stdout(self):
+        return self._pop_output_bytes(len(self._output))
+
     def pop_all_buffered_stderr(self):
         return self._pop_error_bytes(len(self._error))