[Mac] Media tests start to time out after a few days of bot uptime
authorap@apple.com <ap@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 25 Sep 2015 22:36:09 +0000 (22:36 +0000)
committerap@apple.com <ap@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 25 Sep 2015 22:36:09 +0000 (22:36 +0000)
https://bugs.webkit.org/show_bug.cgi?id=149468

Reviewed by Darin Adler.

Source/WebKit2:

Make the fix work with WebKit2, and avoid touching user's media cache for WebKit1.

When UI process is not sandboxed, inherit DIRHELPER_USER_DIR_SUFFIX from it.
This means that during testing, WebContent uses a temporary directory of the form
/private/var/folders/.../T/WebKitTestRunner-VKjmz0/com.apple.WebKit.WebContent.Development,
instead of /private/var/folders/.../T/com.apple.WebKit.WebContent.Development+WebKitTestRunner.

WebKit1 now also better respects a temporary directory of the form
/private/var/folders/.../T/DumpRenderTree-VKjmz0, instead of the default, which
makes CoreMedia put its cache inside it.

As a result, almost everything is isolated per testing process, and deleted at the end.
Same with cache directories. The only outliers that I know of are plug-in directories,
which are per-process, but are not under the testing root, and thus aren't deleted.

* PluginProcess/mac/PluginProcessMac.mm:
(WebKit::PluginProcess::initializeSandbox):
* Shared/ChildProcessProxy.cpp:
(WebKit::ChildProcessProxy::fromConnection):
(WebKit::ChildProcessProxy::getLaunchOptions):
(WebKit::ChildProcessProxy::connect):
* Shared/ChildProcessProxy.h:
* Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.mm:
(WebKit::XPCServiceInitializerDelegate::getExtraInitializationData):
* Shared/SandboxInitializationParameters.h:
(WebKit::SandboxInitializationParameters::sandboxProfile):
(WebKit::SandboxInitializationParameters::setUserDirectorySuffix):
(WebKit::SandboxInitializationParameters::userDirectorySuffix):
(WebKit::SandboxInitializationParameters::setSystemDirectorySuffix): Deleted.
(WebKit::SandboxInitializationParameters::systemDirectorySuffix): Deleted.
* Shared/ios/ChildProcessIOS.mm:
(WebKit::ChildProcess::initializeSandbox):
* Shared/mac/ChildProcessMac.mm:
(WebKit::ChildProcess::initializeSandbox):
* UIProcess/Databases/DatabaseProcessProxy.cpp:
(WebKit::DatabaseProcessProxy::getLaunchOptions):
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::getLaunchOptions):
* UIProcess/Plugins/PluginProcessProxy.cpp:
(WebKit::PluginProcessProxy::getLaunchOptions):
* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::getLaunchOptions):

Tools:

Delete cache directory in addition to the temporary one. Don't delete /private/tmp/MediaCache,
as it's not longer used by tests. Set TMPDIR and DIRHELPER_USER_DIR_SUFFIX to make
all temporary and cache files go to a custom location.

* Scripts/webkitpy/port/base.py:
* Scripts/webkitpy/port/driver.py:
* Scripts/webkitpy/port/mac.py:
* Scripts/webkitpy/port/win.py:
(WinPort._driver_tempdir_for_environment): Deleted unused function.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@190257 268f45cc-cd09-0410-ab3c-d52691b4dbfc

17 files changed:
Source/WebKit2/ChangeLog
Source/WebKit2/PluginProcess/mac/PluginProcessMac.mm
Source/WebKit2/Shared/ChildProcessProxy.cpp
Source/WebKit2/Shared/ChildProcessProxy.h
Source/WebKit2/Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.mm
Source/WebKit2/Shared/SandboxInitializationParameters.h
Source/WebKit2/Shared/ios/ChildProcessIOS.mm
Source/WebKit2/Shared/mac/ChildProcessMac.mm
Source/WebKit2/UIProcess/Databases/DatabaseProcessProxy.cpp
Source/WebKit2/UIProcess/Network/NetworkProcessProxy.cpp
Source/WebKit2/UIProcess/Plugins/PluginProcessProxy.cpp
Source/WebKit2/UIProcess/WebProcessProxy.cpp
Tools/ChangeLog
Tools/Scripts/webkitpy/port/base.py
Tools/Scripts/webkitpy/port/driver.py
Tools/Scripts/webkitpy/port/mac.py
Tools/Scripts/webkitpy/port/win.py

index dfcffd8299c543574ee80d253ec5f0781867b63a..14b787260d72a9598da64f2ecc4bf8d405dd2e1e 100644 (file)
@@ -1,3 +1,53 @@
+2015-09-25  Alexey Proskuryakov  <ap@apple.com>
+
+        [Mac] Media tests start to time out after a few days of bot uptime
+        https://bugs.webkit.org/show_bug.cgi?id=149468
+
+        Reviewed by Darin Adler.
+
+        Make the fix work with WebKit2, and avoid touching user's media cache for WebKit1.
+
+        When UI process is not sandboxed, inherit DIRHELPER_USER_DIR_SUFFIX from it.
+        This means that during testing, WebContent uses a temporary directory of the form
+        /private/var/folders/.../T/WebKitTestRunner-VKjmz0/com.apple.WebKit.WebContent.Development,
+        instead of /private/var/folders/.../T/com.apple.WebKit.WebContent.Development+WebKitTestRunner.
+
+        WebKit1 now also better respects a temporary directory of the form
+        /private/var/folders/.../T/DumpRenderTree-VKjmz0, instead of the default, which
+        makes CoreMedia put its cache inside it.
+
+        As a result, almost everything is isolated per testing process, and deleted at the end.
+        Same with cache directories. The only outliers that I know of are plug-in directories,
+        which are per-process, but are not under the testing root, and thus aren't deleted.
+
+        * PluginProcess/mac/PluginProcessMac.mm:
+        (WebKit::PluginProcess::initializeSandbox):
+        * Shared/ChildProcessProxy.cpp:
+        (WebKit::ChildProcessProxy::fromConnection):
+        (WebKit::ChildProcessProxy::getLaunchOptions):
+        (WebKit::ChildProcessProxy::connect):
+        * Shared/ChildProcessProxy.h:
+        * Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.mm:
+        (WebKit::XPCServiceInitializerDelegate::getExtraInitializationData):
+        * Shared/SandboxInitializationParameters.h:
+        (WebKit::SandboxInitializationParameters::sandboxProfile):
+        (WebKit::SandboxInitializationParameters::setUserDirectorySuffix):
+        (WebKit::SandboxInitializationParameters::userDirectorySuffix):
+        (WebKit::SandboxInitializationParameters::setSystemDirectorySuffix): Deleted.
+        (WebKit::SandboxInitializationParameters::systemDirectorySuffix): Deleted.
+        * Shared/ios/ChildProcessIOS.mm:
+        (WebKit::ChildProcess::initializeSandbox):
+        * Shared/mac/ChildProcessMac.mm:
+        (WebKit::ChildProcess::initializeSandbox):
+        * UIProcess/Databases/DatabaseProcessProxy.cpp:
+        (WebKit::DatabaseProcessProxy::getLaunchOptions):
+        * UIProcess/Network/NetworkProcessProxy.cpp:
+        (WebKit::NetworkProcessProxy::getLaunchOptions):
+        * UIProcess/Plugins/PluginProcessProxy.cpp:
+        (WebKit::PluginProcessProxy::getLaunchOptions):
+        * UIProcess/WebProcessProxy.cpp:
+        (WebKit::WebProcessProxy::getLaunchOptions):
+
 2015-09-25  Beth Dakin  <bdakin@apple.com>
 
         Clicking on a data detected item inside a form control always pops up a map 
index 10a43fb6939867b32eab2e1e5dabcf8c34315c36..30af5a5f8ce19d571d1275360a03bdef7ea454e6 100644 (file)
@@ -508,7 +508,7 @@ void PluginProcess::initializeSandbox(const ChildProcessInitializationParameters
         exit(EX_OSERR);
     }
 
-    sandboxParameters.setSystemDirectorySuffix([[[[NSFileManager defaultManager] stringWithFileSystemRepresentation:temporaryDirectory length:strlen(temporaryDirectory)] lastPathComponent] fileSystemRepresentation]);
+    sandboxParameters.setUserDirectorySuffix([[[[NSFileManager defaultManager] stringWithFileSystemRepresentation:temporaryDirectory length:strlen(temporaryDirectory)] lastPathComponent] fileSystemRepresentation]);
 
     sandboxParameters.addPathParameter("PLUGIN_PATH", m_pluginPath);
     sandboxParameters.addPathParameter("NSURL_CACHE_DIR", m_nsurlCacheDirectory);
index eabeb91eeb1dea71e33f039b8af11dd96131491c..319f802238a325c1e8e545bd3751e81cf04736cb 100644 (file)
@@ -56,6 +56,12 @@ ChildProcessProxy* ChildProcessProxy::fromConnection(IPC::Connection* connection
     return childProcessProxy;
 }
 
+void ChildProcessProxy::getLaunchOptions(ProcessLauncher::LaunchOptions& launchOptions)
+{
+    if (const char* userDirectorySuffix = getenv("DIRHELPER_USER_DIR_SUFFIX"))
+        launchOptions.extraInitializationData.add(ASCIILiteral("user-directory-suffix"), userDirectorySuffix);
+}
+
 void ChildProcessProxy::connect()
 {
     ASSERT(!m_processLauncher);
index 203a023ff1fc3ac3d85b7c451354f967ff7952e1..02edcc13ace27fbbc2a9b4248e2f7dfe3f91165e 100644 (file)
@@ -82,8 +82,9 @@ protected:
     bool dispatchMessage(IPC::Connection&, IPC::MessageDecoder&);
     bool dispatchSyncMessage(IPC::Connection&, IPC::MessageDecoder&, std::unique_ptr<IPC::MessageEncoder>&);
     
+    virtual void getLaunchOptions(ProcessLauncher::LaunchOptions&);
+
 private:
-    virtual void getLaunchOptions(ProcessLauncher::LaunchOptions&) = 0;
     virtual void connectionWillOpen(IPC::Connection&);
     virtual void processWillShutDown(IPC::Connection&) = 0;
 
index 6afef80d687cd971defe44ad00819aab6fa1b764..5fda2d6e22bdb3668c54d8eb682679329b96208b 100644 (file)
@@ -97,6 +97,12 @@ bool XPCServiceInitializerDelegate::getExtraInitializationData(HashMap<String, S
     if (!inspectorProcess.isEmpty())
         extraInitializationData.add(ASCIILiteral("inspector-process"), inspectorProcess);
 
+    if (!isClientSandboxed()) {
+        String userDirectorySuffix = xpc_dictionary_get_string(extraDataInitializationDataObject, "user-directory-suffix");
+        if (!userDirectorySuffix.isEmpty())
+            extraInitializationData.add(ASCIILiteral("user-directory-suffix"), userDirectorySuffix);
+    }
+
     return true;
 }
 
index 5c9bbe31d0f8634e56dde3385dceb084c4faa9b9..a619b7004239d825619256bc90bb56b8db0ae340 100644 (file)
@@ -86,8 +86,8 @@ public:
         return m_overrideSandboxProfilePathOrSandboxProfile;
     }
 
-    void setSystemDirectorySuffix(const String& suffix) { m_systemDirectorySuffix = suffix; }
-    const String& systemDirectorySuffix() const { return m_systemDirectorySuffix; }
+    void setUserDirectorySuffix(const String& suffix) { m_userDirectorySuffix = suffix; }
+    const String& userDirectorySuffix() const { return m_userDirectorySuffix; }
 #endif
 
 private:
@@ -95,7 +95,7 @@ private:
     void appendPathInternal(const char* name, const char* path);
 
     mutable Vector<const char*> m_namedParameters;
-    String m_systemDirectorySuffix;
+    String m_userDirectorySuffix;
 
     ProfileSelectionMode m_profileSelectionMode;
     String m_overrideSandboxProfilePathOrSandboxProfile;
index 5dbeceddf49f65a5e874a13149f4fdf51c7acad2..0978bec25bce6c45e0db3d38fd6ae7501b652722 100644 (file)
@@ -73,9 +73,9 @@ void ChildProcess::initializeSandbox(const ChildProcessInitializationParameters&
 #if ENABLE(MANUAL_SANDBOXING)
     NSBundle *webkit2Bundle = [NSBundle bundleForClass:NSClassFromString(@"WKWebView")];
     String defaultProfilePath = [webkit2Bundle pathForResource:[[NSBundle mainBundle] bundleIdentifier] ofType:@"sb"];
-    if (sandboxParameters.systemDirectorySuffix().isNull()) {
-        String defaultSystemDirectorySuffix = String([[NSBundle mainBundle] bundleIdentifier]) + "+" + parameters.clientIdentifier;
-        sandboxParameters.setSystemDirectorySuffix(defaultSystemDirectorySuffix);
+    if (sandboxParameters.userDirectorySuffix().isNull()) {
+        String defaultUserDirectorySuffix = makeString(String([[NSBundle mainBundle] bundleIdentifier]), '+', parameters.clientIdentifier);
+        sandboxParameters.setUserDirectorySuffix(defaultUserDirectorySuffix);
     }
 
     String sandboxImportPath = "/usr/local/share/sandbox/imports";
index 94ac1ded973dbc6cfde90c803853b85c3c7b781d..380f52a3b325ede9d82e5e83bd5fd95907eb1b89 100644 (file)
@@ -90,9 +90,14 @@ void ChildProcess::initializeSandbox(const ChildProcessInitializationParameters&
     NSBundle *webkit2Bundle = [NSBundle bundleForClass:NSClassFromString(@"WKView")];
     String defaultProfilePath = [webkit2Bundle pathForResource:[[NSBundle mainBundle] bundleIdentifier] ofType:@"sb"];
 
-    if (sandboxParameters.systemDirectorySuffix().isNull()) {
-        String defaultSystemDirectorySuffix = String([[NSBundle mainBundle] bundleIdentifier]) + "+" + parameters.clientIdentifier;
-        sandboxParameters.setSystemDirectorySuffix(defaultSystemDirectorySuffix);
+    if (sandboxParameters.userDirectorySuffix().isNull()) {
+        auto userDirectorySuffix = parameters.extraInitializationData.find("user-directory-suffix");
+        if (userDirectorySuffix != parameters.extraInitializationData.end())
+            sandboxParameters.setUserDirectorySuffix([makeString(userDirectorySuffix->value, '/', String([[NSBundle mainBundle] bundleIdentifier])) fileSystemRepresentation]);
+        else {
+            String defaultUserDirectorySuffix = makeString(String([[NSBundle mainBundle] bundleIdentifier]), '+', parameters.clientIdentifier);
+            sandboxParameters.setUserDirectorySuffix(defaultUserDirectorySuffix);
+        }
     }
 
     Vector<String> osVersionParts;
@@ -106,7 +111,7 @@ void ChildProcess::initializeSandbox(const ChildProcessInitializationParameters&
     sandboxParameters.addParameter("_OS_VERSION", osVersion.utf8().data());
 
     // Use private temporary and cache directories.
-    setenv("DIRHELPER_USER_DIR_SUFFIX", fileSystemRepresentation(sandboxParameters.systemDirectorySuffix()).data(), 0);
+    setenv("DIRHELPER_USER_DIR_SUFFIX", fileSystemRepresentation(sandboxParameters.userDirectorySuffix()).data(), 1);
     char temporaryDirectory[PATH_MAX];
     if (!confstr(_CS_DARWIN_USER_TEMP_DIR, temporaryDirectory, sizeof(temporaryDirectory))) {
         WTFLogAlways("%s: couldn't retrieve private temporary directory path: %d\n", getprogname(), errno);
index 45b95cfa6420dbb70979636cd7d73e3f09524978..cbfab7b3ac2fbe0d48f2e0ab9d7c66ec600473ab 100644 (file)
@@ -67,6 +67,7 @@ DatabaseProcessProxy::~DatabaseProcessProxy()
 void DatabaseProcessProxy::getLaunchOptions(ProcessLauncher::LaunchOptions& launchOptions)
 {
     launchOptions.processType = ProcessLauncher::DatabaseProcess;
+    ChildProcessProxy::getLaunchOptions(launchOptions);
     platformGetLaunchOptions(launchOptions);
 }
 
index 41610abc3551af5c7ec36e2df9bcc071671721fd..be1c0da57c4ad032e5f8e3e692ea7b944a631540 100644 (file)
@@ -83,6 +83,7 @@ NetworkProcessProxy::~NetworkProcessProxy()
 void NetworkProcessProxy::getLaunchOptions(ProcessLauncher::LaunchOptions& launchOptions)
 {
     launchOptions.processType = ProcessLauncher::NetworkProcess;
+    ChildProcessProxy::getLaunchOptions(launchOptions);
     platformGetLaunchOptions(launchOptions);
 }
 
index 7ebf7799af5099a4567779e800922d30a5ee3405..3f85b4cf22c33346ed670c78f5f08a5a08cd1c85 100644 (file)
@@ -85,6 +85,7 @@ PluginProcessProxy::~PluginProcessProxy()
 void PluginProcessProxy::getLaunchOptions(ProcessLauncher::LaunchOptions& launchOptions)
 {
     launchOptions.processType = ProcessLauncher::PluginProcess;
+    ChildProcessProxy::getLaunchOptions(launchOptions);
     platformGetLaunchOptions(launchOptions, m_pluginProcessAttributes);
 }
 
index d00c85ac266e31f0613228f629eda39f1d64ee51..8881ef70568a63b18118811632b5b54f4e2c1f8c 100644 (file)
@@ -130,6 +130,8 @@ void WebProcessProxy::getLaunchOptions(ProcessLauncher::LaunchOptions& launchOpt
 {
     launchOptions.processType = ProcessLauncher::WebProcess;
 
+    ChildProcessProxy::getLaunchOptions(launchOptions);
+
     if (WebInspectorProxy::isInspectorProcessPool(m_processPool))
         launchOptions.extraInitializationData.add(ASCIILiteral("inspector-process"), ASCIILiteral("1"));
 
index 4f8769834fbef8c9d2794ef434f334d8e0861abf..8417355204171a26635af30e9b001603b643e94c 100644 (file)
@@ -1,3 +1,20 @@
+2015-09-25  Alexey Proskuryakov  <ap@apple.com>
+
+        [Mac] Media tests start to time out after a few days of bot uptime
+        https://bugs.webkit.org/show_bug.cgi?id=149468
+
+        Reviewed by Darin Adler.
+
+        Delete cache directory in addition to the temporary one. Don't delete /private/tmp/MediaCache,
+        as it's not longer used by tests. Set TMPDIR and DIRHELPER_USER_DIR_SUFFIX to make
+        all temporary and cache files go to a custom location.
+
+        * Scripts/webkitpy/port/base.py:
+        * Scripts/webkitpy/port/driver.py:
+        * Scripts/webkitpy/port/mac.py:
+        * Scripts/webkitpy/port/win.py:
+        (WinPort._driver_tempdir_for_environment): Deleted unused function.
+
 2015-09-25  Dan Bernstein  <mitz@apple.com>
 
         Removed MiniBrowser.entitlements.
index 1cc2089eedeab72de1a795b223ca17336ec8a0d9..559e2a5c5672758531e13deeca8ca1c3a8d15d26 100644 (file)
@@ -1154,8 +1154,8 @@ class Port(object):
     def _driver_tempdir(self):
         return self._filesystem.mkdtemp(prefix='%s-' % self.driver_name())
 
-    def _driver_tempdir_for_environment(self):
-        return self._driver_tempdir()
+    def remove_cache_directory(self, name):
+        pass
 
     def _path_to_webcore_library(self):
         """Returns the full path to a built copy of WebCore."""
index e647326c385eb1be09e4970bbd7c73d40d096d46..df4114e6e91d3732d95c7e747a58f2add9de03bc 100644 (file)
@@ -313,10 +313,11 @@ class Driver(object):
     def _setup_environ_for_driver(self, environment):
         environment['DYLD_LIBRARY_PATH'] = self._port._build_path()
         environment['DYLD_FRAMEWORK_PATH'] = self._port._build_path()
-        # FIXME: We're assuming that WebKitTestRunner checks this DumpRenderTree-named environment variable.
-        # FIXME: Commented out for now to avoid tests breaking. Re-enable after
-        # we cut over to NRWT
-        #environment['DUMPRENDERTREE_TEMP'] = str(self._port._driver_tempdir_for_environment())
+        # Use an isolated temp directory that can be deleted after testing (especially important on Mac, as
+        # CoreMedia disk cache is in the temp directory).
+        environment['TMPDIR'] = str(self._driver_tempdir)
+        environment['DIRHELPER_USER_DIR_SUFFIX'] = os.path.basename(str(self._driver_tempdir))
+        # Put certain normally persistent files into the temp directory (e.g. IndexedDB storage).
         environment['DUMPRENDERTREE_TEMP'] = str(self._driver_tempdir)
         environment['LOCAL_RESOURCE_ROOT'] = self._port.layout_tests_dir()
         environment['ASAN_OPTIONS'] = "allocator_may_return_null=1"
@@ -358,6 +359,7 @@ class Driver(object):
 
         if self._driver_tempdir:
             self._port._filesystem.rmtree(str(self._driver_tempdir))
+            self._port.remove_cache_directory(os.path.basename(str(self._driver_tempdir)))
             self._driver_tempdir = None
 
     def cmd_line(self, pixel_tests, per_test_args):
index dcf6053e96f7ee5317df108057ea78c954cd8c76..e6a4cb3edfd4623cffdd106c16be26b44c5f0dd9 100644 (file)
@@ -108,12 +108,14 @@ class MacPort(ApplePort):
         return env
 
     def _clear_global_caches_and_temporary_files(self):
-        self._filesystem.rmtree('/private/tmp/MediaCache')
         self._filesystem.rmtree(os.path.expanduser('~/Library/' + self.driver_name()))
         self._filesystem.rmtree(os.path.expanduser('~/Library/Application Support/' + self.driver_name()))
         self._filesystem.rmtree(os.path.expanduser('~/Library/Caches/' + self.driver_name()))
         self._filesystem.rmtree(os.path.expanduser('~/Library/WebKit/' + self.driver_name()))
 
+    def remove_cache_directory(self, name):
+        self._filesystem.rmtree(os.confstr(65538) + name)
+
     def operating_system(self):
         return 'mac'
 
index ab69969729ae70cfd9f8365f874f0c4a5a949430..5894ea3298cd1e3886715b3209a256cc2b56f020 100644 (file)
@@ -138,9 +138,6 @@ class WinPort(ApplePort):
     def _path_to_lighttpd_php(self):
         return "/usr/bin/php-cgi"
 
-    def _driver_tempdir_for_environment(self):
-        return cygpath(self._driver_tempdir())
-
     def test_search_path(self):
         test_fallback_names = [path for path in self.baseline_search_path() if not path.startswith(self._webkit_baseline_path('mac'))]
         return map(self._webkit_baseline_path, test_fallback_names)