[WK2] Add logging to validate the network cache efficacy (Part 1)
authorcdumez@apple.com <cdumez@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sun, 8 Feb 2015 22:06:06 +0000 (22:06 +0000)
committercdumez@apple.com <cdumez@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sun, 8 Feb 2015 22:06:06 +0000 (22:06 +0000)
https://bugs.webkit.org/show_bug.cgi?id=141269
<rdar://problem/19632080>

Reviewed by Antti Koivisto.

Source/WebCore:

Export an extra symbol.

* WebCore.exp.in:

Source/WebKit2:

Add console logging to validate the network cache efficacy. This will
tell us if how the network cache satisties requests, in particular:
- Request cannot be handled by the cache
- Entry was not in the cache but is no longer there (pruned)
- Entry is in the cache but is not usable
- Entry is in the cache and is used.

This patch introduces a SQLite-based network cache statistics storage
that is used to store requests we have seen before, and query if we
have seen a request before. The storage is lightweight as it only
stores hashes in the database, in a background thread.

The statistics cache is initially bootstapped from the network disk
cache so that we have data initially and get as accurate statistics
as possible from the start.

To maintain an acceptable level of performance, we have a hard limit
on the number of unique requests that are retained set to 100000.

Diagnostic logging for this will be added in a follow-up patch.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@179804 268f45cc-cd09-0410-ab3c-d52691b4dbfc

15 files changed:
Source/WebCore/ChangeLog
Source/WebCore/WebCore.exp.in
Source/WebKit2/ChangeLog
Source/WebKit2/NetworkProcess/cache/NetworkCache.cpp
Source/WebKit2/NetworkProcess/cache/NetworkCache.h
Source/WebKit2/NetworkProcess/cache/NetworkCacheFileSystemPosix.h [new file with mode: 0644]
Source/WebKit2/NetworkProcess/cache/NetworkCacheStatistics.h [new file with mode: 0644]
Source/WebKit2/NetworkProcess/cache/NetworkCacheStatisticsCocoa.mm [new file with mode: 0644]
Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.h
Source/WebKit2/NetworkProcess/cache/NetworkCacheStorageCocoa.mm
Source/WebKit2/NetworkProcess/cocoa/NetworkProcessCocoa.mm
Source/WebKit2/Shared/Network/NetworkProcessCreationParameters.cpp
Source/WebKit2/Shared/Network/NetworkProcessCreationParameters.h
Source/WebKit2/UIProcess/Cocoa/WebProcessPoolCocoa.mm
Source/WebKit2/WebKit2.xcodeproj/project.pbxproj

index a353e10..a7859dc 100644 (file)
@@ -1,3 +1,15 @@
+2015-02-08  Chris Dumez  <cdumez@apple.com>
+
+        [WK2] Add logging to validate the network cache efficacy (Part 1)
+        https://bugs.webkit.org/show_bug.cgi?id=141269
+        <rdar://problem/19632080>
+
+        Reviewed by Antti Koivisto.
+
+        Export an extra symbol.
+
+        * WebCore.exp.in:
+
 2015-02-07  Chris Fleizach  <cfleizach@apple.com>
 
         AX: The input element with type="search" has no default focus outline
index 19a040a..c003730 100644 (file)
@@ -2262,6 +2262,7 @@ __ZNK7WebCore7Element26fastAttributeLookupAllowedERKNS_13QualifiedNameE
 #endif
 
 #if !ASSERT_DISABLED
+__ZN7WebCore21SQLiteDatabaseTracker24hasTransactionInProgressEv
 __ZN7WebCore27NoExceptionAssertionCheckerC1EPKci
 __ZN7WebCore27NoExceptionAssertionCheckerD1Ev
 #endif
index 9488e69..3648149 100644 (file)
@@ -1,3 +1,32 @@
+2015-02-08  Chris Dumez  <cdumez@apple.com>
+
+        [WK2] Add logging to validate the network cache efficacy (Part 1)
+        https://bugs.webkit.org/show_bug.cgi?id=141269
+        <rdar://problem/19632080>
+
+        Reviewed by Antti Koivisto.
+
+        Add console logging to validate the network cache efficacy. This will
+        tell us if how the network cache satisties requests, in particular:
+        - Request cannot be handled by the cache
+        - Entry was not in the cache but is no longer there (pruned)
+        - Entry is in the cache but is not usable
+        - Entry is in the cache and is used.
+
+        This patch introduces a SQLite-based network cache statistics storage
+        that is used to store requests we have seen before, and query if we
+        have seen a request before. The storage is lightweight as it only
+        stores hashes in the database, in a background thread.
+
+        The statistics cache is initially bootstapped from the network disk
+        cache so that we have data initially and get as accurate statistics
+        as possible from the start.
+
+        To maintain an acceptable level of performance, we have a hard limit
+        on the number of unique requests that are retained set to 100000.
+
+        Diagnostic logging for this will be added in a follow-up patch.
+
 2015-02-07  Chris Dumez  <cdumez@apple.com>
 
         Add Vector::removeFirstMatching() / removeAllMatching() methods taking lambda functions
index 66b6479..d4770aa 100644 (file)
@@ -30,6 +30,7 @@
 
 #include "Logging.h"
 #include "NetworkCacheCoders.h"
+#include "NetworkCacheStatistics.h"
 #include "NetworkCacheStorage.h"
 #include "NetworkResourceLoader.h"
 #include "WebCoreArgumentCoders.h"
@@ -49,10 +50,13 @@ NetworkCache& NetworkCache::singleton()
     return instance;
 }
 
-bool NetworkCache::initialize(const String& cachePath)
+bool NetworkCache::initialize(const String& cachePath, bool enableEfficacyLogging)
 {
     m_storage = NetworkCacheStorage::open(cachePath);
 
+    if (enableEfficacyLogging)
+        m_statistics = NetworkCacheStatistics::open(cachePath);
+
     LOG(NetworkCache, "(NetworkProcess) opened cache storage, success %d", !!m_storage);
     return !!m_storage;
 }
@@ -220,23 +224,33 @@ void NetworkCache::retrieve(const WebCore::ResourceRequest& originalRequest, std
 
     LOG(NetworkCache, "(NetworkProcess) retrieving %s priority %u", originalRequest.url().string().ascii().data(), originalRequest.priority());
 
+    NetworkCacheKey storageKey = makeCacheKey(originalRequest);
     if (!canRetrieve(originalRequest)) {
+        if (m_statistics)
+            m_statistics->recordNotUsingCacheForRequest(storageKey, originalRequest);
+
         completionHandler(nullptr);
         return;
     }
 
     auto startTime = std::chrono::system_clock::now();
-    NetworkCacheKey storageKey = makeCacheKey(originalRequest);
     unsigned priority = originalRequest.priority();
 
-    m_storage->retrieve(storageKey, priority, [this, originalRequest, completionHandler, startTime](std::unique_ptr<NetworkCacheStorage::Entry> entry) {
+    m_storage->retrieve(storageKey, priority, [this, originalRequest, completionHandler, startTime, storageKey](std::unique_ptr<NetworkCacheStorage::Entry> entry) {
         if (!entry) {
             LOG(NetworkCache, "(NetworkProcess) not found in storage");
+
+            if (m_statistics)
+                m_statistics->recordRetrievalFailure(storageKey, originalRequest);
+
             completionHandler(nullptr);
             return false;
         }
         auto decodedEntry = decodeStorageEntry(*entry, originalRequest);
         bool success = !!decodedEntry;
+        if (m_statistics)
+            m_statistics->recordRetrievedCachedEntry(storageKey, originalRequest, success);
+
 #if !LOG_DISABLED
         auto elapsedMS = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now() - startTime).count();
 #endif
@@ -331,6 +345,13 @@ void NetworkCache::clear()
     LOG(NetworkCache, "(NetworkProcess) clearing cache");
     if (m_storage)
         m_storage->clear();
+    if (m_statistics)
+        m_statistics->clear();
+}
+
+String NetworkCache::storagePath() const
+{
+    return m_storage ? m_storage->directoryPath() : String();
 }
 
 }
index b287969..40e43fc 100644 (file)
@@ -41,13 +41,15 @@ class URL;
 
 namespace WebKit {
 
+class NetworkCacheStatistics;
+
 class NetworkCache {
     WTF_MAKE_NONCOPYABLE(NetworkCache);
     friend class WTF::NeverDestroyed<NetworkCache>;
 public:
     static NetworkCache& singleton();
 
-    bool initialize(const String& cachePath);
+    bool initialize(const String& cachePath, bool enableEfficacyLogging);
     void setMaximumSize(size_t);
 
     bool isEnabled() const { return !!m_storage; }
@@ -75,11 +77,14 @@ public:
 
     void clear();
 
+    String storagePath() const;
+
 private:
     NetworkCache() = default;
     ~NetworkCache() = delete;
 
     std::unique_ptr<NetworkCacheStorage> m_storage;
+    std::unique_ptr<NetworkCacheStatistics> m_statistics;
 };
 
 }
diff --git a/Source/WebKit2/NetworkProcess/cache/NetworkCacheFileSystemPosix.h b/Source/WebKit2/NetworkProcess/cache/NetworkCacheFileSystemPosix.h
new file mode 100644 (file)
index 0000000..ba1f547
--- /dev/null
@@ -0,0 +1,73 @@
+/*
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef NetworkCacheFileSystemPosix_h
+#define NetworkCacheFileSystemPosix_h
+
+#if ENABLE(NETWORK_CACHE)
+
+#include "NetworkCacheKey.h"
+#include <WebCore/FileSystem.h>
+#include <dirent.h>
+#include <wtf/text/CString.h>
+
+namespace WebKit {
+
+template <typename Function>
+static void traverseDirectory(const String& path, uint8_t type, const Function& function)
+{
+    DIR* dir = opendir(WebCore::fileSystemRepresentation(path).data());
+    if (!dir)
+        return;
+    struct dirent* dp;
+    while ((dp = readdir(dir))) {
+        if (dp->d_type != type)
+            continue;
+        const char* name = dp->d_name;
+        if (!strcmp(name, ".") || !strcmp(name, ".."))
+            continue;
+        function(String(name));
+    }
+    closedir(dir);
+}
+
+inline void traverseCacheFiles(const String& cachePath, std::function<void (const String& fileName, const String& partitionPath)> function)
+{
+    traverseDirectory(cachePath, DT_DIR, [&cachePath, &function](const String& subdirName) {
+        String partitionPath = WebCore::pathByAppendingComponent(cachePath, subdirName);
+        traverseDirectory(partitionPath, DT_REG, [&function, &partitionPath](const String& fileName) {
+            if (fileName.length() != NetworkCacheKey::hashStringLength())
+                return;
+            function(fileName, partitionPath);
+        });
+    });
+}
+
+} // namespace WebKit
+
+#endif // ENABLE(NETWORK_CACHE)
+
+#endif // NetworkCacheFileSystemPosix_h
+
diff --git a/Source/WebKit2/NetworkProcess/cache/NetworkCacheStatistics.h b/Source/WebKit2/NetworkProcess/cache/NetworkCacheStatistics.h
new file mode 100644 (file)
index 0000000..bb52387
--- /dev/null
@@ -0,0 +1,82 @@
+/*
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef NetworkCacheStatistics_h
+#define NetworkCacheStatistics_h
+
+#if ENABLE(NETWORK_CACHE)
+
+#include "NetworkCacheKey.h"
+#include "NetworkCacheStorage.h"
+#include <WebCore/SQLiteDatabase.h>
+
+namespace WebCore {
+class ResourceRequest;
+}
+
+namespace WebKit {
+
+class NetworkCacheStatistics {
+public:
+    static std::unique_ptr<NetworkCacheStatistics> open(const String& cachePath);
+
+    void clear();
+
+    void recordNotUsingCacheForRequest(const NetworkCacheKey&, const WebCore::ResourceRequest&);
+    void recordRetrievalFailure(const NetworkCacheKey&, const WebCore::ResourceRequest&);
+    void recordRetrievedCachedEntry(const NetworkCacheKey&, const WebCore::ResourceRequest&, bool success);
+
+private:
+    explicit NetworkCacheStatistics(const String& databasePath);
+
+    void initialize(const String& databasePath);
+    void bootstrapFromNetworkCache(const String& networkCachePath);
+    void shrinkIfNeeded();
+
+    void addHashToDatabase(const String& hash);
+
+    typedef std::function<void (bool wasEverRequested)> RequestedCompletionHandler;
+    void queryWasEverRequested(const String&, const RequestedCompletionHandler&);
+    void markAsRequested(const String& hash);
+
+    struct EverRequestedQuery {
+        String hash;
+        RequestedCompletionHandler completionHandler;
+    };
+
+    std::atomic<size_t> m_approximateEntryCount { 0 };
+
+#if PLATFORM(COCOA)
+    mutable DispatchPtr<dispatch_queue_t> m_backgroundIOQueue;
+#endif
+    mutable HashSet<std::unique_ptr<const EverRequestedQuery>> m_activeQueries;
+    WebCore::SQLiteDatabase m_database;
+};
+
+} // namespace WebKit
+
+#endif // ENABLE(NETWORK_CACHE)
+
+#endif // NetworkCacheStatistics_h
diff --git a/Source/WebKit2/NetworkProcess/cache/NetworkCacheStatisticsCocoa.mm b/Source/WebKit2/NetworkProcess/cache/NetworkCacheStatisticsCocoa.mm
new file mode 100644 (file)
index 0000000..1227946
--- /dev/null
@@ -0,0 +1,274 @@
+/*
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#if ENABLE(NETWORK_CACHE)
+#include "NetworkCacheStatistics.h"
+
+#include "Logging.h"
+#include "NetworkCache.h"
+#include "NetworkCacheFileSystemPosix.h"
+#include <WebCore/ResourceRequest.h>
+#include <WebCore/SQLiteDatabaseTracker.h>
+#include <WebCore/SQLiteStatement.h>
+#include <WebCore/SQLiteTransaction.h>
+#include <wtf/RunLoop.h>
+
+namespace WebKit {
+
+static const char* networkCacheStatisticsDatabaseName = "WebKitCacheStatistics.db";
+
+static bool executeSQLCommand(WebCore::SQLiteDatabase& database, const String& sql)
+{
+    ASSERT(!RunLoop::isMain());
+    ASSERT(WebCore::SQLiteDatabaseTracker::hasTransactionInProgress());
+    ASSERT(database.isOpen());
+
+    bool result = database.executeCommand(sql);
+    if (!result)
+        LOG_ERROR("Network cache statistics: failed to execute statement \"%s\" error \"%s\"", sql.utf8().data(), database.lastErrorMsg());
+
+    return result;
+}
+
+static bool executeSQLStatement(WebCore::SQLiteStatement& statement)
+{
+    ASSERT(!RunLoop::isMain());
+    ASSERT(WebCore::SQLiteDatabaseTracker::hasTransactionInProgress());
+    ASSERT(statement.database().isOpen());
+
+    bool result = statement.executeCommand();
+    if (!result)
+        LOG_ERROR("Network cache statistics: failed to execute statement \"%s\" error \"%s\"", statement.query().utf8().data(), statement.database().lastErrorMsg());
+
+    return result;
+}
+
+std::unique_ptr<NetworkCacheStatistics> NetworkCacheStatistics::open(const String& cachePath)
+{
+    ASSERT(RunLoop::isMain());
+
+    String databasePath = WebCore::pathByAppendingComponent(cachePath, networkCacheStatisticsDatabaseName);
+    return std::unique_ptr<NetworkCacheStatistics>(new NetworkCacheStatistics(databasePath));
+}
+
+NetworkCacheStatistics::NetworkCacheStatistics(const String& databasePath)
+    : m_backgroundIOQueue(adoptDispatch(dispatch_queue_create("com.apple.WebKit.Cache.Statistics.Background", DISPATCH_QUEUE_SERIAL)))
+{
+    dispatch_set_target_queue(m_backgroundIOQueue.get(), dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0));
+
+    initialize(databasePath);
+}
+
+void NetworkCacheStatistics::initialize(const String& databasePath)
+{
+    ASSERT(RunLoop::isMain());
+
+    auto startTime = std::chrono::system_clock::now();
+
+    StringCapture databasePathCapture(databasePath);
+    StringCapture networkCachePathCapture(NetworkCache::singleton().storagePath());
+    dispatch_async(m_backgroundIOQueue.get(), [this, databasePathCapture, networkCachePathCapture, startTime] {
+        WebCore::SQLiteTransactionInProgressAutoCounter transactionCounter;
+
+        String databasePath = databasePathCapture.string();
+        if (!WebCore::makeAllDirectories(WebCore::directoryName(databasePath)))
+            return;
+
+        LOG(NetworkCache, "(NetworkProcess) Opening network cache statistics database at %s...", databasePath.utf8().data());
+        m_database.open(databasePath);
+        m_database.disableThreadingChecks();
+        if (!m_database.isOpen()) {
+            LOG_ERROR("Network cache statistics: Failed to open / create the network cache statistics database");
+            return;
+        }
+
+        executeSQLCommand(m_database, ASCIILiteral("CREATE TABLE IF NOT EXISTS AlreadyRequested (hash TEXT PRIMARY KEY)"));
+        WebCore::SQLiteStatement statement(m_database, ASCIILiteral("SELECT count(*) FROM AlreadyRequested"));
+        if (statement.prepareAndStep() != WebCore::SQLResultRow) {
+            LOG_ERROR("Network cache statistics: Failed to count the number of rows in AlreadyRequested table");
+            return;
+        }
+
+        m_approximateEntryCount = statement.getColumnInt(0);
+
+#if !LOG_DISABLED
+        auto elapsedMS = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now() - startTime).count();
+#endif
+        LOG(NetworkCache, "(NetworkProcess) Network cache statistics database load complete, entries=%lu time=%lldms", static_cast<size_t>(m_approximateEntryCount), elapsedMS);
+
+        if (!m_approximateEntryCount) {
+            bootstrapFromNetworkCache(networkCachePathCapture.string());
+#if !LOG_DISABLED
+            elapsedMS = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now() - startTime).count();
+#endif
+            LOG(NetworkCache, "(NetworkProcess) Network cache statistics database bootstrapping complete, entries=%lu time=%lldms", static_cast<size_t>(m_approximateEntryCount), elapsedMS);
+        }
+    });
+}
+
+void NetworkCacheStatistics::bootstrapFromNetworkCache(const String& networkCachePath)
+{
+    ASSERT(!RunLoop::isMain());
+
+    LOG(NetworkCache, "(NetworkProcess) Bootstrapping the network cache statistics database from the network cache...");
+
+    WebCore::SQLiteTransactionInProgressAutoCounter transactionCounter;
+    WebCore::SQLiteTransaction bootstrapTransaction(m_database);
+    bootstrapTransaction.begin();
+
+    traverseCacheFiles(networkCachePath, [this](const String& hash, const String&) {
+        addHashToDatabase(hash);
+    });
+    bootstrapTransaction.commit();
+}
+
+void NetworkCacheStatistics::shrinkIfNeeded()
+{
+    ASSERT(RunLoop::isMain());
+    const size_t maxEntries = 100000;
+
+    if (m_approximateEntryCount < maxEntries)
+        return;
+
+    LOG(NetworkCache, "(NetworkProcess) shrinking statistics cache m_approximateEntryCount=%lu, maxEntries=%lu", static_cast<size_t>(m_approximateEntryCount), maxEntries);
+
+    clear();
+
+    StringCapture networkCachePathCapture(NetworkCache::singleton().storagePath());
+    dispatch_async(m_backgroundIOQueue.get(), [this, networkCachePathCapture] {
+        bootstrapFromNetworkCache(networkCachePathCapture.string());
+        LOG(NetworkCache, "(NetworkProcess) statistics cache shrink completed m_approximateEntryCount=%lu", static_cast<size_t>(m_approximateEntryCount));
+    });
+}
+
+void NetworkCacheStatistics::recordNotUsingCacheForRequest(const NetworkCacheKey& key, const WebCore::ResourceRequest& request)
+{
+    String hash = key.hashAsString();
+    WebCore::URL requestURL = request.url();
+    queryWasEverRequested(hash, [this, hash, requestURL](bool wasEverRequested) {
+        if (wasEverRequested) {
+            LOG(NetworkCache, "(NetworkProcess) %s was previously requested but is not handled by the cache", requestURL.string().ascii().data());
+            // FIXME: Do diagnostic logging.
+        } else
+            markAsRequested(hash);
+    });
+}
+
+void NetworkCacheStatistics::recordRetrievalFailure(const NetworkCacheKey& key, const WebCore::ResourceRequest& request)
+{
+    String hash = key.hashAsString();
+    WebCore::URL requestURL = request.url();
+    queryWasEverRequested(hash, [this, hash, requestURL](bool wasPreviouslyRequested) {
+        if (wasPreviouslyRequested) {
+            LOG(NetworkCache, "(NetworkProcess) %s was previously cached but is no longer in the cache", requestURL.string().ascii().data());
+            // FIXME: Do diagnostic logging.
+        } else
+            markAsRequested(hash);
+    });
+}
+
+void NetworkCacheStatistics::recordRetrievedCachedEntry(const NetworkCacheKey& key, const WebCore::ResourceRequest& request, bool success)
+{
+    WebCore::URL requestURL = request.url();
+    if (success)
+        LOG(NetworkCache, "(NetworkProcess) %s is in the cache and is used", requestURL.string().ascii().data());
+    else
+        LOG(NetworkCache, "(NetworkProcess) %s is in the cache but wasn't used", requestURL.string().ascii().data());
+    // FIXME: Do diagnostic logging.
+}
+
+void NetworkCacheStatistics::markAsRequested(const String& hash)
+{
+    ASSERT(RunLoop::isMain());
+
+    StringCapture hashCapture(hash);
+    dispatch_async(m_backgroundIOQueue.get(), [this, hashCapture] {
+        WebCore::SQLiteTransactionInProgressAutoCounter transactionCounter;
+        if (m_database.isOpen())
+            addHashToDatabase(hashCapture.string());
+    });
+
+    shrinkIfNeeded();
+}
+
+void NetworkCacheStatistics::queryWasEverRequested(const String& hash, const RequestedCompletionHandler& completionHandler)
+{
+    ASSERT(RunLoop::isMain());
+
+    auto everRequestedQuery = std::make_unique<EverRequestedQuery>(EverRequestedQuery { hash, completionHandler });
+    auto& query = *everRequestedQuery;
+    m_activeQueries.add(WTF::move(everRequestedQuery));
+    dispatch_async(m_backgroundIOQueue.get(), [this, &query] {
+        WebCore::SQLiteTransactionInProgressAutoCounter transactionCounter;
+        bool wasAlreadyRequested = false;
+        if (m_database.isOpen()) {
+            WebCore::SQLiteStatement statement(m_database, ASCIILiteral("SELECT hash FROM AlreadyRequested WHERE hash=?"));
+            if (statement.prepare() == WebCore::SQLResultOk) {
+                statement.bindText(1, query.hash);
+                wasAlreadyRequested = statement.step() == WebCore::SQLResultRow;
+            }
+        }
+        dispatch_async(dispatch_get_main_queue(), [this, &query, wasAlreadyRequested] {
+            query.completionHandler(wasAlreadyRequested);
+            m_activeQueries.remove(&query);
+        });
+    });
+}
+
+void NetworkCacheStatistics::clear()
+{
+    ASSERT(RunLoop::isMain());
+
+    dispatch_async(m_backgroundIOQueue.get(), [this] {
+        WebCore::SQLiteTransactionInProgressAutoCounter transactionCounter;
+        if (m_database.isOpen()) {
+            executeSQLCommand(m_database, ASCIILiteral("DELETE FROM AlreadyRequested"));
+            m_approximateEntryCount = 0;
+        }
+    });
+}
+
+void NetworkCacheStatistics::addHashToDatabase(const String& hash)
+{
+    ASSERT(!RunLoop::isMain());
+    ASSERT(WebCore::SQLiteDatabaseTracker::hasTransactionInProgress());
+    ASSERT(m_database.isOpen());
+
+    WebCore::SQLiteStatement statement(m_database, ASCIILiteral("INSERT OR IGNORE INTO AlreadyRequested (hash) VALUES (?)"));
+    if (statement.prepare() != WebCore::SQLResultOk)
+        return;
+
+    statement.bindText(1, hash);
+    if (!executeSQLStatement(statement))
+        return;
+
+    ++m_approximateEntryCount;
+}
+
+} // namespace WebKit
+
+#endif // ENABLE(NETWORK_CACHE)
index 85cc028..c17b3ec 100644 (file)
@@ -155,6 +155,8 @@ public:
 
     static const unsigned version = 2;
 
+    const String& directoryPath() const { return m_directoryPath; }
+
 private:
     NetworkCacheStorage(const String& directoryPath);
 
index 72c06c6..e5db263 100644 (file)
@@ -30,8 +30,7 @@
 
 #include "Logging.h"
 #include "NetworkCacheCoders.h"
-#include <WebCore/FileSystem.h>
-#include <dirent.h>
+#include "NetworkCacheFileSystemPosix.h"
 #include <dispatch/dispatch.h>
 #include <sys/mman.h>
 #include <sys/stat.h>
@@ -44,36 +43,6 @@ namespace WebKit {
 static const char networkCacheSubdirectory[] = "WebKitCache";
 static const char versionDirectoryPrefix[] = "Version ";
 
-template <typename Function>
-static void traverseDirectory(const String& path, uint8_t type, const Function& function)
-{
-    DIR* dir = opendir(WebCore::fileSystemRepresentation(path).data());
-    if (!dir)
-        return;
-    struct dirent* dp;
-    while ((dp = readdir(dir))) {
-        if (dp->d_type != type)
-            continue;
-        const char* name = dp->d_name;
-        if (!strcmp(name, ".") || !strcmp(name, ".."))
-            continue;
-        function(String(name));
-    }
-    closedir(dir);
-}
-
-static void traverseCacheFiles(const String& cachePath, std::function<void (const String& fileName, const String& partitionPath)> function)
-{
-    traverseDirectory(cachePath, DT_DIR, [&cachePath, &function](const String& subdirName) {
-        String partitionPath = WebCore::pathByAppendingComponent(cachePath, subdirName);
-        traverseDirectory(partitionPath, DT_REG, [&function, &partitionPath](const String& fileName) {
-            if (fileName.length() != NetworkCacheKey::hashStringLength())
-                return;
-            function(fileName, partitionPath);
-        });
-    });
-}
-
 NetworkCacheStorage::Data::Data(const uint8_t* data, size_t size)
     : m_dispatchData(adoptDispatch(dispatch_data_create(data, size, nullptr, DISPATCH_DATA_DESTRUCTOR_DEFAULT)))
     , m_size(size)
index 7604409..eac58e9 100644 (file)
@@ -64,7 +64,7 @@ void NetworkProcess::platformInitializeNetworkProcessCocoa(const NetworkProcessC
     if (!m_diskCacheDirectory.isNull()) {
         SandboxExtension::consumePermanently(parameters.diskCacheDirectoryExtensionHandle);
 #if ENABLE(NETWORK_CACHE)
-        if (parameters.shouldEnableNetworkCache && NetworkCache::singleton().initialize(m_diskCacheDirectory)) {
+        if (parameters.shouldEnableNetworkCache && NetworkCache::singleton().initialize(m_diskCacheDirectory, parameters.shouldEnableNetworkCacheEfficacyLogging)) {
             RetainPtr<NSURLCache> urlCache(adoptNS([[NSURLCache alloc] initWithMemoryCapacity:0 diskCapacity:0 diskPath:nil]));
             [NSURLCache setSharedURLCache:urlCache.get()];
             return;
index 579e36b..021a726 100644 (file)
@@ -45,6 +45,7 @@ void NetworkProcessCreationParameters::encode(IPC::ArgumentEncoder& encoder) con
     encoder << diskCacheDirectoryExtensionHandle;
 #if ENABLE(NETWORK_CACHE)
     encoder << shouldEnableNetworkCache;
+    encoder << shouldEnableNetworkCacheEfficacyLogging;
 #endif
     encoder << cookieStorageDirectory;
 #if PLATFORM(IOS)
@@ -86,6 +87,8 @@ bool NetworkProcessCreationParameters::decode(IPC::ArgumentDecoder& decoder, Net
 #if ENABLE(NETWORK_CACHE)
     if (!decoder.decode(result.shouldEnableNetworkCache))
         return false;
+    if (!decoder.decode(result.shouldEnableNetworkCacheEfficacyLogging))
+        return false;
 #endif
     if (!decoder.decode(result.cookieStorageDirectory))
         return false;
index 4a1e4b7..1a54928 100644 (file)
@@ -58,6 +58,7 @@ struct NetworkProcessCreationParameters {
     SandboxExtension::Handle diskCacheDirectoryExtensionHandle;
 #if ENABLE(NETWORK_CACHE)
     bool shouldEnableNetworkCache;
+    bool shouldEnableNetworkCacheEfficacyLogging;
 #endif
 
     String cookieStorageDirectory;
index 91653dc..f00a5ff 100644 (file)
@@ -81,6 +81,7 @@ static NSString * const WebKit2HTTPSProxyDefaultsKey = @"WebKit2HTTPSProxy";
 
 #if ENABLE(NETWORK_CACHE)
 static NSString * const WebKitNetworkCacheEnabledDefaultsKey = @"WebKitNetworkCacheEnabled";
+static NSString * const WebKitNetworkCacheEfficacyLoggingEnabledDefaultsKey = @"WebKitNetworkCacheEfficacyLoggingEnabled";
 #endif
 
 namespace WebKit {
@@ -106,6 +107,7 @@ static void registerUserDefaultsIfNeeded()
 
 #if ENABLE(NETWORK_CACHE)
     [registrationDictionary setObject:[NSNumber numberWithBool:YES] forKey:WebKitNetworkCacheEnabledDefaultsKey];
+    [registrationDictionary setObject:[NSNumber numberWithBool:YES] forKey:WebKitNetworkCacheEfficacyLoggingEnabledDefaultsKey];
 #endif
 
     [[NSUserDefaults standardUserDefaults] registerDefaults:registrationDictionary];
@@ -252,6 +254,7 @@ void WebProcessPool::platformInitializeNetworkProcess(NetworkProcessCreationPara
 
 #if ENABLE(NETWORK_CACHE)
     parameters.shouldEnableNetworkCache = [[NSUserDefaults standardUserDefaults] boolForKey:WebKitNetworkCacheEnabledDefaultsKey];
+    parameters.shouldEnableNetworkCacheEfficacyLogging = [[NSUserDefaults standardUserDefaults] boolForKey:WebKitNetworkCacheEfficacyLoggingEnabledDefaultsKey];
 #endif
 }
 #endif
index 6dd8a29..0a0fafe 100644 (file)
                7CF47FFE17276AE3008ACB91 /* WKBundlePageBannerMac.mm in Sources */ = {isa = PBXBuildFile; fileRef = 7CF47FFC17276AE3008ACB91 /* WKBundlePageBannerMac.mm */; };
                7CF47FFF17276AE3008ACB91 /* WKBundlePageBannerMac.h in Headers */ = {isa = PBXBuildFile; fileRef = 7CF47FFD17276AE3008ACB91 /* WKBundlePageBannerMac.h */; settings = {ATTRIBUTES = (Private, ); }; };
                7EC4F0FB18E4ACBB008056AF /* NetworkProcessCocoa.mm in Sources */ = {isa = PBXBuildFile; fileRef = 7EC4F0F918E4A945008056AF /* NetworkProcessCocoa.mm */; };
+               834B250F1A831A8D00CFB150 /* NetworkCacheFileSystemPosix.h in Headers */ = {isa = PBXBuildFile; fileRef = 834B250E1A831A8D00CFB150 /* NetworkCacheFileSystemPosix.h */; };
+               834B25121A842C8700CFB150 /* NetworkCacheStatistics.h in Headers */ = {isa = PBXBuildFile; fileRef = 834B25101A842C8700CFB150 /* NetworkCacheStatistics.h */; };
+               834B25131A842C8700CFB150 /* NetworkCacheStatisticsCocoa.mm in Sources */ = {isa = PBXBuildFile; fileRef = 834B25111A842C8700CFB150 /* NetworkCacheStatisticsCocoa.mm */; };
                8372DB251A674C8F00C697C5 /* WKPageDiagnosticLoggingClient.h in Headers */ = {isa = PBXBuildFile; fileRef = 8372DB241A674C8F00C697C5 /* WKPageDiagnosticLoggingClient.h */; settings = {ATTRIBUTES = (Private, ); }; };
                8372DB281A67562800C697C5 /* WebPageDiagnosticLoggingClient.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 8372DB261A67562800C697C5 /* WebPageDiagnosticLoggingClient.cpp */; };
                8372DB291A67562800C697C5 /* WebPageDiagnosticLoggingClient.h in Headers */ = {isa = PBXBuildFile; fileRef = 8372DB271A67562800C697C5 /* WebPageDiagnosticLoggingClient.h */; };
                7CF47FFC17276AE3008ACB91 /* WKBundlePageBannerMac.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WKBundlePageBannerMac.mm; sourceTree = "<group>"; };
                7CF47FFD17276AE3008ACB91 /* WKBundlePageBannerMac.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WKBundlePageBannerMac.h; sourceTree = "<group>"; };
                7EC4F0F918E4A945008056AF /* NetworkProcessCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = NetworkProcessCocoa.mm; path = NetworkProcess/cocoa/NetworkProcessCocoa.mm; sourceTree = "<group>"; };
+               834B250E1A831A8D00CFB150 /* NetworkCacheFileSystemPosix.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NetworkCacheFileSystemPosix.h; sourceTree = "<group>"; };
+               834B25101A842C8700CFB150 /* NetworkCacheStatistics.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NetworkCacheStatistics.h; sourceTree = "<group>"; };
+               834B25111A842C8700CFB150 /* NetworkCacheStatisticsCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = NetworkCacheStatisticsCocoa.mm; sourceTree = "<group>"; };
                8372DB241A674C8F00C697C5 /* WKPageDiagnosticLoggingClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WKPageDiagnosticLoggingClient.h; sourceTree = "<group>"; };
                8372DB261A67562800C697C5 /* WebPageDiagnosticLoggingClient.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WebPageDiagnosticLoggingClient.cpp; sourceTree = "<group>"; };
                8372DB271A67562800C697C5 /* WebPageDiagnosticLoggingClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebPageDiagnosticLoggingClient.h; sourceTree = "<group>"; };
                                E489D2871A0A2DB80078C06A /* NetworkCacheDecoder.h */,
                                E489D2881A0A2DB80078C06A /* NetworkCacheEncoder.cpp */,
                                E489D2891A0A2DB80078C06A /* NetworkCacheEncoder.h */,
+                               834B250E1A831A8D00CFB150 /* NetworkCacheFileSystemPosix.h */,
                                E4436EC01A0CFDB200EAD204 /* NetworkCacheKey.cpp */,
                                E4436EC11A0CFDB200EAD204 /* NetworkCacheKey.h */,
+                               834B25101A842C8700CFB150 /* NetworkCacheStatistics.h */,
+                               834B25111A842C8700CFB150 /* NetworkCacheStatisticsCocoa.mm */,
                                E4436EC21A0CFDB200EAD204 /* NetworkCacheStorage.h */,
                                E4436EC31A0CFDB200EAD204 /* NetworkCacheStorageCocoa.mm */,
                        );
                                1AC1336C18565C7A00F3EC05 /* APIPageHandle.h in Headers */,
                                1AFDD3151891B54000153970 /* APIPolicyClient.h in Headers */,
                                7CE4D2201A4914CA00C7F152 /* APIProcessPoolConfiguration.h in Headers */,
+                               834B25121A842C8700CFB150 /* NetworkCacheStatistics.h in Headers */,
                                F634445612A885C8000612D8 /* APISecurityOrigin.h in Headers */,
                                75A8D2E1187DEC1A00C39C9E /* APISession.h in Headers */,
                                1AFDE6621954E9B100C48FFA /* APISessionState.h in Headers */,
                                1A2161B011F37664008AD0F5 /* NPRuntimeObjectMap.h in Headers */,
                                1A2162B111F38971008AD0F5 /* NPRuntimeUtilities.h in Headers */,
                                1A2D84A3127F6AD1001EB962 /* NPVariantData.h in Headers */,
+                               834B250F1A831A8D00CFB150 /* NetworkCacheFileSystemPosix.h in Headers */,
                                BC8ACA1316670D89004C1941 /* ObjCObjectGraph.h in Headers */,
                                BCCF672D12C7EDF7008F9C35 /* OriginAndDatabases.h in Headers */,
                                7CF47FFB17275C57008ACB91 /* PageBanner.h in Headers */,
                                1C8E293A12761E5B00BC7BD0 /* WKInspector.cpp in Sources */,
                                0F3C725C196F605200AEDD0C /* WKInspectorHighlightView.mm in Sources */,
                                A54293A5195A43DD002782C7 /* WKInspectorNodeSearchGestureRecognizer.mm in Sources */,
+                               834B25131A842C8700CFB150 /* NetworkCacheStatisticsCocoa.mm in Sources */,
                                51A9E10A1315CD18009E7031 /* WKKeyValueStorageManager.cpp in Sources */,
                                33D3A3B51339600B00709BE4 /* WKMediaCacheManager.cpp in Sources */,
                                BC4075FD124FF0270068F20A /* WKMutableArray.cpp in Sources */,