Compute quota after network process restart based on default quota and space used
authoryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 21 Mar 2019 04:38:29 +0000 (04:38 +0000)
committeryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 21 Mar 2019 04:38:29 +0000 (04:38 +0000)
https://bugs.webkit.org/show_bug.cgi?id=195804

Reviewed by Chris Dumez.

Source/WebCore:

At creation of quota manager, a default quota will be assigned.
This value is the same for all origins.
Some origins may have been granted a bigger quota by the user.
In that case, the space used might be greater for these origins.
Update at initialization time the quota according the space used as follows:
- If space used is below default quota, stick with default quota.
- If space used is above, set quota to space used rounded by one tenth of the default quota.
The rounding ensures that quota requests will not happen too quickly after a page is loaded.

Test: http/wpt/cache-storage/cache-quota-after-restart.any.html

* Modules/cache/CacheStorageConnection.h:
(WebCore::CacheStorageConnection::setQuotaBasedOnSpaceUsage):
* storage/StorageQuotaManager.cpp:
(WebCore::StorageQuotaManager::setQuotaBasedOnSpaceUsage):
(WebCore::StorageQuotaManager::addUser):
* storage/StorageQuotaManager.h:
* testing/Internals.cpp:
(WebCore::Internals::updateQuotaBasedOnSpaceUsage):
* testing/Internals.h:
* testing/Internals.idl:

Source/WebKit:

Make sure that Cache Storage quota user waits to declare as initialized to its manager
until all data is loaded so that it can report a valid space used from the start.

Add test API to reset the quota to its default value and compute it according current space use.

* NetworkProcess/NetworkProcess.cpp:
(WebKit::NetworkProcess::updateQuotaBasedOnSpaceUsageForTesting):
* NetworkProcess/NetworkProcess.h:
* NetworkProcess/NetworkProcess.messages.in:
* NetworkProcess/cache/CacheStorageEngineCaches.cpp:
(WebKit::CacheStorage::Caches::create):
(WebKit::CacheStorage::Caches::Caches):
(WebKit::CacheStorage::Caches::whenInitialized):
* NetworkProcess/cache/CacheStorageEngineCaches.h:
* NetworkProcess/cache/CacheStorageEngineConnection.cpp:
(WebKit::CacheStorageEngineConnection::dereference):
* WebProcess/Cache/WebCacheStorageConnection.cpp:
(WebKit::WebCacheStorageConnection::setQuotaBasedOnSpaceUsage):
* WebProcess/Cache/WebCacheStorageConnection.h:

LayoutTests:

* http/wpt/cache-storage/cache-quota-after-restart.any-expected.txt: Added.
* http/wpt/cache-storage/cache-quota-after-restart.any.html: Added.
* http/wpt/cache-storage/cache-quota-after-restart.any.js: Added.
(promise_test.async):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243276 268f45cc-cd09-0410-ab3c-d52691b4dbfc

20 files changed:
LayoutTests/ChangeLog
LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any-expected.txt [new file with mode: 0644]
LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any.html [new file with mode: 0644]
LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any.js [new file with mode: 0644]
Source/WebCore/ChangeLog
Source/WebCore/Modules/cache/CacheStorageConnection.h
Source/WebCore/storage/StorageQuotaManager.cpp
Source/WebCore/storage/StorageQuotaManager.h
Source/WebCore/testing/Internals.cpp
Source/WebCore/testing/Internals.h
Source/WebCore/testing/Internals.idl
Source/WebKit/ChangeLog
Source/WebKit/NetworkProcess/NetworkProcess.cpp
Source/WebKit/NetworkProcess/NetworkProcess.h
Source/WebKit/NetworkProcess/NetworkProcess.messages.in
Source/WebKit/NetworkProcess/cache/CacheStorageEngineCaches.cpp
Source/WebKit/NetworkProcess/cache/CacheStorageEngineCaches.h
Source/WebKit/NetworkProcess/cache/CacheStorageEngineConnection.cpp
Source/WebKit/WebProcess/Cache/WebCacheStorageConnection.cpp
Source/WebKit/WebProcess/Cache/WebCacheStorageConnection.h

index f57d105..61f92e6 100644 (file)
@@ -1,5 +1,17 @@
 2019-03-20  Youenn Fablet  <youenn@apple.com>
 
+        Compute quota after network process restart based on default quota and space used
+        https://bugs.webkit.org/show_bug.cgi?id=195804
+
+        Reviewed by Chris Dumez.
+
+        * http/wpt/cache-storage/cache-quota-after-restart.any-expected.txt: Added.
+        * http/wpt/cache-storage/cache-quota-after-restart.any.html: Added.
+        * http/wpt/cache-storage/cache-quota-after-restart.any.js: Added.
+        (promise_test.async):
+
+2019-03-20  Youenn Fablet  <youenn@apple.com>
+
         Include WAL and SHM file size in IDB database size computation
         https://bugs.webkit.org/show_bug.cgi?id=195688
 
diff --git a/LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any-expected.txt b/LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any-expected.txt
new file mode 100644 (file)
index 0000000..e204314
--- /dev/null
@@ -0,0 +1,7 @@
+CONSOLE MESSAGE: Cache API operation failed: Quota exceeded
+CONSOLE MESSAGE: Cache API operation failed: Quota exceeded
+
+PASS Increasing quota 
+PASS After network process restart, verify quota is computed according space being used 
+PASS After network process restart, verify quota is computed according space being used and does not increase 
+
diff --git a/LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any.html b/LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any.html
new file mode 100644 (file)
index 0000000..2382913
--- /dev/null
@@ -0,0 +1 @@
+<!-- This file is required for WebKit test infrastructure to run the templated test -->
\ No newline at end of file
diff --git a/LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any.js b/LayoutTests/http/wpt/cache-storage/cache-quota-after-restart.any.js
new file mode 100644 (file)
index 0000000..11cd7de
--- /dev/null
@@ -0,0 +1,43 @@
+var response1 = new Response(new ArrayBuffer(1 * 1024));
+var response30ko = new Response(new ArrayBuffer(30 * 1024));
+var response400 = new Response(new ArrayBuffer(400 * 1024));
+let cache;
+
+promise_test(async (test) => {
+    if (!window.testRunner)
+        return Promise.reject("Test requires internals");
+
+    testRunner.setAllowStorageQuotaIncrease(true);
+
+    cache = await self.caches.open("temp1");
+    await cache.put("400-v1", response400.clone());
+    await cache.put("1", response1.clone());
+    // quota should be equal to 801ko now.
+    await cache.put("400-v2", response400.clone());
+
+    testRunner.setAllowStorageQuotaIncrease(false);
+
+    return promise_rejects(test, "QuotaExceededError", cache.put("30ko", response30ko.clone()), "put should fail");
+}, 'Increasing quota');
+
+promise_test(async (test) => {
+    if (!window.internals)
+        return Promise.reject("Test requires internals");
+
+    // Space used is around 801ko. After rounding, quota should be 840ko.
+    internals.updateQuotaBasedOnSpaceUsage();
+
+    return cache.put("30ko1", response30ko.clone());
+}, 'After network process restart, verify quota is computed according space being used');
+
+promise_test(async (test) => {
+    if (!window.internals)
+        return Promise.reject("Test requires internals");
+
+    // Space used is around 831ko. After rounding, quota should be 840ko.
+    internals.updateQuotaBasedOnSpaceUsage();
+
+    return promise_rejects(test, "QuotaExceededError", cache.put("30ko2", response30ko.clone()), "put should fail");
+}, 'After network process restart, verify quota is computed according space being used and does not increase');
+
+done();
index 484fb3d..9f0b533 100644 (file)
@@ -1,3 +1,32 @@
+2019-03-20  Youenn Fablet  <youenn@apple.com>
+
+        Compute quota after network process restart based on default quota and space used
+        https://bugs.webkit.org/show_bug.cgi?id=195804
+
+        Reviewed by Chris Dumez.
+
+        At creation of quota manager, a default quota will be assigned.
+        This value is the same for all origins.
+        Some origins may have been granted a bigger quota by the user.
+        In that case, the space used might be greater for these origins.
+        Update at initialization time the quota according the space used as follows:
+        - If space used is below default quota, stick with default quota.
+        - If space used is above, set quota to space used rounded by one tenth of the default quota.
+        The rounding ensures that quota requests will not happen too quickly after a page is loaded.
+
+        Test: http/wpt/cache-storage/cache-quota-after-restart.any.html
+
+        * Modules/cache/CacheStorageConnection.h:
+        (WebCore::CacheStorageConnection::setQuotaBasedOnSpaceUsage):
+        * storage/StorageQuotaManager.cpp:
+        (WebCore::StorageQuotaManager::setQuotaBasedOnSpaceUsage):
+        (WebCore::StorageQuotaManager::addUser):
+        * storage/StorageQuotaManager.h:
+        * testing/Internals.cpp:
+        (WebCore::Internals::updateQuotaBasedOnSpaceUsage):
+        * testing/Internals.h:
+        * testing/Internals.idl:
+
 2019-03-20  Simon Fraser  <simon.fraser@apple.com>
 
         Rename ENABLE_ACCELERATED_OVERFLOW_SCROLLING macro to ENABLE_OVERFLOW_SCROLLING_TOUCH
index 4daa344..2a060fb 100644 (file)
@@ -55,6 +55,7 @@ public:
     // Used only for testing purposes.
     virtual void clearMemoryRepresentation(const ClientOrigin&, DOMCacheEngine::CompletionCallback&& callback) { callback(DOMCacheEngine::Error::NotImplemented); }
     virtual void engineRepresentation(WTF::Function<void(const String&)>&& callback) { callback(String { }); }
+    virtual void updateQuotaBasedOnSpaceUsage(const ClientOrigin&) { }
 
 protected:
     CacheStorageConnection() =  default;
index 3d34834..b7e71be 100644 (file)
@@ -44,6 +44,15 @@ uint64_t StorageQuotaManager::spaceUsage() const
     return usage;
 }
 
+void StorageQuotaManager::updateQuotaBasedOnSpaceUsage()
+{
+    if (!m_quota)
+        return;
+
+    auto defaultQuotaStep = m_quota / 10;
+    m_quota = std::max(m_quota, defaultQuotaStep * ((spaceUsage() / defaultQuotaStep) + 1));
+}
+
 void StorageQuotaManager::addUser(StorageQuotaUser& user)
 {
     ASSERT(!m_pendingInitializationUsers.contains(&user));
@@ -52,8 +61,14 @@ void StorageQuotaManager::addUser(StorageQuotaUser& user)
     user.whenInitialized([this, &user, weakThis = makeWeakPtr(this)]() {
         if (!weakThis)
             return;
-        m_pendingInitializationUsers.remove(&user);
-        m_users.add(&user);
+
+        if (m_pendingInitializationUsers.remove(&user))
+            m_users.add(&user);
+
+        if (!m_pendingInitializationUsers.isEmpty())
+            return;
+
+        updateQuotaBasedOnSpaceUsage();
         processPendingRequests({ });
     });
 }
index 0aab548..a0585ab 100644 (file)
@@ -62,6 +62,8 @@ public:
     WEBCORE_EXPORT void requestSpace(uint64_t, RequestCallback&&);
     void resetQuota(uint64_t newQuota) { m_quota = newQuota; }
 
+    WEBCORE_EXPORT void updateQuotaBasedOnSpaceUsage();
+
 private:
     uint64_t spaceUsage() const;
     bool shouldAskForMoreSpace(uint64_t spaceIncrease) const;
index b2e0f4f..07de946 100644 (file)
@@ -4757,6 +4757,22 @@ void Internals::cacheStorageEngineRepresentation(DOMPromiseDeferred<IDLDOMString
     });
 }
 
+void Internals::updateQuotaBasedOnSpaceUsage()
+{
+    auto* document = contextDocument();
+    if (!document)
+        return;
+
+    if (!m_cacheStorageConnection) {
+        if (auto* page = contextDocument()->page())
+            m_cacheStorageConnection = page->cacheStorageProvider().createCacheStorageConnection(page->sessionID());
+        if (!m_cacheStorageConnection)
+            return;
+    }
+
+    m_cacheStorageConnection->updateQuotaBasedOnSpaceUsage(ClientOrigin { document->topOrigin().data(), document->securityOrigin().data() });
+}
+
 void Internals::setConsoleMessageListener(RefPtr<StringCallback>&& listener)
 {
     if (!contextDocument())
index 76f0e9f..311db6a 100644 (file)
@@ -706,6 +706,8 @@ public:
     void setResponseSizeWithPadding(FetchResponse&, uint64_t size);
     uint64_t responseSizeWithPadding(FetchResponse&) const;
 
+    void updateQuotaBasedOnSpaceUsage();
+
     void setConsoleMessageListener(RefPtr<StringCallback>&&);
 
 #if ENABLE(SERVICE_WORKER)
index a90593b..26025f7 100644 (file)
@@ -689,6 +689,8 @@ enum CompositingPolicy {
     void setResponseSizeWithPadding(FetchResponse response, unsigned long long size);
     unsigned long long responseSizeWithPadding(FetchResponse response);
 
+    void updateQuotaBasedOnSpaceUsage();
+
     void setConsoleMessageListener(StringCallback callback);
 
     DOMString audioSessionCategory();
index 5386370..4613bc9 100644 (file)
@@ -1,3 +1,30 @@
+2019-03-20  Youenn Fablet  <youenn@apple.com>
+
+        Compute quota after network process restart based on default quota and space used
+        https://bugs.webkit.org/show_bug.cgi?id=195804
+
+        Reviewed by Chris Dumez.
+
+        Make sure that Cache Storage quota user waits to declare as initialized to its manager
+        until all data is loaded so that it can report a valid space used from the start.
+
+        Add test API to reset the quota to its default value and compute it according current space use.
+
+        * NetworkProcess/NetworkProcess.cpp:
+        (WebKit::NetworkProcess::updateQuotaBasedOnSpaceUsageForTesting):
+        * NetworkProcess/NetworkProcess.h:
+        * NetworkProcess/NetworkProcess.messages.in:
+        * NetworkProcess/cache/CacheStorageEngineCaches.cpp:
+        (WebKit::CacheStorage::Caches::create):
+        (WebKit::CacheStorage::Caches::Caches):
+        (WebKit::CacheStorage::Caches::whenInitialized):
+        * NetworkProcess/cache/CacheStorageEngineCaches.h:
+        * NetworkProcess/cache/CacheStorageEngineConnection.cpp:
+        (WebKit::CacheStorageEngineConnection::dereference):
+        * WebProcess/Cache/WebCacheStorageConnection.cpp:
+        (WebKit::WebCacheStorageConnection::setQuotaBasedOnSpaceUsage):
+        * WebProcess/Cache/WebCacheStorageConnection.h:
+
 2019-03-20  Simon Fraser  <simon.fraser@apple.com>
 
         Rename ENABLE_ACCELERATED_OVERFLOW_SCROLLING macro to ENABLE_OVERFLOW_SCROLLING_TOUCH
index 58b34a0..937a97d 100644 (file)
@@ -2228,6 +2228,13 @@ void NetworkProcess::setIDBPerOriginQuota(uint64_t quota)
 }
 #endif // ENABLE(INDEXED_DATABASE)
 
+void NetworkProcess::updateQuotaBasedOnSpaceUsageForTesting(PAL::SessionID sessionID, const ClientOrigin& origin)
+{
+    auto& manager = storageQuotaManager(sessionID, origin);
+    manager.resetQuota(m_storageQuotaManagers.find(sessionID)->value.defaultQuota);
+    manager.updateQuotaBasedOnSpaceUsage();
+}
+
 #if ENABLE(SANDBOX_EXTENSIONS)
 void NetworkProcess::getSandboxExtensionsForBlobFiles(const Vector<String>& filenames, CompletionHandler<void(SandboxExtension::HandleArray&&)>&& completionHandler)
 {
index 44f8197..133beaa 100644 (file)
@@ -281,6 +281,7 @@ public:
     void accessToTemporaryFileComplete(const String& path) final;
     void setIDBPerOriginQuota(uint64_t);
 #endif
+    void updateQuotaBasedOnSpaceUsageForTesting(PAL::SessionID, const WebCore::ClientOrigin&);
 
 #if ENABLE(SANDBOX_EXTENSIONS)
     void getSandboxExtensionsForBlobFiles(const Vector<String>& filenames, CompletionHandler<void(SandboxExtension::HandleArray&&)>&&);
index 987760f..013325f 100644 (file)
@@ -164,6 +164,7 @@ messages -> NetworkProcess LegacyReceiver {
 #if ENABLE(INDEXED_DATABASE)
     SetIDBPerOriginQuota(uint64_t quota)
 #endif
+    UpdateQuotaBasedOnSpaceUsageForTesting(PAL::SessionID sessionID, struct WebCore::ClientOrigin origin)
 
     StoreAdClickAttribution(PAL::SessionID sessionID, WebCore::AdClickAttribution adClickAttribution)
     DumpAdClickAttribution(PAL::SessionID sessionID) -> (String adClickAttributionState) Async
index 9c2aa37..c532f1c 100644 (file)
@@ -51,13 +51,19 @@ static inline String cachesOriginFilename(const String& cachesRootPath)
     return FileSystem::pathByAppendingComponent(cachesRootPath, "origin"_s);
 }
 
+Ref<Caches> Caches::create(Engine& engine, WebCore::ClientOrigin&& origin, String&& rootPath, WebCore::StorageQuotaManager& quotaManager)
+{
+    auto caches = adoptRef(*new Caches { engine, WTFMove(origin), WTFMove(rootPath), quotaManager });
+    quotaManager.addUser(caches.get());
+    return caches;
+}
+
 Caches::Caches(Engine& engine, WebCore::ClientOrigin&& origin, String&& rootPath, WebCore::StorageQuotaManager& quotaManager)
     : m_engine(&engine)
     , m_origin(WTFMove(origin))
     , m_rootPath(WTFMove(rootPath))
     , m_quotaManager(makeWeakPtr(quotaManager))
 {
-    quotaManager.addUser(*this);
 }
 
 Caches::~Caches()
@@ -68,6 +74,15 @@ Caches::~Caches()
         m_quotaManager->removeUser(*this);
 }
 
+void Caches::whenInitialized(CompletionHandler<void()>&& callback)
+{
+    initialize([callback = WTFMove(callback)](auto&& error) mutable {
+        if (error)
+            RELEASE_LOG_ERROR(CacheStorage, "Caches::initialize failed, reported space used will be zero");
+        callback();
+    });
+}
+
 void Caches::retrieveOriginFromDirectory(const String& folderPath, WorkQueue& queue, WTF::CompletionHandler<void(Optional<WebCore::ClientOrigin>&&)>&& completionHandler)
 {
     queue.dispatch([completionHandler = WTFMove(completionHandler), filename = cachesOriginFilename(folderPath)]() mutable {
index eee8267..33a162c 100644 (file)
@@ -44,7 +44,7 @@ class Engine;
 
 class Caches final : public RefCounted<Caches>, private WebCore::StorageQuotaUser {
 public:
-    static Ref<Caches> create(Engine& engine, WebCore::ClientOrigin&& origin, String&& rootPath, WebCore::StorageQuotaManager& quotaManager) { return adoptRef(*new Caches { engine, WTFMove(origin), WTFMove(rootPath), quotaManager }); }
+    static Ref<Caches> create(Engine&, WebCore::ClientOrigin&&, String&& rootPath, WebCore::StorageQuotaManager&);
     ~Caches();
 
     static void retrieveOriginFromDirectory(const String& folderPath, WorkQueue&, WTF::CompletionHandler<void(Optional<WebCore::ClientOrigin>&&)>&&);
@@ -91,6 +91,8 @@ private:
     void readCachesFromDisk(WTF::Function<void(Expected<Vector<Cache>, WebCore::DOMCacheEngine::Error>&&)>&&);
     void writeCachesToDisk(WebCore::DOMCacheEngine::CompletionCallback&&);
 
+    void whenInitialized(CompletionHandler<void()>&&) final;
+
     void storeOrigin(WebCore::DOMCacheEngine::CompletionCallback&&);
     static Optional<WebCore::ClientOrigin> readOrigin(const NetworkCache::Data&);
 
index 6ca46ad..5a22234 100644 (file)
@@ -138,7 +138,6 @@ void CacheStorageEngineConnection::dereference(PAL::SessionID sessionID, uint64_
     }).iterator->value;
 
     auto referenceResult = references.find(cacheIdentifier);
-    ASSERT(referenceResult != references.end());
     if (referenceResult == references.end())
         return;
 
index 22a00ad..fd254dc 100644 (file)
@@ -30,6 +30,7 @@
 #include "CacheStorageEngineConnectionMessages.h"
 #include "NetworkConnectionToWebProcessMessages.h"
 #include "NetworkProcessConnection.h"
+#include "NetworkProcessMessages.h"
 #include "WebCacheStorageProvider.h"
 #include "WebCoreArgumentCoders.h"
 #include "WebProcess.h"
@@ -151,4 +152,9 @@ void WebCacheStorageConnection::engineRepresentationCompleted(uint64_t requestId
         callback(result);
 }
 
+void WebCacheStorageConnection::updateQuotaBasedOnSpaceUsage(const ClientOrigin& origin)
+{
+    connection().send(Messages::NetworkProcess::UpdateQuotaBasedOnSpaceUsageForTesting(m_sessionID, origin), 0);
+}
+
 }
index 84d9e4d..c47fc74 100644 (file)
@@ -65,6 +65,7 @@ private:
 
     void clearMemoryRepresentation(const WebCore::ClientOrigin&, WebCore::DOMCacheEngine::CompletionCallback&&) final;
     void engineRepresentation(WTF::Function<void(const String&)>&&) final;
+    void updateQuotaBasedOnSpaceUsage(const WebCore::ClientOrigin&) final;
 
     void openCompleted(uint64_t requestIdentifier, const WebCore::DOMCacheEngine::CacheIdentifierOrError&);
     void removeCompleted(uint64_t requestIdentifier, const WebCore::DOMCacheEngine::CacheIdentifierOrError&);