ResourceLoadStatistics grandfathering happens much too often.
authorbeidson@apple.com <beidson@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 26 Jul 2017 05:44:05 +0000 (05:44 +0000)
committerbeidson@apple.com <beidson@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 26 Jul 2017 05:44:05 +0000 (05:44 +0000)
<rdar://problem/32655834> and https://bugs.webkit.org/show_bug.cgi?id=174825

Reviewed by Chris Dumez.

Source/WebKit:

The ResourceLoadStatistics grandfathering procedure happens too often.
- With an empty memory store, even though an empty memory store is a perfectly valid state.
- On each launch, even if grandfathering happened on the last launch - because grandfathering
  data would not automatically be saved to disk.
- After clearing all website data, at which point no grandfathering can possibly be relevant
  because there is no data to grandfather.

This patch fixes those cases and adds API tests to verify they remain fixed.

* Shared/WebsiteData/WebsiteDataType.h:

* UIProcess/API/Cocoa/WKWebsiteDataStore.mm:
(+[WKWebsiteDataStore _allWebsiteDataTypesIncludingPrivate]): allWebsiteDataTypes, but even with the private ones.
(-[WKWebsiteDataStore _resourceLoadStatisticsClearInMemoryAndPersistentStore]): If the types being cleared cover all of
  the types that ResourceLoadStatistics care about, don't grandfather afterwards.
(-[WKWebsiteDataStore _resourceLoadStatisticsClearInMemoryAndPersistentStoreModifiedSinceHours:]): Ditto.
(-[WKWebsiteDataStore _setResourceLoadStatisticsTestingCallback:]):
* UIProcess/API/Cocoa/WKWebsiteDataStorePrivate.h:

* UIProcess/Storage/ResourceLoadStatisticsPersistentStorage.cpp:
(WebKit::ResourceLoadStatisticsPersistentStorage::populateMemoryStoreFromDisk): Don't grandfather if the store read from
  disk is empty (as being empty is perfectly fine), and also log the event of "populated without grandfathering".
* UIProcess/Storage/ResourceLoadStatisticsPersistentStorage.h:

* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::topPrivatelyControlledDomainsWithWebsiteData):

* UIProcess/WebResourceLoadStatisticsStore.cpp:
(WebKit::WebResourceLoadStatisticsStore::monitoredDataTypes):
(WebKit::WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore):
(WebKit::WebResourceLoadStatisticsStore::~WebResourceLoadStatisticsStore):
(WebKit::WebResourceLoadStatisticsStore::removeDataRecords):
(WebKit::WebResourceLoadStatisticsStore::grandfatherExistingWebsiteData): Schedule a write right away so we don't re-grandfather
  on next launch, and also log the grandfathering event.
(WebKit::WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent): Takes a ShouldGrandfather flag
  to tell whether or not data should be re-grandfathered after the store is cleared.
(WebKit::WebResourceLoadStatisticsStore::logTestingEvent): Log the event to the testing client.
(WebKit::dataTypesToRemove): Deleted.
* UIProcess/WebResourceLoadStatisticsStore.h:

* UIProcess/WebsiteData/WebsiteDataStore.cpp:
(WebKit::WebsiteDataStore::removeData):
(WebKit::WebsiteDataStore::setResourceLoadStatisticsEnabled):
(WebKit::WebsiteDataStore::enableResourceLoadStatisticsAndSetTestingCallback): Handles enabling ResourceLoadStatistics both
  with and without a testing callback.
* UIProcess/WebsiteData/WebsiteDataStore.h:

Tools:

* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WebKit2Cocoa/EmptyGrandfatheredResourceLoadStatistics.plist: Added.
* TestWebKitAPI/Tests/WebKit2Cocoa/ResourceLoadStatistics.mm: Added.
(TEST):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@219904 268f45cc-cd09-0410-ab3c-d52691b4dbfc

15 files changed:
Source/WebKit/ChangeLog
Source/WebKit/Shared/WebsiteData/WebsiteDataType.h
Source/WebKit/UIProcess/API/Cocoa/WKWebsiteDataStore.mm
Source/WebKit/UIProcess/API/Cocoa/WKWebsiteDataStorePrivate.h
Source/WebKit/UIProcess/Storage/ResourceLoadStatisticsPersistentStorage.cpp
Source/WebKit/UIProcess/Storage/ResourceLoadStatisticsPersistentStorage.h
Source/WebKit/UIProcess/WebProcessProxy.cpp
Source/WebKit/UIProcess/WebResourceLoadStatisticsStore.cpp
Source/WebKit/UIProcess/WebResourceLoadStatisticsStore.h
Source/WebKit/UIProcess/WebsiteData/WebsiteDataStore.cpp
Source/WebKit/UIProcess/WebsiteData/WebsiteDataStore.h
Tools/ChangeLog
Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/EmptyGrandfatheredResourceLoadStatistics.plist [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/ResourceLoadStatistics.mm [new file with mode: 0644]

index d021812..9101ab9 100644 (file)
@@ -1,3 +1,57 @@
+2017-07-25  Brady Eidson  <beidson@apple.com>
+
+        ResourceLoadStatistics grandfathering happens much too often.
+        <rdar://problem/32655834> and https://bugs.webkit.org/show_bug.cgi?id=174825
+
+        Reviewed by Chris Dumez.
+
+        The ResourceLoadStatistics grandfathering procedure happens too often.
+        - With an empty memory store, even though an empty memory store is a perfectly valid state.
+        - On each launch, even if grandfathering happened on the last launch - because grandfathering
+          data would not automatically be saved to disk.
+        - After clearing all website data, at which point no grandfathering can possibly be relevant
+          because there is no data to grandfather.
+
+        This patch fixes those cases and adds API tests to verify they remain fixed.
+
+        * Shared/WebsiteData/WebsiteDataType.h:
+
+        * UIProcess/API/Cocoa/WKWebsiteDataStore.mm:
+        (+[WKWebsiteDataStore _allWebsiteDataTypesIncludingPrivate]): allWebsiteDataTypes, but even with the private ones.
+        (-[WKWebsiteDataStore _resourceLoadStatisticsClearInMemoryAndPersistentStore]): If the types being cleared cover all of 
+          the types that ResourceLoadStatistics care about, don't grandfather afterwards.
+        (-[WKWebsiteDataStore _resourceLoadStatisticsClearInMemoryAndPersistentStoreModifiedSinceHours:]): Ditto.
+        (-[WKWebsiteDataStore _setResourceLoadStatisticsTestingCallback:]):
+        * UIProcess/API/Cocoa/WKWebsiteDataStorePrivate.h:
+
+        * UIProcess/Storage/ResourceLoadStatisticsPersistentStorage.cpp:
+        (WebKit::ResourceLoadStatisticsPersistentStorage::populateMemoryStoreFromDisk): Don't grandfather if the store read from 
+          disk is empty (as being empty is perfectly fine), and also log the event of "populated without grandfathering".
+        * UIProcess/Storage/ResourceLoadStatisticsPersistentStorage.h:
+
+        * UIProcess/WebProcessProxy.cpp:
+        (WebKit::WebProcessProxy::topPrivatelyControlledDomainsWithWebsiteData):
+
+        * UIProcess/WebResourceLoadStatisticsStore.cpp:
+        (WebKit::WebResourceLoadStatisticsStore::monitoredDataTypes):
+        (WebKit::WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore):
+        (WebKit::WebResourceLoadStatisticsStore::~WebResourceLoadStatisticsStore):
+        (WebKit::WebResourceLoadStatisticsStore::removeDataRecords): 
+        (WebKit::WebResourceLoadStatisticsStore::grandfatherExistingWebsiteData): Schedule a write right away so we don't re-grandfather
+          on next launch, and also log the grandfathering event.
+        (WebKit::WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent): Takes a ShouldGrandfather flag
+          to tell whether or not data should be re-grandfathered after the store is cleared.
+        (WebKit::WebResourceLoadStatisticsStore::logTestingEvent): Log the event to the testing client.
+        (WebKit::dataTypesToRemove): Deleted.
+        * UIProcess/WebResourceLoadStatisticsStore.h:
+
+        * UIProcess/WebsiteData/WebsiteDataStore.cpp:
+        (WebKit::WebsiteDataStore::removeData):
+        (WebKit::WebsiteDataStore::setResourceLoadStatisticsEnabled):
+        (WebKit::WebsiteDataStore::enableResourceLoadStatisticsAndSetTestingCallback): Handles enabling ResourceLoadStatistics both
+          with and without a testing callback.
+        * UIProcess/WebsiteData/WebsiteDataStore.h:
+
 2017-07-25  Andy Estes  <aestes@apple.com>
 
         [Apple Pay] Add "carteBancaire" as a supported payment network
index cab8452..30cc033 100644 (file)
@@ -43,11 +43,8 @@ enum class WebsiteDataType {
 #if ENABLE(NETSCAPE_PLUGIN_API)
     PlugInData = 1 << 11,
 #endif
-#if ENABLE(MEDIA_STREAM)
-    MediaDeviceIdentifier = 1 << 12,
-#endif
-    ResourceLoadStatistics = 1 << 13,
-    Credentials = 1 << 14,
+    ResourceLoadStatistics = 1 << 12,
+    Credentials = 1 << 13,
 };
 
 };
index aa3a56b..41b18d0 100644 (file)
@@ -157,6 +157,23 @@ static Vector<WebKit::WebsiteDataRecord> toWebsiteDataRecords(NSArray *dataRecor
 
 @implementation WKWebsiteDataStore (WKPrivate)
 
++ (NSSet<NSString *> *)_allWebsiteDataTypesIncludingPrivate
+{
+    static dispatch_once_t onceToken;
+    static NSSet *allWebsiteDataTypes;
+    dispatch_once(&onceToken, ^ {
+        auto *privateTypes = @[_WKWebsiteDataTypeHSTSCache, _WKWebsiteDataTypeMediaKeys, _WKWebsiteDataTypeSearchFieldRecentSearches, _WKWebsiteDataTypeResourceLoadStatistics, _WKWebsiteDataTypeCredentials
+#if !TARGET_OS_IPHONE
+        , _WKWebsiteDataTypePlugInData
+#endif
+        ];
+
+        allWebsiteDataTypes = [[[self allWebsiteDataTypes] setByAddingObjectsFromArray:privateTypes] retain];
+    });
+
+    return allWebsiteDataTypes;
+}
+
 - (instancetype)_initWithConfiguration:(_WKWebsiteDataStoreConfiguration *)configuration
 {
     if (!(self = [super init]))
@@ -449,7 +466,7 @@ static Vector<WebKit::WebsiteDataRecord> toWebsiteDataRecords(NSArray *dataRecor
     if (!store)
         return;
 
-    store->scheduleClearInMemoryAndPersistent();
+    store->scheduleClearInMemoryAndPersistent(WebKit::WebResourceLoadStatisticsStore::ShouldGrandfather::Yes);
 }
 
 - (void)_resourceLoadStatisticsClearInMemoryAndPersistentStoreModifiedSinceHours:(unsigned)hours
@@ -458,7 +475,7 @@ static Vector<WebKit::WebsiteDataRecord> toWebsiteDataRecords(NSArray *dataRecor
     if (!store)
         return;
 
-    store->scheduleClearInMemoryAndPersistent(std::chrono::system_clock::now() - std::chrono::hours(hours));
+    store->scheduleClearInMemoryAndPersistent(std::chrono::system_clock::now() - std::chrono::hours(hours), WebKit::WebResourceLoadStatisticsStore::ShouldGrandfather::Yes);
 }
 
 - (void)_resourceLoadStatisticsResetToConsistentState
@@ -473,6 +490,22 @@ static Vector<WebKit::WebsiteDataRecord> toWebsiteDataRecords(NSArray *dataRecor
     store->scheduleClearInMemory();
 }
 
+- (void)_setResourceLoadStatisticsTestingCallback:(void (^)(WKWebsiteDataStore *, NSString *))callback
+{
+    if (callback) {
+        _websiteDataStore->websiteDataStore().enableResourceLoadStatisticsAndSetTestingCallback([callback = makeBlockPtr(callback), self](const String& event) {
+            callback(self, (NSString *)event);
+        });
+        return;
+    }
+
+    auto* store = _websiteDataStore->websiteDataStore().resourceLoadStatistics();
+    if (!store)
+        return;
+
+    store->setStatisticsTestingCallback(nullptr);
+}
+
 @end
 
 #endif // WK_API_ENABLED
index 4ee6c14..50f630c 100644 (file)
@@ -37,6 +37,8 @@ typedef NS_OPTIONS(NSUInteger, _WKWebsiteDataStoreFetchOptions) {
 
 @interface WKWebsiteDataStore (WKPrivate)
 
++ (NSSet<NSString *> *)_allWebsiteDataTypesIncludingPrivate;
+
 - (instancetype)_initWithConfiguration:(_WKWebsiteDataStoreConfiguration *)configuration WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));
 
 - (void)_fetchDataRecordsOfTypes:(NSSet<NSString *> *)dataTypes withOptions:(_WKWebsiteDataStoreFetchOptions)options completionHandler:(void (^)(NSArray<WKWebsiteDataRecord *> *))completionHandler;
@@ -71,6 +73,7 @@ typedef NS_OPTIONS(NSUInteger, _WKWebsiteDataStoreFetchOptions) {
 - (void)_resourceLoadStatisticsClearInMemoryAndPersistentStore WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));
 - (void)_resourceLoadStatisticsClearInMemoryAndPersistentStoreModifiedSinceHours:(unsigned)hours WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));
 - (void)_resourceLoadStatisticsResetToConsistentState WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));
+- (void)_setResourceLoadStatisticsTestingCallback:(nullable void (^)(WKWebsiteDataStore *, NSString *))callback;
 
 @end
 
index b4d92be..d304525 100644 (file)
@@ -234,8 +234,7 @@ void ResourceLoadStatisticsPersistentStorage::populateMemoryStoreFromDisk()
 
     m_lastStatisticsFileSyncTime = readTime;
 
-    if (m_memoryStore.isEmpty())
-        m_memoryStore.grandfatherExistingWebsiteData();
+    m_memoryStore.logTestingEvent(ASCIILiteral("PopulatedWithoutGrandfathering"));
 }
 
 void ResourceLoadStatisticsPersistentStorage::asyncWriteTimerFired()
@@ -280,12 +279,12 @@ void ResourceLoadStatisticsPersistentStorage::writeMemoryStoreToDisk()
     startMonitoringDisk();
 }
 
-void ResourceLoadStatisticsPersistentStorage::scheduleOrWriteMemoryStore()
+void ResourceLoadStatisticsPersistentStorage::scheduleOrWriteMemoryStore(ForceImmediateWrite forceImmediateWrite)
 {
     ASSERT(!RunLoop::isMain());
 
     auto timeSinceLastWrite = MonotonicTime::now() - m_lastStatisticsWriteTime;
-    if (timeSinceLastWrite < minimumWriteInterval) {
+    if (forceImmediateWrite != ForceImmediateWrite::Yes && timeSinceLastWrite < minimumWriteInterval) {
         if (!m_hasPendingWrite) {
             m_hasPendingWrite = true;
             Seconds delay = minimumWriteInterval - timeSinceLastWrite + 1_s;
index 81f7ccd..c5d2ed4 100644 (file)
@@ -45,7 +45,6 @@ public:
     ~ResourceLoadStatisticsPersistentStorage();
 
     void initialize();
-    void scheduleOrWriteMemoryStore();
     void clear();
 
     void finishAllPendingWorkSynchronously();
@@ -53,6 +52,12 @@ public:
     void ref();
     void deref();
 
+    enum class ForceImmediateWrite {
+        No,
+        Yes,
+    };
+    void scheduleOrWriteMemoryStore(ForceImmediateWrite);
+
 private:
     String storageDirectoryPath() const;
     String resourceLogFilePath() const;
index 4275056..5561167 100644 (file)
@@ -327,6 +327,10 @@ void WebProcessProxy::topPrivatelyControlledDomainsWithWebsiteData(OptionSet<Web
             callbackAggregator->removePendingCallback();
         });
     }
+
+    // FIXME: It's bizarre that this call is on WebProcessProxy and that it doesn't work if there are no visited pages.
+    // This should actually be a function of WebsiteDataStore and it should work even if there are no WebViews instances.
+    callbackAggregator->callIfNeeded();
 }
     
 void WebProcessProxy::notifyPageStatisticsAndDataRecordsProcessed()
index 7e61913..c60d7d5 100644 (file)
@@ -33,7 +33,6 @@
 #include "WebResourceLoadStatisticsTelemetry.h"
 #include "WebsiteDataFetchOption.h"
 #include "WebsiteDataStore.h"
-#include "WebsiteDataType.h"
 #include <WebCore/KeyedCoding.h>
 #include <WebCore/ResourceLoadStatistics.h>
 #include <wtf/CrossThreadCopier.h>
@@ -54,15 +53,12 @@ template<typename T> static inline String isolatedPrimaryDomain(const T& value)
     return ResourceLoadStatistics::primaryDomain(value).isolatedCopy();
 }
 
-static const OptionSet<WebsiteDataType>& dataTypesToRemove()
+const OptionSet<WebsiteDataType>& WebResourceLoadStatisticsStore::monitoredDataTypes()
 {
     static NeverDestroyed<OptionSet<WebsiteDataType>> dataTypes(std::initializer_list<WebsiteDataType>({
         WebsiteDataType::Cookies,
         WebsiteDataType::IndexedDBDatabases,
         WebsiteDataType::LocalStorage,
-#if ENABLE(MEDIA_STREAM)
-        WebsiteDataType::MediaDeviceIdentifier,
-#endif
         WebsiteDataType::MediaKeys,
         WebsiteDataType::OfflineWebApplicationCache,
 #if ENABLE(NETSCAPE_PLUGIN_API)
@@ -149,11 +145,12 @@ static Vector<OperatingDate> mergeOperatingDates(const Vector<OperatingDate>& ex
     return mergedDates;
 }
 
-WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore(const String& resourceLoadStatisticsDirectory, UpdateCookiePartitioningForDomainsHandler&& updateCookiePartitioningForDomainsHandler)
+WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore(const String& resourceLoadStatisticsDirectory, Function<void (const String&)>&& testingCallback, UpdateCookiePartitioningForDomainsHandler&& updateCookiePartitioningForDomainsHandler)
     : m_statisticsQueue(WorkQueue::create("WebResourceLoadStatisticsStore Process Data Queue", WorkQueue::Type::Serial, WorkQueue::QOS::Utility))
     , m_persistentStorage(*this, resourceLoadStatisticsDirectory)
     , m_updateCookiePartitioningForDomainsHandler(WTFMove(updateCookiePartitioningForDomainsHandler))
     , m_dailyTasksTimer(RunLoop::main(), this, &WebResourceLoadStatisticsStore::performDailyTasks)
+    , m_statisticsTestingCallback(WTFMove(testingCallback))
 {
     ASSERT(RunLoop::isMain());
 
@@ -192,7 +189,7 @@ void WebResourceLoadStatisticsStore::removeDataRecords()
     setDataRecordsBeingRemoved(true);
 
     RunLoop::main().dispatch([prevalentResourceDomains = CrossThreadCopier<Vector<String>>::copy(prevalentResourceDomains), this, protectedThis = makeRef(*this)] () mutable {
-        WebProcessProxy::deleteWebsiteDataForTopPrivatelyControlledDomainsInAllPersistentDataStores(dataTypesToRemove(), WTFMove(prevalentResourceDomains), m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, [this, protectedThis = WTFMove(protectedThis)](const HashSet<String>& domainsWithDeletedWebsiteData) mutable {
+        WebProcessProxy::deleteWebsiteDataForTopPrivatelyControlledDomainsInAllPersistentDataStores(WebResourceLoadStatisticsStore::monitoredDataTypes(), WTFMove(prevalentResourceDomains), m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, [this, protectedThis = WTFMove(protectedThis)](const HashSet<String>& domainsWithDeletedWebsiteData) mutable {
             m_statisticsQueue->dispatch([this, protectedThis = WTFMove(protectedThis), topDomains = CrossThreadCopier<HashSet<String>>::copy(domainsWithDeletedWebsiteData)] () mutable {
                 for (auto& prevalentResourceDomain : topDomains) {
                     auto& statistic = ensureResourceStatisticsForPrimaryDomain(prevalentResourceDomain);
@@ -232,7 +229,7 @@ void WebResourceLoadStatisticsStore::processStatisticsAndDataRecords()
         });
     }
 
-    m_persistentStorage.scheduleOrWriteMemoryStore();
+    m_persistentStorage.scheduleOrWriteMemoryStore(ResourceLoadStatisticsPersistentStorage::ForceImmediateWrite::No);
 }
 
 void WebResourceLoadStatisticsStore::resourceLoadStatisticsUpdated(Vector<WebCore::ResourceLoadStatistics>&& origins)
@@ -250,14 +247,19 @@ void WebResourceLoadStatisticsStore::grandfatherExistingWebsiteData()
     ASSERT(!RunLoop::isMain());
 
     RunLoop::main().dispatch([this, protectedThis = makeRef(*this)] () mutable {
-        WebProcessProxy::topPrivatelyControlledDomainsWithWebsiteData(dataTypesToRemove(), m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, [this, protectedThis = WTFMove(protectedThis)] (HashSet<String>&& topPrivatelyControlledDomainsWithWebsiteData) mutable {
+        // FIXME: This method being a static call on WebProcessProxy is wrong.
+        // It should be on the data store that this object belongs to.
+        WebProcessProxy::topPrivatelyControlledDomainsWithWebsiteData(WebResourceLoadStatisticsStore::monitoredDataTypes(), m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, [this, protectedThis = WTFMove(protectedThis)] (HashSet<String>&& topPrivatelyControlledDomainsWithWebsiteData) mutable {
             m_statisticsQueue->dispatch([this, protectedThis = WTFMove(protectedThis), topDomains = CrossThreadCopier<HashSet<String>>::copy(topPrivatelyControlledDomainsWithWebsiteData)] () mutable {
                 for (auto& topPrivatelyControlledDomain : topDomains) {
                     auto& statistic = ensureResourceStatisticsForPrimaryDomain(topPrivatelyControlledDomain);
                     statistic.grandfathered = true;
                 }
                 m_endOfGrandfatheringTimestamp = WallTime::now() + m_parameters.grandfatheringTime;
+                m_persistentStorage.scheduleOrWriteMemoryStore(ResourceLoadStatisticsPersistentStorage::ForceImmediateWrite::Yes);
             });
+
+            logTestingEvent(ASCIILiteral("Grandfathered"));
         });
     });
 }
@@ -483,21 +485,23 @@ void WebResourceLoadStatisticsStore::scheduleClearInMemory()
     });
 }
 
-void WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent()
+void WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent(ShouldGrandfather shouldGrandfather)
 {
     ASSERT(RunLoop::isMain());
-    m_statisticsQueue->dispatch([this, protectedThis = makeRef(*this)] {
+    m_statisticsQueue->dispatch([this, protectedThis = makeRef(*this), shouldGrandfather] {
         clearInMemory();
         m_persistentStorage.clear();
-        grandfatherExistingWebsiteData();
+        
+        if (shouldGrandfather == ShouldGrandfather::Yes)
+            grandfatherExistingWebsiteData();
     });
 }
 
-void WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent(std::chrono::system_clock::time_point modifiedSince)
+void WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent(std::chrono::system_clock::time_point modifiedSince, ShouldGrandfather shouldGrandfather)
 {
     // For now, be conservative and clear everything regardless of modifiedSince.
     UNUSED_PARAM(modifiedSince);
-    scheduleClearInMemoryAndPersistent();
+    scheduleClearInMemoryAndPersistent(shouldGrandfather);
 }
 
 void WebResourceLoadStatisticsStore::setTimeToLiveUserInteraction(Seconds seconds)
@@ -824,5 +828,20 @@ void WebResourceLoadStatisticsStore::resetParametersToDefaultValues()
 {
     m_parameters = { };
 }
-    
+
+void WebResourceLoadStatisticsStore::logTestingEvent(const String& event)
+{
+    if (!m_statisticsTestingCallback)
+        return;
+
+    if (RunLoop::isMain())
+        m_statisticsTestingCallback(event);
+    else {
+        RunLoop::main().dispatch([this, protectedThis = makeRef(*this), event = event.isolatedCopy()] {
+            if (m_statisticsTestingCallback)
+                m_statisticsTestingCallback(event);
+        });
+    }
+}
+
 } // namespace WebKit
index cd5b509..521bd32 100644 (file)
@@ -28,6 +28,7 @@
 #include "Connection.h"
 #include "ResourceLoadStatisticsClassifier.h"
 #include "ResourceLoadStatisticsPersistentStorage.h"
+#include "WebsiteDataType.h"
 #include <wtf/MonotonicTime.h>
 #include <wtf/RunLoop.h>
 #include <wtf/Vector.h>
@@ -59,13 +60,15 @@ enum class ShouldClearFirst;
 class WebResourceLoadStatisticsStore final : public IPC::Connection::WorkQueueMessageReceiver {
 public:
     using UpdateCookiePartitioningForDomainsHandler = WTF::Function<void(const Vector<String>& domainsToRemove, const Vector<String>& domainsToAdd, ShouldClearFirst)>;
-    static Ref<WebResourceLoadStatisticsStore> create(const String& resourceLoadStatisticsDirectory, UpdateCookiePartitioningForDomainsHandler&& updateCookiePartitioningForDomainsHandler = { })
+    static Ref<WebResourceLoadStatisticsStore> create(const String& resourceLoadStatisticsDirectory, Function<void (const String&)>&& testingCallback, UpdateCookiePartitioningForDomainsHandler&& updateCookiePartitioningForDomainsHandler = { })
     {
-        return adoptRef(*new WebResourceLoadStatisticsStore(resourceLoadStatisticsDirectory, WTFMove(updateCookiePartitioningForDomainsHandler)));
+        return adoptRef(*new WebResourceLoadStatisticsStore(resourceLoadStatisticsDirectory, WTFMove(testingCallback), WTFMove(updateCookiePartitioningForDomainsHandler)));
     }
 
     ~WebResourceLoadStatisticsStore();
 
+    static const OptionSet<WebsiteDataType>& monitoredDataTypes();
+
     bool isEmpty() const { return m_resourceStatisticsMap.isEmpty(); }
     WorkQueue& statisticsQueue() { return m_statisticsQueue.get(); }
 
@@ -98,8 +101,13 @@ public:
     void scheduleCookiePartitioningStateReset();
 
     void scheduleClearInMemory();
-    void scheduleClearInMemoryAndPersistent();
-    void scheduleClearInMemoryAndPersistent(std::chrono::system_clock::time_point modifiedSince);
+    
+    enum class ShouldGrandfather {
+        No,
+        Yes,
+    };
+    void scheduleClearInMemoryAndPersistent(ShouldGrandfather);
+    void scheduleClearInMemoryAndPersistent(std::chrono::system_clock::time_point modifiedSince, ShouldGrandfather);
 
     void setTimeToLiveUserInteraction(Seconds);
     void setTimeToLiveCookiePartitionFree(Seconds);
@@ -117,9 +125,12 @@ public:
     void mergeWithDataFromDecoder(WebCore::KeyedDecoder&);
     void clearInMemory();
     void grandfatherExistingWebsiteData();
-    
+
+    void setStatisticsTestingCallback(Function<void (const String&)>&& callback) { m_statisticsTestingCallback = WTFMove(callback); }
+    void logTestingEvent(const String&);
+
 private:
-    WebResourceLoadStatisticsStore(const String&, UpdateCookiePartitioningForDomainsHandler&&);
+    WebResourceLoadStatisticsStore(const String&, Function<void (const String&)>&& testingCallback, UpdateCookiePartitioningForDomainsHandler&&);
 
     void removeDataRecords();
 
@@ -178,6 +189,8 @@ private:
     Parameters m_parameters;
 
     bool m_dataRecordsBeingRemoved { false };
+
+    Function<void (const String&)> m_statisticsTestingCallback;
 };
 
 } // namespace WebKit
index c99a20e..f640b4e 100644 (file)
@@ -813,8 +813,19 @@ void WebsiteDataStore::removeData(OptionSet<WebsiteDataType> dataTypes, std::chr
     }
 #endif
 
-    if (dataTypes.contains(WebsiteDataType::ResourceLoadStatistics) && m_resourceLoadStatistics)
-        m_resourceLoadStatistics->scheduleClearInMemoryAndPersistent(modifiedSince);
+    // FIXME <rdar://problem/33491222>; scheduleClearInMemoryAndPersistent does not have a completion handler,
+    // so the completion handler for this removeData() call can be called prematurely.
+    if (dataTypes.contains(WebsiteDataType::ResourceLoadStatistics) && m_resourceLoadStatistics) {
+        auto deletedTypesRaw = dataTypes.toRaw();
+        auto monitoredTypesRaw = WebResourceLoadStatisticsStore::monitoredDataTypes().toRaw();
+
+        // If we are deleting all of the data types that the resource load statistics store monitors
+        // we do not need to re-grandfather old data.
+        if ((monitoredTypesRaw & deletedTypesRaw) == monitoredTypesRaw)
+            m_resourceLoadStatistics->scheduleClearInMemoryAndPersistent(modifiedSince, WebResourceLoadStatisticsStore::ShouldGrandfather::No);
+        else
+            m_resourceLoadStatistics->scheduleClearInMemoryAndPersistent(modifiedSince, WebResourceLoadStatisticsStore::ShouldGrandfather::Yes);
+    }
 
     // There's a chance that we don't have any pending callbacks. If so, we want to dispatch the completion handler right away.
     callbackAggregator->callIfNeeded();
@@ -1082,8 +1093,19 @@ void WebsiteDataStore::removeData(OptionSet<WebsiteDataType> dataTypes, const Ve
     }
 #endif
 
-    if (dataTypes.contains(WebsiteDataType::ResourceLoadStatistics) && m_resourceLoadStatistics)
-        m_resourceLoadStatistics->scheduleClearInMemoryAndPersistent();
+    // FIXME <rdar://problem/33491222>; scheduleClearInMemoryAndPersistent does not have a completion handler,
+    // so the completion handler for this removeData() call can be called prematurely.
+    if (dataTypes.contains(WebsiteDataType::ResourceLoadStatistics) && m_resourceLoadStatistics) {
+        auto deletedTypesRaw = dataTypes.toRaw();
+        auto monitoredTypesRaw = WebResourceLoadStatisticsStore::monitoredDataTypes().toRaw();
+
+        // If we are deleting all of the data types that the resource load statistics store monitors
+        // we do not need to re-grandfather old data.
+        if ((monitoredTypesRaw & deletedTypesRaw) == monitoredTypesRaw)
+            m_resourceLoadStatistics->scheduleClearInMemoryAndPersistent(WebResourceLoadStatisticsStore::ShouldGrandfather::No);
+        else
+            m_resourceLoadStatistics->scheduleClearInMemoryAndPersistent(WebResourceLoadStatisticsStore::ShouldGrandfather::Yes);
+    }
 
     // There's a chance that we don't have any pending callbacks. If so, we want to dispatch the completion handler right away.
     callbackAggregator->callIfNeeded();
@@ -1270,18 +1292,33 @@ void WebsiteDataStore::setResourceLoadStatisticsEnabled(bool enabled)
         return;
 
     if (enabled) {
+        ASSERT(!m_resourceLoadStatistics);
+        enableResourceLoadStatisticsAndSetTestingCallback(nullptr);
+        return;
+    }
+
+    m_resourceLoadStatistics = nullptr;
+    for (auto& processPool : processPools())
+        processPool->setResourceLoadStatisticsEnabled(false);
+}
+
+void WebsiteDataStore::enableResourceLoadStatisticsAndSetTestingCallback(Function<void (const String&)>&& callback)
+{
+    if (m_resourceLoadStatistics) {
+        m_resourceLoadStatistics->setStatisticsTestingCallback(WTFMove(callback));
+        return;
+    }
+
 #if HAVE(CFNETWORK_STORAGE_PARTITIONING)
-        m_resourceLoadStatistics = WebResourceLoadStatisticsStore::create(m_configuration.resourceLoadStatisticsDirectory, [this] (const Vector<String>& domainsToRemove, const Vector<String>& domainsToAdd, ShouldClearFirst shouldClearFirst) {
-            updateCookiePartitioningForTopPrivatelyOwnedDomains(domainsToRemove, domainsToAdd, shouldClearFirst);
-        });
+    m_resourceLoadStatistics = WebResourceLoadStatisticsStore::create(m_configuration.resourceLoadStatisticsDirectory, WTFMove(callback), [this] (const Vector<String>& domainsToRemove, const Vector<String>& domainsToAdd, ShouldClearFirst shouldClearFirst) {
+        updateCookiePartitioningForTopPrivatelyOwnedDomains(domainsToRemove, domainsToAdd, shouldClearFirst);
+    });
 #else
-        m_resourceLoadStatistics = WebResourceLoadStatisticsStore::create(m_configuration.resourceLoadStatisticsDirectory);
+    m_resourceLoadStatistics = WebResourceLoadStatisticsStore::create(m_configuration.resourceLoadStatisticsDirectory, nullptr);
 #endif
-    } else
-        m_resourceLoadStatistics = nullptr;
 
     for (auto& processPool : processPools())
-        processPool->setResourceLoadStatisticsEnabled(enabled);
+        processPool->setResourceLoadStatisticsEnabled(true);
 }
 
 DatabaseProcessCreationParameters WebsiteDataStore::databaseProcessParameters()
index 2869b9d..8e7749e 100644 (file)
@@ -127,6 +127,8 @@ public:
     void addPendingCookie(const WebCore::Cookie&);
     void removePendingCookie(const WebCore::Cookie&);
 
+    void enableResourceLoadStatisticsAndSetTestingCallback(Function<void (const String&)>&& callback);
+
 private:
     explicit WebsiteDataStore(WebCore::SessionID);
     explicit WebsiteDataStore(Configuration, WebCore::SessionID);
index 227395e..bd3a7b0 100644 (file)
@@ -1,3 +1,15 @@
+2017-07-25  Brady Eidson  <beidson@apple.com>
+
+        ResourceLoadStatistics grandfathering happens much too often.
+        <rdar://problem/32655834> and https://bugs.webkit.org/show_bug.cgi?id=174825
+
+        Reviewed by Chris Dumez.
+
+        * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
+        * TestWebKitAPI/Tests/WebKit2Cocoa/EmptyGrandfatheredResourceLoadStatistics.plist: Added.
+        * TestWebKitAPI/Tests/WebKit2Cocoa/ResourceLoadStatistics.mm: Added.
+        (TEST):
+
 2017-07-25  Matthew Stewart  <matthew_r_stewart@apple.com>
 
         Fix autoinstaller failing on autoinstall_everything
index d73c380..c52e547 100644 (file)
                51BCEE4E1C84F53B0042C82E /* IndexedDBMultiProcess-1.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 51BCEE4C1C84F52C0042C82E /* IndexedDBMultiProcess-1.html */; };
                51BCEE4F1C84F53B0042C82E /* IndexedDBMultiProcess-2.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 51BCEE4D1C84F52C0042C82E /* IndexedDBMultiProcess-2.html */; };
                51C683DE1EA134E800650183 /* WKURLSchemeHandler-1.mm in Sources */ = {isa = PBXBuildFile; fileRef = 51C683DD1EA134DB00650183 /* WKURLSchemeHandler-1.mm */; };
+               51C8E1A51F26AF4C00BF731B /* ResourceLoadStatistics.mm in Sources */ = {isa = PBXBuildFile; fileRef = 51C8E1A41F26AC5400BF731B /* ResourceLoadStatistics.mm */; };
+               51C8E1A91F27F49600BF731B /* EmptyGrandfatheredResourceLoadStatistics.plist in Copy Resources */ = {isa = PBXBuildFile; fileRef = 51C8E1A81F27F47300BF731B /* EmptyGrandfatheredResourceLoadStatistics.plist */; };
                51CD1C6C1B38CE4300142CA5 /* ModalAlerts.mm in Sources */ = {isa = PBXBuildFile; fileRef = 51CD1C6A1B38CE3600142CA5 /* ModalAlerts.mm */; };
                51CD1C721B38D48400142CA5 /* modal-alerts-in-new-about-blank-window.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 51CD1C711B38D48400142CA5 /* modal-alerts-in-new-about-blank-window.html */; };
                51D124981E763B02002B2820 /* WKHTTPCookieStore.mm in Sources */ = {isa = PBXBuildFile; fileRef = 51D124971E763AF8002B2820 /* WKHTTPCookieStore.mm */; };
                7CCE7EC11A411A7E00447C4C /* HTMLCollectionNamedItem.mm in Sources */ = {isa = PBXBuildFile; fileRef = 9B4F8FA3159D52B1002D9F94 /* HTMLCollectionNamedItem.mm */; };
                7CCE7EC21A411A7E00447C4C /* HTMLFormCollectionNamedItem.mm in Sources */ = {isa = PBXBuildFile; fileRef = 9B26FC6B159D061000CC3765 /* HTMLFormCollectionNamedItem.mm */; };
                7CCE7EC31A411A7E00447C4C /* InspectorBar.mm in Sources */ = {isa = PBXBuildFile; fileRef = C507E8A614C6545B005D6B3B /* InspectorBar.mm */; };
-               95646E5B1F1DB60E00DE0EB9 /* InspectorValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 953F47911F1DB40300E3D1E3 /* InspectorValue.cpp */; };
                7CCE7EC41A411A7E00447C4C /* JSWrapperForNodeInWebFrame.mm in Sources */ = {isa = PBXBuildFile; fileRef = 4BB4160116815B2600824238 /* JSWrapperForNodeInWebFrame.mm */; };
                7CCE7EC51A411A7E00447C4C /* MemoryCacheDisableWithinResourceLoadDelegate.mm in Sources */ = {isa = PBXBuildFile; fileRef = E1220D9F155B25480013E2FC /* MemoryCacheDisableWithinResourceLoadDelegate.mm */; };
                7CCE7EC61A411A7E00447C4C /* MemoryCachePruneWithinResourceLoadDelegate.mm in Sources */ = {isa = PBXBuildFile; fileRef = 517E7DFB15110EA600D0B008 /* MemoryCachePruneWithinResourceLoadDelegate.mm */; };
                93F56DA71E5F9174003EDE84 /* libicucore.dylib in Frameworks */ = {isa = PBXBuildFile; fileRef = 7C83E0331D0A5F2700FEBCF3 /* libicucore.dylib */; };
                93F56DA91E5F919D003EDE84 /* WKWebViewSnapshot.mm in Sources */ = {isa = PBXBuildFile; fileRef = 93F56DA81E5F9181003EDE84 /* WKWebViewSnapshot.mm */; };
                93F7E86F14DC8E5C00C84A99 /* NewFirstVisuallyNonEmptyLayoutFrames_Bundle.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 93F7E86E14DC8E5B00C84A99 /* NewFirstVisuallyNonEmptyLayoutFrames_Bundle.cpp */; };
+               95646E5B1F1DB60E00DE0EB9 /* InspectorValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 953F47911F1DB40300E3D1E3 /* InspectorValue.cpp */; };
                9984FACC1CFFAF60008D198C /* WKWebViewTextInput.mm in Sources */ = {isa = PBXBuildFile; fileRef = 9984FACA1CFFAEEE008D198C /* WKWebViewTextInput.mm */; };
                9984FACE1CFFB090008D198C /* editable-body.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = 9984FACD1CFFB038008D198C /* editable-body.html */; };
                9B0786A51C5885C300D159E3 /* InjectedBundleMakeAllShadowRootsOpen_Bundle.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9B0786A41C5885C300D159E3 /* InjectedBundleMakeAllShadowRootsOpen_Bundle.cpp */; };
                        dstPath = TestWebKitAPI.resources;
                        dstSubfolderSpec = 7;
                        files = (
+                               51C8E1A91F27F49600BF731B /* EmptyGrandfatheredResourceLoadStatistics.plist in Copy Resources */,
                                F45B63FB1F197F4A009D38B9 /* image-map.html in Copy Resources */,
                                F407FE391F1D0DFC0017CF25 /* enormous.svg in Copy Resources */,
                                F4D5E4E81F0C5D38008C1A49 /* dragstart-clear-selection.html in Copy Resources */,
                51BCEE4C1C84F52C0042C82E /* IndexedDBMultiProcess-1.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "IndexedDBMultiProcess-1.html"; sourceTree = "<group>"; };
                51BCEE4D1C84F52C0042C82E /* IndexedDBMultiProcess-2.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "IndexedDBMultiProcess-2.html"; sourceTree = "<group>"; };
                51C683DD1EA134DB00650183 /* WKURLSchemeHandler-1.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = "WKURLSchemeHandler-1.mm"; sourceTree = "<group>"; };
+               51C8E1A41F26AC5400BF731B /* ResourceLoadStatistics.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = ResourceLoadStatistics.mm; sourceTree = "<group>"; };
+               51C8E1A81F27F47300BF731B /* EmptyGrandfatheredResourceLoadStatistics.plist */ = {isa = PBXFileReference; lastKnownFileType = file.bplist; path = EmptyGrandfatheredResourceLoadStatistics.plist; sourceTree = "<group>"; };
                51CB4AD71B3A079C00C1B1C6 /* ModalAlertsSPI.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ModalAlertsSPI.cpp; sourceTree = "<group>"; };
                51CD1C6A1B38CE3600142CA5 /* ModalAlerts.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = ModalAlerts.mm; sourceTree = "<group>"; };
                51CD1C711B38D48400142CA5 /* modal-alerts-in-new-about-blank-window.html */ = {isa = PBXFileReference; lastKnownFileType = text.html; path = "modal-alerts-in-new-about-blank-window.html"; sourceTree = "<group>"; };
                93F56DA81E5F9181003EDE84 /* WKWebViewSnapshot.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WKWebViewSnapshot.mm; sourceTree = "<group>"; };
                93F7E86B14DC8E4D00C84A99 /* NewFirstVisuallyNonEmptyLayoutFrames.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = NewFirstVisuallyNonEmptyLayoutFrames.cpp; sourceTree = "<group>"; };
                93F7E86E14DC8E5B00C84A99 /* NewFirstVisuallyNonEmptyLayoutFrames_Bundle.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = NewFirstVisuallyNonEmptyLayoutFrames_Bundle.cpp; sourceTree = "<group>"; };
+               953F47911F1DB40300E3D1E3 /* InspectorValue.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = InspectorValue.cpp; sourceTree = "<group>"; };
                9984FACA1CFFAEEE008D198C /* WKWebViewTextInput.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WKWebViewTextInput.mm; sourceTree = "<group>"; };
                9984FACD1CFFB038008D198C /* editable-body.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "editable-body.html"; sourceTree = "<group>"; };
                9B0786A21C58830F00D159E3 /* InjectedBundleMakeAllShadowRootsOpen.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InjectedBundleMakeAllShadowRootsOpen.cpp; sourceTree = "<group>"; };
                C2CF975916CEC69E0054E99D /* JSContextBackForwardCache2.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = JSContextBackForwardCache2.html; sourceTree = "<group>"; };
                C2EB2DD116CAC7AC009B52EE /* WebViewDidCreateJavaScriptContext.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebViewDidCreateJavaScriptContext.mm; sourceTree = "<group>"; };
                C507E8A614C6545B005D6B3B /* InspectorBar.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = InspectorBar.mm; sourceTree = "<group>"; };
-               953F47911F1DB40300E3D1E3 /* InspectorValue.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = InspectorValue.cpp; sourceTree = "<group>"; };
                C5101C4E176B8BB900EE9B15 /* findRanges.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = findRanges.html; sourceTree = "<group>"; };
                C51AFB98169F49FF009CCF66 /* FindMatches.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = FindMatches.mm; sourceTree = "<group>"; };
                C540F775152E4DA000A40C8C /* SimplifyMarkup.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = SimplifyMarkup.mm; sourceTree = "<group>"; };
                                A12DDBFC1E836FF100CF6CAE /* RenderedImageWithOptionsPlugIn.mm */,
                                A12DDC011E8374F500CF6CAE /* RenderedImageWithOptionsProtocol.h */,
                                CD9E292B1C90A71F000BB800 /* RequiresUserActionForPlayback.mm */,
+                               51C8E1A41F26AC5400BF731B /* ResourceLoadStatistics.mm */,
                                A180C0F91EE67DF000468F47 /* RunOpenPanel.mm */,
                                37BCA61B1B596BA9002012CA /* ShouldOpenExternalURLsInNewWindowActions.mm */,
                                2D9A53AE1B31FA8D0074D5AA /* ShrinkToFit.mm */,
                                5714ECBC1CA8C21800051AC8 /* DownloadRequestOriginalURL2.html */,
                                5714ECBA1CA8BFD100051AC8 /* DownloadRequestOriginalURLFrame.html */,
                                A155022B1E050BC500A24C57 /* duplicate-completion-handler-calls.html */,
+                               51C8E1A81F27F47300BF731B /* EmptyGrandfatheredResourceLoadStatistics.plist */,
                                9984FACD1CFFB038008D198C /* editable-body.html */,
                                F407FE381F1D0DE60017CF25 /* enormous.svg */,
                                F4C2AB211DD6D94100E06D5B /* enormous-video-with-sound.html */,
                                7C83E0BB1D0A650000FEBCF3 /* FindInPage.mm in Sources */,
                                7CCE7EF41A411AE600447C4C /* FindMatches.mm in Sources */,
                                7C83E0401D0A63E300FEBCF3 /* FirstResponderScrollingPosition.mm in Sources */,
+                               51C8E1A51F26AF4C00BF731B /* ResourceLoadStatistics.mm in Sources */,
                                7C83E0BC1D0A650700FEBCF3 /* FixedLayoutSize.mm in Sources */,
                                7A909A7E1D877480007E10F8 /* FloatPoint.cpp in Sources */,
                                7A909A7F1D877480007E10F8 /* FloatRect.cpp in Sources */,
diff --git a/Tools/TestWebKitAPI/Tests/WebKit2Cocoa/EmptyGrandfatheredResourceLoadStatistics.plist b/Tools/TestWebKitAPI/Tests/WebKit2Cocoa/EmptyGrandfatheredResourceLoadStatistics.plist
new file mode 100644 (file)
index 0000000..3114edc
Binary files /dev/null and b/Tools/TestWebKitAPI/Tests/WebKit2Cocoa/EmptyGrandfatheredResourceLoadStatistics.plist differ
diff --git a/Tools/TestWebKitAPI/Tests/WebKit2Cocoa/ResourceLoadStatistics.mm b/Tools/TestWebKitAPI/Tests/WebKit2Cocoa/ResourceLoadStatistics.mm
new file mode 100644 (file)
index 0000000..a9a69c4
--- /dev/null
@@ -0,0 +1,120 @@
+/*
+ * Copyright (C) 2017 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#import "PlatformUtilities.h"
+#import <WebKit/WKFoundation.h>
+#import <WebKit/WKWebsiteDataRecordPrivate.h>
+#import <WebKit/WKWebsiteDataStorePrivate.h>
+#import <wtf/RetainPtr.h>
+
+#if WK_API_ENABLED
+
+TEST(ResourceLoadStatistics, GrandfatherCallback)
+{
+    auto *dataStore = [WKWebsiteDataStore defaultDataStore];
+    [dataStore _setResourceLoadStatisticsEnabled:NO];
+
+    NSURL *statisticsDirectoryURL = [NSURL fileURLWithPath:[@"~/Library/WebKit/TestWebKitAPI/WebsiteData/ResourceLoadStatistics" stringByExpandingTildeInPath] isDirectory:YES];
+    NSURL *fileURL = [statisticsDirectoryURL URLByAppendingPathComponent:@"full_browsing_session_resourceLog.plist"];
+    [[NSFileManager defaultManager] removeItemAtURL:fileURL error:nil];
+    [[NSFileManager defaultManager] removeItemAtURL:statisticsDirectoryURL error:nil];
+    EXPECT_FALSE([[NSFileManager defaultManager] fileExistsAtPath:statisticsDirectoryURL.path]);
+
+    static bool doneFlag;
+    static bool grandfatheredFlag;
+
+    [dataStore _setResourceLoadStatisticsTestingCallback:^(WKWebsiteDataStore *, NSString *message) {
+        ASSERT_TRUE([message isEqualToString:@"Grandfathered"]);
+        grandfatheredFlag = true;
+    }];
+
+    TestWebKitAPI::Util::run(&grandfatheredFlag);
+
+    // Spin the runloop until the resource load statistics file has written to disk.
+    // If the test enters a spin loop here, it has failed.
+    while (true) {
+        if ([[NSFileManager defaultManager] fileExistsAtPath:fileURL.path])
+            break;
+        TestWebKitAPI::Util::spinRunLoop(1);
+    }
+
+    grandfatheredFlag = false;
+    [dataStore removeDataOfTypes:[WKWebsiteDataStore _allWebsiteDataTypesIncludingPrivate]  modifiedSince:[NSDate distantPast] completionHandler:^ {
+        doneFlag = true;
+    }];
+
+    TestWebKitAPI::Util::run(&doneFlag);
+    TestWebKitAPI::Util::spinRunLoop(10);
+
+    // The website data store remove should have completed, but since we removed all of the data types that are monitored by resource load statistics,
+    // no grandfathering call should have been made.
+    EXPECT_FALSE(grandfatheredFlag);
+    EXPECT_FALSE([[NSFileManager defaultManager] fileExistsAtPath:fileURL.path]);
+
+    doneFlag = false;
+    [dataStore removeDataOfTypes:[NSSet setWithObjects:WKWebsiteDataTypeCookies, _WKWebsiteDataTypeResourceLoadStatistics, nil] modifiedSince:[NSDate distantPast] completionHandler:^ {
+        doneFlag = true;
+    }];
+
+    // Since we did not remove every data type covered by resource load statistics, we do expect that grandfathering took place again.
+    // If the test hangs waiting on either of these conditions, it has failed.
+    TestWebKitAPI::Util::run(&grandfatheredFlag);
+    TestWebKitAPI::Util::run(&doneFlag);
+
+    // Spin the runloop until the resource load statistics file has written to disk.
+    // If the test enters a spin loop here, it has failed.
+    while (true) {
+        if ([[NSFileManager defaultManager] fileExistsAtPath:fileURL.path])
+            break;
+        TestWebKitAPI::Util::spinRunLoop(1);
+    }
+}
+
+TEST(ResourceLoadStatistics, ShouldNotGrandfatherOnStartup)
+{
+    auto *dataStore = [WKWebsiteDataStore defaultDataStore];
+    [dataStore _setResourceLoadStatisticsEnabled:NO];
+
+    NSURL *statisticsDirectoryURL = [NSURL fileURLWithPath:[@"~/Library/WebKit/TestWebKitAPI/WebsiteData/ResourceLoadStatistics" stringByExpandingTildeInPath] isDirectory:YES];
+    NSURL *targetURL = [statisticsDirectoryURL URLByAppendingPathComponent:@"full_browsing_session_resourceLog.plist"];
+    NSURL *testResourceURL = [[NSBundle mainBundle] URLForResource:@"EmptyGrandfatheredResourceLoadStatistics" withExtension:@"plist" subdirectory:@"TestWebKitAPI.resources"];
+
+    [[NSFileManager defaultManager] createDirectoryAtURL:statisticsDirectoryURL withIntermediateDirectories:YES attributes:nil error:nil];
+    [[NSFileManager defaultManager] copyItemAtURL:testResourceURL toURL:targetURL error:nil];
+
+    EXPECT_TRUE([[NSFileManager defaultManager] fileExistsAtPath:targetURL.path]);
+
+    static bool callbackFlag;
+    [dataStore _setResourceLoadStatisticsTestingCallback:^(WKWebsiteDataStore *, NSString *message) {
+        ASSERT_TRUE([message isEqualToString:@"PopulatedWithoutGrandfathering"]);
+        callbackFlag = true;
+    }];
+
+    TestWebKitAPI::Util::run(&callbackFlag);
+}
+#endif
+