Only report prevalent domains in the ResourceLoadStatistics data summary API
authorkatherine_cheney@apple.com <katherine_cheney@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 18 Dec 2019 19:02:44 +0000 (19:02 +0000)
committerkatherine_cheney@apple.com <katherine_cheney@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 18 Dec 2019 19:02:44 +0000 (19:02 +0000)
https://bugs.webkit.org/show_bug.cgi?id=205281
<rdar://problem/57975017>

Reviewed by Brent Fulgham.

Source/WebKit:

ResourceLoadStatisticsStore::aggregatedThirdPartyData() should only
return prevalent third party domains.

* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
* NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp:

Tools:

Update the GetResourceLoadStatisticsDataSummary to set third parties
as prevalent to maintain test functionality.

* TestWebKitAPI/Tests/WebKitCocoa/ResourceLoadStatistics.mm:
(TEST):

LayoutTests:

The tests for the ITP data summary had to be updated to set their
third party domains to be prevalent to maintain test functionality.

All http/tests/resourceLoadStatistics/(dont-)count* tests should
no longer output the ITP data summary because the domains are not
set as prevalent.

* http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-database-expected.txt:
* http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-database.html:
* http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-expected.txt:
* http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access.html:
* http/tests/resourceLoadStatistics/count-third-party-script-import-in-worker-database-expected.txt:
* http/tests/resourceLoadStatistics/count-third-party-script-import-in-worker-expected.txt:
* http/tests/resourceLoadStatistics/count-third-party-script-loads-database-expected.txt:
* http/tests/resourceLoadStatistics/count-third-party-script-loads-expected.txt:
* http/tests/resourceLoadStatistics/dont-count-third-party-image-as-third-party-script-database-expected.txt:
* http/tests/resourceLoadStatistics/dont-count-third-party-image-as-third-party-script-expected.txt:
* http/tests/storageAccess/aggregate-sorted-data-with-storage-access-database-expected.txt:
* http/tests/storageAccess/aggregate-sorted-data-with-storage-access-database.html:
* http/tests/storageAccess/aggregate-sorted-data-with-storage-access-expected.txt:
* http/tests/storageAccess/aggregate-sorted-data-with-storage-access.html:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253697 268f45cc-cd09-0410-ab3c-d52691b4dbfc

20 files changed:
LayoutTests/ChangeLog
LayoutTests/http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-database-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-database.html
LayoutTests/http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access.html
LayoutTests/http/tests/resourceLoadStatistics/count-third-party-script-import-in-worker-database-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/count-third-party-script-import-in-worker-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/count-third-party-script-loads-database-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/count-third-party-script-loads-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/dont-count-third-party-image-as-third-party-script-database-expected.txt
LayoutTests/http/tests/resourceLoadStatistics/dont-count-third-party-image-as-third-party-script-expected.txt
LayoutTests/http/tests/storageAccess/aggregate-sorted-data-with-storage-access-database-expected.txt
LayoutTests/http/tests/storageAccess/aggregate-sorted-data-with-storage-access-database.html
LayoutTests/http/tests/storageAccess/aggregate-sorted-data-with-storage-access-expected.txt
LayoutTests/http/tests/storageAccess/aggregate-sorted-data-with-storage-access.html
Source/WebKit/ChangeLog
Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp
Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp
Tools/ChangeLog
Tools/TestWebKitAPI/Tests/WebKitCocoa/ResourceLoadStatistics.mm

index c02ac5e..b74cd12 100644 (file)
@@ -1,3 +1,33 @@
+2019-12-18  Kate Cheney  <katherine_cheney@apple.com>
+
+        Only report prevalent domains in the ResourceLoadStatistics data summary API
+        https://bugs.webkit.org/show_bug.cgi?id=205281
+        <rdar://problem/57975017>
+
+        Reviewed by Brent Fulgham.
+
+        The tests for the ITP data summary had to be updated to set their
+        third party domains to be prevalent to maintain test functionality.
+
+        All http/tests/resourceLoadStatistics/(dont-)count* tests should
+        no longer output the ITP data summary because the domains are not
+        set as prevalent.
+
+        * http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-database-expected.txt:
+        * http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-database.html:
+        * http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access-expected.txt:
+        * http/tests/resourceLoadStatistics/aggregate-sorted-data-no-storage-access.html:
+        * http/tests/resourceLoadStatistics/count-third-party-script-import-in-worker-database-expected.txt:
+        * http/tests/resourceLoadStatistics/count-third-party-script-import-in-worker-expected.txt:
+        * http/tests/resourceLoadStatistics/count-third-party-script-loads-database-expected.txt:
+        * http/tests/resourceLoadStatistics/count-third-party-script-loads-expected.txt:
+        * http/tests/resourceLoadStatistics/dont-count-third-party-image-as-third-party-script-database-expected.txt:
+        * http/tests/resourceLoadStatistics/dont-count-third-party-image-as-third-party-script-expected.txt:
+        * http/tests/storageAccess/aggregate-sorted-data-with-storage-access-database-expected.txt:
+        * http/tests/storageAccess/aggregate-sorted-data-with-storage-access-database.html:
+        * http/tests/storageAccess/aggregate-sorted-data-with-storage-access-expected.txt:
+        * http/tests/storageAccess/aggregate-sorted-data-with-storage-access.html:
+
 2019-12-18  Diego Pino Garcia  <dpino@igalia.com>
 
         [GTK][WPE] Renderization of Conic gradients
index acea0ed..26dac94 100644 (file)
@@ -13,7 +13,7 @@ Registrable domain: subframe1
         topframe1
     SubresourceUniqueRedirectsTo:
         topframe2
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 Registrable domain: subframe2
@@ -23,7 +23,7 @@ Registrable domain: subframe2
     IsScheduledForAllButCookieDataRemoval: No
     SubframeUnderTopFrameDomains:
         topframe1
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 Registrable domain: subframe3
@@ -38,7 +38,7 @@ Registrable domain: subframe3
         topframe3
     SubresourceUniqueRedirectsTo:
         topframe2
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 Registrable domain: topframe1
index 9e1c9ef..a736921 100644 (file)
     function runTest() {
         testRunner.setUseITPDatabase(true);
         setEnableFeature(true, function () {
-
-            // subOrigin1 will be logged under 2 top frames. Should appear second
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin1, topFrameOrigin1);
-            testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin1, topFrameOrigin2);
-
-            // subOrigin2 will be logged under 1 top frames. Should appear third
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin2, topFrameOrigin1);
-
-            // subOrigin3 will be logged under 4 top frames. Should appear first
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin1);
-            testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin3, topFrameOrigin2);
-            testRunner.setStatisticsSubresourceUnderTopFrameOrigin(subOrigin3, topFrameOrigin3);
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin4);
-
-            setTimeout(function () { finishTest() }, 500);
+            testRunner.setStatisticsPrevalentResource(subOrigin1, true, function () {
+                testRunner.setStatisticsPrevalentResource(subOrigin2, true, function () {
+                    testRunner.setStatisticsPrevalentResource(subOrigin3, true, function () {
+                       // subOrigin1 will be logged under 2 top frames. Should appear second
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin1, topFrameOrigin1);
+                       testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin1, topFrameOrigin2);
+
+                       // subOrigin2 will be logged under 1 top frame. Should appear third
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin2, topFrameOrigin1);
+
+                       // subOrigin3 will be logged under 4 top frames. Should appear first
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin1);
+                       testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin3, topFrameOrigin2);
+                       testRunner.setStatisticsSubresourceUnderTopFrameOrigin(subOrigin3, topFrameOrigin3);
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin4);
+
+                       setTimeout(function () { finishTest() }, 500);
+                    });
+                });
+            });
         });
     }
 </script>
index be0fa7b..2d84376 100644 (file)
@@ -45,7 +45,7 @@ Registrable domain: subframe1
         topframe1
     subresourceUniqueRedirectsTo:
         topframe2
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 Registrable domain: subframe2
@@ -55,7 +55,7 @@ Registrable domain: subframe2
     gotLinkDecorationFromPrevalentResource: No
     subframeUnderTopFrameDomains:
         topframe1
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 Registrable domain: subframe3
@@ -70,7 +70,7 @@ Registrable domain: subframe3
         topframe3
     subresourceUniqueRedirectsTo:
         topframe2
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 
index 54961eb..4212744 100644 (file)
 
     function runTest() {
         setEnableFeature(true, function () {
+            testRunner.setStatisticsPrevalentResource(subOrigin1, true, function () {
+                testRunner.setStatisticsPrevalentResource(subOrigin2, true, function () {
+                    testRunner.setStatisticsPrevalentResource(subOrigin3, true, function () {
+                       // subOrigin1 will be logged under 2 top frames. Should appear second
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin1, topFrameOrigin1);
+                       testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin1, topFrameOrigin2);
 
-            // subOrigin1 will be logged under 2 top frames. Should appear second
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin1, topFrameOrigin1);
-            testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin1, topFrameOrigin2);
+                       // subOrigin2 will be logged under 1 top frame. Should appear third
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin2, topFrameOrigin1);
 
-            // subOrigin2 will be logged under 1 top frames. Should appear third
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin2, topFrameOrigin1);
+                       // subOrigin3 will be logged under 4 top frames. Should appear first
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin1);
+                       testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin3, topFrameOrigin2);
+                       testRunner.setStatisticsSubresourceUnderTopFrameOrigin(subOrigin3, topFrameOrigin3);
+                       testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin4);
 
-            // subOrigin3 will be logged under 4 top frames. Should appear first
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin1);
-            testRunner.setStatisticsSubresourceUniqueRedirectTo(subOrigin3, topFrameOrigin2);
-            testRunner.setStatisticsSubresourceUnderTopFrameOrigin(subOrigin3, topFrameOrigin3);
-            testRunner.setStatisticsSubframeUnderTopFrameOrigin(subOrigin3, topFrameOrigin4);
-
-            setTimeout(function () { finishTest() }, 500);
+                       setTimeout(function () { finishTest() }, 500);
+                    });
+                });
+            });
         });
     }
 </script>
index 4c9e586..369c1e4 100644 (file)
@@ -24,7 +24,3 @@ Registrable domain: localhost
     isPrevalentResource: No
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
-
-ITP Data:
-Third Party Registrable Domain: localhost
-    {{ Has been granted storage access under 127.0.0.1: 0 },}
index 03025ce..1211af0 100644 (file)
@@ -24,7 +24,3 @@ Registrable domain: 127.0.0.1
     isPrevalentResource: No
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
-
-ITP Data:
-Third Party Registrable Domain: localhost
-    {{ Has been granted storage access under 127.0.0.1: 0 },}
index 4c9e586..369c1e4 100644 (file)
@@ -24,7 +24,3 @@ Registrable domain: localhost
     isPrevalentResource: No
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
-
-ITP Data:
-Third Party Registrable Domain: localhost
-    {{ Has been granted storage access under 127.0.0.1: 0 },}
index 03025ce..1211af0 100644 (file)
@@ -24,7 +24,3 @@ Registrable domain: 127.0.0.1
     isPrevalentResource: No
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
-
-ITP Data:
-Third Party Registrable Domain: localhost
-    {{ Has been granted storage access under 127.0.0.1: 0 },}
index 92bbba9..025c23f 100644 (file)
@@ -22,7 +22,3 @@ Registrable domain: localhost
     isPrevalentResource: No
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
-
-ITP Data:
-Third Party Registrable Domain: localhost
-    {{ Has been granted storage access under 127.0.0.1: 0 },}
index 19ed851..4ecb973 100644 (file)
@@ -14,7 +14,3 @@ Registrable domain: localhost
     isPrevalentResource: No
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
-
-ITP Data:
-Third Party Registrable Domain: localhost
-    {{ Has been granted storage access under 127.0.0.1: 0 },}
index 173a63a..bbda8d9 100644 (file)
@@ -27,7 +27,7 @@ Registrable domain: localhost
     IsScheduledForAllButCookieDataRemoval: No
     SubframeUnderTopFrameDomains:
         127.0.0.1
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 
index 58233d0..41a5c93 100644 (file)
         function receiveMessage(event) {
             if (event.origin === "http://localhost:8000") {
                 if (event.data.indexOf("PASS ") !== -1) {
-                    testRunner.dumpResourceLoadStatistics();
-                    testPassed(event.data.replace("PASS ", ""));
+                    testRunner.setStatisticsPrevalentResource(thirdPartyUrl, true, function () {
+                        testRunner.dumpResourceLoadStatistics();
+                        testPassed(event.data.replace("PASS ", ""));
+                    });
                 } else
                     testFailed(event.data.replace("FAIL ", ""));
             } else
index f948f2d..9ae26dd 100644 (file)
@@ -19,7 +19,7 @@ Registrable domain: localhost
     gotLinkDecorationFromPrevalentResource: No
     subframeUnderTopFrameDomains:
         127.0.0.1
-    isPrevalentResource: No
+    isPrevalentResource: Yes
     isVeryPrevalentResource: No
     dataRecordsRemoved: 0
 Registrable domain: 127.0.0.1
index d3d301f..9e7599c 100644 (file)
         function receiveMessage(event) {
             if (event.origin === "http://localhost:8000") {
                 if (event.data.indexOf("PASS ") !== -1) {
-                    testRunner.dumpResourceLoadStatistics();
-                    testPassed(event.data.replace("PASS ", ""));
+                    testRunner.setStatisticsPrevalentResource(thirdPartyUrl, true, function () {
+                        testRunner.dumpResourceLoadStatistics();
+                        testPassed(event.data.replace("PASS ", ""));
+                    });
                 } else
                     testFailed(event.data.replace("FAIL ", ""));
             } else
index fc7ade5..fb5353e 100644 (file)
@@ -1,3 +1,17 @@
+2019-12-18  Kate Cheney  <katherine_cheney@apple.com>
+
+        Only report prevalent domains in the ResourceLoadStatistics data summary API
+        https://bugs.webkit.org/show_bug.cgi?id=205281
+        <rdar://problem/57975017>
+
+        Reviewed by Brent Fulgham.
+
+        ResourceLoadStatisticsStore::aggregatedThirdPartyData() should only
+        return prevalent third party domains.
+
+        * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
+        * NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp:
+
 2019-12-18  Brent Fulgham  <bfulgham@apple.com>
 
         Add syscall to GPU Process sandbox
index ae5f518..0bc3579 100644 (file)
@@ -815,7 +815,7 @@ Vector<WebResourceLoadStatisticsStore::ThirdPartyData> ResourceLoadStatisticsDat
     Vector<WebResourceLoadStatisticsStore::ThirdPartyData> thirdPartyDataList;
     SQLiteStatement sortedStatistics(m_database, makeString("SELECT ", joinSubStatisticsForSorting()));
     if (sortedStatistics.prepare() != SQLITE_OK
-        || sortedStatistics.bindText(1, "%")
+        || sortedStatistics.bindText(1, "1")
         || sortedStatistics.bindText(2, "%") != SQLITE_OK) {
         RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::aggregatedThirdPartyData, error message: %{public}s", m_database.lastErrorMsg());
         ASSERT_NOT_REACHED();
index 61671d6..08f699e 100644 (file)
@@ -136,7 +136,7 @@ Vector<WebResourceLoadStatisticsStore::ThirdPartyData> ResourceLoadStatisticsMem
 
     Vector<WebResourceLoadStatisticsStore::ThirdPartyData> thirdPartyDataList;
     for (auto& statistic : m_resourceStatisticsMap.values()) {
-        if (hasBeenThirdParty(statistic))
+        if (hasBeenThirdParty(statistic) && statistic.isPrevalentResource)
             thirdPartyDataList.append(WebResourceLoadStatisticsStore::ThirdPartyData { statistic.registrableDomain, getThirdPartyDataForSpecificFirstPartyDomains(statistic) });
     }
     std::sort(thirdPartyDataList.rbegin(), thirdPartyDataList.rend());
index 800afde..45a6c76 100644 (file)
@@ -1,3 +1,17 @@
+2019-12-18  Kate Cheney  <katherine_cheney@apple.com>
+
+        Only report prevalent domains in the ResourceLoadStatistics data summary API
+        https://bugs.webkit.org/show_bug.cgi?id=205281
+        <rdar://problem/57975017>
+
+        Reviewed by Brent Fulgham.
+
+        Update the GetResourceLoadStatisticsDataSummary to set third parties
+        as prevalent to maintain test functionality.
+
+        * TestWebKitAPI/Tests/WebKitCocoa/ResourceLoadStatistics.mm:
+        (TEST):
+
 2019-12-18  Carlos Garcia Campos  <cgarcia@igalia.com>
 
         Unreviewed GTK gardening. Update expectations of unit tests
index d685339..daf6109 100644 (file)
@@ -697,6 +697,19 @@ TEST(ResourceLoadStatistics, GetResourceLoadStatisticsDataSummary)
 
     TestWebKitAPI::Util::run(&doneFlag);
 
+    // set third parties to be prevalent
+    [dataStore _setPrevalentDomain:[NSURL URLWithString:@"http://evil1.com"] completionHandler: ^(void) {
+        doneFlag = true;
+    }];
+    doneFlag = false;
+    [dataStore _setPrevalentDomain:[NSURL URLWithString:@"http://evil2.com"] completionHandler: ^(void) {
+        doneFlag = true;
+    }];
+    doneFlag = false;
+    [dataStore _setPrevalentDomain:[NSURL URLWithString:@"http://evil3.com"] completionHandler: ^(void) {
+        doneFlag = true;
+    }];
+
     // Seed test data in the web process' observer.
 
     // evil1