Implement Telemetry and Dumping Routines for SQLite backend (195088)
authorcommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 8 Oct 2019 22:13:39 +0000 (22:13 +0000)
committercommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 8 Oct 2019 22:13:39 +0000 (22:13 +0000)
https://bugs.webkit.org/show_bug.cgi?id=195088
<rdar://problem/54213407>

Patch by Kate Cheney <katherine_cheney@apple.com> on 2019-10-08
Reviewed by John Wilander.

Source/WebKit:

Implemented database telemetry calculating for ITP. Mimicked
ResourceLoadStatisticsMemoryStore telemetry logging behavior using
SQLite Queries as opposed to vector sorting/manipulation. Once fully
integrated, this will simplify analysis of ITP data by transitioning
ITP data storage from a plist to a SQLite database.

* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
(WebKit::ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore):
Added SQL queries to be initialized in the constructor. These queries
are needed to mimic the telemetry calculations done in
ResourceLoadStatisticsMemoryStore.

(WebKit::resetStatement):
To reduce code duplication, this function holds common code to reset
a SQL query.

(WebKit::ResourceLoadStatisticsDatabaseStore::prepareStatements):
Added SQL queries needed for telemetry calculations to be prepared.

(WebKit::joinSubStatisticsForSorting):
This function returns the query string for sorting resources that is
shared by many queries.

(WebKit::ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction const):
Implemented a function to take the median days since user interaction
from all prevalent resources in the database with user interaction
(sorted by max count of subframes, subresources and unique redirects under
the top frame domain).

(WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResources const):
Executes a SQL query to get the number of prevalent resources to log as
telemetry.

(WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithUI const):
Executes a SQL query to get the number of prevalent resources with user
interaction to log as telemetry.

(WebKit::ResourceLoadStatisticsDatabaseStore::getTopPrevelentResourceDaysSinceUI const):
Prepares and executes a SQL query to get the days since user
interaction from the top prevalent resource to be recorded as
telemetry data.

(WebKit::ResourceLoadStatisticsDatabaseStore::getMedianStatisticOfPrevalentResourceWithoutUserInteraction const):
Implemented a function which takes a statistic and returns the value
of that statistic for the median prevalent resource without user
interaction to be recorded as telemetry data.

(WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesInTopResources const):
Returns the count of prevalent resources in the top x resources
sorted by sum of substatistics again to be logged as telemetry.

(WebKit::ResourceLoadStatisticsDatabaseStore::calculateTelemetryData const):
Function which executes all functions which populate the struct with
telemetry data. This struct will then be passed to
WebResourceLoadStatisticsTelemetry to be logged.

(WebKit::ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry const):
Initializes the telemetry struct and calls the function to populate
it, then passes it to the WebResourceLoadStatisticsTelemetry object
to be recorded.

* NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h:
Describe PrevalentResourceDatabaseTelemetry Struct to be passed and
logged as telemetry data.

* NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp:
(WebKit::databaseSubmitTopLists):
Logging telemetry data by looping through 2D array of statistics for
the top 1, 3, 10, 50, and 100 prevalent resources sorted by the sum
of substatistics under the top frame domain. This matches the logging
already done in ResourceLoadStatisticsMemoryStore.

(WebKit::WebResourceLoadStatisticsTelemetry::submitTelemetry):
Submits data to the webPageProxy logs and plists.

* NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.h:
Added new submitTopList function to accomodate database telemetry
logging.

* NetworkProcess/NetworkSession.cpp:
(WebKit::NetworkSession::notifyPageStatisticsTelemetryFinished):
* NetworkProcess/NetworkSession.h:
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::notifyResourceLoadStatisticsTelemetryFinished):
* UIProcess/Network/NetworkProcessProxy.h:
* UIProcess/Network/NetworkProcessProxy.messages.in:
Updated the current testing for telemetry which only tested 3
statistics. With this patch it now tests 10 statistics.

Tools:

Updated the current testing for telemetry which only tested 3
statistics. With this patch it now tests 10 statistics.
* WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:
(WTR::InjectedBundle::didReceiveMessageToPage):
* WebKitTestRunner/InjectedBundle/TestRunner.cpp:
(WTR::TestRunner::statisticsDidRunTelemetryCallback):
* WebKitTestRunner/InjectedBundle/TestRunner.h:

LayoutTests:

Updated the current testing for telemetry which only tested 3
statistics. With this patch it now tests 10 statistics.
* http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database-expected.txt: Added.
* http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database.html: Added.
* http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database-expected.txt: Added.
* http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database.html: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@250866 268f45cc-cd09-0410-ab3c-d52691b4dbfc

21 files changed:
LayoutTests/ChangeLog
LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database-expected.txt [new file with mode: 0644]
LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database.html [new file with mode: 0644]
LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database-expected.txt [new file with mode: 0644]
LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database.html [new file with mode: 0644]
Source/WebKit/ChangeLog
Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp
Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h
Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp
Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h
Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp
Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.h
Source/WebKit/NetworkProcess/NetworkSession.cpp
Source/WebKit/NetworkProcess/NetworkSession.h
Source/WebKit/UIProcess/Network/NetworkProcessProxy.cpp
Source/WebKit/UIProcess/Network/NetworkProcessProxy.h
Source/WebKit/UIProcess/Network/NetworkProcessProxy.messages.in
Tools/ChangeLog
Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.cpp
Tools/WebKitTestRunner/InjectedBundle/TestRunner.cpp
Tools/WebKitTestRunner/InjectedBundle/TestRunner.h

index 5ce2de3..18fde3d 100644 (file)
@@ -1,3 +1,18 @@
+2019-10-08  Kate Cheney  <katherine_cheney@apple.com>
+
+        Implement Telemetry and Dumping Routines for SQLite backend (195088)
+        https://bugs.webkit.org/show_bug.cgi?id=195088
+        <rdar://problem/54213407>
+
+        Reviewed by John Wilander.
+
+        Updated the current testing for telemetry which only tested 3
+        statistics. With this patch it now tests 10 statistics.
+        * http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database-expected.txt: Added.
+        * http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database.html: Added.
+        * http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database-expected.txt: Added.
+        * http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database.html: Added.
+
 2019-10-08  Antti Koivisto  <antti@apple.com>
 
         [CSS Shadow Parts] Add test for exportparts list syntax
diff --git a/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database-expected.txt b/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database-expected.txt
new file mode 100644 (file)
index 0000000..c687150
--- /dev/null
@@ -0,0 +1,20 @@
+Tests that advanced telemetry for prevalent resources in the SQLite Database is calculated correctly.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS Hosts classified correctly.
+PASS testResult.numberOfPrevalentResources is 5
+PASS testResult.numberOfPrevalentResourcesWithUserInteraction is 2
+PASS testResult.numberOfPrevalentResourcesWithoutUserInteraction is 3
+PASS testResult.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction is 0
+PASS testResult.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction is 1
+PASS testResult.top3NumberOfPrevalentResourcesWithUI is 2
+PASS testResult.top3MedianSubFrameWithoutUI is 5
+PASS testResult.top3MedianSubResourceWithoutUI is 5
+PASS testResult.top3MedianUniqueRedirectsWithoutUI is 4
+PASS testResult.top3MedianDataRecordsRemovedWithoutUI is 0
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
diff --git a/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database.html b/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-advanced-functionality-database.html
new file mode 100644 (file)
index 0000000..5942909
--- /dev/null
@@ -0,0 +1,157 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="UTF-8">
+    <title>Test for Advanced Telemetry Generation</title>
+    <script src="/js-test-resources/js-test.js"></script>
+    <script src="resources/util.js"></script>
+</head>
+<body>
+<script>
+    testRunner.setUseITPDatabase(true);
+    description("Tests that advanced telemetry for prevalent resources in the SQLite Database is calculated correctly.");
+    jsTestIsAsync = true;
+
+    const topFrameUrl = "http://127.0.0.5:8000/temp";
+    const topResourceUrl = "http://127.0.0.6:8000/temp";
+    const toUniqueRedirectUrl = "http://127.0.0.7:8000/temp";
+    
+    const topFrameUrl1 = "http://127.0.0.1:8000/temp";
+    const topFrameUrl2 = "http://127.0.0.2:8000/temp";
+    const topFrameUrl3 = "http://127.0.0.3:8000/temp";
+    const topFrameUrl4 = "http://127.0.0.4:8000/temp";
+    
+    
+    const url1 = "http://127.0.1.1:8000/temp";
+    const url2 = "http://127.0.1.2:8000/temp";
+    const url3 = "http://127.0.1.3:8000/temp";
+    const url4 = "http://127.0.1.4:8000/temp";
+    const url5 = "http://127.0.1.5:8000/temp";
+    
+    const twoDaysOld = Math.round((new Date()).getTime() / 1000) - 180000;
+
+    function checkClassificationAndContinue() {
+        if (!testRunner.isStatisticsPrevalentResource(url1)) {
+            testFailed("Host 1 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(url2)) {
+            testFailed("Host 2 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(url3)) {
+            testFailed("Host 3 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(url4)) {
+            testFailed("Host 4 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(url5)) {
+            testFailed("Host 5 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsHasHadUserInteraction(url1)) {
+            testFailed("Host 1 did not get classified as having user interaction.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsHasHadUserInteraction(url2)) {
+            testFailed("Host 2 did not get classified as having user interaction.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsRegisteredAsSubFrameUnder(url1, topFrameUrl)) {
+            testFailed("Host 1 did not get set as subframe.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsRegisteredAsSubresourceUnder(url1, topResourceUrl)) {
+            testFailed("Host 1 did not get set as subresource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsRegisteredAsSubFrameUnder(url2, topFrameUrl)) {
+            testFailed("Host 2 did not get set as subframe.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsRegisteredAsSubFrameUnder(url3, topFrameUrl)) {
+            testFailed("Host 3 did not get set as subframe.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsRegisteredAsSubresourceUnder(url3, topResourceUrl)) {
+            testFailed("Host 3 did not get set as subresource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsRegisteredAsSubFrameUnder(url4, topFrameUrl)) {
+            testFailed("Host 4 did not get set as subframe.");
+            setEnableFeature(false, finishJSTest);
+        } else {
+            testPassed("Hosts classified correctly.");
+            runTelemetryAndContinue();
+        }
+    }
+                             
+    function makeUrlPrevalent(prevalentResourceUrl) {
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl1);
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl2);
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl3);
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl4);
+
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl1);
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl2);
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl3);
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl4);
+
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl1);
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl2);
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl3);
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl4);
+    }
+
+    function setUpStatisticsAndContinue() {
+        // merge statistic arguments: hostName, topFrameDomain, lastSeen, hadUserInteraction, mostRecentUserInteraction, isGrandfathered, isPrevalent, isVeryPrevalent, dataRecordsRemoved
+         makeUrlPrevalent(url1);
+         testRunner.setStatisticsSubframeUnderTopFrameOrigin(url1, topFrameUrl);
+         testRunner.setStatisticsSubresourceUnderTopFrameOrigin(url1, topResourceUrl);
+         testRunner.setStatisticsSubresourceUniqueRedirectTo(url1, toUniqueRedirectUrl);
+                             
+         makeUrlPrevalent(url2);
+         testRunner.setStatisticsSubframeUnderTopFrameOrigin(url2, topFrameUrl);
+         testRunner.setStatisticsSubresourceUniqueRedirectTo(url2, toUniqueRedirectUrl);
+
+         makeUrlPrevalent(url3);
+         testRunner.setStatisticsSubframeUnderTopFrameOrigin(url3, topFrameUrl);
+         testRunner.setStatisticsSubresourceUnderTopFrameOrigin(url3, topResourceUrl);
+         testRunner.setStatisticsSubresourceUniqueRedirectTo(url3, toUniqueRedirectUrl);
+                             
+         makeUrlPrevalent(url4);
+         testRunner.setStatisticsSubframeUnderTopFrameOrigin(url4, topFrameUrl);
+         testRunner.setStatisticsSubresourceUnderTopFrameOrigin(url4, topResourceUrl);
+
+         makeUrlPrevalent(url5);
+
+        // set URLs 1 and 2 to have user interaction. UI for url 2 will be set to be 2 days old.
+        testRunner.setStatisticsHasHadUserInteraction(url1, true, function() {
+            testRunner.setStatisticsMergeStatistic(url2, topFrameUrl, "", twoDaysOld, true, twoDaysOld, false, true, false, 0, function() {
+                testRunner.installStatisticsDidScanDataRecordsCallback(checkClassificationAndContinue);
+                testRunner.installStatisticsDidRunTelemetryCallback(checkTelemetry);
+                testRunner.statisticsProcessStatisticsAndDataRecords();
+            });
+        });
+    }
+
+    function runTelemetryAndContinue() {
+        testRunner.statisticsSubmitTelemetry();
+    }
+
+    var testResult;
+    function checkTelemetry(result) {
+        testResult = result;
+        shouldBe("testResult.numberOfPrevalentResources", "5");
+        shouldBe("testResult.numberOfPrevalentResourcesWithUserInteraction", "2");
+        shouldBe("testResult.numberOfPrevalentResourcesWithoutUserInteraction", "3");
+        shouldBe("testResult.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction", "0");
+        shouldBe("testResult.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction", "1");
+        shouldBe("testResult.top3NumberOfPrevalentResourcesWithUI", "2");
+        shouldBe("testResult.top3MedianSubFrameWithoutUI", "5");
+        shouldBe("testResult.top3MedianSubResourceWithoutUI", "5");
+        shouldBe("testResult.top3MedianUniqueRedirectsWithoutUI", "4");
+        shouldBe("testResult.top3MedianDataRecordsRemovedWithoutUI", "0");
+        setEnableFeature(false, finishJSTest);
+    }
+
+    if (window.testRunner) {
+        setEnableFeature(true, function() {
+            testRunner.setStatisticsNotifyPagesWhenDataRecordsWereScanned(true);
+            testRunner.setStatisticsNotifyPagesWhenTelemetryWasCaptured(true);
+            setUpStatisticsAndContinue();
+        });
+    }
+</script>
+</body>
+</html>
diff --git a/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database-expected.txt b/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database-expected.txt
new file mode 100644 (file)
index 0000000..fe438c9
--- /dev/null
@@ -0,0 +1,31 @@
+Tests that basic telemetry for prevalent resources in the SQLite Database is calculated correctly.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS Hosts classified as prevalent resources.
+PASS testResult.numberOfPrevalentResources is 0
+PASS testResult.numberOfPrevalentResourcesWithUserInteraction is 0
+PASS testResult.numberOfPrevalentResourcesWithoutUserInteraction is 0
+PASS testResult.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction is 0
+PASS testResult.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction is 0
+PASS testResult.top3NumberOfPrevalentResourcesWithUI is 0
+PASS testResult.top3MedianSubFrameWithoutUI is 0
+PASS testResult.top3MedianSubResourceWithoutUI is 0
+PASS testResult.top3MedianUniqueRedirectsWithoutUI is 0
+PASS testResult.top3MedianDataRecordsRemovedWithoutUI is 0
+PASS Hosts classified as prevalent resources.
+PASS testResult.numberOfPrevalentResources is 4
+PASS testResult.numberOfPrevalentResourcesWithUserInteraction is 1
+PASS testResult.numberOfPrevalentResourcesWithoutUserInteraction is 3
+PASS testResult.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction is 0
+PASS testResult.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction is 0
+PASS testResult.top3NumberOfPrevalentResourcesWithUI is 0
+PASS testResult.top3MedianSubFrameWithoutUI is 4
+PASS testResult.top3MedianSubResourceWithoutUI is 4
+PASS testResult.top3MedianUniqueRedirectsWithoutUI is 4
+PASS testResult.top3MedianDataRecordsRemovedWithoutUI is 0
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
diff --git a/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database.html b/LayoutTests/http/tests/resourceLoadStatistics/telemetry-generation-basic-functionality-database.html
new file mode 100644 (file)
index 0000000..97b5579
--- /dev/null
@@ -0,0 +1,137 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="UTF-8">
+    <title>Test for Basic Telemetry Generation</title>
+    <script src="/js-test-resources/js-test.js"></script>
+    <script src="resources/util.js"></script>
+</head>
+<body>
+<script>
+    testRunner.setUseITPDatabase(true);
+    description("Tests that basic telemetry for prevalent resources in the SQLite Database is calculated correctly.");
+    jsTestIsAsync = true;
+
+    const topFrameUrl1 = "http://127.0.0.1:8000/temp";
+    const topFrameUrl2 = "http://127.0.0.2:8000/temp";
+    const topFrameUrl3 = "http://127.0.0.3:8000/temp";
+    const topFrameUrl4 = "http://127.0.0.4:8000/temp";
+    const prevalentResourceUrl1 = "http://127.0.1.1:8000/temp";
+    const prevalentResourceUrl2 = "http://127.0.1.2:8000/temp";
+    const prevalentResourceUrl3 = "http://127.0.1.3:8000/temp";
+    const prevalentResourceUrl4 = "http://127.0.1.4:8000/temp";
+
+    function checkInsufficientClassificationAndContinue() {
+        if (!testRunner.isStatisticsPrevalentResource(prevalentResourceUrl1)) {
+            testFailed("Host 1 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(prevalentResourceUrl2)) {
+            testFailed("Host 2 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else {
+            testPassed("Hosts classified as prevalent resources.");
+            runTelemetryAndContinue();
+        }
+    }
+
+    function checkSufficientClassificationAndContinue() {
+        if (!testRunner.isStatisticsPrevalentResource(prevalentResourceUrl1)) {
+            testFailed("Host 1 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(prevalentResourceUrl2)) {
+            testFailed("Host 2 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(prevalentResourceUrl3)) {
+            testFailed("Host 3 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else if (!testRunner.isStatisticsPrevalentResource(prevalentResourceUrl4)) {
+            testFailed("Host 4 did not get classified as prevalent resource.");
+            setEnableFeature(false, finishJSTest);
+        } else {
+            testPassed("Hosts classified as prevalent resources.");
+            runTelemetryAndContinue();
+        }
+    }
+
+    function makeUrlPrevalent(prevalentResourceUrl) {
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl1);
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl2);
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl3);
+        testRunner.setStatisticsSubframeUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl4);
+
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl1);
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl2);
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl3);
+        testRunner.setStatisticsSubresourceUniqueRedirectTo(prevalentResourceUrl, topFrameUrl4);
+
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl1);
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl2);
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl3);
+        testRunner.setStatisticsSubresourceUnderTopFrameOrigin(prevalentResourceUrl, topFrameUrl4);
+    }
+
+    function setUpInsufficientStatisticsAndContinue() {
+        makeUrlPrevalent(prevalentResourceUrl1);
+        makeUrlPrevalent(prevalentResourceUrl2);
+
+        testRunner.installStatisticsDidScanDataRecordsCallback(checkInsufficientClassificationAndContinue);
+        testRunner.installStatisticsDidRunTelemetryCallback(checkInsufficientTelemetry);
+        testRunner.statisticsProcessStatisticsAndDataRecords();
+    }
+
+    function setUpSufficientStatisticsAndContinue() {
+        makeUrlPrevalent(prevalentResourceUrl3);
+        makeUrlPrevalent(prevalentResourceUrl4);
+
+        testRunner.setStatisticsHasHadUserInteraction(prevalentResourceUrl4, true, function() {
+            testRunner.installStatisticsDidScanDataRecordsCallback(checkSufficientClassificationAndContinue);
+            testRunner.installStatisticsDidRunTelemetryCallback(checkSufficientTelemetry);
+            testRunner.statisticsProcessStatisticsAndDataRecords();
+        });
+    }
+
+    function runTelemetryAndContinue() {
+        testRunner.statisticsSubmitTelemetry();
+    }
+
+    var testResult;
+    function checkInsufficientTelemetry(result) {
+        testResult = result;
+        shouldBe("testResult.numberOfPrevalentResources", "0");
+        shouldBe("testResult.numberOfPrevalentResourcesWithUserInteraction", "0");
+        shouldBe("testResult.numberOfPrevalentResourcesWithoutUserInteraction", "0");
+        shouldBe("testResult.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction", "0");
+        shouldBe("testResult.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction", "0");
+        shouldBe("testResult.top3NumberOfPrevalentResourcesWithUI", "0");
+        shouldBe("testResult.top3MedianSubFrameWithoutUI", "0");
+        shouldBe("testResult.top3MedianSubResourceWithoutUI", "0");
+        shouldBe("testResult.top3MedianUniqueRedirectsWithoutUI", "0");
+        shouldBe("testResult.top3MedianDataRecordsRemovedWithoutUI", "0");
+        setUpSufficientStatisticsAndContinue();
+    }
+
+    function checkSufficientTelemetry(result) {
+        testResult = result;
+        shouldBe("testResult.numberOfPrevalentResources", "4");
+        shouldBe("testResult.numberOfPrevalentResourcesWithUserInteraction", "1");
+        shouldBe("testResult.numberOfPrevalentResourcesWithoutUserInteraction", "3");
+        shouldBe("testResult.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction", "0");
+        shouldBe("testResult.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction", "0");
+        shouldBe("testResult.top3NumberOfPrevalentResourcesWithUI", "0");
+        shouldBe("testResult.top3MedianSubFrameWithoutUI", "4");
+        shouldBe("testResult.top3MedianSubResourceWithoutUI", "4");
+        shouldBe("testResult.top3MedianUniqueRedirectsWithoutUI", "4");
+        shouldBe("testResult.top3MedianDataRecordsRemovedWithoutUI", "0");
+        setEnableFeature(false, finishJSTest);
+    }
+
+    if (window.testRunner) {
+        setEnableFeature(true, function() {
+            testRunner.setStatisticsNotifyPagesWhenDataRecordsWereScanned(true);
+            testRunner.setStatisticsNotifyPagesWhenTelemetryWasCaptured(true);
+            setUpInsufficientStatisticsAndContinue();
+        });
+    }
+</script>
+</body>
+</html>
index 0eab047..d546789 100644 (file)
@@ -1,3 +1,100 @@
+2019-10-08  Kate Cheney  <katherine_cheney@apple.com>
+
+        Implement Telemetry and Dumping Routines for SQLite backend (195088)
+        https://bugs.webkit.org/show_bug.cgi?id=195088
+        <rdar://problem/54213407>
+
+        Reviewed by John Wilander.
+
+        Implemented database telemetry calculating for ITP. Mimicked
+        ResourceLoadStatisticsMemoryStore telemetry logging behavior using
+        SQLite Queries as opposed to vector sorting/manipulation. Once fully
+        integrated, this will simplify analysis of ITP data by transitioning
+        ITP data storage from a plist to a SQLite database.
+
+        * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp:
+        (WebKit::ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore):
+        Added SQL queries to be initialized in the constructor. These queries
+        are needed to mimic the telemetry calculations done in
+        ResourceLoadStatisticsMemoryStore.
+        
+        (WebKit::resetStatement):
+        To reduce code duplication, this function holds common code to reset
+        a SQL query.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::prepareStatements):
+        Added SQL queries needed for telemetry calculations to be prepared.
+        
+        (WebKit::joinSubStatisticsForSorting):
+        This function returns the query string for sorting resources that is
+        shared by many queries.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction const):
+        Implemented a function to take the median days since user interaction
+        from all prevalent resources in the database with user interaction
+        (sorted by max count of subframes, subresources and unique redirects under
+        the top frame domain).
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResources const):
+        Executes a SQL query to get the number of prevalent resources to log as
+        telemetry.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithUI const):
+        Executes a SQL query to get the number of prevalent resources with user
+        interaction to log as telemetry.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::getTopPrevelentResourceDaysSinceUI const):
+        Prepares and executes a SQL query to get the days since user
+        interaction from the top prevalent resource to be recorded as
+        telemetry data.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::getMedianStatisticOfPrevalentResourceWithoutUserInteraction const):
+        Implemented a function which takes a statistic and returns the value
+        of that statistic for the median prevalent resource without user
+        interaction to be recorded as telemetry data.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesInTopResources const):
+        Returns the count of prevalent resources in the top x resources
+        sorted by sum of substatistics again to be logged as telemetry.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::calculateTelemetryData const):
+        Function which executes all functions which populate the struct with
+        telemetry data. This struct will then be passed to
+        WebResourceLoadStatisticsTelemetry to be logged.
+
+        (WebKit::ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry const):
+        Initializes the telemetry struct and calls the function to populate
+        it, then passes it to the WebResourceLoadStatisticsTelemetry object
+        to be recorded.
+
+        * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h:
+        Describe PrevalentResourceDatabaseTelemetry Struct to be passed and
+        logged as telemetry data.
+
+        * NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp:
+        (WebKit::databaseSubmitTopLists):
+        Logging telemetry data by looping through 2D array of statistics for
+        the top 1, 3, 10, 50, and 100 prevalent resources sorted by the sum
+        of substatistics under the top frame domain. This matches the logging
+        already done in ResourceLoadStatisticsMemoryStore.
+
+        (WebKit::WebResourceLoadStatisticsTelemetry::submitTelemetry):
+        Submits data to the webPageProxy logs and plists.
+
+        * NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.h:
+        Added new submitTopList function to accomodate database telemetry
+        logging.
+        
+        * NetworkProcess/NetworkSession.cpp:
+        (WebKit::NetworkSession::notifyPageStatisticsTelemetryFinished):
+        * NetworkProcess/NetworkSession.h:
+        * UIProcess/Network/NetworkProcessProxy.cpp:
+        (WebKit::NetworkProcessProxy::notifyResourceLoadStatisticsTelemetryFinished):
+        * UIProcess/Network/NetworkProcessProxy.h:
+        * UIProcess/Network/NetworkProcessProxy.messages.in:
+        Updated the current testing for telemetry which only tested 3
+        statistics. With this patch it now tests 10 statistics.
+
 2019-10-08  Adrian Perez de Castro  <aperez@igalia.com>
 
         [GTK][WPE] Fix non-unified builds after r250486
index 55c3dfb..61695c8 100644 (file)
@@ -67,6 +67,10 @@ constexpr auto observedDomainCountQuery = "SELECT COUNT(*) FROM ObservedDomains"
 constexpr auto countSubframeUnderTopFrameQuery = "SELECT COUNT(*) FROM SubframeUnderTopFrameDomains WHERE subFrameDomainID = ? AND topFrameDomainID = ?;"_s;
 constexpr auto countSubresourceUnderTopFrameQuery = "SELECT COUNT(*) FROM SubresourceUnderTopFrameDomains WHERE subresourceDomainID = ? AND topFrameDomainID = ?;"_s;
 constexpr auto countSubresourceUniqueRedirectsToQuery = "SELECT COUNT(*) FROM SubresourceUniqueRedirectsTo WHERE subresourceDomainID = ? AND toDomainID = ?;"_s;
+constexpr auto countPrevalentResourcesQuery = "SELECT COUNT(DISTINCT registrableDomain) FROM ObservedDomains WHERE isPrevalent = 1;"_s;
+constexpr auto countPrevalentResourcesWithUserInteractionQuery = "SELECT COUNT(DISTINCT registrableDomain) FROM ObservedDomains WHERE isPrevalent = 1 AND hadUserInteraction = 1;"_s;
+
+constexpr auto countPrevalentResourcesWithoutUserInteractionQuery = "SELECT COUNT(DISTINCT registrableDomain) FROM ObservedDomains WHERE isPrevalent = 1 AND hadUserInteraction = 0;"_s;
 
 // INSERT Queries
 constexpr auto insertObservedDomainQuery = "INSERT INTO ObservedDomains (registrableDomain, lastSeen, hadUserInteraction,"
@@ -186,7 +190,8 @@ constexpr auto createUniqueIndexSubresourceUnderTopFrameDomains = "CREATE UNIQUE
 constexpr auto createUniqueIndexSubresourceUniqueRedirectsTo = "CREATE UNIQUE INDEX IF NOT EXISTS SubresourceUniqueRedirectsTo_subresourceDomainID_toDomainID on SubresourceUniqueRedirectsTo ( subresourceDomainID, toDomainID );"_s;
 constexpr auto createUniqueIndexSubresourceUniqueRedirectsFrom = "CREATE UNIQUE INDEX IF NOT EXISTS SubresourceUniqueRedirectsFrom_subresourceDomainID_fromDomainID on SubresourceUnderTopFrameDomains ( subresourceDomainID, fromDomainID );"_s;
 
-    
+const unsigned minimumPrevalentResourcesForTelemetry = 3;
+
 ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore(WebResourceLoadStatisticsStore& store, WorkQueue& workQueue, ShouldIncludeLocalhost shouldIncludeLocalhost, const String& storageDirectoryPath, PAL::SessionID sessionID)
     : ResourceLoadStatisticsStore(store, workQueue, shouldIncludeLocalhost)
     , m_storageDirectoryPath(storageDirectoryPath + "/observations.db")
@@ -210,6 +215,9 @@ ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore(WebReso
     , m_updateGrandfatheredStatement(m_database, updateGrandfatheredQuery)
     , m_isGrandfatheredStatement(m_database, isGrandfatheredQuery)
     , m_findExpiredUserInteractionStatement(m_database, findExpiredUserInteractionQuery)
+    , m_countPrevalentResourcesStatement(m_database, countPrevalentResourcesQuery)
+    , m_countPrevalentResourcesWithUserInteractionStatement(m_database, countPrevalentResourcesWithUserInteractionQuery)
+    , m_countPrevalentResourcesWithoutUserInteractionStatement(m_database, countPrevalentResourcesWithoutUserInteractionQuery)
     , m_sessionID(sessionID)
 {
     ASSERT(!RunLoop::isMain());
@@ -246,6 +254,12 @@ ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore(WebReso
     });
 }
 
+static void resetStatement(SQLiteStatement& statement)
+{
+    int resetResult = statement.reset();
+    ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK);
+}
+
 bool ResourceLoadStatisticsDatabaseStore::isEmpty() const
 {
     ASSERT(!RunLoop::isMain());
@@ -359,7 +373,11 @@ bool ResourceLoadStatisticsDatabaseStore::prepareStatements()
         || m_updateGrandfatheredStatement.prepare() != SQLITE_OK
         || m_isGrandfatheredStatement.prepare() != SQLITE_OK
         || m_findExpiredUserInteractionStatement.prepare() != SQLITE_OK
-        || m_topFrameLinkDecorationsFromExists.prepare() != SQLITE_OK) {
+        || m_topFrameLinkDecorationsFromExists.prepare() != SQLITE_OK
+        || m_countPrevalentResourcesStatement.prepare() != SQLITE_OK
+        || m_countPrevalentResourcesWithUserInteractionStatement.prepare() != SQLITE_OK
+        || m_countPrevalentResourcesWithoutUserInteractionStatement.prepare() != SQLITE_OK
+        ) {
         RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::prepareStatements failed to prepare, error message: %{public}s", this, m_database.lastErrorMsg());
         ASSERT_NOT_REACHED();
         return false;
@@ -602,11 +620,286 @@ void ResourceLoadStatisticsDatabaseStore::mergeStatistics(Vector<ResourceLoadSta
         insertDomainRelationships(statistic);
 }
 
+static const StringView joinSubStatisticsForSorting()
+{
+    return R"query(
+        domainID,
+        (cnt1 + cnt2 + cnt3) as sum
+        FROM (
+        SELECT
+            domainID,
+            COUNT(DISTINCT f.topFrameDomainID) as cnt1,
+            COUNT(DISTINCT r.topFrameDomainID) as cnt2,
+            COUNT(DISTINCT toDomainID) as cnt3
+        FROM
+        ObservedDomains o
+        LEFT JOIN SubframeUnderTopFrameDomains f ON o.domainID = f.subFrameDomainID
+        LEFT JOIN SubresourceUnderTopFrameDomains r ON o.domainID = r.subresourceDomainID
+        LEFT JOIN SubresourceUniqueRedirectsTo u ON o.domainID = u.subresourceDomainID
+        WHERE isPrevalent = 1
+        and hadUserInteraction LIKE ?
+        GROUP BY domainID) ORDER BY sum DESC
+        )query";
+}
+
+static SQLiteStatement makeMedianWithUIQuery(SQLiteDatabase& database)
+{
+    return SQLiteStatement(database, makeString("SELECT mostRecentUserInteractionTime FROM ObservedDomains INNER JOIN (SELECT ", joinSubStatisticsForSorting(), ") as q ON ObservedDomains.domainID = q.domainID LIMIT 1 OFFSET ?"));
+}
+
+static std::pair<StringView, StringView> buildQueryStartAndEnd(PrevalentResourceDatabaseTelemetry::Statistic statistic)
+{
+    switch (statistic) {
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubFrameWithoutUI:
+        return std::make_pair("SELECT cnt1 FROM ObservedDomains o INNER JOIN(SELECT cnt1, ", ") as q ON o.domainID = q.domainID LIMIT 1 OFFSET ?");
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubResourceWithoutUI:
+        return std::make_pair("SELECT cnt2 FROM ObservedDomains o INNER JOIN(SELECT cnt2, ", ") as q ON o.domainID = q.domainID LIMIT 1 OFFSET ?");
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianUniqueRedirectsWithoutUI:
+        return std::make_pair("SELECT cnt3 FROM ObservedDomains o INNER JOIN(SELECT cnt3, ", ") as q ON o.domainID = q.domainID LIMIT 1 OFFSET ?");
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianDataRecordsRemovedWithoutUI:
+        return std::make_pair("SELECT dataRecordsRemoved FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", ") as q ON o.domainID = q.domainID) LIMIT 1 OFFSET ?");
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToUserInteractionWithoutUI:
+        return std::make_pair("SELECT timesAccessedAsFirstPartyDueToUserInteraction FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", ") as q ON o.domainID = q.domainID) LIMIT 1 OFFSET ?");
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToStorageAccessAPIWithoutUI:
+        return std::make_pair("SELECT timesAccessedAsFirstPartyDueToStorageAccessAPI FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", ") as q ON o.domainID = q.domainID) LIMIT 1 OFFSET ?");
+    case PrevalentResourceDatabaseTelemetry::Statistic::NumberOfPrevalentResourcesWithUI:
+        LOG_ERROR("ResourceLoadStatisticsDatabaseStore::makeMedianWithoutUIQuery was called for an incorrect statistic, undetermined query behavior will result.");
+        RELEASE_ASSERT_NOT_REACHED();
+    }
+}
+
+static SQLiteStatement makeMedianWithoutUIQuery(SQLiteDatabase& database, PrevalentResourceDatabaseTelemetry::Statistic statistic)
+{
+    auto[queryStart, queryEnd] = buildQueryStartAndEnd(statistic);
+
+    return SQLiteStatement(database, makeString(queryStart, joinSubStatisticsForSorting(), queryEnd));
+}
+
+static unsigned getMedianOfPrevalentResourcesWithUserInteraction(SQLiteDatabase& database, unsigned prevalentResourcesWithUserInteractionCount)
+{
+    SQLiteStatement medianDaysSinceUIStatement = makeMedianWithUIQuery(database);
+
+    // Prepare
+    if (medianDaysSinceUIStatement.prepare() != SQLITE_OK) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %{public}s", database.lastErrorMsg());
+        ASSERT_NOT_REACHED();
+        return 0;
+    }
+
+    // Bind
+    if (medianDaysSinceUIStatement.bindInt(1, 1) != SQLITE_OK || medianDaysSinceUIStatement.bindInt(2, (prevalentResourcesWithUserInteractionCount / 2) != SQLITE_OK)) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %{public}s", database.lastErrorMsg());
+        ASSERT_NOT_REACHED();
+        return 0;
+    }
+
+    // Step
+    if (medianDaysSinceUIStatement.step() != SQLITE_ROW) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %{public}s", database.lastErrorMsg());
+        return 0;
+    }
+
+    double rawSeconds = medianDaysSinceUIStatement.getColumnDouble(0);
+    WallTime wallTime = WallTime::fromRawSeconds(rawSeconds);
+    unsigned median = wallTime <= WallTime() ? 0 : std::floor((WallTime::now() - wallTime) / 24_h);
+
+    if (prevalentResourcesWithUserInteractionCount & 1)
+        return median;
+
+    SQLiteStatement lowerMedianDaysSinceUIStatement = makeMedianWithUIQuery(database);
+
+    // Prepare
+    if (lowerMedianDaysSinceUIStatement.prepare() != SQLITE_OK) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %{public}s", database.lastErrorMsg());
+        ASSERT_NOT_REACHED();
+        return 0;
+    }
+
+    // Bind
+    if (lowerMedianDaysSinceUIStatement.bindInt(1, 1) != SQLITE_OK || lowerMedianDaysSinceUIStatement.bindInt(2, ((prevalentResourcesWithUserInteractionCount - 1) / 2)) != SQLITE_OK) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %{public}s", database.lastErrorMsg());
+        ASSERT_NOT_REACHED();
+        return 0;
+    }
+
+    // Step
+    if (lowerMedianDaysSinceUIStatement.step() != SQLITE_ROW) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getMedianOfPrevalentResourcesWithUserInteraction, error message: %{public}s", database.lastErrorMsg());
+        return 0;
+    }
+
+    double rawSecondsLower = lowerMedianDaysSinceUIStatement.getColumnDouble(0);
+    WallTime wallTimeLower = WallTime::fromRawSeconds(rawSecondsLower);
+    return ((wallTimeLower <= WallTime() ? 0 : std::floor((WallTime::now() - wallTimeLower) / 24_h)) + median) / 2;
+}
+
+unsigned ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResources() const
+{
+    if (m_countPrevalentResourcesStatement.step() == SQLITE_ROW) {
+        unsigned prevalentResourceCount = m_countPrevalentResourcesStatement.getColumnInt(0);
+        if (prevalentResourceCount >= minimumPrevalentResourcesForTelemetry) {
+            resetStatement(m_countPrevalentResourcesStatement);
+            return prevalentResourceCount;
+        }
+    }
+    resetStatement(m_countPrevalentResourcesStatement);
+    return 0;
+}
+
+unsigned ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithUI() const
+{
+    if (m_countPrevalentResourcesWithUserInteractionStatement.step() == SQLITE_ROW) {
+        int count = m_countPrevalentResourcesWithUserInteractionStatement.getColumnInt(0);
+        resetStatement(m_countPrevalentResourcesWithUserInteractionStatement);
+        return count;
+    }
+    resetStatement(m_countPrevalentResourcesWithUserInteractionStatement);
+    return 0;
+}
+
+unsigned ResourceLoadStatisticsDatabaseStore::getTopPrevelentResourceDaysSinceUI() const
+{
+    SQLiteStatement topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement(m_database, makeString("SELECT mostRecentUserInteractionTime FROM ObservedDomains INNER JOIN (SELECT ", joinSubStatisticsForSorting(), " LIMIT 1) as q ON ObservedDomains.domainID = q.domainID;"));
+    
+    // Prepare
+    if (topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.prepare() != SQLITE_OK) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement query failed to prepare, error message: %{public}s", m_database.lastErrorMsg());
+        ASSERT_NOT_REACHED();
+        return 0;
+    }
+    
+    // Bind
+    if (topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.bindInt(1, 1) != SQLITE_OK) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement query failed to bind, error message: %{public}s", m_database.lastErrorMsg());
+        ASSERT_NOT_REACHED();
+        return 0;
+    }
+    
+    // Step
+    if (topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.step() != SQLITE_ROW) {
+        RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement query failed to step, error message: %{public}s", m_database.lastErrorMsg());
+        return 0;
+    }
+    
+    double rawSeconds = topPrevalentResourceWithUserInteractionDaysSinceUserInteractionStatement.getColumnDouble(0);
+    WallTime wallTime = WallTime::fromRawSeconds(rawSeconds);
+    
+    return wallTime <= WallTime() ? 0 : std::floor((WallTime::now() - wallTime) / 24_h);
+}
+
+static unsigned getMedianOfPrevalentResourceWithoutUserInteraction(SQLiteDatabase& database, unsigned bucketSize, PrevalentResourceDatabaseTelemetry::Statistic statistic, unsigned numberOfPrevalentResourcesWithoutUI)
+{
+    if (numberOfPrevalentResourcesWithoutUI < bucketSize)
+        return 0;
+
+    unsigned median;
+    SQLiteStatement getMedianStatistic = makeMedianWithoutUIQuery(database, statistic);
+
+    if (getMedianStatistic.prepare() == SQLITE_OK) {
+        if (getMedianStatistic.bindInt(1, 0) != SQLITE_OK
+            || getMedianStatistic.bindInt(2, (bucketSize / 2)) != SQLITE_OK) {
+            RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::makeMedianWithoutUIQuery, error message: %{public}s", database.lastErrorMsg());
+            ASSERT_NOT_REACHED();
+            return 0;
+        }
+        if (getMedianStatistic.step() == SQLITE_ROW)
+            median = getMedianStatistic.getColumnDouble(0);
+    }
+
+    if (bucketSize & 1)
+        return median;
+
+    SQLiteStatement getLowerMedianStatistic = makeMedianWithoutUIQuery(database, statistic);
+
+    if (getLowerMedianStatistic.prepare() == SQLITE_OK) {
+        if (getLowerMedianStatistic.bindInt(1, 0) != SQLITE_OK
+            || getLowerMedianStatistic.bindInt(2, ((bucketSize-1) / 2)) != SQLITE_OK) {
+            RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::makeMedianWithoutUIQuery, error message: %{public}s", database.lastErrorMsg());
+            ASSERT_NOT_REACHED();
+            return 0;
+        }
+        if (getLowerMedianStatistic.step() == SQLITE_ROW)
+            return (getLowerMedianStatistic.getColumnDouble(0) + median) / 2;
+    }
+
+    return 0;
+}
+
+static unsigned getNumberOfPrevalentResourcesInTopResources(SQLiteDatabase& database, unsigned bucketSize)
+{
+    SQLiteStatement prevalentResourceCountInTop(database, makeString("SELECT COUNT(*) FROM (SELECT * FROM ObservedDomains o INNER JOIN(SELECT ", joinSubStatisticsForSorting(), ") as q on q.domainID = o.domainID LIMIT ?) as p WHERE p.hadUserInteraction = 1;"));
+
+    if (prevalentResourceCountInTop.prepare() == SQLITE_OK) {
+        if (prevalentResourceCountInTop.bindText(1, "%") != SQLITE_OK
+            || prevalentResourceCountInTop.bindInt(2, bucketSize) != SQLITE_OK) {
+            RELEASE_LOG_ERROR(Network, "ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesInTopResources, error message: %{public}s", database.lastErrorMsg());
+            ASSERT_NOT_REACHED();
+            return 0;
+        }
+
+        if (prevalentResourceCountInTop.step() == SQLITE_ROW)
+            return prevalentResourceCountInTop.getColumnInt(0);
+    }
+
+    return 0;
+}
+
+static unsigned makeStatisticQuery(SQLiteDatabase& database, PrevalentResourceDatabaseTelemetry::Statistic statistic, int bucketSize, unsigned totalWithUI, unsigned totalWithoutUI)
+{
+    switch (statistic) {
+    case PrevalentResourceDatabaseTelemetry::Statistic::NumberOfPrevalentResourcesWithUI:
+        return getNumberOfPrevalentResourcesInTopResources(database, bucketSize);
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubFrameWithoutUI:
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubResourceWithoutUI:
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianUniqueRedirectsWithoutUI:
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianDataRecordsRemovedWithoutUI:
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToUserInteractionWithoutUI:
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToStorageAccessAPIWithoutUI:
+        return getMedianOfPrevalentResourceWithoutUserInteraction(database, bucketSize, statistic, totalWithoutUI);
+    }
+}
+
+unsigned ResourceLoadStatisticsDatabaseStore::getNumberOfPrevalentResourcesWithoutUI() const
+{
+    if (m_countPrevalentResourcesWithoutUserInteractionStatement.step() == SQLITE_ROW) {
+        int count = m_countPrevalentResourcesWithoutUserInteractionStatement.getColumnInt(0);
+        resetStatement(m_countPrevalentResourcesWithoutUserInteractionStatement);
+        return count;
+    }
+    resetStatement(m_countPrevalentResourcesWithoutUserInteractionStatement);
+    return 0;
+}
+
+void ResourceLoadStatisticsDatabaseStore::calculateTelemetryData(PrevalentResourceDatabaseTelemetry& data) const
+{
+    data.numberOfPrevalentResources = getNumberOfPrevalentResources();
+    data.numberOfPrevalentResourcesWithUserInteraction = getNumberOfPrevalentResourcesWithUI();
+    data.numberOfPrevalentResourcesWithoutUserInteraction = getNumberOfPrevalentResourcesWithoutUI();
+    data.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction = getTopPrevelentResourceDaysSinceUI();
+    data.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction = getMedianOfPrevalentResourcesWithUserInteraction(m_database, data.numberOfPrevalentResourcesWithUserInteraction);
+
+    for (unsigned bucketIndex = 0; bucketIndex < bucketSizes.size(); bucketIndex++) {
+        unsigned bucketSize = bucketSizes[bucketIndex];
+
+        if (data.numberOfPrevalentResourcesWithoutUserInteraction < bucketSize)
+            return;
+
+        for (unsigned statisticIndex = 0; statisticIndex < numberOfStatistics; statisticIndex++) {
+            auto statistic = static_cast<PrevalentResourceDatabaseTelemetry::Statistic>(statisticIndex);
+            data.statistics[statisticIndex][bucketIndex] = makeStatisticQuery(m_database, statistic, bucketSize, data.numberOfPrevalentResourcesWithUserInteraction, data.numberOfPrevalentResourcesWithoutUserInteraction);
+        }
+    }
+}
+
 void ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry() const
 {
     ASSERT(!RunLoop::isMain());
 
-    // FIXME(195088): Implement for Database version.
+    if (parameters().shouldSubmitTelemetry) {
+        PrevalentResourceDatabaseTelemetry prevalentResourceDatabaseTelemetry;
+        calculateTelemetryData(prevalentResourceDatabaseTelemetry);
+        WebResourceLoadStatisticsTelemetry::submitTelemetry(*this, prevalentResourceDatabaseTelemetry);
+    }
 }
 
 static String domainsToString(const HashSet<RegistrableDomain>& domains)
index 4e95f54..beed834 100644 (file)
@@ -47,6 +47,32 @@ struct ResourceLoadStatistics;
 
 namespace WebKit {
 
+static constexpr size_t numberOfBucketsPerStatistic = 5;
+static constexpr size_t numberOfStatistics = 7;
+static constexpr std::array<unsigned, numberOfBucketsPerStatistic> bucketSizes {{ 1, 3, 10, 50, 100 }};
+
+struct PrevalentResourceDatabaseTelemetry {
+    using Buckets = std::array<unsigned, numberOfBucketsPerStatistic>;
+
+    enum class Statistic {
+        NumberOfPrevalentResourcesWithUI,
+        MedianSubFrameWithoutUI,
+        MedianSubResourceWithoutUI,
+        MedianUniqueRedirectsWithoutUI,
+        MedianDataRecordsRemovedWithoutUI,
+        MedianTimesAccessedDueToUserInteractionWithoutUI,
+        MedianTimesAccessedDueToStorageAccessAPIWithoutUI
+    };
+
+    unsigned numberOfPrevalentResources;
+    unsigned numberOfPrevalentResourcesWithUserInteraction;
+    unsigned numberOfPrevalentResourcesWithoutUserInteraction;
+    unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction;
+    unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction;
+
+    std::array<Buckets, numberOfStatistics> statistics;
+};
+
 class ResourceLoadStatisticsMemoryStore;
 
 // This is always constructed / used / destroyed on the WebResourceLoadStatisticsStore's statistics queue.
@@ -110,6 +136,13 @@ private:
     void mergeStatistic(const ResourceLoadStatistics&);
     void merge(WebCore::SQLiteStatement&, const ResourceLoadStatistics&);
     void clearDatabaseContents();
+    unsigned getNumberOfPrevalentResources() const;
+    unsigned getNumberOfPrevalentResourcesWithUI() const;
+    unsigned getNumberOfPrevalentResourcesWithoutUI() const;
+    unsigned getTopPrevelentResourceDaysSinceUI() const;
+    void resetTelemetryPreparedStatements() const;
+    void resetTelemetryStatements() const;
+    void calculateTelemetryData(PrevalentResourceDatabaseTelemetry&) const;
     bool insertObservedDomain(const ResourceLoadStatistics&);
     void insertDomainRelationships(const ResourceLoadStatistics&);
     void insertDomainRelationshipList(const String&, const HashSet<RegistrableDomain>&, unsigned);
@@ -197,6 +230,9 @@ private:
     WebCore::SQLiteStatement m_updateGrandfatheredStatement;
     mutable WebCore::SQLiteStatement m_isGrandfatheredStatement;
     mutable WebCore::SQLiteStatement m_findExpiredUserInteractionStatement;
+    mutable WebCore::SQLiteStatement m_countPrevalentResourcesStatement;
+    mutable WebCore::SQLiteStatement m_countPrevalentResourcesWithUserInteractionStatement;
+    mutable WebCore::SQLiteStatement m_countPrevalentResourcesWithoutUserInteractionStatement;
     PAL::SessionID m_sessionID;
 };
 
index ea00bc1..8f35745 100644 (file)
@@ -492,9 +492,13 @@ void WebResourceLoadStatisticsStore::submitTelemetry(CompletionHandler<void()>&&
     ASSERT(RunLoop::isMain());
 
     postTask([this, completionHandler = WTFMove(completionHandler)]() mutable  {
-        if (m_statisticsStore && is<ResourceLoadStatisticsMemoryStore>(*m_statisticsStore))
+        if (!m_statisticsStore)
+            return;
+        
+        if (is<ResourceLoadStatisticsMemoryStore>(*m_statisticsStore))
             WebResourceLoadStatisticsTelemetry::calculateAndSubmit(downcast<ResourceLoadStatisticsMemoryStore>(*m_statisticsStore));
-
+        else
+            m_statisticsStore->calculateAndSubmitTelemetry();
         postTaskReply(WTFMove(completionHandler));
     });
 }
@@ -1077,11 +1081,11 @@ void WebResourceLoadStatisticsStore::sendDiagnosticMessageWithValue(const String
         const_cast<WebResourceLoadStatisticsStore*>(this)->networkSession()->logDiagnosticMessageWithValue(message, description, value, sigDigits, shouldSample);
 }
 
-void WebResourceLoadStatisticsStore::notifyPageStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins) const
+void WebResourceLoadStatisticsStore::notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI) const
 {
     ASSERT(RunLoop::isMain());
     if (m_networkSession)
-        const_cast<WebResourceLoadStatisticsStore*>(this)->networkSession()->notifyPageStatisticsTelemetryFinished(totalPrevalentResources, totalPrevalentResourcesWithUserInteraction, top3SubframeUnderTopFrameOrigins);
+        const_cast<WebResourceLoadStatisticsStore*>(this)->networkSession()->notifyPageStatisticsTelemetryFinished(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI);
 }
 
 } // namespace WebKit
index 92fe157..4164ab6 100644 (file)
@@ -188,7 +188,7 @@ public:
     void invalidateAndCancel();
 
     void sendDiagnosticMessageWithValue(const String& message, const String& description, unsigned value, unsigned sigDigits, WebCore::ShouldSample) const;
-    void notifyPageStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins) const;
+    void notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI) const;
 
     void resourceLoadStatisticsUpdated(Vector<WebCore::ResourceLoadStatistics>&&);
     void requestStorageAccessUnderOpener(DomainInNeedOfStorageAccess&&, WebCore::PageIdentifier openerID, OpenerDomain&&);
index 1c0e776..b9e9f5e 100644 (file)
@@ -28,6 +28,7 @@
 
 #if ENABLE(RESOURCE_LOAD_STATISTICS)
 
+#include "ResourceLoadStatisticsDatabaseStore.h"
 #include "ResourceLoadStatisticsMemoryStore.h"
 #include "WebPageProxy.h"
 #include "WebProcessPool.h"
@@ -199,10 +200,10 @@ static void submitTopLists(const Vector<PrevalentResourceTelemetry>& sortedPreva
 }
     
 // This function is for testing purposes.
-void static notifyPages(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins, const WebResourceLoadStatisticsStore& store)
+void static notifyPages(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI, const WebResourceLoadStatisticsStore& store)
 {
-    RunLoop::main().dispatch([totalPrevalentResources, totalPrevalentResourcesWithUserInteraction, top3SubframeUnderTopFrameOrigins, store = makeRef(store)] {
-        store->notifyPageStatisticsTelemetryFinished(totalPrevalentResources, totalPrevalentResourcesWithUserInteraction, top3SubframeUnderTopFrameOrigins);
+    RunLoop::main().dispatch([numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI, store = makeRef(store)] {
+        store->notifyPageStatisticsTelemetryFinished(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI);
     });
 }
     
@@ -212,8 +213,7 @@ void static notifyPages(const Vector<PrevalentResourceTelemetry>& sortedPrevalen
     WTF::Function<unsigned(const PrevalentResourceTelemetry& telemetry)> subframeUnderTopFrameOriginsGetter = [] (const PrevalentResourceTelemetry& t) {
         return t.subframeUnderTopFrameOrigins;
     };
-    
-    notifyPages(sortedPrevalentResources.size(), totalNumberOfPrevalentResourcesWithUserInteraction, median(sortedPrevalentResourcesWithoutUserInteraction, 0, 2, subframeUnderTopFrameOriginsGetter), store);
+    notifyPages(sortedPrevalentResources.size(), totalNumberOfPrevalentResourcesWithUserInteraction, 0, 0, 0, 0, median(sortedPrevalentResourcesWithoutUserInteraction, 0, 2, subframeUnderTopFrameOriginsGetter), 0, 0, 0,  store);
 }
     
 void WebResourceLoadStatisticsTelemetry::calculateAndSubmit(const ResourceLoadStatisticsMemoryStore& resourceLoadStatisticsStore)
@@ -222,7 +222,7 @@ void WebResourceLoadStatisticsTelemetry::calculateAndSubmit(const ResourceLoadSt
     
     auto sortedPrevalentResources = sortedPrevalentResourceTelemetry(resourceLoadStatisticsStore);
     if (notifyPagesWhenTelemetryWasCaptured && sortedPrevalentResources.isEmpty()) {
-        notifyPages(0, 0, 0, resourceLoadStatisticsStore.store());
+        notifyPages(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, resourceLoadStatisticsStore.store());
         return;
     }
     
@@ -242,7 +242,7 @@ void WebResourceLoadStatisticsTelemetry::calculateAndSubmit(const ResourceLoadSt
         auto webPageProxy = WebPageProxy::nonEphemeralWebPageProxy();
         if (!webPageProxy) {
             if (notifyPagesWhenTelemetryWasCaptured)
-                notifyPages(0, 0, 0, store);
+                notifyPages(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, store);
             return;
         }
         
@@ -263,6 +263,90 @@ void WebResourceLoadStatisticsTelemetry::calculateAndSubmit(const ResourceLoadSt
         submitTopLists(sortedPrevalentResources, sortedPrevalentResourcesWithoutUserInteraction, store);
     });
 }
+
+static StringView makeDescription(PrevalentResourceDatabaseTelemetry::Statistic statistic)
+{
+    switch (statistic) {
+    case PrevalentResourceDatabaseTelemetry::Statistic::NumberOfPrevalentResourcesWithUI:
+        return "PrevalentResourcesWithUserInteraction";
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubFrameWithoutUI:
+        return "SubframeUnderTopFrameOrigins";
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianSubResourceWithoutUI:
+        return "SubresourceUnderTopFrameOrigins";
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianUniqueRedirectsWithoutUI:
+        return "SubresourceUniqueRedirectsTo";
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianDataRecordsRemovedWithoutUI:
+        return "NumberOfTimesDataRecordsRemoved";
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToUserInteractionWithoutUI:
+        return "NumberOfTimesAccessedAsFirstPartyDueToUserInteraction";
+    case PrevalentResourceDatabaseTelemetry::Statistic::MedianTimesAccessedDueToStorageAccessAPIWithoutUI:
+        return "NumberOfTimesAccessedAsFirstPartyDueToStorageAccessAPI";
+    }
+}
+
+static void databaseSubmitTopLists(const PrevalentResourceDatabaseTelemetry& telemetry, const WebResourceLoadStatisticsStore& store)
+{
+    for (unsigned bucketIndex = 0; bucketIndex < bucketSizes.size(); bucketIndex++) {
+
+        if (telemetry.numberOfPrevalentResourcesWithoutUserInteraction < bucketSizes[bucketIndex])
+            return;
+
+        String descriptionPreamble = makeString("top", bucketSizes[bucketIndex]);
+
+        for (unsigned statisticIndex = 0; statisticIndex < numberOfStatistics; statisticIndex++) {
+            auto statistic = static_cast<PrevalentResourceDatabaseTelemetry::Statistic>(statisticIndex);
+
+            store.sendDiagnosticMessageWithValue(DiagnosticLoggingKeys::resourceLoadStatisticsTelemetryKey(), descriptionPreamble + makeDescription(statistic), telemetry.statistics[statisticIndex][bucketIndex], significantFiguresForLoggedValues, ShouldSample::No);
+        }
+    }
+}
+
+void WebResourceLoadStatisticsTelemetry::submitTelemetry(const ResourceLoadStatisticsDatabaseStore& resourceLoadStatisticsStore, PrevalentResourceDatabaseTelemetry& prevalentResourceDatabaseTelemetry)
+{
+    ASSERT(!RunLoop::isMain());
+
+    if (notifyPagesWhenTelemetryWasCaptured && !prevalentResourceDatabaseTelemetry.numberOfPrevalentResources) {
+        notifyPages(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, resourceLoadStatisticsStore.store());
+        return;
+    }
+
+    // Dispatch on the main thread to make sure the WebPageProxy we're using doesn't go away.
+    RunLoop::main().dispatch([telemetry = WTFMove(prevalentResourceDatabaseTelemetry), store = makeRef(resourceLoadStatisticsStore.store())] () {
+
+        // The notify pages function is for testing so we don't need to do an actual submission.
+        if (notifyPagesWhenTelemetryWasCaptured) {
+            if (telemetry.numberOfPrevalentResourcesWithoutUserInteraction < 3) {
+                notifyPages(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, store);
+                return;
+            }
+            notifyPages(telemetry.numberOfPrevalentResources,
+                telemetry.numberOfPrevalentResourcesWithUserInteraction,
+                telemetry.numberOfPrevalentResourcesWithoutUserInteraction,
+                telemetry.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction,
+                telemetry.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction,
+                telemetry.statistics[0][1], // bucket 1 -> top3.
+                telemetry.statistics[1][1],
+                telemetry.statistics[2][1],
+                telemetry.statistics[3][1],
+                telemetry.statistics[4][1],
+                store);
+            return;
+        }
+        
+        auto webPageProxy = WebPageProxy::nonEphemeralWebPageProxy();
+        if (!webPageProxy)
+            return;
+
+        webPageProxy->logDiagnosticMessageWithValue(DiagnosticLoggingKeys::resourceLoadStatisticsTelemetryKey(), "totalNumberOfPrevalentResources"_s, telemetry.numberOfPrevalentResources, significantFiguresForLoggedValues, ShouldSample::No);
+        webPageProxy->logDiagnosticMessageWithValue(DiagnosticLoggingKeys::resourceLoadStatisticsTelemetryKey(), "totalNumberOfPrevalentResourcesWithUserInteraction"_s, telemetry.numberOfPrevalentResourcesWithUserInteraction, significantFiguresForLoggedValues, ShouldSample::No);
+        if (telemetry.numberOfPrevalentResourcesWithUserInteraction > 0)
+            webPageProxy->logDiagnosticMessageWithValue(DiagnosticLoggingKeys::resourceLoadStatisticsTelemetryKey(), "topPrevalentResourceWithUserInteractionDaysSinceUserInteraction"_s, telemetry.topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, significantFiguresForLoggedValues, ShouldSample::No);
+        if (telemetry.numberOfPrevalentResourcesWithUserInteraction > 1)
+            webPageProxy->logDiagnosticMessageWithValue(DiagnosticLoggingKeys::resourceLoadStatisticsTelemetryKey(), "medianPrevalentResourcesWithUserInteractionDaysSinceUserInteraction"_s, telemetry.medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, significantFiguresForLoggedValues, ShouldSample::No);
+
+        databaseSubmitTopLists(telemetry, store);
+    });
+}
     
 void WebResourceLoadStatisticsTelemetry::setNotifyPagesWhenTelemetryWasCaptured(bool always)
 {
index b92d821..d7ee107 100644 (file)
 namespace WebKit {
 
 class ResourceLoadStatisticsMemoryStore;
+class ResourceLoadStatisticsDatabaseStore;
+struct PrevalentResourceDatabaseTelemetry;
 
 namespace WebResourceLoadStatisticsTelemetry {
     
 void calculateAndSubmit(const ResourceLoadStatisticsMemoryStore&);
+void submitTelemetry(const ResourceLoadStatisticsDatabaseStore&, PrevalentResourceDatabaseTelemetry&);
 void setNotifyPagesWhenTelemetryWasCaptured(bool);
     
 }
index 0c9baaf..112cd2b 100644 (file)
@@ -193,9 +193,9 @@ void NetworkSession::logDiagnosticMessageWithValue(const String& message, const
     m_networkProcess->parentProcessConnection()->send(Messages::WebPageProxy::LogDiagnosticMessageWithValue(message, description, value, significantFigures, shouldSample), 0);
 }
 
-void NetworkSession::notifyPageStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins)
+void NetworkSession::notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)
 {
-    m_networkProcess->parentProcessConnection()->send(Messages::NetworkProcessProxy::NotifyResourceLoadStatisticsTelemetryFinished(totalPrevalentResources, totalPrevalentResourcesWithUserInteraction, top3SubframeUnderTopFrameOrigins), 0);
+    m_networkProcess->parentProcessConnection()->send(Messages::NetworkProcessProxy::NotifyResourceLoadStatisticsTelemetryFinished(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI), 0);
 }
 
 void NetworkSession::deleteWebsiteDataForRegistrableDomains(OptionSet<WebsiteDataType> dataTypes, Vector<std::pair<RegistrableDomain, WebsiteDataToRemove>>&& domains, bool shouldNotifyPage, CompletionHandler<void(const HashSet<RegistrableDomain>&)>&& completionHandler)
index 7ce2dbb..9102279 100644 (file)
@@ -90,7 +90,7 @@ public:
     void deleteWebsiteDataForRegistrableDomains(OptionSet<WebsiteDataType>, Vector<std::pair<WebCore::RegistrableDomain, WebsiteDataToRemove>>&&, bool shouldNotifyPage, CompletionHandler<void(const HashSet<WebCore::RegistrableDomain>&)>&&);
     void registrableDomainsWithWebsiteData(OptionSet<WebsiteDataType>, bool shouldNotifyPage, CompletionHandler<void(HashSet<WebCore::RegistrableDomain>&&)>&&);
     void logDiagnosticMessageWithValue(const String& message, const String& description, unsigned value, unsigned significantFigures, WebCore::ShouldSample);
-    void notifyPageStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins);
+    void notifyPageStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI);
     bool enableResourceLoadStatisticsLogTestingEvent() const { return m_enableResourceLoadStatisticsLogTestingEvent; }
     void setResourceLoadStatisticsLogTestingEvent(bool log) { m_enableResourceLoadStatisticsLogTestingEvent = log; }
     bool shouldIsolateSessionsForPrevalentTopFrames() const { return m_enableResourceLoadStatisticsNSURLSessionSwitching == EnableResourceLoadStatisticsNSURLSessionSwitching::Yes; }
index 50c40a5..6e30a07 100644 (file)
@@ -956,12 +956,19 @@ void NetworkProcessProxy::notifyWebsiteDataScanForRegistrableDomainsFinished()
     WebProcessProxy::notifyWebsiteDataScanForRegistrableDomainsFinished();
 }
 
-void NetworkProcessProxy::notifyResourceLoadStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins)
+void NetworkProcessProxy::notifyResourceLoadStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)
 {
     API::Dictionary::MapType messageBody;
-    messageBody.set("TotalPrevalentResources"_s, API::UInt64::create(totalPrevalentResources));
-    messageBody.set("TotalPrevalentResourcesWithUserInteraction"_s, API::UInt64::create(totalPrevalentResourcesWithUserInteraction));
-    messageBody.set("Top3SubframeUnderTopFrameOrigins"_s, API::UInt64::create(top3SubframeUnderTopFrameOrigins));
+    messageBody.set("NumberOfPrevalentResources"_s, API::UInt64::create(numberOfPrevalentResources));
+    messageBody.set("NumberOfPrevalentResourcesWithUserInteraction"_s, API::UInt64::create(numberOfPrevalentResourcesWithUserInteraction));
+    messageBody.set("NumberOfPrevalentResourcesWithoutUserInteraction"_s, API::UInt64::create(numberOfPrevalentResourcesWithoutUserInteraction));
+    messageBody.set("TopPrevalentResourceWithUserInteractionDaysSinceUserInteraction"_s, API::UInt64::create(topPrevalentResourceWithUserInteractionDaysSinceUserInteraction));
+    messageBody.set("MedianDaysSinceUserInteractionPrevalentResourceWithUserInteraction"_s, API::UInt64::create(medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction));
+    messageBody.set("Top3NumberOfPrevalentResourcesWithUI"_s, API::UInt64::create(top3NumberOfPrevalentResourcesWithUI));
+    messageBody.set("Top3MedianSubFrameWithoutUI"_s, API::UInt64::create(top3MedianSubFrameWithoutUI));
+    messageBody.set("Top3MedianSubResourceWithoutUI"_s, API::UInt64::create(top3MedianSubResourceWithoutUI));
+    messageBody.set("Top3MedianUniqueRedirectsWithoutUI"_s, API::UInt64::create(top3MedianUniqueRedirectsWithoutUI));
+    messageBody.set("Top3MedianDataRecordsRemovedWithoutUI"_s, API::UInt64::create(top3MedianDataRecordsRemovedWithoutUI));
 
     WebProcessProxy::notifyPageStatisticsTelemetryFinished(API::Dictionary::create(messageBody).ptr());
 }
index 04cd511..e65aaf4 100644 (file)
@@ -232,7 +232,7 @@ private:
     void notifyResourceLoadStatisticsProcessed();
     void notifyWebsiteDataDeletionForRegistrableDomainsFinished();
     void notifyWebsiteDataScanForRegistrableDomainsFinished();
-    void notifyResourceLoadStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins);
+    void notifyResourceLoadStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI);
 #endif
     void retrieveCacheStorageParameters(PAL::SessionID);
 
index a6dbb7c..3ab0db0 100644 (file)
@@ -45,7 +45,7 @@ messages -> NetworkProcessProxy LegacyReceiver {
     NotifyResourceLoadStatisticsProcessed()
     NotifyWebsiteDataDeletionForRegistrableDomainsFinished()
     NotifyWebsiteDataScanForRegistrableDomainsFinished()
-    NotifyResourceLoadStatisticsTelemetryFinished(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins)
+    NotifyResourceLoadStatisticsTelemetryFinished(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)
     RequestStorageAccessConfirm(WebKit::WebPageProxyIdentifier pageID, WebCore::FrameIdentifier frameID, WebCore::RegistrableDomain subFrameDomain, WebCore::RegistrableDomain topFrameDomain) -> (bool userDidGrantAccess) Async
     DeleteWebsiteDataInUIProcessForRegistrableDomains(PAL::SessionID sessionID, OptionSet<WebKit::WebsiteDataType> dataTypes, OptionSet<WebKit::WebsiteDataFetchOption> fetchOptions, Vector<WebCore::RegistrableDomain> domains) -> (HashSet<WebCore::RegistrableDomain> domainsWithMatchingDataRecords) Async
     DidCommitCrossSiteLoadWithDataTransferFromPrevalentResource(WebKit::WebPageProxyIdentifier pageID)
index 54b25db..8361366 100644 (file)
@@ -1,3 +1,19 @@
+2019-10-08  Kate Cheney  <katherine_cheney@apple.com>
+
+        Implement Telemetry and Dumping Routines for SQLite backend (195088)
+        https://bugs.webkit.org/show_bug.cgi?id=195088
+        <rdar://problem/54213407>
+
+        Reviewed by John Wilander.
+
+        Updated the current testing for telemetry which only tested 3
+        statistics. With this patch it now tests 10 statistics.
+        * WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:
+        (WTR::InjectedBundle::didReceiveMessageToPage):
+        * WebKitTestRunner/InjectedBundle/TestRunner.cpp:
+        (WTR::TestRunner::statisticsDidRunTelemetryCallback):
+        * WebKitTestRunner/InjectedBundle/TestRunner.h:
+
 2019-10-08  Timothy Hatcher  <timothy@apple.com>
 
         Copying white text from dark mode WebKit apps and pasting in a light mode app results in white (invisible) text.
index 01b7562..eb8735c 100644 (file)
@@ -440,15 +440,29 @@ void InjectedBundle::didReceiveMessageToPage(WKBundlePageRef page, WKStringRef m
     if (WKStringIsEqualToUTF8CString(messageName, "ResourceLoadStatisticsTelemetryFinished")) {
         WKDictionaryRef messageBodyDictionary = static_cast<WKDictionaryRef>(messageBody);
 
-        WKRetainPtr<WKStringRef> totalPrevalentResourcesKey = adoptWK(WKStringCreateWithUTF8CString("TotalPrevalentResources"));
-        WKRetainPtr<WKStringRef> totalPrevalentResourcesWithUserInteractionKey = adoptWK(WKStringCreateWithUTF8CString("TotalPrevalentResourcesWithUserInteraction"));
-        WKRetainPtr<WKStringRef> top3SubframeUnderTopFrameOriginsKey = adoptWK(WKStringCreateWithUTF8CString("Top3SubframeUnderTopFrameOrigins"));
-
-        unsigned totalPrevalentResources = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, totalPrevalentResourcesKey.get())));
-        unsigned totalPrevalentResourcesWithUserInteraction = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, totalPrevalentResourcesWithUserInteractionKey.get())));
-        unsigned top3SubframeUnderTopFrameOrigins = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, top3SubframeUnderTopFrameOriginsKey.get())));
-
-        m_testRunner->statisticsDidRunTelemetryCallback(totalPrevalentResources, totalPrevalentResourcesWithUserInteraction, top3SubframeUnderTopFrameOrigins);
+        WKRetainPtr<WKStringRef> numberOfPrevalentResourcesKey = adoptWK(WKStringCreateWithUTF8CString("NumberOfPrevalentResources"));
+        WKRetainPtr<WKStringRef> numberOfPrevalentResourcesWithUserInteractionKey = adoptWK(WKStringCreateWithUTF8CString("NumberOfPrevalentResourcesWithUserInteraction"));
+        WKRetainPtr<WKStringRef> numberOfPrevalentResourcesWithoutUserInteractionKey = adoptWK(WKStringCreateWithUTF8CString("NumberOfPrevalentResourcesWithoutUserInteraction"));
+        WKRetainPtr<WKStringRef> topPrevalentResourceWithUserInteractionDaysSinceUserInteractionKey = adoptWK(WKStringCreateWithUTF8CString("TopPrevalentResourceWithUserInteractionDaysSinceUserInteraction"));
+        WKRetainPtr<WKStringRef> medianDaysSinceUserInteractionPrevalentResourceWithUserInteractionKey = adoptWK(WKStringCreateWithUTF8CString("MedianDaysSinceUserInteractionPrevalentResourceWithUserInteraction"));
+        WKRetainPtr<WKStringRef> top3NumberOfPrevalentResourcesWithUIKey = adoptWK(WKStringCreateWithUTF8CString("Top3NumberOfPrevalentResourcesWithUI"));
+        WKRetainPtr<WKStringRef> top3MedianSubFrameWithoutUIKey = adoptWK(WKStringCreateWithUTF8CString("Top3MedianSubFrameWithoutUI"));
+        WKRetainPtr<WKStringRef> top3MedianSubResourceWithoutUIKey = adoptWK(WKStringCreateWithUTF8CString("Top3MedianSubResourceWithoutUI"));
+        WKRetainPtr<WKStringRef> top3MedianUniqueRedirectsWithoutUIKey = adoptWK(WKStringCreateWithUTF8CString("Top3MedianUniqueRedirectsWithoutUI"));
+        WKRetainPtr<WKStringRef> top3MedianDataRecordsRemovedWithoutUIKey = adoptWK(WKStringCreateWithUTF8CString("Top3MedianDataRecordsRemovedWithoutUI"));
+        
+        unsigned numberOfPrevalentResources = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, numberOfPrevalentResourcesKey.get())));
+        unsigned numberOfPrevalentResourcesWithUserInteraction = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, numberOfPrevalentResourcesWithUserInteractionKey.get())));
+        unsigned numberOfPrevalentResourcesWithoutUserInteraction = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, numberOfPrevalentResourcesWithoutUserInteractionKey.get())));
+        unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, topPrevalentResourceWithUserInteractionDaysSinceUserInteractionKey.get())));
+        unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, medianDaysSinceUserInteractionPrevalentResourceWithUserInteractionKey.get())));
+        unsigned top3NumberOfPrevalentResourcesWithUI = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, top3NumberOfPrevalentResourcesWithUIKey.get())));
+        unsigned top3MedianSubFrameWithoutUI = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, top3MedianSubFrameWithoutUIKey.get())));
+        unsigned top3MedianSubResourceWithoutUI = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, top3MedianSubResourceWithoutUIKey.get())));
+        unsigned top3MedianUniqueRedirectsWithoutUI = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, top3MedianUniqueRedirectsWithoutUIKey.get())));
+        unsigned top3MedianDataRecordsRemovedWithoutUI = (unsigned)WKUInt64GetValue(static_cast<WKUInt64Ref>(WKDictionaryGetItemForKey(messageBodyDictionary, top3MedianDataRecordsRemovedWithoutUIKey.get())));
+
+        m_testRunner->statisticsDidRunTelemetryCallback(numberOfPrevalentResources, numberOfPrevalentResourcesWithUserInteraction, numberOfPrevalentResourcesWithoutUserInteraction, topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, top3NumberOfPrevalentResourcesWithUI, top3MedianSubFrameWithoutUI, top3MedianSubResourceWithoutUI, top3MedianUniqueRedirectsWithoutUI, top3MedianDataRecordsRemovedWithoutUI);
         return;
     }
     
index 8dbc6ba..46131cc 100644 (file)
@@ -2008,12 +2008,12 @@ void TestRunner::installStatisticsDidRunTelemetryCallback(JSValueRef callback)
     cacheTestRunnerCallback(StatisticsDidRunTelemetryCallbackID, callback);
 }
     
-void TestRunner::statisticsDidRunTelemetryCallback(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins)
+void TestRunner::statisticsDidRunTelemetryCallback(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI)
 {
     WKBundleFrameRef mainFrame = WKBundlePageGetMainFrame(InjectedBundle::singleton().page()->page());
     JSContextRef context = WKBundleFrameGetJavaScriptContext(mainFrame);
     
-    String string = makeString("{ \"totalPrevalentResources\" : ", totalPrevalentResources, ", \"totalPrevalentResourcesWithUserInteraction\" : ", totalPrevalentResourcesWithUserInteraction, ", \"top3SubframeUnderTopFrameOrigins\" : ", top3SubframeUnderTopFrameOrigins, " }");
+    String string = makeString("{ \"numberOfPrevalentResources\" : ", numberOfPrevalentResources, ", \"numberOfPrevalentResourcesWithUserInteraction\" : ", numberOfPrevalentResourcesWithUserInteraction, ", \"numberOfPrevalentResourcesWithoutUserInteraction\" : ", numberOfPrevalentResourcesWithoutUserInteraction, ", \"topPrevalentResourceWithUserInteractionDaysSinceUserInteraction\" : ", topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, ", \"medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction\" : ", medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, ", \"top3NumberOfPrevalentResourcesWithUI\" : ", top3NumberOfPrevalentResourcesWithUI, ", \"top3MedianSubFrameWithoutUI\" : ", top3MedianSubFrameWithoutUI, ", \"top3MedianSubResourceWithoutUI\" : ", top3MedianSubResourceWithoutUI, ", \"top3MedianUniqueRedirectsWithoutUI\" : ", top3MedianUniqueRedirectsWithoutUI, ", \"top3MedianDataRecordsRemovedWithoutUI\" : ", top3MedianDataRecordsRemovedWithoutUI, " }");
     
     JSValueRef result = JSValueMakeFromJSONString(context, adopt(JSStringCreateWithUTF8CString(string.utf8().data())).get());
 
index 3c06353..ff9f972 100644 (file)
@@ -384,7 +384,7 @@ public:
     void installStatisticsDidRunTelemetryCallback(JSValueRef callback);
     void statisticsDidModifyDataRecordsCallback();
     void statisticsDidScanDataRecordsCallback();
-    void statisticsDidRunTelemetryCallback(unsigned totalPrevalentResources, unsigned totalPrevalentResourcesWithUserInteraction, unsigned top3SubframeUnderTopFrameOrigins);
+    void statisticsDidRunTelemetryCallback(unsigned numberOfPrevalentResources, unsigned numberOfPrevalentResourcesWithUserInteraction, unsigned numberOfPrevalentResourcesWithoutUserInteraction, unsigned topPrevalentResourceWithUserInteractionDaysSinceUserInteraction, unsigned medianDaysSinceUserInteractionPrevalentResourceWithUserInteraction, unsigned top3NumberOfPrevalentResourcesWithUI, unsigned top3MedianSubFrameWithoutUI, unsigned top3MedianSubResourceWithoutUI, unsigned top3MedianUniqueRedirectsWithoutUI, unsigned top3MedianDataRecordsRemovedWithoutUI);
     bool statisticsNotifyObserver();
     void statisticsProcessStatisticsAndDataRecords();
     void statisticsUpdateCookieBlocking(JSValueRef completionHandler);