Handle remote audio capture IPC messages in a background thread
authoryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 8 May 2020 12:42:03 +0000 (12:42 +0000)
committeryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 8 May 2020 12:42:03 +0000 (12:42 +0000)
https://bugs.webkit.org/show_bug.cgi?id=211583

Reviewed by Eric Carlson.

Source/WebCore:

* platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.cpp:
(WebCore::AudioMediaStreamTrackRendererCocoa::pushSamples):
Add assertion to check that we are not running on the main thread.

Source/WebKit:

We introduce RemoteCaptureSampleManager which is responsible to get remote capture sample IPC messages and treat them from a background thread.
It is currently used to handle audio capture IPC messages.
We move UserMediaCaptureManager::Source in its own file and rename it to RemoteRealtimeMediaSource.

Covered by existing tests.

* DerivedSources-input.xcfilelist:
* DerivedSources-output.xcfilelist:
* DerivedSources.make:
* PlatformMac.cmake:
* SourcesCocoa.txt:
* UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
* WebKit.xcodeproj/project.pbxproj:
* WebProcess/cocoa/RemoteCaptureSampleManager.cpp: Added.
(WebKit::RemoteCaptureSampleManager::RemoteCaptureSampleManager):
(WebKit::RemoteCaptureSampleManager::~RemoteCaptureSampleManager):
(WebKit::RemoteCaptureSampleManager::setConnection):
(WebKit::RemoteCaptureSampleManager::addSource):
(WebKit::RemoteCaptureSampleManager::removeSource):
(WebKit::RemoteCaptureSampleManager::dispatchToThread):
(WebKit::RemoteCaptureSampleManager::storageChanged):
(WebKit::RemoteCaptureSampleManager::audioSamplesAvailable):
* WebProcess/cocoa/RemoteCaptureSampleManager.h: Added.
* WebProcess/cocoa/RemoteCaptureSampleManager.messages.in: Added.
* WebProcess/cocoa/RemoteRealtimeMediaSource.cpp: Added.
(WebKit::RemoteRealtimeMediaSource::RemoteRealtimeMediaSource):
(WebKit::RemoteRealtimeMediaSource::~RemoteRealtimeMediaSource):
(WebKit::RemoteRealtimeMediaSource::whenReady):
(WebKit::RemoteRealtimeMediaSource::didFail):
(WebKit::RemoteRealtimeMediaSource::setAsReady):
(WebKit::RemoteRealtimeMediaSource::clone):
(WebKit::RemoteRealtimeMediaSource::setCapabilities):
(WebKit::RemoteRealtimeMediaSource::setSettings):
(WebKit::RemoteRealtimeMediaSource::remoteAudioSamplesAvailable):
(WebKit::RemoteRealtimeMediaSource::remoteVideoSampleAvailable):
(WebKit::RemoteRealtimeMediaSource::connection):
(WebKit::RemoteRealtimeMediaSource::startProducingData):
(WebKit::RemoteRealtimeMediaSource::stopProducingData):
(WebKit::RemoteRealtimeMediaSource::setShouldApplyRotation):
(WebKit::RemoteRealtimeMediaSource::capabilities):
(WebKit::RemoteRealtimeMediaSource::applyConstraints):
(WebKit::RemoteRealtimeMediaSource::applyConstraintsSucceeded):
(WebKit::RemoteRealtimeMediaSource::applyConstraintsFailed):
(WebKit::RemoteRealtimeMediaSource::hasEnded):
(WebKit::RemoteRealtimeMediaSource::stopBeingObserved):
(WebKit::RemoteRealtimeMediaSource::requestToEnd):
* WebProcess/cocoa/RemoteRealtimeMediaSource.h: Added.
(WebKit::RemoteRealtimeMediaSource::identifier const):
(WebKit::RemoteRealtimeMediaSource::settings const):
(WebKit::RemoteRealtimeMediaSource::setShouldCaptureInGPUProcess):
(WebKit::RemoteRealtimeMediaSource::shouldCaptureInGPUProcess const):
* WebProcess/cocoa/UserMediaCaptureManager.cpp:
(WebKit::UserMediaCaptureManager::createCaptureSource):
(WebKit::UserMediaCaptureManager::cloneSource):
(WebKit::UserMediaCaptureManager::cloneVideoSource):
(WebKit::UserMediaCaptureManager::AudioFactory::createAudioCaptureSource):
(WebKit::UserMediaCaptureManager::AudioFactory::setShouldCaptureInGPUProcess):
(WebKit::UserMediaCaptureManager::VideoFactory::createVideoCaptureSource):
(WebKit::UserMediaCaptureManager::DisplayFactory::createDisplayCaptureSource):
* WebProcess/cocoa/UserMediaCaptureManager.h:
* WebProcess/cocoa/UserMediaCaptureManager.messages.in:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@261375 268f45cc-cd09-0410-ab3c-d52691b4dbfc

18 files changed:
Source/WebCore/ChangeLog
Source/WebCore/platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.cpp
Source/WebKit/ChangeLog
Source/WebKit/DerivedSources-input.xcfilelist
Source/WebKit/DerivedSources-output.xcfilelist
Source/WebKit/DerivedSources.make
Source/WebKit/PlatformMac.cmake
Source/WebKit/SourcesCocoa.txt
Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp
Source/WebKit/WebKit.xcodeproj/project.pbxproj
Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.cpp [new file with mode: 0644]
Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.h [new file with mode: 0644]
Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.messages.in [new file with mode: 0644]
Source/WebKit/WebProcess/cocoa/RemoteRealtimeMediaSource.cpp [new file with mode: 0644]
Source/WebKit/WebProcess/cocoa/RemoteRealtimeMediaSource.h [new file with mode: 0644]
Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.cpp
Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.h
Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.messages.in

index 024a6ab..a9cc70e 100644 (file)
@@ -1,5 +1,16 @@
 2020-05-08  Youenn Fablet  <youenn@apple.com>
 
+        Handle remote audio capture IPC messages in a background thread
+        https://bugs.webkit.org/show_bug.cgi?id=211583
+
+        Reviewed by Eric Carlson.
+
+        * platform/mediastream/mac/AudioMediaStreamTrackRendererCocoa.cpp:
+        (WebCore::AudioMediaStreamTrackRendererCocoa::pushSamples):
+        Add assertion to check that we are not running on the main thread.
+
+2020-05-08  Youenn Fablet  <youenn@apple.com>
+
         Video capture does not get unmuted in case of tab switch on iOS
         https://bugs.webkit.org/show_bug.cgi?id=211509
 
index 0b66875..bee3eed 100644 (file)
@@ -143,9 +143,9 @@ AudioComponentInstance AudioMediaStreamTrackRendererCocoa::createAudioUnit(CAAud
     return remoteIOUnit;
 }
 
-// May get called on a background thread.
 void AudioMediaStreamTrackRendererCocoa::pushSamples(const MediaTime& sampleTime, const PlatformAudioData& audioData, const AudioStreamDescription& description, size_t sampleCount)
 {
+    ASSERT(!isMainThread());
     ASSERT(description.platformDescription().type == PlatformDescription::CAAudioStreamBasicType);
     if (!m_remoteIOUnit)
         return;
index b8f883b..960015a 100644 (file)
@@ -1,5 +1,74 @@
 2020-05-08  Youenn Fablet  <youenn@apple.com>
 
+        Handle remote audio capture IPC messages in a background thread
+        https://bugs.webkit.org/show_bug.cgi?id=211583
+
+        Reviewed by Eric Carlson.
+
+        We introduce RemoteCaptureSampleManager which is responsible to get remote capture sample IPC messages and treat them from a background thread.
+        It is currently used to handle audio capture IPC messages.
+        We move UserMediaCaptureManager::Source in its own file and rename it to RemoteRealtimeMediaSource.
+
+        Covered by existing tests.
+
+        * DerivedSources-input.xcfilelist:
+        * DerivedSources-output.xcfilelist:
+        * DerivedSources.make:
+        * PlatformMac.cmake:
+        * SourcesCocoa.txt:
+        * UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
+        * WebKit.xcodeproj/project.pbxproj:
+        * WebProcess/cocoa/RemoteCaptureSampleManager.cpp: Added.
+        (WebKit::RemoteCaptureSampleManager::RemoteCaptureSampleManager):
+        (WebKit::RemoteCaptureSampleManager::~RemoteCaptureSampleManager):
+        (WebKit::RemoteCaptureSampleManager::setConnection):
+        (WebKit::RemoteCaptureSampleManager::addSource):
+        (WebKit::RemoteCaptureSampleManager::removeSource):
+        (WebKit::RemoteCaptureSampleManager::dispatchToThread):
+        (WebKit::RemoteCaptureSampleManager::storageChanged):
+        (WebKit::RemoteCaptureSampleManager::audioSamplesAvailable):
+        * WebProcess/cocoa/RemoteCaptureSampleManager.h: Added.
+        * WebProcess/cocoa/RemoteCaptureSampleManager.messages.in: Added.
+        * WebProcess/cocoa/RemoteRealtimeMediaSource.cpp: Added.
+        (WebKit::RemoteRealtimeMediaSource::RemoteRealtimeMediaSource):
+        (WebKit::RemoteRealtimeMediaSource::~RemoteRealtimeMediaSource):
+        (WebKit::RemoteRealtimeMediaSource::whenReady):
+        (WebKit::RemoteRealtimeMediaSource::didFail):
+        (WebKit::RemoteRealtimeMediaSource::setAsReady):
+        (WebKit::RemoteRealtimeMediaSource::clone):
+        (WebKit::RemoteRealtimeMediaSource::setCapabilities):
+        (WebKit::RemoteRealtimeMediaSource::setSettings):
+        (WebKit::RemoteRealtimeMediaSource::remoteAudioSamplesAvailable):
+        (WebKit::RemoteRealtimeMediaSource::remoteVideoSampleAvailable):
+        (WebKit::RemoteRealtimeMediaSource::connection):
+        (WebKit::RemoteRealtimeMediaSource::startProducingData):
+        (WebKit::RemoteRealtimeMediaSource::stopProducingData):
+        (WebKit::RemoteRealtimeMediaSource::setShouldApplyRotation):
+        (WebKit::RemoteRealtimeMediaSource::capabilities):
+        (WebKit::RemoteRealtimeMediaSource::applyConstraints):
+        (WebKit::RemoteRealtimeMediaSource::applyConstraintsSucceeded):
+        (WebKit::RemoteRealtimeMediaSource::applyConstraintsFailed):
+        (WebKit::RemoteRealtimeMediaSource::hasEnded):
+        (WebKit::RemoteRealtimeMediaSource::stopBeingObserved):
+        (WebKit::RemoteRealtimeMediaSource::requestToEnd):
+        * WebProcess/cocoa/RemoteRealtimeMediaSource.h: Added.
+        (WebKit::RemoteRealtimeMediaSource::identifier const):
+        (WebKit::RemoteRealtimeMediaSource::settings const):
+        (WebKit::RemoteRealtimeMediaSource::setShouldCaptureInGPUProcess):
+        (WebKit::RemoteRealtimeMediaSource::shouldCaptureInGPUProcess const):
+        * WebProcess/cocoa/UserMediaCaptureManager.cpp:
+        (WebKit::UserMediaCaptureManager::createCaptureSource):
+        (WebKit::UserMediaCaptureManager::cloneSource):
+        (WebKit::UserMediaCaptureManager::cloneVideoSource):
+        (WebKit::UserMediaCaptureManager::AudioFactory::createAudioCaptureSource):
+        (WebKit::UserMediaCaptureManager::AudioFactory::setShouldCaptureInGPUProcess):
+        (WebKit::UserMediaCaptureManager::VideoFactory::createVideoCaptureSource):
+        (WebKit::UserMediaCaptureManager::DisplayFactory::createDisplayCaptureSource):
+        * WebProcess/cocoa/UserMediaCaptureManager.h:
+        * WebProcess/cocoa/UserMediaCaptureManager.messages.in:
+
+2020-05-08  Youenn Fablet  <youenn@apple.com>
+
         Video capture does not get unmuted in case of tab switch on iOS
         https://bugs.webkit.org/show_bug.cgi?id=211509
 
index ad99d33..6a29111 100644 (file)
@@ -163,7 +163,9 @@ $(PROJECT_DIR)/WebProcess/WebPage/VisitedLinkTableController.messages.in
 $(PROJECT_DIR)/WebProcess/WebPage/WebPage.messages.in
 $(PROJECT_DIR)/WebProcess/WebProcess.messages.in
 $(PROJECT_DIR)/WebProcess/WebStorage/StorageAreaMap.messages.in
+$(PROJECT_DIR)/WebProcess/cocoa/AudioCaptureSampleManager.messages.in
 $(PROJECT_DIR)/WebProcess/cocoa/PlaybackSessionManager.messages.in
+$(PROJECT_DIR)/WebProcess/cocoa/RemoteCaptureSampleManager.messages.in
 $(PROJECT_DIR)/WebProcess/cocoa/UserMediaCaptureManager.messages.in
 $(PROJECT_DIR)/WebProcess/cocoa/VideoFullscreenManager.messages.in
 $(PROJECT_DIR)/WebProcess/com.apple.WebProcess.sb.in
index 7d70da4..bd8933e 100644 (file)
@@ -1,4 +1,6 @@
 # This file is generated by the generate-xcfilelists script.
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/AudioCaptureSampleManagerMessageReceiver.cpp
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/AudioCaptureSampleManagerMessagesReplies.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/AudioSessionRoutingArbitratorProxyMessageReceiver.cpp
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/AudioSessionRoutingArbitratorProxyMessages.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/AudioSessionRoutingArbitratorProxyMessagesReplies.h
@@ -169,6 +171,8 @@ $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteCDMInstanceSessionProxyMessag
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteCDMProxyMessageReceiver.cpp
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteCDMProxyMessages.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteCDMProxyMessagesReplies.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteCaptureSampleManagerMessageReceiver.cpp
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteCaptureSampleManagerMessagesReplies.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteLayerTreeDrawingAreaProxyMessageReceiver.cpp
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteLayerTreeDrawingAreaProxyMessages.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/RemoteLayerTreeDrawingAreaProxyMessagesReplies.h
index c9d6363..a12b786 100644 (file)
@@ -199,6 +199,7 @@ MESSAGE_RECEIVERS = \
        WebProcess/Storage/WebSWClientConnection \
        WebProcess/WebProcess \
        WebProcess/cocoa/PlaybackSessionManager \
+       WebProcess/cocoa/RemoteCaptureSampleManager \
        WebProcess/cocoa/UserMediaCaptureManager \
        WebProcess/cocoa/VideoFullscreenManager \
        WebProcess/Geolocation/WebGeolocationManager \
index cc66c92..48e1601 100644 (file)
@@ -208,6 +208,7 @@ list(APPEND WebKit_MESSAGES_IN_FILES
     WebProcess/ApplePay/WebPaymentCoordinator.messages.in
 
     WebProcess/cocoa/PlaybackSessionManager.messages.in
+    WebProcess/cocoa/RemoteCaptureSampleManager.messages.in
     WebProcess/cocoa/UserMediaCaptureManager.messages.in
     WebProcess/cocoa/VideoFullscreenManager.messages.in
 
index 9567ae9..168f1db 100644 (file)
@@ -555,6 +555,8 @@ WebProcess/ApplePay/WebPaymentCoordinator.cpp
 WebProcess/ApplePay/cocoa/WebPaymentCoordinatorCocoa.mm
 
 WebProcess/cocoa/PlaybackSessionManager.mm
+WebProcess/cocoa/RemoteCaptureSampleManager.cpp
+WebProcess/cocoa/RemoteRealtimeMediaSource.cpp
 WebProcess/cocoa/UserMediaCaptureManager.cpp
 WebProcess/cocoa/VideoFullscreenManager.mm
 WebProcess/cocoa/WebProcessCocoa.mm
index b2343f4..e893cd2 100644 (file)
@@ -29,6 +29,7 @@
 #if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
 
 #include "Connection.h"
+#include "RemoteCaptureSampleManagerMessages.h"
 #include "SharedRingBufferStorage.h"
 #include "UserMediaCaptureManagerMessages.h"
 #include "UserMediaCaptureManagerProxyMessages.h"
@@ -138,7 +139,7 @@ private:
         uint64_t startFrame;
         uint64_t endFrame;
         m_ringBuffer.getCurrentFrameBounds(startFrame, endFrame);
-        m_connection->send(Messages::UserMediaCaptureManager::AudioSamplesAvailable(m_id, time, numberOfFrames, startFrame, endFrame), 0);
+        m_connection->send(Messages::RemoteCaptureSampleManager::AudioSamplesAvailable(m_id, time, numberOfFrames, startFrame, endFrame), 0);
     }
 
     void videoSampleAvailable(MediaSample& sample) final
@@ -181,7 +182,7 @@ private:
         SharedMemory::Handle handle;
         if (storage)
             storage->createHandle(handle, SharedMemory::Protection::ReadOnly);
-        m_connection->send(Messages::UserMediaCaptureManager::StorageChanged(m_id, handle, m_description, m_numberOfFrames), 0);
+        m_connection->send(Messages::RemoteCaptureSampleManager::AudioStorageChanged(m_id, handle, m_description, m_numberOfFrames), 0);
     }
 
     bool preventSourceFromStopping()
index 717e8df..970c6df 100644 (file)
                0FCB4E6C18BBF26A000FCFC9 /* WKContentViewInteraction.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FCB4E6A18BBF26A000FCFC9 /* WKContentViewInteraction.h */; };
                0FDCD7F71D47E92A009F08BC /* LogInitialization.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FDCD7F61D47E92A009F08BC /* LogInitialization.h */; };
                0FF24A2D1879E4BC003ABF0C /* RemoteLayerTreeDrawingAreaProxyMessageReceiver.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FF24A2B1879E4BC003ABF0C /* RemoteLayerTreeDrawingAreaProxyMessageReceiver.cpp */; };
+               0FF24A2D1879E4BC003ABF0D /* RemoteCaptureSampleManagerMessageReceiver.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FF24A2B1879E4BC003ABF0D /* RemoteCaptureSampleManagerMessageReceiver.cpp */; };
                0FF24A2E1879E4BC003ABF0C /* RemoteLayerTreeDrawingAreaProxyMessages.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FF24A2C1879E4BC003ABF0C /* RemoteLayerTreeDrawingAreaProxyMessages.h */; };
                0FF264A01A1FF9CC001FE759 /* RemoteLayerTreeScrollingPerformanceData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F707C791A1FEEA300DA7A45 /* RemoteLayerTreeScrollingPerformanceData.h */; };
                0FFED98E23A3200A00EEF459 /* WKWebViewIOS.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FFED98C23A3200400EEF459 /* WKWebViewIOS.h */; };
                0FCB4E6B18BBF26A000FCFC9 /* WKContentViewInteraction.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = WKContentViewInteraction.mm; path = ios/WKContentViewInteraction.mm; sourceTree = "<group>"; };
                0FDCD7F61D47E92A009F08BC /* LogInitialization.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LogInitialization.h; sourceTree = "<group>"; };
                0FF24A2B1879E4BC003ABF0C /* RemoteLayerTreeDrawingAreaProxyMessageReceiver.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = RemoteLayerTreeDrawingAreaProxyMessageReceiver.cpp; path = DerivedSources/WebKit2/RemoteLayerTreeDrawingAreaProxyMessageReceiver.cpp; sourceTree = BUILT_PRODUCTS_DIR; };
+               0FF24A2B1879E4BC003ABF0D /* RemoteCaptureSampleManagerMessageReceiver.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = RemoteCaptureSampleManagerMessageReceiver.cpp; path = DerivedSources/WebKit2/RemoteCaptureSampleManagerMessageReceiver.cpp; sourceTree = BUILT_PRODUCTS_DIR; };
                0FF24A2C1879E4BC003ABF0C /* RemoteLayerTreeDrawingAreaProxyMessages.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = RemoteLayerTreeDrawingAreaProxyMessages.h; path = DerivedSources/WebKit2/RemoteLayerTreeDrawingAreaProxyMessages.h; sourceTree = BUILT_PRODUCTS_DIR; };
                0FF24A2F1879E4FE003ABF0C /* RemoteLayerTreeDrawingAreaProxy.messages.in */ = {isa = PBXFileReference; lastKnownFileType = text; path = RemoteLayerTreeDrawingAreaProxy.messages.in; sourceTree = "<group>"; };
                0FFED98C23A3200400EEF459 /* WKWebViewIOS.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = WKWebViewIOS.h; sourceTree = "<group>"; };
                4157853021276B6F00DD3800 /* copy-webcontent-resources-to-private-headers.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = "copy-webcontent-resources-to-private-headers.sh"; sourceTree = "<group>"; };
                4157E4AF20E2EC9800A6C0D7 /* com.google.o1dbrowserplugin.sb */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = com.google.o1dbrowserplugin.sb; sourceTree = "<group>"; };
                4158649A23BE092400A0A61E /* RTCEncoderIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RTCEncoderIdentifier.h; sourceTree = "<group>"; };
+               41684C4D24643BFA001CE8ED /* RemoteCaptureSampleManager.messages.in */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = RemoteCaptureSampleManager.messages.in; sourceTree = "<group>"; };
+               41684C4E24643C81001CE8ED /* RemoteCaptureSampleManager.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteCaptureSampleManager.h; sourceTree = "<group>"; };
+               41684C4F24643D3F001CE8ED /* RemoteCaptureSampleManager.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteCaptureSampleManager.cpp; sourceTree = "<group>"; };
+               41684C50246450FD001CE8ED /* RemoteRealtimeMediaSource.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteRealtimeMediaSource.cpp; sourceTree = "<group>"; };
+               41684C51246450FD001CE8ED /* RemoteRealtimeMediaSource.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteRealtimeMediaSource.h; sourceTree = "<group>"; };
                416CD74523FB178B00661D99 /* MDNSRegisterIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = MDNSRegisterIdentifier.h; path = Network/webrtc/MDNSRegisterIdentifier.h; sourceTree = "<group>"; };
                416F8086245B397400B68F02 /* LibWebRTCNetwork.messages.in */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; name = LibWebRTCNetwork.messages.in; path = Network/webrtc/LibWebRTCNetwork.messages.in; sourceTree = "<group>"; };
                416F8089245C7FF500B68F02 /* LibWebRTCNetwork.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = LibWebRTCNetwork.cpp; path = Network/webrtc/LibWebRTCNetwork.cpp; sourceTree = "<group>"; };
                7C6E70F818B2D47E00F24E2E /* cocoa */ = {
                        isa = PBXGroup;
                        children = (
+                               41684C4F24643D3F001CE8ED /* RemoteCaptureSampleManager.cpp */,
+                               41684C4E24643C81001CE8ED /* RemoteCaptureSampleManager.h */,
+                               41684C4D24643BFA001CE8ED /* RemoteCaptureSampleManager.messages.in */,
                                CD4570CB2440FB2A00A3DCEB /* AudioSessionRoutingArbitrator.cpp */,
                                CD4570CA2440FB2A00A3DCEB /* AudioSessionRoutingArbitrator.h */,
                                CDA29A191CBDBF4100901CCF /* PlaybackSessionManager.h */,
                                CDA29A1C1CBDBF5B00901CCF /* PlaybackSessionManager.messages.in */,
                                CDA29A181CBDBF4100901CCF /* PlaybackSessionManager.mm */,
+                               41684C50246450FD001CE8ED /* RemoteRealtimeMediaSource.cpp */,
+                               41684C51246450FD001CE8ED /* RemoteRealtimeMediaSource.h */,
                                CD491B051E70D05F00009066 /* UserMediaCaptureManager.cpp */,
                                CD491B061E70D05F00009066 /* UserMediaCaptureManager.h */,
                                CD491B0A1E732D1200009066 /* UserMediaCaptureManager.messages.in */,
                        isa = PBXGroup;
                        children = (
                                2D7DEBE221269E4B00B9F73C /* unified-sources */,
+                               0FF24A2B1879E4BC003ABF0D /* RemoteCaptureSampleManagerMessageReceiver.cpp */,
                                CD4570D02441014A00A3DCEB /* AudioSessionRoutingArbitratorProxyMessageReceiver.cpp */,
                                CD4570D12441014B00A3DCEB /* AudioSessionRoutingArbitratorProxyMessages.h */,
                                CD4570D22441014B00A3DCEB /* AudioSessionRoutingArbitratorProxyMessagesReplies.h */,
                                2D92A77B212B6A7100F493FD /* ArgumentCoders.cpp in Sources */,
                                2DEB1D2E2127473600933906 /* ArgumentCodersCF.cpp in Sources */,
                                2D92A77C212B6A7100F493FD /* Attachment.cpp in Sources */,
+                               0FF24A2D1879E4BC003ABF0D /* RemoteCaptureSampleManagerMessageReceiver.cpp in Sources */,
                                CD4570D424411D0F00A3DCEB /* AudioSessionRoutingArbitrator.cpp in Sources */,
                                CD7CF3612444CCA6003BC4D5 /* AudioSessionRoutingArbitratorProxyCocoa.mm in Sources */,
                                CD4570D3244113B500A3DCEB /* AudioSessionRoutingArbitratorProxyMessageReceiver.cpp in Sources */,
diff --git a/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.cpp b/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.cpp
new file mode 100644 (file)
index 0000000..cbc3dd5
--- /dev/null
@@ -0,0 +1,156 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "RemoteCaptureSampleManager.h"
+
+#include "RemoteCaptureSampleManagerMessages.h"
+#include "SharedRingBufferStorage.h"
+#include <WebCore/WebAudioBufferList.h>
+
+#if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
+
+namespace WebKit {
+using namespace PAL;
+using namespace WebCore;
+
+RemoteCaptureSampleManager::RemoteCaptureSampleManager()
+    : m_queue(WorkQueue::create("RemoteCaptureSampleManager", WorkQueue::Type::Serial, WorkQueue::QOS::UserInteractive))
+{
+}
+
+RemoteCaptureSampleManager::~RemoteCaptureSampleManager()
+{
+    setConnection(nullptr);
+}
+
+void RemoteCaptureSampleManager::setConnection(IPC::Connection* connection)
+{
+    if (m_connection == connection)
+        return;
+
+    if (m_connection)
+        m_connection->removeThreadMessageReceiver(Messages::RemoteCaptureSampleManager::messageReceiverName());
+
+    m_connection = WTFMove(connection);
+
+    if (m_connection)
+        m_connection->addThreadMessageReceiver(Messages::RemoteCaptureSampleManager::messageReceiverName(), this);
+}
+
+void RemoteCaptureSampleManager::addSource(Ref<RemoteRealtimeMediaSource>&& source)
+{
+    ASSERT(WTF::isMainRunLoop());
+    setConnection(source->connection());
+
+    dispatchToThread([this, source = WTFMove(source)]() mutable {
+        auto identifier = source->identifier();
+
+        ASSERT(!m_sources.contains(identifier));
+        m_sources.add(identifier, makeUnique<RemoteAudio>(WTFMove(source)));
+    });
+}
+
+void RemoteCaptureSampleManager::removeSource(WebCore::RealtimeMediaSourceIdentifier identifier)
+{
+    ASSERT(WTF::isMainRunLoop());
+    dispatchToThread([this, identifier] {
+        ASSERT(m_sources.contains(identifier));
+        m_sources.remove(identifier);
+    });
+}
+
+void RemoteCaptureSampleManager::dispatchToThread(Function<void()>&& callback)
+{
+    m_queue->dispatch(WTFMove(callback));
+}
+
+void RemoteCaptureSampleManager::audioStorageChanged(WebCore::RealtimeMediaSourceIdentifier identifier, const SharedMemory::Handle& handle, const WebCore::CAAudioStreamDescription& description, uint64_t numberOfFrames)
+{
+    ASSERT(!WTF::isMainRunLoop());
+
+    auto iterator = m_sources.find(identifier);
+    if (iterator == m_sources.end()) {
+        RELEASE_LOG_ERROR(WebRTC, "Unable to find source %llu for storageChanged", identifier.toUInt64());
+        return;
+    }
+    iterator->value->setStorage(handle, description, numberOfFrames);
+}
+
+void RemoteCaptureSampleManager::audioSamplesAvailable(WebCore::RealtimeMediaSourceIdentifier identifier, MediaTime time, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame)
+{
+    ASSERT(!WTF::isMainRunLoop());
+
+    auto iterator = m_sources.find(identifier);
+    if (iterator == m_sources.end()) {
+        RELEASE_LOG_ERROR(WebRTC, "Unable to find source %llu for audioSamplesAvailable", identifier.toUInt64());
+        return;
+    }
+    iterator->value->audioSamplesAvailable(time, numberOfFrames, startFrame, endFrame);
+}
+
+RemoteCaptureSampleManager::RemoteAudio::RemoteAudio(Ref<RemoteRealtimeMediaSource>&& source)
+    : m_source(WTFMove(source))
+    , m_ringBuffer(makeUnique<CARingBuffer>(makeUniqueRef<SharedRingBufferStorage>(nullptr)))
+{
+}
+
+void RemoteCaptureSampleManager::RemoteAudio::setStorage(const SharedMemory::Handle& handle, const WebCore::CAAudioStreamDescription& description, uint64_t numberOfFrames)
+{
+    m_description = description;
+
+    RefPtr<SharedMemory> memory;
+    if (!handle.isNull()) {
+        memory = SharedMemory::map(handle, SharedMemory::Protection::ReadOnly);
+        RELEASE_LOG_ERROR_IF(!memory, WebRTC, "Unable to create shared memory for audio source %llu", m_source->identifier().toUInt64());
+    }
+
+    auto& storage = static_cast<SharedRingBufferStorage&>(m_ringBuffer->storage());
+    if (!memory) {
+        m_ringBuffer->deallocate();
+        storage.setReadOnly(false);
+        storage.setStorage(nullptr);
+        return;
+    }
+
+    storage.setStorage(memory.releaseNonNull());
+    storage.setReadOnly(true);
+    m_ringBuffer->allocate(description, numberOfFrames);
+}
+
+void RemoteCaptureSampleManager::RemoteAudio::audioSamplesAvailable(MediaTime time, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame)
+{
+    // FIXME: We should allocate this buffer once and resize it as needed.
+    WebAudioBufferList audioData(m_description, numberOfFrames);
+
+    m_ringBuffer->setCurrentFrameBounds(startFrame, endFrame);
+    m_ringBuffer->fetch(audioData.list(), numberOfFrames, time.timeValue());
+
+    m_source->remoteAudioSamplesAvailable(time, audioData, m_description, numberOfFrames);
+}
+
+}
+
+#endif
diff --git a/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.h b/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.h
new file mode 100644 (file)
index 0000000..65ba564
--- /dev/null
@@ -0,0 +1,85 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
+
+#include "Connection.h"
+#include "MessageReceiver.h"
+#include "RemoteRealtimeMediaSource.h"
+#include "SharedMemory.h"
+#include <WebCore/CAAudioStreamDescription.h>
+#include <WebCore/CARingBuffer.h>
+#include <wtf/HashMap.h>
+#include <wtf/WorkQueue.h>
+
+namespace WebKit {
+
+class RemoteCaptureSampleManager : public IPC::Connection::ThreadMessageReceiver {
+    WTF_MAKE_FAST_ALLOCATED;
+public:
+    RemoteCaptureSampleManager();
+    ~RemoteCaptureSampleManager();
+
+    void addSource(Ref<RemoteRealtimeMediaSource>&&);
+    void removeSource(WebCore::RealtimeMediaSourceIdentifier);
+
+    void didReceiveMessage(IPC::Connection&, IPC::Decoder&);
+
+private:
+    // IPC::Connection::ThreadMessageReceiver
+    void dispatchToThread(Function<void()>&&) final;
+
+    // Messages
+    void audioStorageChanged(WebCore::RealtimeMediaSourceIdentifier, const SharedMemory::Handle&, const WebCore::CAAudioStreamDescription&, uint64_t numberOfFrames);
+    void audioSamplesAvailable(WebCore::RealtimeMediaSourceIdentifier, MediaTime, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame);
+
+    void setConnection(IPC::Connection*);
+
+    class RemoteAudio {
+        WTF_MAKE_FAST_ALLOCATED;
+    public:
+        explicit RemoteAudio(Ref<RemoteRealtimeMediaSource>&&);
+
+        void setStorage(const SharedMemory::Handle&, const WebCore::CAAudioStreamDescription&, uint64_t numberOfFrames);
+        void audioSamplesAvailable(MediaTime, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame);
+
+    private:
+        Ref<RemoteRealtimeMediaSource> m_source;
+        WebCore::CAAudioStreamDescription m_description;
+        std::unique_ptr<WebCore::CARingBuffer> m_ringBuffer;
+    };
+
+    Ref<WorkQueue> m_queue;
+    RefPtr<IPC::Connection> m_connection;
+
+    // background thread member
+    HashMap<WebCore::RealtimeMediaSourceIdentifier, std::unique_ptr<RemoteAudio>> m_sources;
+};
+
+} // namespace WebKit
+
+#endif
diff --git a/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.messages.in b/Source/WebKit/WebProcess/cocoa/RemoteCaptureSampleManager.messages.in
new file mode 100644 (file)
index 0000000..f5a98d4
--- /dev/null
@@ -0,0 +1,31 @@
+# Copyright (C) 2020 Apple Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+# THE POSSIBILITY OF SUCH DAMAGE.
+
+#if ENABLE(MEDIA_STREAM)
+
+messages -> RemoteCaptureSampleManager NotRefCounted {
+    AudioStorageChanged(WebCore::RealtimeMediaSourceIdentifier id, WebKit::SharedMemory::Handle storageHandle, WebCore::CAAudioStreamDescription description, uint64_t numberOfFrames)
+    AudioSamplesAvailable(WebCore::RealtimeMediaSourceIdentifier id, MediaTime time, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame)
+}
+
+#endif
diff --git a/Source/WebKit/WebProcess/cocoa/RemoteRealtimeMediaSource.cpp b/Source/WebKit/WebProcess/cocoa/RemoteRealtimeMediaSource.cpp
new file mode 100644 (file)
index 0000000..a7f0400
--- /dev/null
@@ -0,0 +1,305 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "RemoteRealtimeMediaSource.h"
+
+#if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
+
+#include "GPUProcessConnection.h"
+#include "SharedRingBufferStorage.h"
+#include "UserMediaCaptureManager.h"
+#include "UserMediaCaptureManagerMessages.h"
+#include "UserMediaCaptureManagerProxyMessages.h"
+#include "WebProcess.h"
+#include <WebCore/ImageTransferSessionVT.h>
+#include <WebCore/MediaConstraints.h>
+#include <WebCore/RealtimeMediaSourceCenter.h>
+#include <WebCore/RemoteVideoSample.h>
+#include <WebCore/WebAudioBufferList.h>
+
+namespace WebKit {
+using namespace PAL;
+using namespace WebCore;
+
+Ref<RealtimeMediaSource> RemoteRealtimeMediaSource::create(const WebCore::CaptureDevice& device, const WebCore::MediaConstraints& constraints, String&& name, String&& hashSalt, UserMediaCaptureManager& manager, bool shouldCaptureInGPUProcess)
+{
+    auto source = adoptRef(*new RemoteRealtimeMediaSource(RealtimeMediaSourceIdentifier::generate(), device.type(), WTFMove(name), WTFMove(hashSalt), manager, shouldCaptureInGPUProcess));
+    manager.addSource(source.copyRef());
+    source->connection()->sendWithAsyncReply(Messages::UserMediaCaptureManagerProxy::CreateMediaSourceForCaptureDeviceWithConstraints(source->identifier(), device, source->deviceIDHashSalt(), constraints), [source = source.copyRef()](bool succeeded, auto&& errorMessage, auto&& settings, auto&& capabilities) {
+        if (!succeeded) {
+            source->didFail(WTFMove(errorMessage));
+            return;
+        }
+        source->setName(String { settings.label().string() });
+        source->setSettings(WTFMove(settings));
+        source->setCapabilities(WTFMove(capabilities));
+        source->setAsReady();
+    });
+    return source;
+}
+
+static inline RealtimeMediaSource::Type sourceTypeFromDeviceType(CaptureDevice::DeviceType deviceType)
+{
+    switch (deviceType) {
+    case CaptureDevice::DeviceType::Microphone:
+        return RealtimeMediaSource::Type::Audio;
+    case CaptureDevice::DeviceType::Camera:
+    case CaptureDevice::DeviceType::Screen:
+    case CaptureDevice::DeviceType::Window:
+        return RealtimeMediaSource::Type::Video;
+    case CaptureDevice::DeviceType::Unknown:
+        ASSERT_NOT_REACHED();
+    }
+    return RealtimeMediaSource::Type::None;
+}
+
+RemoteRealtimeMediaSource::RemoteRealtimeMediaSource(RealtimeMediaSourceIdentifier identifier, CaptureDevice::DeviceType deviceType, String&& name, String&& hashSalt, UserMediaCaptureManager& manager, bool shouldCaptureInGPUProcess)
+    : RealtimeMediaSource(sourceTypeFromDeviceType(deviceType), WTFMove(name), String::number(identifier.toUInt64()), WTFMove(hashSalt))
+    , m_identifier(identifier)
+    , m_manager(manager)
+    , m_deviceType(deviceType)
+    , m_shouldCaptureInGPUProcess(shouldCaptureInGPUProcess)
+{
+    switch (m_deviceType) {
+    case CaptureDevice::DeviceType::Microphone:
+#if PLATFORM(IOS_FAMILY)
+        RealtimeMediaSourceCenter::singleton().audioCaptureFactory().setActiveSource(*this);
+#endif
+        break;
+    case CaptureDevice::DeviceType::Camera:
+#if PLATFORM(IOS_FAMILY)
+        RealtimeMediaSourceCenter::singleton().videoCaptureFactory().setActiveSource(*this);
+#endif
+        break;
+    case CaptureDevice::DeviceType::Screen:
+    case CaptureDevice::DeviceType::Window:
+        break;
+    case CaptureDevice::DeviceType::Unknown:
+        ASSERT_NOT_REACHED();
+    }
+}
+
+RemoteRealtimeMediaSource::~RemoteRealtimeMediaSource()
+{
+    switch (m_deviceType) {
+    case CaptureDevice::DeviceType::Microphone:
+#if PLATFORM(IOS_FAMILY)
+        RealtimeMediaSourceCenter::singleton().audioCaptureFactory().unsetActiveSource(*this);
+#endif
+        break;
+    case CaptureDevice::DeviceType::Camera:
+#if PLATFORM(IOS_FAMILY)
+        RealtimeMediaSourceCenter::singleton().videoCaptureFactory().unsetActiveSource(*this);
+#endif
+        break;
+    case CaptureDevice::DeviceType::Screen:
+    case CaptureDevice::DeviceType::Window:
+        break;
+    case CaptureDevice::DeviceType::Unknown:
+        ASSERT_NOT_REACHED();
+    }
+}
+
+void RemoteRealtimeMediaSource::whenReady(CompletionHandler<void(String)>&& callback)
+{
+    if (m_isReady)
+        return callback(WTFMove(m_errorMessage));
+    m_callback = WTFMove(callback);
+}
+
+void RemoteRealtimeMediaSource::didFail(String&& errorMessage)
+{
+    m_isReady = true;
+    m_errorMessage = WTFMove(errorMessage);
+    if (m_callback)
+        m_callback(m_errorMessage);
+}
+
+void RemoteRealtimeMediaSource::setAsReady()
+{
+    m_isReady = true;
+    if (m_callback)
+        m_callback({ });
+}
+
+Ref<RealtimeMediaSource> RemoteRealtimeMediaSource::clone()
+{
+    switch (type()) {
+    case RealtimeMediaSource::Type::Video:
+        return cloneVideoSource();
+    case RealtimeMediaSource::Type::Audio:
+        break;
+    case RealtimeMediaSource::Type::None:
+        ASSERT_NOT_REACHED();
+    }
+    return *this;
+}
+
+Ref<RealtimeMediaSource> RemoteRealtimeMediaSource::cloneVideoSource()
+{
+    auto identifier = RealtimeMediaSourceIdentifier::generate();
+    if (!connection()->send(Messages::UserMediaCaptureManagerProxy::Clone { m_identifier, identifier }, 0))
+        return *this;
+
+    auto cloneSource = adoptRef(*new RemoteRealtimeMediaSource(identifier, deviceType(), String { m_settings.label().string() }, deviceIDHashSalt(), m_manager, m_shouldCaptureInGPUProcess));
+    cloneSource->setSettings(RealtimeMediaSourceSettings { m_settings });
+    m_manager.addSource(cloneSource.copyRef());
+    return cloneSource;
+}
+
+void RemoteRealtimeMediaSource::setCapabilities(RealtimeMediaSourceCapabilities&& capabilities)
+{
+    m_capabilities = WTFMove(capabilities);
+}
+
+void RemoteRealtimeMediaSource::setSettings(RealtimeMediaSourceSettings&& settings)
+{
+    auto changed = m_settings.difference(settings);
+    m_settings = WTFMove(settings);
+    notifySettingsDidChangeObservers(changed);
+}
+
+void RemoteRealtimeMediaSource::remoteAudioSamplesAvailable(const WTF::MediaTime& time, const PlatformAudioData& data, const AudioStreamDescription& description, size_t size)
+{
+    ASSERT(!isMainThread());
+    audioSamplesAvailable(time, data, description, size);
+}
+
+void RemoteRealtimeMediaSource::remoteVideoSampleAvailable(RemoteVideoSample&& remoteSample)
+{
+    ASSERT(type() == Type::Video);
+
+    setIntrinsicSize(remoteSample.size());
+
+    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != remoteSample.videoFormat())
+        m_imageTransferSession = ImageTransferSessionVT::create(remoteSample.videoFormat());
+
+    if (!m_imageTransferSession) {
+        ASSERT_NOT_REACHED();
+        return;
+    }
+
+    auto sampleRef = m_imageTransferSession->createMediaSample(remoteSample);
+    if (!sampleRef) {
+        ASSERT_NOT_REACHED();
+        return;
+    }
+
+    RealtimeMediaSource::videoSampleAvailable(*sampleRef);
+}
+
+IPC::Connection* RemoteRealtimeMediaSource::connection()
+{
+    ASSERT(isMainThread());
+#if ENABLE(GPU_PROCESS)
+    if (m_shouldCaptureInGPUProcess)
+        return &WebProcess::singleton().ensureGPUProcessConnection().connection();
+#endif
+    return WebProcess::singleton().parentProcessConnection();
+}
+
+void RemoteRealtimeMediaSource::startProducingData()
+{
+    connection()->send(Messages::UserMediaCaptureManagerProxy::StartProducingData { m_identifier }, 0);
+}
+
+void RemoteRealtimeMediaSource::stopProducingData()
+{
+    connection()->send(Messages::UserMediaCaptureManagerProxy::StopProducingData { m_identifier }, 0);
+}
+
+bool RemoteRealtimeMediaSource::setShouldApplyRotation(bool shouldApplyRotation)
+{
+    connection()->send(Messages::UserMediaCaptureManagerProxy::SetShouldApplyRotation { m_identifier, shouldApplyRotation }, 0);
+    return true;
+}
+
+const WebCore::RealtimeMediaSourceCapabilities& RemoteRealtimeMediaSource::capabilities()
+{
+    return m_capabilities;
+}
+
+void RemoteRealtimeMediaSource::applyConstraints(const WebCore::MediaConstraints& constraints, ApplyConstraintsHandler&& completionHandler)
+{
+    m_pendingApplyConstraintsCallbacks.append(WTFMove(completionHandler));
+    // FIXME: Use sendAsyncWithReply.
+    connection()->send(Messages::UserMediaCaptureManagerProxy::ApplyConstraints { m_identifier, constraints }, 0);
+}
+
+void RemoteRealtimeMediaSource::applyConstraintsSucceeded(const WebCore::RealtimeMediaSourceSettings& settings)
+{
+    setSettings(WebCore::RealtimeMediaSourceSettings(settings));
+
+    auto callback = m_pendingApplyConstraintsCallbacks.takeFirst();
+    callback({ });
+}
+
+void RemoteRealtimeMediaSource::applyConstraintsFailed(String&& failedConstraint, String&& errorMessage)
+{
+    auto callback = m_pendingApplyConstraintsCallbacks.takeFirst();
+    callback(ApplyConstraintsError { WTFMove(failedConstraint), WTFMove(errorMessage) });
+}
+
+void RemoteRealtimeMediaSource::hasEnded()
+{
+    connection()->send(Messages::UserMediaCaptureManagerProxy::End { m_identifier }, 0);
+    m_manager.removeSource(m_identifier);
+}
+
+void RemoteRealtimeMediaSource::captureStopped()
+{
+    stop();
+    hasEnded();
+}
+
+void RemoteRealtimeMediaSource::captureFailed()
+{
+    RealtimeMediaSource::captureFailed();
+    hasEnded();
+}
+
+void RemoteRealtimeMediaSource::stopBeingObserved()
+{
+    connection()->send(Messages::UserMediaCaptureManagerProxy::RequestToEnd { m_identifier }, 0);
+}
+
+void RemoteRealtimeMediaSource::requestToEnd(Observer& observer)
+{
+    switch (type()) {
+    case Type::Audio:
+        RealtimeMediaSource::requestToEnd(observer);
+        break;
+    case Type::Video:
+        stopBeingObserved();
+        break;
+    case Type::None:
+        ASSERT_NOT_REACHED();
+    }
+}
+
+}
+
+#endif
diff --git a/Source/WebKit/WebProcess/cocoa/RemoteRealtimeMediaSource.h b/Source/WebKit/WebProcess/cocoa/RemoteRealtimeMediaSource.h
new file mode 100644 (file)
index 0000000..375408c
--- /dev/null
@@ -0,0 +1,106 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
+
+#include <WebCore/CaptureDevice.h>
+#include <WebCore/RealtimeMediaSource.h>
+#include <WebCore/RealtimeMediaSourceIdentifier.h>
+
+namespace WebCore {
+class CAAudioStreamDescription;
+class ImageTransferSessionVT;
+struct MediaConstraints;
+class RemoteVideoSample;
+}
+
+namespace WebKit {
+
+class UserMediaCaptureManager;
+
+class RemoteRealtimeMediaSource : public WebCore::RealtimeMediaSource {
+public:
+    static Ref<WebCore::RealtimeMediaSource> create(const WebCore::CaptureDevice&, const WebCore::MediaConstraints&, String&& name, String&& hashSalt, UserMediaCaptureManager&, bool shouldCaptureInGPUProcess = false);
+    ~RemoteRealtimeMediaSource();
+
+    WebCore::RealtimeMediaSourceIdentifier identifier() const { return m_identifier; }
+    IPC::Connection* connection();
+
+    void setSettings(WebCore::RealtimeMediaSourceSettings&&);
+
+    void applyConstraintsSucceeded(const WebCore::RealtimeMediaSourceSettings&);
+    void applyConstraintsFailed(String&& failedConstraint, String&& errorMessage);
+
+    void captureStopped();
+    void captureFailed() final;
+
+    void remoteVideoSampleAvailable(WebCore::RemoteVideoSample&&);
+    void remoteAudioSamplesAvailable(const WTF::MediaTime&, const WebCore::PlatformAudioData&, const WebCore::AudioStreamDescription&, size_t);
+
+private:
+    RemoteRealtimeMediaSource(WebCore::RealtimeMediaSourceIdentifier, WebCore::CaptureDevice::DeviceType, String&& name, String&& hashSalt, UserMediaCaptureManager&, bool shouldCaptureInGPUProcess);
+
+    // RealtimeMediaSource
+    void startProducingData() final;
+    void stopProducingData() final;
+    bool isCaptureSource() const final { return true; }
+    void beginConfiguration() final { }
+    void commitConfiguration() final { }
+    bool setShouldApplyRotation(bool /* shouldApplyRotation */) final;
+    void applyConstraints(const WebCore::MediaConstraints&, ApplyConstraintsHandler&&) final;
+    void requestToEnd(Observer&) final;
+    void stopBeingObserved() final;
+    void hasEnded() final;
+    const WebCore::RealtimeMediaSourceSettings& settings() final { return m_settings; }
+    const WebCore::RealtimeMediaSourceCapabilities& capabilities() final;
+    void whenReady(CompletionHandler<void(String)>&&) final;
+    WebCore::CaptureDevice::DeviceType deviceType() const final { return m_deviceType; }
+    Ref<RealtimeMediaSource> clone() final;
+
+    void didFail(String&& errorMessage);
+    void setAsReady();
+    void setCapabilities(WebCore::RealtimeMediaSourceCapabilities&&);
+    Ref<RealtimeMediaSource> cloneVideoSource();
+
+    WebCore::RealtimeMediaSourceIdentifier m_identifier;
+    UserMediaCaptureManager& m_manager;
+    WebCore::RealtimeMediaSourceCapabilities m_capabilities;
+    WebCore::RealtimeMediaSourceSettings m_settings;
+
+    std::unique_ptr<WebCore::ImageTransferSessionVT> m_imageTransferSession;
+    WebCore::CaptureDevice::DeviceType m_deviceType { WebCore::CaptureDevice::DeviceType::Unknown };
+
+    Deque<ApplyConstraintsHandler> m_pendingApplyConstraintsCallbacks;
+    bool m_shouldCaptureInGPUProcess { false };
+    bool m_isReady { false };
+    String m_errorMessage;
+    CompletionHandler<void(String)> m_callback;
+};
+
+} // namespace WebKit
+
+#endif
index 637ee91..083f829 100644 (file)
 
 #include "AudioMediaStreamTrackRenderer.h"
 #include "GPUProcessConnection.h"
-#include "SharedRingBufferStorage.h"
+#include "RemoteRealtimeMediaSource.h"
 #include "UserMediaCaptureManagerMessages.h"
-#include "UserMediaCaptureManagerProxyMessages.h"
 #include "WebCoreArgumentCoders.h"
 #include "WebProcess.h"
-#include "WebProcessCreationParameters.h"
-#include <WebCore/CaptureDevice.h>
 #include <WebCore/DeprecatedGlobalSettings.h>
-#include <WebCore/ImageTransferSessionVT.h>
-#include <WebCore/MediaConstraints.h>
 #include <WebCore/MockRealtimeMediaSourceCenter.h>
 #include <WebCore/RealtimeMediaSourceCenter.h>
-#include <WebCore/RemoteVideoSample.h>
-#include <WebCore/WebAudioBufferList.h>
-#include <WebCore/WebAudioSourceProviderAVFObjC.h>
 #include <wtf/Assertions.h>
 
 namespace WebKit {
 using namespace PAL;
 using namespace WebCore;
 
-class UserMediaCaptureManager::Source : public RealtimeMediaSource {
-public:
-    Source(String&& sourceID, Type type, CaptureDevice::DeviceType deviceType, String&& name, String&& hashSalt, RealtimeMediaSourceIdentifier id, UserMediaCaptureManager& manager)
-        : RealtimeMediaSource(type, WTFMove(name), WTFMove(sourceID), WTFMove(hashSalt))
-        , m_id(id)
-        , m_manager(manager)
-        , m_deviceType(deviceType)
-    {
-        switch (m_deviceType) {
-        case CaptureDevice::DeviceType::Microphone:
-            m_ringBuffer = makeUnique<CARingBuffer>(makeUniqueRef<SharedRingBufferStorage>(nullptr));
-#if PLATFORM(IOS_FAMILY)
-            RealtimeMediaSourceCenter::singleton().audioCaptureFactory().setActiveSource(*this);
-#endif
-            break;
-        case CaptureDevice::DeviceType::Camera:
-#if PLATFORM(IOS_FAMILY)
-            RealtimeMediaSourceCenter::singleton().videoCaptureFactory().setActiveSource(*this);
-#endif
-            break;
-        case CaptureDevice::DeviceType::Screen:
-        case CaptureDevice::DeviceType::Window:
-            break;
-        case CaptureDevice::DeviceType::Unknown:
-            ASSERT_NOT_REACHED();
-        }
-    }
-
-    ~Source()
-    {
-        switch (m_deviceType) {
-        case CaptureDevice::DeviceType::Microphone:
-            storage().invalidate();
-#if PLATFORM(IOS_FAMILY)
-            RealtimeMediaSourceCenter::singleton().audioCaptureFactory().unsetActiveSource(*this);
-#endif
-            break;
-        case CaptureDevice::DeviceType::Camera:
-#if PLATFORM(IOS_FAMILY)
-            RealtimeMediaSourceCenter::singleton().videoCaptureFactory().unsetActiveSource(*this);
-#endif
-            break;
-        case CaptureDevice::DeviceType::Screen:
-        case CaptureDevice::DeviceType::Window:
-            break;
-        case CaptureDevice::DeviceType::Unknown:
-            ASSERT_NOT_REACHED();
-        }
-    }
-
-    void whenReady(CompletionHandler<void(String)>&& callback) final
-    {
-        if (m_isReady)
-            return callback(WTFMove(m_errorMessage));
-        m_callback = WTFMove(callback);
-    }
-
-    void didFail(String&& errorMessage)
-    {
-        m_isReady = true;
-        m_errorMessage = WTFMove(errorMessage);
-        if (m_callback)
-            m_callback(m_errorMessage);
-    }
-
-    void setAsReady()
-    {
-        m_isReady = true;
-        if (m_callback)
-            m_callback({ });
-    }
-
-    SharedRingBufferStorage& storage()
-    {
-        ASSERT(type() == Type::Audio);
-        return static_cast<SharedRingBufferStorage&>(m_ringBuffer->storage());
-    }
-
-    Ref<RealtimeMediaSource> clone() final
-    {
-        return m_manager.cloneSource(*this);
-    }
-
-    RealtimeMediaSourceIdentifier sourceID() const
-    {
-        return m_id;
-    }
-
-    const RealtimeMediaSourceSettings& settings() const
-    {
-        return m_settings;
-    }
-
-    const RealtimeMediaSourceCapabilities& capabilities() final;
-    void setCapabilities(RealtimeMediaSourceCapabilities&& capabilities)
-    {
-        m_capabilities = WTFMove(capabilities);
-    }
-
-    const RealtimeMediaSourceSettings& settings() final { return m_settings; }
-    void setSettings(RealtimeMediaSourceSettings&& settings)
-    {
-        auto changed = m_settings.difference(settings);
-        m_settings = WTFMove(settings);
-        notifySettingsDidChangeObservers(changed);
-    }
-
-    const CAAudioStreamDescription& description() const { return m_description; }
-    void setStorage(const SharedMemory::Handle& handle, const WebCore::CAAudioStreamDescription& description, uint64_t numberOfFrames)
-    {
-        ASSERT(type() == Type::Audio);
-        m_description = description;
-
-        if (handle.isNull()) {
-            m_ringBuffer->deallocate();
-            storage().setReadOnly(false);
-            storage().setStorage(nullptr);
-            return;
-        }
-
-        RefPtr<SharedMemory> memory = SharedMemory::map(handle, SharedMemory::Protection::ReadOnly);
-        storage().setStorage(WTFMove(memory));
-        storage().setReadOnly(true);
-
-        m_ringBuffer->allocate(description, numberOfFrames);
-    }
-
-    void setRingBufferFrameBounds(uint64_t startFrame, uint64_t endFrame)
-    {
-        ASSERT(type() == Type::Audio);
-        m_ringBuffer->setCurrentFrameBounds(startFrame, endFrame);
-    }
-
-    void audioSamplesAvailable(MediaTime time, uint64_t numberOfFrames)
-    {
-        ASSERT(type() == Type::Audio);
-        WebAudioBufferList audioData(m_description, numberOfFrames);
-        m_ringBuffer->fetch(audioData.list(), numberOfFrames, time.timeValue());
-
-        RealtimeMediaSource::audioSamplesAvailable(time, audioData, m_description, numberOfFrames);
-    }
-
-    void remoteVideoSampleAvailable(RemoteVideoSample&& remoteSample)
-    {
-        ASSERT(type() == Type::Video);
-
-        setIntrinsicSize(remoteSample.size());
-
-        if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != remoteSample.videoFormat())
-            m_imageTransferSession = ImageTransferSessionVT::create(remoteSample.videoFormat());
-
-        if (!m_imageTransferSession) {
-            ASSERT_NOT_REACHED();
-            return;
-        }
-
-        auto sampleRef = m_imageTransferSession->createMediaSample(remoteSample);
-        if (!sampleRef) {
-            ASSERT_NOT_REACHED();
-            return;
-        }
-
-        RealtimeMediaSource::videoSampleAvailable(*sampleRef);
-    }
-
-    void applyConstraintsSucceeded(const WebCore::RealtimeMediaSourceSettings& settings)
-    {
-        setSettings(WebCore::RealtimeMediaSourceSettings(settings));
-
-        auto callback = m_pendingApplyConstraintsCallbacks.takeFirst();
-        callback({ });
-    }
-
-    void applyConstraintsFailed(String&& failedConstraint, String&& errorMessage)
-    {
-        auto callback = m_pendingApplyConstraintsCallbacks.takeFirst();
-        callback(ApplyConstraintsError { WTFMove(failedConstraint), WTFMove(errorMessage) });
-    }
-
-    CaptureDevice::DeviceType deviceType() const final { return m_deviceType; }
-
-    void setShouldCaptureInGPUProcess(bool value) { m_shouldCaptureInGPUProcess = value; }
-    bool shouldCaptureInGPUProcess() const { return m_shouldCaptureInGPUProcess; }
-
-    IPC::Connection* connection();
-
-    void hasEnded() final;
-
-private:
-    void startProducingData() final;
-    void stopProducingData() final;
-    bool isCaptureSource() const final { return true; }
-
-    // RealtimeMediaSource
-    void beginConfiguration() final { }
-    void commitConfiguration() final { }
-    bool setShouldApplyRotation(bool /* shouldApplyRotation */) final;
-    void applyConstraints(const WebCore::MediaConstraints&, ApplyConstraintsHandler&&) final;
-    void requestToEnd(Observer&) final;
-    void stopBeingObserved() final;
-
-    RealtimeMediaSourceIdentifier m_id;
-    UserMediaCaptureManager& m_manager;
-    RealtimeMediaSourceCapabilities m_capabilities;
-    RealtimeMediaSourceSettings m_settings;
-
-    CAAudioStreamDescription m_description;
-    std::unique_ptr<CARingBuffer> m_ringBuffer;
-
-    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
-    CaptureDevice::DeviceType m_deviceType { CaptureDevice::DeviceType::Unknown };
-
-    Deque<ApplyConstraintsHandler> m_pendingApplyConstraintsCallbacks;
-    bool m_shouldCaptureInGPUProcess { false };
-    bool m_isReady { false };
-    String m_errorMessage;
-    CompletionHandler<void(String)> m_callback;
-};
-
 UserMediaCaptureManager::UserMediaCaptureManager(WebProcess& process)
     : m_process(process)
     , m_audioFactory(*this)
@@ -302,7 +75,7 @@ void UserMediaCaptureManager::setupCaptureProcesses(bool shouldCaptureAudioInUIP
     m_videoFactory.setShouldCaptureInGPUProcess(shouldCaptureVideoInGPUProcess);
 
     if (shouldCaptureAudioInGPUProcess)
-        WebCore::AudioMediaStreamTrackRenderer::setCreator(WebKit::AudioMediaStreamTrackRenderer::create);
+        AudioMediaStreamTrackRenderer::setCreator(WebKit::AudioMediaStreamTrackRenderer::create);
 
     if (shouldCaptureAudioInUIProcess || shouldCaptureAudioInGPUProcess)
         RealtimeMediaSourceCenter::singleton().setAudioCaptureFactory(m_audioFactory);
@@ -312,54 +85,31 @@ void UserMediaCaptureManager::setupCaptureProcesses(bool shouldCaptureAudioInUIP
         RealtimeMediaSourceCenter::singleton().setDisplayCaptureFactory(m_displayFactory);
 }
 
-WebCore::CaptureSourceOrError UserMediaCaptureManager::createCaptureSource(const CaptureDevice& device, String&& hashSalt, const WebCore::MediaConstraints* constraints, bool shouldCaptureInGPUProcess)
+void UserMediaCaptureManager::addSource(Ref<RemoteRealtimeMediaSource>&& source)
 {
-    if (!constraints)
-        return { };
+    if (source->type() == RealtimeMediaSource::Type::Audio)
+        m_remoteCaptureSampleManager.addSource(source.copyRef());
 
-    auto id = RealtimeMediaSourceIdentifier::generate();
+    auto identifier = source->identifier();
+    ASSERT(!m_sources.contains(identifier));
+    m_sources.add(identifier, WTFMove(source));
+}
 
-#if ENABLE(GPU_PROCESS)
-    auto* connection = shouldCaptureInGPUProcess ? &m_process.ensureGPUProcessConnection().connection() : m_process.parentProcessConnection();
-#else
-    ASSERT(!shouldCaptureInGPUProcess);
-    auto* connection = m_process.parentProcessConnection();
-#endif
-    
-    auto type = device.type() == CaptureDevice::DeviceType::Microphone ? WebCore::RealtimeMediaSource::Type::Audio : WebCore::RealtimeMediaSource::Type::Video;
-    auto source = adoptRef(*new Source(String::number(id.toUInt64()), type, device.type(), String { }, String { hashSalt }, id, *this));
-    if (shouldCaptureInGPUProcess)
-        source->setShouldCaptureInGPUProcess(shouldCaptureInGPUProcess);
-    m_sources.add(id, source.copyRef());
-
-    connection->sendWithAsyncReply(Messages::UserMediaCaptureManagerProxy::CreateMediaSourceForCaptureDeviceWithConstraints(id, device, hashSalt, *constraints), [source = source.copyRef()](bool succeeded, auto&& errorMessage, auto&& settings, auto&& capabilities) {
-        if (!succeeded) {
-            source->didFail(WTFMove(errorMessage));
-            return;
-        }
-        source->setName(String { settings.label().string() });
-        source->setSettings(WTFMove(settings));
-        source->setCapabilities(WTFMove(capabilities));
-        source->setAsReady();
-    });
-
-    return WebCore::CaptureSourceOrError(WTFMove(source));
+void UserMediaCaptureManager::removeSource(RealtimeMediaSourceIdentifier id)
+{
+    m_sources.remove(id);
 }
 
 void UserMediaCaptureManager::sourceStopped(RealtimeMediaSourceIdentifier id)
 {
-    if (auto source = m_sources.get(id)) {
-        source->stop();
-        source->hasEnded();
-    }
+    if (auto source = m_sources.get(id))
+        source->captureStopped();
 }
 
 void UserMediaCaptureManager::captureFailed(RealtimeMediaSourceIdentifier id)
 {
-    if (auto source = m_sources.get(id)) {
+    if (auto source = m_sources.get(id))
         source->captureFailed();
-        source->hasEnded();
-    }
 }
 
 void UserMediaCaptureManager::sourceMutedChanged(RealtimeMediaSourceIdentifier id, bool muted)
@@ -374,80 +124,13 @@ void UserMediaCaptureManager::sourceSettingsChanged(RealtimeMediaSourceIdentifie
         source->setSettings(RealtimeMediaSourceSettings(settings));
 }
 
-void UserMediaCaptureManager::storageChanged(RealtimeMediaSourceIdentifier id, const SharedMemory::Handle& handle, const WebCore::CAAudioStreamDescription& description, uint64_t numberOfFrames)
-{
-    if (auto source = m_sources.get(id))
-        source->setStorage(handle, description, numberOfFrames);
-}
-
-void UserMediaCaptureManager::ringBufferFrameBoundsChanged(RealtimeMediaSourceIdentifier id, uint64_t startFrame, uint64_t endFrame)
-{
-    if (auto source = m_sources.get(id))
-        source->setRingBufferFrameBounds(startFrame, endFrame);
-}
-
-void UserMediaCaptureManager::audioSamplesAvailable(RealtimeMediaSourceIdentifier id, MediaTime time, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame)
-{
-    if (auto source = m_sources.get(id)) {
-        source->setRingBufferFrameBounds(startFrame, endFrame);
-        source->audioSamplesAvailable(time, numberOfFrames);
-    }
-}
-
 void UserMediaCaptureManager::remoteVideoSampleAvailable(RealtimeMediaSourceIdentifier id, RemoteVideoSample&& sample)
 {
     if (auto source = m_sources.get(id))
         source->remoteVideoSampleAvailable(WTFMove(sample));
 }
 
-IPC::Connection* UserMediaCaptureManager::Source::connection()
-{
-#if ENABLE(GPU_PROCESS)
-    if (m_shouldCaptureInGPUProcess)
-        return &m_manager.m_process.ensureGPUProcessConnection().connection();
-#endif
-    return m_manager.m_process.parentProcessConnection();
-}
-
-void UserMediaCaptureManager::Source::startProducingData()
-{
-    connection()->send(Messages::UserMediaCaptureManagerProxy::StartProducingData(m_id), 0);
-}
-
-void UserMediaCaptureManager::Source::stopProducingData()
-{
-    connection()->send(Messages::UserMediaCaptureManagerProxy::StopProducingData(m_id), 0);
-}
-
-bool UserMediaCaptureManager::Source::setShouldApplyRotation(bool shouldApplyRotation)
-{
-    connection()->send(Messages::UserMediaCaptureManagerProxy::SetShouldApplyRotation(m_id, shouldApplyRotation), 0);
-    return true;
-}
-
-const WebCore::RealtimeMediaSourceCapabilities& UserMediaCaptureManager::Source::capabilities()
-{
-    return m_capabilities;
-}
-
-void UserMediaCaptureManager::Source::applyConstraints(const WebCore::MediaConstraints& constraints, ApplyConstraintsHandler&& completionHandler)
-{
-    m_pendingApplyConstraintsCallbacks.append(WTFMove(completionHandler));
-    connection()->send(Messages::UserMediaCaptureManagerProxy::ApplyConstraints(m_id, constraints), 0);
-}
-
-void UserMediaCaptureManager::sourceEnded(RealtimeMediaSourceIdentifier id)
-{
-    m_sources.remove(id);
-}
-
-void UserMediaCaptureManager::Source::hasEnded()
-{
-    connection()->send(Messages::UserMediaCaptureManagerProxy::End { m_id }, 0);
-    m_manager.sourceEnded(m_id);
-}
-
-void UserMediaCaptureManager::applyConstraintsSucceeded(RealtimeMediaSourceIdentifier id, const WebCore::RealtimeMediaSourceSettings& settings)
+void UserMediaCaptureManager::applyConstraintsSucceeded(RealtimeMediaSourceIdentifier id, const RealtimeMediaSourceSettings& settings)
 {
     if (auto source = m_sources.get(id))
         source->applyConstraintsSucceeded(settings);
@@ -459,74 +142,41 @@ void UserMediaCaptureManager::applyConstraintsFailed(RealtimeMediaSourceIdentifi
         source->applyConstraintsFailed(WTFMove(failedConstraint), WTFMove(message));
 }
 
-Ref<RealtimeMediaSource> UserMediaCaptureManager::cloneSource(Source& source)
-{
-    switch (source.type()) {
-    case RealtimeMediaSource::Type::Video:
-        return cloneVideoSource(source);
-    case RealtimeMediaSource::Type::Audio:
-        break;
-    case RealtimeMediaSource::Type::None:
-        ASSERT_NOT_REACHED();
-    }
-    return makeRef(source);
-}
-
-Ref<RealtimeMediaSource> UserMediaCaptureManager::cloneVideoSource(Source& source)
-{
-    auto id = RealtimeMediaSourceIdentifier::generate();
-    if (!source.connection()->send(Messages::UserMediaCaptureManagerProxy::Clone { source.sourceID(), id }, 0))
-        return makeRef(source);
-
-    auto settings = source.settings();
-    auto cloneSource = adoptRef(*new Source(String::number(id.toUInt64()), source.type(), source.deviceType(), String { settings.label().string() }, source.deviceIDHashSalt(), id, *this));
-    if (source.shouldCaptureInGPUProcess())
-        cloneSource->setShouldCaptureInGPUProcess(true);
-    cloneSource->setSettings(WTFMove(settings));
-    m_sources.add(id, cloneSource.copyRef());
-    return cloneSource;
-}
-
-void UserMediaCaptureManager::Source::stopBeingObserved()
-{
-    connection()->send(Messages::UserMediaCaptureManagerProxy::RequestToEnd { m_id }, 0);
-}
-
-void UserMediaCaptureManager::Source::requestToEnd(Observer& observer)
-{
-    switch (type()) {
-    case Type::Audio:
-        RealtimeMediaSource::requestToEnd(observer);
-        break;
-    case Type::Video:
-        stopBeingObserved();
-        break;
-    case Type::None:
-        ASSERT_NOT_REACHED();
-    }
-}
-
 CaptureSourceOrError UserMediaCaptureManager::AudioFactory::createAudioCaptureSource(const CaptureDevice& device, String&& hashSalt, const MediaConstraints* constraints)
 {
+    if (!constraints)
+        return { };
+
 #if !ENABLE(GPU_PROCESS)
     if (m_shouldCaptureInGPUProcess)
         return CaptureSourceOrError { "Audio capture in GPUProcess is not implemented"_s };
 #endif
+
 #if PLATFORM(IOS_FAMILY) || ENABLE(ROUTING_ARBITRATION)
-    // FIXME: Remove disabling of the audio session category managemeent once we move all media playing to GPUProcess.
+    // FIXME: Remove disabling of the audio session category management once we move all media playing to GPUProcess.
     if (m_shouldCaptureInGPUProcess)
         DeprecatedGlobalSettings::setShouldManageAudioSessionCategory(true);
 #endif
-    return m_manager.createCaptureSource(device, WTFMove(hashSalt), constraints, m_shouldCaptureInGPUProcess);
+
+    return RemoteRealtimeMediaSource::create(device, *constraints, { }, WTFMove(hashSalt), m_manager);
+}
+
+void UserMediaCaptureManager::AudioFactory::setShouldCaptureInGPUProcess(bool value)
+{
+    m_shouldCaptureInGPUProcess = value;
 }
 
 CaptureSourceOrError UserMediaCaptureManager::VideoFactory::createVideoCaptureSource(const CaptureDevice& device, String&& hashSalt, const MediaConstraints* constraints)
 {
+    if (!constraints)
+        return { };
+
 #if !ENABLE(GPU_PROCESS)
     if (m_shouldCaptureInGPUProcess)
         return CaptureSourceOrError { "Video capture in GPUProcess is not implemented"_s };
 #endif
-    return m_manager.createCaptureSource(device, WTFMove(hashSalt), constraints, m_shouldCaptureInGPUProcess);
+
+    return RemoteRealtimeMediaSource::create(device, *constraints, { }, WTFMove(hashSalt), m_manager);
 }
 
 #if PLATFORM(IOS_FAMILY)
@@ -536,6 +186,14 @@ void UserMediaCaptureManager::VideoFactory::setActiveSource(RealtimeMediaSource&
 }
 #endif
 
+CaptureSourceOrError UserMediaCaptureManager::DisplayFactory::createDisplayCaptureSource(const CaptureDevice& device, const MediaConstraints* constraints)
+{
+    if (!constraints)
+        return { };
+
+    return RemoteRealtimeMediaSource::create(device, *constraints, { }, { }, m_manager);
+}
+
 }
 
 #endif
index 3bc51ea..6ed0a0f 100644 (file)
@@ -27,7 +27,7 @@
 
 #if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
 
-#include "MessageReceiver.h"
+#include "RemoteCaptureSampleManager.h"
 #include "SharedMemory.h"
 #include "WebProcessSupplement.h"
 #include <WebCore/CaptureDeviceManager.h>
@@ -43,7 +43,7 @@ class RemoteVideoSample;
 
 namespace WebKit {
 
-class CrossProcessRealtimeAudioSource;
+class RemoteRealtimeMediaSource;
 class WebProcess;
 
 class UserMediaCaptureManager : public WebProcessSupplement, public IPC::MessageReceiver {
@@ -57,12 +57,15 @@ public:
     void didReceiveMessageFromGPUProcess(IPC::Connection& connection, IPC::Decoder& decoder) { didReceiveMessage(connection, decoder); }
     void setupCaptureProcesses(bool shouldCaptureAudioInUIProcess, bool shouldCaptureAudioInGPUProcess, bool shouldCaptureVideoInUIProcess, bool shouldCaptureVideoInGPUProcess, bool shouldCaptureDisplayInUIProcess);
 
+    void addSource(Ref<RemoteRealtimeMediaSource>&&);
+    void removeSource(WebCore::RealtimeMediaSourceIdentifier);
+
 private:
     // WebCore::RealtimeMediaSource factories
     class AudioFactory : public WebCore::AudioCaptureFactory {
     public:
         explicit AudioFactory(UserMediaCaptureManager& manager) : m_manager(manager) { }
-        void setShouldCaptureInGPUProcess(bool value) { m_shouldCaptureInGPUProcess = value; }
+        void setShouldCaptureInGPUProcess(bool);
 
     private:
         WebCore::CaptureSourceOrError createAudioCaptureSource(const WebCore::CaptureDevice&, String&& hashSalt, const WebCore::MediaConstraints*) final;
@@ -91,14 +94,12 @@ private:
         explicit DisplayFactory(UserMediaCaptureManager& manager) : m_manager(manager) { }
 
     private:
-        WebCore::CaptureSourceOrError createDisplayCaptureSource(const WebCore::CaptureDevice& device, const WebCore::MediaConstraints* constraints) final  { return m_manager.createCaptureSource(device, { }, constraints); }
+        WebCore::CaptureSourceOrError createDisplayCaptureSource(const WebCore::CaptureDevice&, const WebCore::MediaConstraints*) final;
         WebCore::CaptureDeviceManager& displayCaptureDeviceManager() final { return m_manager.m_noOpCaptureDeviceManager; }
 
         UserMediaCaptureManager& m_manager;
     };
 
-    WebCore::CaptureSourceOrError createCaptureSource(const WebCore::CaptureDevice&, String&&, const WebCore::MediaConstraints*, bool shouldCaptureInGPUProcess = false);
-
     class NoOpCaptureDeviceManager : public WebCore::CaptureDeviceManager {
     public:
         NoOpCaptureDeviceManager() = default;
@@ -118,28 +119,22 @@ private:
     // Messages::UserMediaCaptureManager
     void captureFailed(WebCore::RealtimeMediaSourceIdentifier);
     void sourceStopped(WebCore::RealtimeMediaSourceIdentifier);
-    void sourceEnded(WebCore::RealtimeMediaSourceIdentifier);
+    void sourceEnded(WebCore::RealtimeMediaSourceIdentifier identifier) { removeSource(identifier); }
     void sourceMutedChanged(WebCore::RealtimeMediaSourceIdentifier, bool muted);
     void sourceSettingsChanged(WebCore::RealtimeMediaSourceIdentifier, const WebCore::RealtimeMediaSourceSettings&);
-    void storageChanged(WebCore::RealtimeMediaSourceIdentifier, const SharedMemory::Handle&, const WebCore::CAAudioStreamDescription&, uint64_t numberOfFrames);
-    void ringBufferFrameBoundsChanged(WebCore::RealtimeMediaSourceIdentifier, uint64_t startFrame, uint64_t endFrame);
-    void audioSamplesAvailable(WebCore::RealtimeMediaSourceIdentifier, MediaTime, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame);
     void remoteVideoSampleAvailable(WebCore::RealtimeMediaSourceIdentifier, WebCore::RemoteVideoSample&&);
     void applyConstraintsSucceeded(WebCore::RealtimeMediaSourceIdentifier, const WebCore::RealtimeMediaSourceSettings&);
     void applyConstraintsFailed(WebCore::RealtimeMediaSourceIdentifier, String&&, String&&);
 
-    class Source;
-    friend class Source;
-
-    Ref<WebCore::RealtimeMediaSource> cloneSource(Source&);
-    Ref<WebCore::RealtimeMediaSource> cloneVideoSource(Source&);
+    Ref<WebCore::RealtimeMediaSource> cloneVideoSource(RemoteRealtimeMediaSource&);
 
-    HashMap<WebCore::RealtimeMediaSourceIdentifier, Ref<Source>> m_sources;
+    HashMap<WebCore::RealtimeMediaSourceIdentifier, Ref<RemoteRealtimeMediaSource>> m_sources;
     WebProcess& m_process;
     NoOpCaptureDeviceManager m_noOpCaptureDeviceManager;
     AudioFactory m_audioFactory;
     VideoFactory m_videoFactory;
     DisplayFactory m_displayFactory;
+    RemoteCaptureSampleManager m_remoteCaptureSampleManager;
 };
 
 } // namespace WebKit
index 3010eee..10466ba 100644 (file)
@@ -28,8 +28,6 @@ messages -> UserMediaCaptureManager NotRefCounted {
     SourceStopped(WebCore::RealtimeMediaSourceIdentifier id)
     SourceMutedChanged(WebCore::RealtimeMediaSourceIdentifier id, bool muted)
     SourceSettingsChanged(WebCore::RealtimeMediaSourceIdentifier id, WebCore::RealtimeMediaSourceSettings settings)
-    StorageChanged(WebCore::RealtimeMediaSourceIdentifier id, WebKit::SharedMemory::Handle storageHandle, WebCore::CAAudioStreamDescription description, uint64_t numberOfFrames)
-    AudioSamplesAvailable(WebCore::RealtimeMediaSourceIdentifier id, MediaTime time, uint64_t numberOfFrames, uint64_t startFrame, uint64_t endFrame)
     ApplyConstraintsSucceeded(WebCore::RealtimeMediaSourceIdentifier id, WebCore::RealtimeMediaSourceSettings settings)
     ApplyConstraintsFailed(WebCore::RealtimeMediaSourceIdentifier id, String failedConstraint, String message)
     RemoteVideoSampleAvailable(WebCore::RealtimeMediaSourceIdentifier id, WebCore::RemoteVideoSample sample)