Implement RTC VTB decoders in GPUProcess
authoryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 31 Dec 2019 10:00:48 +0000 (10:00 +0000)
committeryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 31 Dec 2019 10:00:48 +0000 (10:00 +0000)
https://bugs.webkit.org/show_bug.cgi?id=205607

Reviewed by Eric Carlson.

Source/ThirdParty/libwebrtc:

Expose remote decoder abilities with C like functions.
This allows WebProcess to implement IPC-based decoders.
Expose VTB H264 decoder as C like functions.
This allows GPU process to instantiate wasily H2664 decoders.

* Configurations/libwebrtc.iOS.exp:
* Configurations/libwebrtc.iOSsim.exp:
* Configurations/libwebrtc.mac.exp:
* Source/webrtc/sdk/WebKit/WebKitUtilities.h:
* Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
(webrtc::videoDecoderCallbacks):
(webrtc::setVideoDecoderCallbacks):
(webrtc::RemoteVideoDecoder::RemoteVideoDecoder):
(webrtc::RemoteVideoDecoder::decodeComplete):
(webrtc::RemoteVideoDecoder::InitDecode):
(webrtc::RemoteVideoDecoder::Decode):
(webrtc::RemoteVideoDecoder::RegisterDecodeCompleteCallback):
(webrtc::RemoteVideoDecoder::Release):
(webrtc::RemoteVideoDecoderFactory::RemoteVideoDecoderFactory):
(webrtc::RemoteVideoDecoderFactory::GetSupportedFormats const):
(webrtc::RemoteVideoDecoderFactory::CreateVideoDecoder):
(webrtc::createWebKitDecoderFactory):
(webrtc::createLocalDecoder):
(webrtc::releaseLocalDecoder):
(webrtc::decodeFrame):
* Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.h:
* Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.mm:
(-[RTCVideoDecoderH264 decode:missingFrames:codecSpecificInfo:renderTimeMs:]):
(-[RTCVideoDecoderH264 decodeData:size:timeStamp:]):

Source/WebCore:

Add routine to create a RemoveVideoSample from a pixel buffer.
Update LibWebRTCProvider to enable/disable decoding in GPU Process and add internals API.

Test: webrtc/video-gpuProcess.html

* platform/graphics/RemoteVideoSample.cpp:
(WebCore::RemoteVideoSample::create):
* platform/graphics/RemoteVideoSample.h:
* platform/graphics/cv/ImageTransferSessionVT.h:
* platform/mediastream/libwebrtc/LibWebRTCProvider.cpp:
(WebCore::LibWebRTCProvider::setUseGPUProcess):
* platform/mediastream/libwebrtc/LibWebRTCProvider.h:
* platform/mediastream/libwebrtc/LibWebRTCProviderCocoa.h:
* testing/Internals.cpp:
(WebCore::Internals::resetToConsistentState):
(WebCore::Internals::setUseGPUProcessForWebRTC):
* testing/Internals.h:
* testing/Internals.idl:

Source/WebKit:

Implement decoder factory callbacks.
Implement WebProcess codecs by sending IPC for creating/releasing/decoding a frame.
WebProcess receives IPC messages from GPU Process whenever a frame is decoded.

* Configurations/WebKit.xcconfig:
* DerivedSources-input.xcfilelist:
* DerivedSources-output.xcfilelist:
* DerivedSources.make:
* GPUProcess/GPUConnectionToWebProcess.cpp:
(WebKit::GPUConnectionToWebProcess::libWebRTCCodecsProxy):
(WebKit::GPUConnectionToWebProcess::didReceiveMessage):
* GPUProcess/GPUConnectionToWebProcess.h:
* GPUProcess/webrtc/LibWebRTCCodecsProxy.h: Added.
(WebKit::LibWebRTCCodecsProxy::didReceiveMessageFromWebProcess):
* GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in: Added.
* GPUProcess/webrtc/LibWebRTCCodecsProxy.mm: Added.
(WebKit::LibWebRTCCodecsProxy::LibWebRTCCodecsProxy):
(WebKit::LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy):
(WebKit::LibWebRTCCodecsProxy::createDecoder):
(WebKit::LibWebRTCCodecsProxy::releaseDecoder):
(WebKit::LibWebRTCCodecsProxy::decodeFrame):
* Scripts/webkit/messages.py:
* Sources.txt:
* SourcesCocoa.txt:
* WebKit.xcodeproj/project.pbxproj:
* WebProcess/GPU/GPUProcessConnection.cpp:
(WebKit::GPUProcessConnection::didReceiveMessage):
* WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp: Added.
(WebKit::createVideoDecoder):
(WebKit::releaseVideoDecoder):
(WebKit::decodeVideoFrame):
(WebKit::registerDecodeCompleteCallback):
(WebKit::LibWebRTCCodecs::setVideoDecoderCallbacks):
(WebKit::LibWebRTCCodecs::createDecoder):
(WebKit::LibWebRTCCodecs::releaseDecoder):
(WebKit::LibWebRTCCodecs::decodeFrame):
(WebKit::LibWebRTCCodecs::registerDecodeFrameCallback):
(WebKit::LibWebRTCCodecs::failedDecoding):
(WebKit::LibWebRTCCodecs::completedDecoding):
* WebProcess/GPU/webrtc/LibWebRTCCodecs.h: Added.
* WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in: Added.
* WebProcess/GPU/webrtc/RTCDecoderIdentifier.h: Added.
* WebProcess/Network/webrtc/LibWebRTCProvider.cpp:
(WebKit::LibWebRTCProvider::createDecoderFactory):
* WebProcess/Network/webrtc/LibWebRTCProvider.h:
* WebProcess/WebProcess.cpp:
(WebKit::WebProcess::libWebRTCCodecs):
* WebProcess/WebProcess.h:

LayoutTests:

* webrtc/video-gpuProcess-expected.txt: Added.
* webrtc/video-gpuProcess.html: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253957 268f45cc-cd09-0410-ab3c-d52691b4dbfc

43 files changed:
LayoutTests/ChangeLog
LayoutTests/webrtc/video-gpuProcess-expected.txt [new file with mode: 0644]
LayoutTests/webrtc/video-gpuProcess.html [new file with mode: 0644]
Source/ThirdParty/libwebrtc/ChangeLog
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOS.exp
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOSsim.exp
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.mac.exp
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.h
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.mm
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.h
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.mm
Source/WebCore/ChangeLog
Source/WebCore/platform/graphics/RemoteVideoSample.cpp
Source/WebCore/platform/graphics/RemoteVideoSample.h
Source/WebCore/platform/graphics/cv/ImageTransferSessionVT.h
Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCProvider.cpp
Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCProvider.h
Source/WebCore/platform/mediastream/libwebrtc/LibWebRTCProviderCocoa.h
Source/WebCore/testing/Internals.cpp
Source/WebCore/testing/Internals.h
Source/WebCore/testing/Internals.idl
Source/WebKit/ChangeLog
Source/WebKit/DerivedSources-input.xcfilelist
Source/WebKit/DerivedSources-output.xcfilelist
Source/WebKit/DerivedSources.make
Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp
Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h
Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.h [new file with mode: 0644]
Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in [new file with mode: 0644]
Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.mm [new file with mode: 0644]
Source/WebKit/Scripts/webkit/messages.py
Source/WebKit/Sources.txt
Source/WebKit/SourcesCocoa.txt
Source/WebKit/WebKit.xcodeproj/project.pbxproj
Source/WebKit/WebProcess/GPU/GPUProcessConnection.cpp
Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp [new file with mode: 0644]
Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.h [new file with mode: 0644]
Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in [new file with mode: 0644]
Source/WebKit/WebProcess/GPU/webrtc/RTCDecoderIdentifier.h [new file with mode: 0644]
Source/WebKit/WebProcess/Network/webrtc/LibWebRTCProvider.cpp
Source/WebKit/WebProcess/Network/webrtc/LibWebRTCProvider.h
Source/WebKit/WebProcess/WebProcess.cpp
Source/WebKit/WebProcess/WebProcess.h

index 35ffc5e..6903f56 100644 (file)
@@ -1,3 +1,13 @@
+2019-12-31  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB decoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205607
+
+        Reviewed by Eric Carlson.
+
+        * webrtc/video-gpuProcess-expected.txt: Added.
+        * webrtc/video-gpuProcess.html: Added.
+
 2019-12-30  youenn fablet  <youenn@apple.com>
 
         Ignore URL host for schemes that are not using host information
diff --git a/LayoutTests/webrtc/video-gpuProcess-expected.txt b/LayoutTests/webrtc/video-gpuProcess-expected.txt
new file mode 100644 (file)
index 0000000..1c5554d
--- /dev/null
@@ -0,0 +1,6 @@
+
+
+PASS Basic video exchange 
+PASS Call setParameters to disable sending a given encoding 
+PASS Call setParameters to reenable sending a given encoding 
+
diff --git a/LayoutTests/webrtc/video-gpuProcess.html b/LayoutTests/webrtc/video-gpuProcess.html
new file mode 100644 (file)
index 0000000..d7ddfb2
--- /dev/null
@@ -0,0 +1,129 @@
+<!doctype html>
+<html>
+    <head>
+        <meta charset="utf-8">
+        <title>Video exchange using GPU process</title>
+        <script src="../resources/testharness.js"></script>
+        <script src="../resources/testharnessreport.js"></script>
+    </head>
+    <body>
+        <video id="video" autoplay=""></video>
+        <canvas id="canvas" width="640" height="480"></canvas>
+        <script src ="routines.js"></script>
+        <script>
+if (window.internals)
+    internals.setUseGPUProcessForWebRTC(true);
+
+video = document.getElementById("video");
+canvas = document.getElementById("canvas");
+
+function grabFrameData(x, y, w, h)
+{
+    canvas.width = video.videoWidth;
+    canvas.height = video.videoHeight;
+
+    canvas.getContext('2d').drawImage(video, x, y, w, h, x, y, w, h);
+    return canvas.getContext('2d').getImageData(x, y, w, h).data;
+}
+
+function testImage()
+{
+    const data = grabFrameData(10, 325, 250, 1);
+
+    var index = 20;
+    assert_true(data[index] < 100);
+    assert_true(data[index + 1] < 100);
+    assert_true(data[index + 2] < 100);
+
+    index = 80;
+    assert_true(data[index] > 200);
+    assert_true(data[index + 1] > 200);
+    assert_true(data[index + 2] > 200);
+
+    index += 80;
+    assert_true(data[index] > 200);
+    assert_true(data[index + 1] > 200);
+    assert_true(data[index + 2] < 100);
+}
+
+var pc1, pc2;
+promise_test(async (test) => {
+    if (window.testRunner)
+        testRunner.setUserMediaPermission(true);
+
+    const localStream = await navigator.mediaDevices.getUserMedia({video: {advanced: [{width:{min:640}}, {height:{min:480} } ]}});
+    if (window.internals)
+        assert_true(internals.pageMediaState().includes('HasActiveVideoCaptureDevice'), "Unexpected HasActiveVideoCaptureDevice");
+    const stream = await new Promise((resolve, reject) => {
+        createConnections((firstConnection) => {
+            pc1 = firstConnection;
+            firstConnection.addTrack(localStream.getVideoTracks()[0], localStream);
+        }, (secondConnection) => {
+            pc2 = secondConnection;
+            secondConnection.ontrack = (trackEvent) => {
+                assert_true(trackEvent.track instanceof MediaStreamTrack);
+                assert_true(trackEvent.receiver instanceof RTCRtpReceiver);
+                assert_true(Array.isArray(trackEvent.streams), "Array.isArray() should return true");
+                assert_true(Object.isFrozen(trackEvent.streams), "Object.isFrozen() should return true");
+                assert_equals(trackEvent.track.id, localStream.getVideoTracks()[0].id);
+                assert_equals(trackEvent.track, trackEvent.streams[0].getVideoTracks()[0]);
+                resolve(trackEvent.streams[0]);
+            };
+        });
+        setTimeout(() => reject("Test timed out"), 5000);
+    });
+
+    video.srcObject = stream;
+    await video.play();
+
+    testImage();
+}, "Basic video exchange");
+
+function getCircleImageData()
+{
+    return grabFrameData(450, 100, 150, 100);
+}
+
+async function checkVideoIsUpdated(shouldBeUpdated, count, referenceData)
+{
+    if (count === undefined)
+        count = 0;
+    else if (count >= 20)
+        return Promise.reject("checkVideoIsUpdated timed out :" + shouldBeUpdated + " " + count);
+
+    if (referenceData === undefined)
+        referenceData = getCircleImageData();
+
+    await waitFor(200);
+    const newData = getCircleImageData();
+
+    if (shouldBeUpdated === (JSON.stringify(referenceData) !== JSON.stringify(newData)))
+        return;
+
+    await checkVideoIsUpdated(shouldBeUpdated, ++count, newData);
+}
+
+promise_test(async (test) => {
+    const sender = pc1.getSenders()[0];
+    let p = sender.getParameters();
+    p.encodings[0].active = false;
+    await sender.setParameters(p);
+
+    assert_false(sender.getParameters().encodings[0].active, "encodings[0].active should be false");
+
+    await checkVideoIsUpdated(false);
+}, "Call setParameters to disable sending a given encoding");
+
+promise_test(async (test) => {
+    const sender = pc1.getSenders()[0];
+    let p = sender.getParameters();
+    p.encodings[0].active = true;
+    await sender.setParameters(p);
+
+    assert_true(sender.getParameters().encodings[0].active, "encodings[0].active should be true");
+
+    await checkVideoIsUpdated(true);
+}, "Call setParameters to reenable sending a given encoding");
+        </script>
+    </body>
+</html>
index 1724491..9f92107 100644 (file)
@@ -1,3 +1,40 @@
+2019-12-31  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB decoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205607
+
+        Reviewed by Eric Carlson.
+
+        Expose remote decoder abilities with C like functions.
+        This allows WebProcess to implement IPC-based decoders.
+        Expose VTB H264 decoder as C like functions.
+        This allows GPU process to instantiate wasily H2664 decoders.
+
+        * Configurations/libwebrtc.iOS.exp:
+        * Configurations/libwebrtc.iOSsim.exp:
+        * Configurations/libwebrtc.mac.exp:
+        * Source/webrtc/sdk/WebKit/WebKitUtilities.h:
+        * Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
+        (webrtc::videoDecoderCallbacks):
+        (webrtc::setVideoDecoderCallbacks):
+        (webrtc::RemoteVideoDecoder::RemoteVideoDecoder):
+        (webrtc::RemoteVideoDecoder::decodeComplete):
+        (webrtc::RemoteVideoDecoder::InitDecode):
+        (webrtc::RemoteVideoDecoder::Decode):
+        (webrtc::RemoteVideoDecoder::RegisterDecodeCompleteCallback):
+        (webrtc::RemoteVideoDecoder::Release):
+        (webrtc::RemoteVideoDecoderFactory::RemoteVideoDecoderFactory):
+        (webrtc::RemoteVideoDecoderFactory::GetSupportedFormats const):
+        (webrtc::RemoteVideoDecoderFactory::CreateVideoDecoder):
+        (webrtc::createWebKitDecoderFactory):
+        (webrtc::createLocalDecoder):
+        (webrtc::releaseLocalDecoder):
+        (webrtc::decodeFrame):
+        * Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.h:
+        * Source/webrtc/sdk/objc/components/video_codec/RTCVideoDecoderH264.mm:
+        (-[RTCVideoDecoderH264 decode:missingFrames:codecSpecificInfo:renderTimeMs:]):
+        (-[RTCVideoDecoderH264 decodeData:size:timeStamp:]):
+
 2019-12-30  youenn fablet  <youenn@apple.com>
 
         Do not build yasm for iOS and iOS simulator
index db083c9..70fe963 100644 (file)
@@ -262,3 +262,8 @@ __ZNK6webrtc20DataChannelInterface17maxRetransmitsOptEv
 __ZNK6webrtc20RtpReceiverInterface14dtls_transportEv
 __ZNK6webrtc23PeerConnectionInterface16GetSctpTransportEv
 __ZNK6webrtc23RtpTransceiverInterface17codec_preferencesEv
+__ZN6webrtc18createLocalDecoderEU13block_pointerFvP10__CVBufferjjE
+__ZN6webrtc19releaseLocalDecoderEPv
+__ZN6webrtc11decodeFrameEPvjPKhm
+__ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
index db083c9..70fe963 100644 (file)
@@ -262,3 +262,8 @@ __ZNK6webrtc20DataChannelInterface17maxRetransmitsOptEv
 __ZNK6webrtc20RtpReceiverInterface14dtls_transportEv
 __ZNK6webrtc23PeerConnectionInterface16GetSctpTransportEv
 __ZNK6webrtc23RtpTransceiverInterface17codec_preferencesEv
+__ZN6webrtc18createLocalDecoderEU13block_pointerFvP10__CVBufferjjE
+__ZN6webrtc19releaseLocalDecoderEPv
+__ZN6webrtc11decodeFrameEPvjPKhm
+__ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
index db083c9..70fe963 100644 (file)
@@ -262,3 +262,8 @@ __ZNK6webrtc20DataChannelInterface17maxRetransmitsOptEv
 __ZNK6webrtc20RtpReceiverInterface14dtls_transportEv
 __ZNK6webrtc23PeerConnectionInterface16GetSctpTransportEv
 __ZNK6webrtc23RtpTransceiverInterface17codec_preferencesEv
+__ZN6webrtc18createLocalDecoderEU13block_pointerFvP10__CVBufferjjE
+__ZN6webrtc19releaseLocalDecoderEPv
+__ZN6webrtc11decodeFrameEPvjPKhm
+__ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
index 6effc5a..6a58ba1 100644 (file)
@@ -69,4 +69,47 @@ private:
     const std::unique_ptr<VideoEncoderFactory> m_internalEncoderFactory;
 };
 
+using WebKitVideoDecoder = void*;
+using VideoDecoderCreateCallback = WebKitVideoDecoder(*)(const SdpVideoFormat& format);
+using VideoDecoderReleaseCallback = int32_t(*)(WebKitVideoDecoder);
+using VideoDecoderDecodeCallback = int32_t(*)(WebKitVideoDecoder, uint32_t timeStamp, const uint8_t*, size_t length);
+using VideoDecoderRegisterDecodeCompleteCallback = int32_t(*)(WebKitVideoDecoder, void* decodedImageCallback);
+
+void setVideoDecoderCallbacks(VideoDecoderCreateCallback, VideoDecoderReleaseCallback, VideoDecoderDecodeCallback, VideoDecoderRegisterDecodeCompleteCallback);
+
+class RemoteVideoDecoderFactory final : public VideoDecoderFactory {
+public:
+    explicit RemoteVideoDecoderFactory(std::unique_ptr<VideoDecoderFactory>&&);
+    ~RemoteVideoDecoderFactory() = default;
+
+private:
+    std::vector<SdpVideoFormat> GetSupportedFormats() const final;
+    std::unique_ptr<VideoDecoder> CreateVideoDecoder(const SdpVideoFormat& format) final;
+
+    std::unique_ptr<VideoDecoderFactory> m_internalFactory;
+};
+
+class RemoteVideoDecoder final : public webrtc::VideoDecoder {
+public:
+    explicit RemoteVideoDecoder(WebKitVideoDecoder);
+    ~RemoteVideoDecoder() = default;
+
+    static void decodeComplete(void* callback, uint32_t timeStamp, CVPixelBufferRef, uint32_t timeStampRTP);
+
+private:
+    int32_t InitDecode(const VideoCodec*, int32_t number_of_cores) final;
+    int32_t Decode(const EncodedImage&, bool missing_frames, int64_t render_time_ms) final;
+    int32_t RegisterDecodeCompleteCallback(DecodedImageCallback*) final;
+    int32_t Release() final;
+    const char* ImplementationName() const final { return "RemoteVideoToolBox"; }
+
+    WebKitVideoDecoder m_internalDecoder;
+};
+
+using LocalDecoder = void*;
+using LocalDecoderCallback = void (^)(CVPixelBufferRef, uint32_t timeStamp, uint32_t timeStampNs);
+void* createLocalDecoder(LocalDecoderCallback);
+void releaseLocalDecoder(LocalDecoder);
+int32_t decodeFrame(LocalDecoder, uint32_t timeStamp, const uint8_t*, size_t);
+
 }
index 2fe57b2..121eda9 100644 (file)
 #import "WebRTC/RTCVideoCodecH264.h"
 
 #include "api/video/video_frame.h"
+#include "helpers/scoped_cftyperef.h"
 #include "native/src/objc_frame_buffer.h"
+#include "rtc_base/time_utils.h"
+#include "sdk/objc/components/video_codec/nalu_rewriter.h"
 #include "third_party/libyuv/include/libyuv/convert_from.h"
 #include "webrtc/rtc_base/checks.h"
 #include "Framework/Headers/WebRTC/RTCVideoCodecFactory.h"
@@ -116,11 +119,6 @@ std::unique_ptr<webrtc::VideoEncoderFactory> createWebKitEncoderFactory(WebKitCo
     return std::make_unique<VideoEncoderFactoryWithSimulcast>(std::move(internalFactory));
 }
 
-std::unique_ptr<webrtc::VideoDecoderFactory> createWebKitDecoderFactory(WebKitCodecSupport codecSupport)
-{
-    return ObjCToNativeVideoDecoderFactory(codecSupport == WebKitCodecSupport::H264AndVP8 ? [[RTCDefaultVideoDecoderFactory alloc] init] : [[RTCVideoDecoderFactoryH264 alloc] init]);
-}
-
 static bool h264HardwareEncoderAllowed = true;
 void setH264HardwareEncoderAllowed(bool allowed)
 {
@@ -168,7 +166,6 @@ static bool CopyVideoFrameToPixelBuffer(const webrtc::I420BufferInterface* frame
     return true;
 }
 
-
 CVPixelBufferRef pixelBufferFromFrame(const VideoFrame& frame, const std::function<CVPixelBufferRef(size_t, size_t)>& makePixelBuffer)
 {
     if (frame.video_frame_buffer()->type() != VideoFrameBuffer::Type::kNative) {
@@ -188,4 +185,117 @@ CVPixelBufferRef pixelBufferFromFrame(const VideoFrame& frame, const std::functi
     return rtcPixelBuffer.pixelBuffer;
 }
 
+struct VideoDecoderCallbacks {
+    VideoDecoderCreateCallback createCallback;
+    VideoDecoderReleaseCallback releaseCallback;
+    VideoDecoderDecodeCallback decodeCallback;
+    VideoDecoderRegisterDecodeCompleteCallback registerDecodeCompleteCallback;
+};
+
+static VideoDecoderCallbacks& videoDecoderCallbacks() {
+    static VideoDecoderCallbacks callbacks;
+    return callbacks;
+}
+
+void setVideoDecoderCallbacks(VideoDecoderCreateCallback createCallback, VideoDecoderReleaseCallback releaseCallback, VideoDecoderDecodeCallback decodeCallback, VideoDecoderRegisterDecodeCompleteCallback registerDecodeCompleteCallback)
+{
+    auto& callbacks = videoDecoderCallbacks();
+    callbacks.createCallback = createCallback;
+    callbacks.releaseCallback = releaseCallback;
+    callbacks.decodeCallback = decodeCallback;
+    callbacks.registerDecodeCompleteCallback = registerDecodeCompleteCallback;
+}
+
+RemoteVideoDecoder::RemoteVideoDecoder(WebKitVideoDecoder internalDecoder)
+    : m_internalDecoder(internalDecoder)
+{
+}
+
+void RemoteVideoDecoder::decodeComplete(void* callback, uint32_t timeStamp, CVPixelBufferRef pixelBuffer, uint32_t timeStampRTP)
+{
+    auto videoFrame = VideoFrame::Builder().set_video_frame_buffer(pixelBufferToFrame(pixelBuffer))
+        .set_timestamp_rtp(timeStampRTP)
+        .set_timestamp_ms(0)
+        .set_rotation((VideoRotation)RTCVideoRotation_0)
+        .build();
+    videoFrame.set_timestamp(timeStamp);
+
+    static_cast<DecodedImageCallback*>(callback)->Decoded(videoFrame);
+}
+
+int32_t RemoteVideoDecoder::InitDecode(const VideoCodec* codec_settings, int32_t number_of_cores)
+{
+    return WEBRTC_VIDEO_CODEC_OK;
+}
+
+int32_t RemoteVideoDecoder::Decode(const EncodedImage& input_image, bool missing_frames, int64_t render_time_ms)
+{
+    return videoDecoderCallbacks().decodeCallback(m_internalDecoder, input_image.Timestamp(), input_image.data(), input_image.size());
+}
+
+int32_t RemoteVideoDecoder::RegisterDecodeCompleteCallback(DecodedImageCallback* callback)
+{
+    return videoDecoderCallbacks().registerDecodeCompleteCallback(m_internalDecoder, callback);
+}
+
+int32_t RemoteVideoDecoder::Release()
+{
+    return videoDecoderCallbacks().releaseCallback(m_internalDecoder);
+}
+
+RemoteVideoDecoderFactory::RemoteVideoDecoderFactory(std::unique_ptr<VideoDecoderFactory>&& internalFactory)
+    : m_internalFactory(std::move(internalFactory))
+{
+}
+
+std::vector<SdpVideoFormat> RemoteVideoDecoderFactory::GetSupportedFormats() const
+{
+    std::vector<SdpVideoFormat> supported_formats;
+    supported_formats.push_back(SdpVideoFormat { "H264" });
+    supported_formats.push_back(SdpVideoFormat { "VP8" });
+    return supported_formats;
+}
+
+std::unique_ptr<VideoDecoder> RemoteVideoDecoderFactory::CreateVideoDecoder(const SdpVideoFormat& format)
+{
+    if (!videoDecoderCallbacks().createCallback)
+        return m_internalFactory->CreateVideoDecoder(format);
+
+    auto identifier = videoDecoderCallbacks().createCallback(format);
+    if (!identifier)
+        return m_internalFactory->CreateVideoDecoder(format);
+
+    return std::make_unique<RemoteVideoDecoder>(identifier);
+}
+
+std::unique_ptr<webrtc::VideoDecoderFactory> createWebKitDecoderFactory(WebKitCodecSupport codecSupport)
+{
+    auto internalFactory = ObjCToNativeVideoDecoderFactory(codecSupport == WebKitCodecSupport::H264AndVP8 ? [[RTCDefaultVideoDecoderFactory alloc] init] : [[RTCVideoDecoderFactoryH264 alloc] init]);
+    if (videoDecoderCallbacks().createCallback)
+        return std::make_unique<RemoteVideoDecoderFactory>(std::move(internalFactory));
+    return internalFactory;
+}
+
+void* createLocalDecoder(LocalDecoderCallback callback)
+{
+    auto decoder = [[RTCVideoDecoderH264 alloc] init];
+    [decoder setCallback:^(RTCVideoFrame *frame) {
+        auto *buffer = (RTCCVPixelBuffer *)frame.buffer;
+        callback(buffer.pixelBuffer, frame.timeStampNs, frame.timeStamp);
+    }];
+    return (__bridge_retained void*)decoder;
+}
+
+void releaseLocalDecoder(LocalDecoder localDecoder)
+{
+    RTCVideoDecoderH264* decoder = (__bridge_transfer RTCVideoDecoderH264 *)(localDecoder);
+    [decoder releaseDecoder];
+}
+
+int32_t decodeFrame(LocalDecoder localDecoder, uint32_t timeStamp, const uint8_t* data, size_t size)
+{
+    RTCVideoDecoderH264* decoder = (__bridge RTCVideoDecoderH264 *)(localDecoder);
+    return [decoder decodeData: data size: size timeStamp: timeStamp];
+}
+
 }
index b10d1e6..22dcc80 100644 (file)
@@ -16,4 +16,7 @@
 RTC_OBJC_EXPORT
 __attribute__((objc_runtime_name("WK_RTCVideoDecoderH264")))
 @interface RTCVideoDecoderH264 : NSObject <RTCVideoDecoder>
+- (NSInteger)decodeData:(const uint8_t *)data
+    size:(size_t)size
+    timeStamp:(uint32_t)timeStamp;
 @end
index 8e2a7d9..4ce0bc4 100644 (file)
@@ -102,6 +102,13 @@ void decompressionOutputCallback(void *decoderRef,
          renderTimeMs:(int64_t)renderTimeMs {
   RTC_DCHECK(inputImage.buffer);
 
+  return [self decodeData: (uint8_t *)inputImage.buffer.bytes size: inputImage.buffer.length timeStamp: inputImage.timeStamp];
+}
+
+- (NSInteger)decodeData:(const uint8_t *)data
+        size:(size_t)size
+        timeStamp:(uint32_t)timeStamp {
+
   if (_error != noErr) {
     RTC_LOG(LS_WARNING) << "Last frame decode failed.";
     _error = noErr;
@@ -109,8 +116,7 @@ void decompressionOutputCallback(void *decoderRef,
   }
 
   rtc::ScopedCFTypeRef<CMVideoFormatDescriptionRef> inputFormat =
-      rtc::ScopedCF(webrtc::CreateVideoFormatDescription((uint8_t *)inputImage.buffer.bytes,
-                                                         inputImage.buffer.length));
+      rtc::ScopedCF(webrtc::CreateVideoFormatDescription(data, size));
   if (inputFormat) {
     // Check if the video format has changed, and reinitialize decoder if
     // needed.
@@ -132,8 +138,7 @@ void decompressionOutputCallback(void *decoderRef,
     return WEBRTC_VIDEO_CODEC_ERROR;
   }
   CMSampleBufferRef sampleBuffer = nullptr;
-  if (!webrtc::H264AnnexBBufferToCMSampleBuffer((uint8_t *)inputImage.buffer.bytes,
-                                                inputImage.buffer.length,
+  if (!webrtc::H264AnnexBBufferToCMSampleBuffer(data, size,
                                                 _videoFormat,
                                                 &sampleBuffer,
                                                 _memoryPool)) {
@@ -142,14 +147,14 @@ void decompressionOutputCallback(void *decoderRef,
   RTC_DCHECK(sampleBuffer);
   VTDecodeFrameFlags decodeFlags = kVTDecodeFrame_EnableAsynchronousDecompression;
   std::unique_ptr<RTCFrameDecodeParams> frameDecodeParams;
-  frameDecodeParams.reset(new RTCFrameDecodeParams(_callback, inputImage.timeStamp));
+  frameDecodeParams.reset(new RTCFrameDecodeParams(_callback, timeStamp));
   OSStatus status = VTDecompressionSessionDecodeFrame(
       _decompressionSession, sampleBuffer, decodeFlags, frameDecodeParams.release(), nullptr);
 #if defined(WEBRTC_IOS)
   // Re-initialize the decoder if we have an invalid session while the app is
   // active and retry the decode request.
   if (status == kVTInvalidSessionErr && [self resetDecompressionSession] == WEBRTC_VIDEO_CODEC_OK) {
-    frameDecodeParams.reset(new RTCFrameDecodeParams(_callback, inputImage.timeStamp));
+    frameDecodeParams.reset(new RTCFrameDecodeParams(_callback, timeStamp));
     status = VTDecompressionSessionDecodeFrame(
         _decompressionSession, sampleBuffer, decodeFlags, frameDecodeParams.release(), nullptr);
   }
index 0334d89..af91797 100644 (file)
@@ -1,3 +1,29 @@
+2019-12-31  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB decoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205607
+
+        Reviewed by Eric Carlson.
+
+        Add routine to create a RemoveVideoSample from a pixel buffer.
+        Update LibWebRTCProvider to enable/disable decoding in GPU Process and add internals API.
+
+        Test: webrtc/video-gpuProcess.html
+
+        * platform/graphics/RemoteVideoSample.cpp:
+        (WebCore::RemoteVideoSample::create):
+        * platform/graphics/RemoteVideoSample.h:
+        * platform/graphics/cv/ImageTransferSessionVT.h:
+        * platform/mediastream/libwebrtc/LibWebRTCProvider.cpp:
+        (WebCore::LibWebRTCProvider::setUseGPUProcess):
+        * platform/mediastream/libwebrtc/LibWebRTCProvider.h:
+        * platform/mediastream/libwebrtc/LibWebRTCProviderCocoa.h:
+        * testing/Internals.cpp:
+        (WebCore::Internals::resetToConsistentState):
+        (WebCore::Internals::setUseGPUProcessForWebRTC):
+        * testing/Internals.h:
+        * testing/Internals.idl:
+
 2019-12-30  Eric Carlson  <eric.carlson@apple.com>
 
         Create media mime type cache base class to reduce duplicate code
index 93c9305..9fe824b 100644 (file)
@@ -61,6 +61,17 @@ std::unique_ptr<RemoteVideoSample> RemoteVideoSample::create(MediaSample&& sampl
     return std::unique_ptr<RemoteVideoSample>(new RemoteVideoSample(surface, sRGBColorSpaceRef(), sample.presentationTime(), sample.videoRotation(), sample.videoMirrored()));
 }
 
+std::unique_ptr<RemoteVideoSample> RemoteVideoSample::create(CVPixelBufferRef imageBuffer, MediaTime&& presentationTime)
+{
+    auto surface = CVPixelBufferGetIOSurface(imageBuffer);
+    if (!surface) {
+        RELEASE_LOG_ERROR(Media, "RemoteVideoSample::create: CVPixelBufferGetIOSurface returned nullptr");
+        return nullptr;
+    }
+
+    return std::unique_ptr<RemoteVideoSample>(new RemoteVideoSample(surface, sRGBColorSpaceRef(), WTFMove(presentationTime), MediaSample::VideoRotation::None, false));
+}
+
 RemoteVideoSample::RemoteVideoSample(IOSurfaceRef surface, CGColorSpaceRef colorSpace, MediaTime&& time, MediaSample::VideoRotation rotation, bool mirrored)
     : m_ioSurface(WebCore::IOSurface::createFromSurface(surface, colorSpace))
     , m_rotation(rotation)
index f02d4fc..bd255f7 100644 (file)
@@ -36,6 +36,8 @@
 #include "IOSurface.h"
 #endif
 
+typedef struct __CVBuffer* CVPixelBufferRef;
+
 namespace WebCore {
 
 class RemoteVideoSample {
@@ -47,6 +49,7 @@ public:
 
 #if HAVE(IOSURFACE)
     WEBCORE_EXPORT static std::unique_ptr<RemoteVideoSample> create(MediaSample&&);
+    WEBCORE_EXPORT static std::unique_ptr<RemoteVideoSample> create(CVPixelBufferRef, MediaTime&& presentationTime);
     WEBCORE_EXPORT IOSurfaceRef surface();
 #endif
 
index 16e5576..3b7430f 100644 (file)
@@ -55,6 +55,10 @@ public:
     WEBCORE_EXPORT RefPtr<MediaSample> createMediaSample(IOSurfaceRef, const MediaTime&, const IntSize&, MediaSample::VideoRotation = MediaSample::VideoRotation::None, bool mirrored = false);
 #endif
 
+#if HAVE(IOSURFACE) && !PLATFORM(MACCATALYST)
+    WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> createPixelBuffer(IOSurfaceRef, const IntSize&);
+#endif
+
     uint32_t pixelFormat() const { return m_pixelFormat; }
 
 private:
@@ -63,7 +67,6 @@ private:
 #if HAVE(IOSURFACE) && !PLATFORM(MACCATALYST)
     CFDictionaryRef ioSurfacePixelBufferCreationOptions(IOSurfaceRef);
     RetainPtr<CMSampleBufferRef> createCMSampleBuffer(IOSurfaceRef, const MediaTime&, const IntSize&);
-    RetainPtr<CVPixelBufferRef> createPixelBuffer(IOSurfaceRef, const IntSize&);
 #endif
 
     RetainPtr<CMSampleBufferRef> convertCMSampleBuffer(CMSampleBufferRef, const IntSize&);
index e48da18..afa298a 100644 (file)
@@ -328,6 +328,15 @@ void LibWebRTCProvider::setUseDTLS10(bool useDTLS10)
     m_factory->SetOptions(options);
 }
 
+void LibWebRTCProvider::setUseGPUProcess(bool value)
+{
+    if (m_useGPUProcess == value)
+        return;
+
+    m_useGPUProcess = value;
+    m_factory = nullptr;
+}
+
 rtc::scoped_refptr<webrtc::PeerConnectionInterface> LibWebRTCProvider::createPeerConnection(webrtc::PeerConnectionObserver& observer, rtc::NetworkManager& networkManager, rtc::PacketSocketFactory& packetSocketFactory, webrtc::PeerConnectionInterface::RTCConfiguration&& configuration, std::unique_ptr<webrtc::AsyncResolverFactory>&& asyncResolveFactory)
 {
     auto& factoryAndThreads = getStaticFactoryAndThreads(m_useNetworkThreadWithSocketServer);
index e8f99c7..67859b5 100644 (file)
@@ -119,6 +119,7 @@ public:
     void setEnableLogging(bool);
     void setEnableWebRTCEncryption(bool);
     void setUseDTLS10(bool);
+    void setUseGPUProcess(bool);
 
     class SuspendableSocketFactory : public rtc::PacketSocketFactory {
     public:
@@ -146,6 +147,7 @@ protected:
     bool m_supportsVP8 { false };
     bool m_enableLogging { true };
     bool m_useDTLS10 { false };
+    bool m_useGPUProcess { false };
 #endif
 };
 
index d3c3839..239e34a 100644 (file)
@@ -41,9 +41,10 @@ public:
     LibWebRTCProviderCocoa() = default;
     ~LibWebRTCProviderCocoa();
 
+    std::unique_ptr<webrtc::VideoDecoderFactory> createDecoderFactory() override;
+
 private:
     void setActive(bool) final;
-    std::unique_ptr<webrtc::VideoDecoderFactory> createDecoderFactory() final;
     std::unique_ptr<webrtc::VideoEncoderFactory> createEncoderFactory() final;
 
     void setH264HardwareEncoderAllowed(bool allowed) final;
index 00fad12..7920814 100644 (file)
@@ -538,6 +538,7 @@ void Internals::resetToConsistentState(Page& page)
     rtcProvider.disableNonLocalhostConnections();
     RuntimeEnabledFeatures::sharedFeatures().setWebRTCVP8CodecEnabled(true);
     page.settings().setWebRTCEncryptionEnabled(true);
+    rtcProvider.setUseGPUProcess(false);
 #endif
 
     page.settings().setStorageAccessAPIEnabled(false);
@@ -1551,6 +1552,15 @@ void Internals::setUseDTLS10(bool useDTLS10)
 #endif
 }
 
+void Internals::setUseGPUProcessForWebRTC(bool useGPUProcess)
+{
+#if USE(LIBWEBRTC)
+    auto* document = contextDocument();
+    if (!document || !document->page())
+        return;
+    document->page()->libWebRTCProvider().setUseGPUProcess(useGPUProcess);
+#endif
+}
 #endif
 
 #if ENABLE(MEDIA_STREAM)
index e2a120f..02a2ea2 100644 (file)
@@ -559,6 +559,7 @@ public:
     void applyRotationForOutgoingVideoSources(RTCPeerConnection&);
     void setEnableWebRTCEncryption(bool);
     void setUseDTLS10(bool);
+    void setUseGPUProcessForWebRTC(bool);
 #endif
 
     String getImageSourceURL(Element&);
index cfbfe46..9f83c32 100644 (file)
@@ -632,6 +632,7 @@ enum CompositingPolicy {
     [Conditional=WEB_RTC] void clearPeerConnectionFactory();
     [Conditional=WEB_RTC] void setEnableWebRTCEncryption(boolean enabled);
     [Conditional=WEB_RTC] void setUseDTLS10(boolean use);
+    [Conditional=WEB_RTC] void setUseGPUProcessForWebRTC(boolean use);
 
     [Conditional=VIDEO] void simulateSystemSleep();
     [Conditional=VIDEO] void simulateSystemWake();
index ef40232..7981604 100644 (file)
@@ -1,3 +1,59 @@
+2019-12-31  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB decoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205607
+
+        Reviewed by Eric Carlson.
+
+        Implement decoder factory callbacks.
+        Implement WebProcess codecs by sending IPC for creating/releasing/decoding a frame.
+        WebProcess receives IPC messages from GPU Process whenever a frame is decoded.
+
+        * Configurations/WebKit.xcconfig:
+        * DerivedSources-input.xcfilelist:
+        * DerivedSources-output.xcfilelist:
+        * DerivedSources.make:
+        * GPUProcess/GPUConnectionToWebProcess.cpp:
+        (WebKit::GPUConnectionToWebProcess::libWebRTCCodecsProxy):
+        (WebKit::GPUConnectionToWebProcess::didReceiveMessage):
+        * GPUProcess/GPUConnectionToWebProcess.h:
+        * GPUProcess/webrtc/LibWebRTCCodecsProxy.h: Added.
+        (WebKit::LibWebRTCCodecsProxy::didReceiveMessageFromWebProcess):
+        * GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in: Added.
+        * GPUProcess/webrtc/LibWebRTCCodecsProxy.mm: Added.
+        (WebKit::LibWebRTCCodecsProxy::LibWebRTCCodecsProxy):
+        (WebKit::LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy):
+        (WebKit::LibWebRTCCodecsProxy::createDecoder):
+        (WebKit::LibWebRTCCodecsProxy::releaseDecoder):
+        (WebKit::LibWebRTCCodecsProxy::decodeFrame):
+        * Scripts/webkit/messages.py:
+        * Sources.txt:
+        * SourcesCocoa.txt:
+        * WebKit.xcodeproj/project.pbxproj:
+        * WebProcess/GPU/GPUProcessConnection.cpp:
+        (WebKit::GPUProcessConnection::didReceiveMessage):
+        * WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp: Added.
+        (WebKit::createVideoDecoder):
+        (WebKit::releaseVideoDecoder):
+        (WebKit::decodeVideoFrame):
+        (WebKit::registerDecodeCompleteCallback):
+        (WebKit::LibWebRTCCodecs::setVideoDecoderCallbacks):
+        (WebKit::LibWebRTCCodecs::createDecoder):
+        (WebKit::LibWebRTCCodecs::releaseDecoder):
+        (WebKit::LibWebRTCCodecs::decodeFrame):
+        (WebKit::LibWebRTCCodecs::registerDecodeFrameCallback):
+        (WebKit::LibWebRTCCodecs::failedDecoding):
+        (WebKit::LibWebRTCCodecs::completedDecoding):
+        * WebProcess/GPU/webrtc/LibWebRTCCodecs.h: Added.
+        * WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in: Added.
+        * WebProcess/GPU/webrtc/RTCDecoderIdentifier.h: Added.
+        * WebProcess/Network/webrtc/LibWebRTCProvider.cpp:
+        (WebKit::LibWebRTCProvider::createDecoderFactory):
+        * WebProcess/Network/webrtc/LibWebRTCProvider.h:
+        * WebProcess/WebProcess.cpp:
+        (WebKit::WebProcess::libWebRTCCodecs):
+        * WebProcess/WebProcess.h:
+
 2019-12-30  Brady Eidson  <beidson@apple.com>
 
         Add WKWebView SPI to evaluate a function with arguments
index 07d3305..f37f3bd 100644 (file)
@@ -17,6 +17,7 @@ $(PROJECT_DIR)/GPUProcess/GPUConnectionToWebProcess.messages.in
 $(PROJECT_DIR)/GPUProcess/GPUProcess.messages.in
 $(PROJECT_DIR)/GPUProcess/mac/com.apple.WebKit.GPUProcess.sb.in
 $(PROJECT_DIR)/GPUProcess/media/RemoteMediaPlayerManagerProxy.messages.in
+$(PROJECT_DIR)/GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in
 $(PROJECT_DIR)/NetworkProcess/Cookies/WebCookieManager.messages.in
 $(PROJECT_DIR)/NetworkProcess/CustomProtocols/LegacyCustomProtocolManager.messages.in
 $(PROJECT_DIR)/NetworkProcess/IndexedDB/WebIDBServer.messages.in
@@ -99,6 +100,8 @@ $(PROJECT_DIR)/WebProcess/Databases/IndexedDB/WebIDBConnectionToServer.messages.
 $(PROJECT_DIR)/WebProcess/FullScreen/WebFullScreenManager.messages.in
 $(PROJECT_DIR)/WebProcess/GPU/GPUProcessConnection.messages.in
 $(PROJECT_DIR)/WebProcess/GPU/media/RemoteMediaPlayerManager.messages.in
+$(PROJECT_DIR)/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in
+$(PROJECT_DIR)/WebProcess/GPU/webrtc/LibWebRTCRemoteCodecs.messages.in
 $(PROJECT_DIR)/WebProcess/Geolocation/WebGeolocationManager.messages.in
 $(PROJECT_DIR)/WebProcess/Network/NetworkProcessConnection.messages.in
 $(PROJECT_DIR)/WebProcess/Network/WebResourceLoader.messages.in
index 9869cf1..70aeb7b 100644 (file)
@@ -48,6 +48,15 @@ $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LegacyCustomProtocolManagerMessages
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LegacyCustomProtocolManagerProxyMessageReceiver.cpp
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LegacyCustomProtocolManagerProxyMessages.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LegacyCustomProtocolManagerProxyMessagesReplies.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCCodecsMessageReceiver.cpp
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCCodecsMessages.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCCodecsMessagesReplies.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCCodecsProxyMessageReceiver.cpp
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCCodecsProxyMessages.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCCodecsProxyMessagesReplies.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCRemoteCodecsMessageReceiver.cpp
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCRemoteCodecsMessages.h
+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/LibWebRTCRemoteCodecsMessagesReplies.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/NPObjectMessageReceiverMessageReceiver.cpp
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/NPObjectMessageReceiverMessages.h
 $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit2/NPObjectMessageReceiverMessagesReplies.h
index 17bdcb5..3850277 100644 (file)
@@ -25,6 +25,7 @@ VPATH = \
     $(WebKit2)/GPUProcess \
     $(WebKit2)/GPUProcess/mac \
     $(WebKit2)/GPUProcess/media \
+    $(WebKit2)/GPUProcess/webrtc \
     $(WebKit2)/NetworkProcess \
     $(WebKit2)/NetworkProcess/Cookies \
     $(WebKit2)/NetworkProcess/cache \
@@ -51,6 +52,7 @@ VPATH = \
     $(WebKit2)/WebProcess/Geolocation \
     $(WebKit2)/WebProcess/GPU \
     $(WebKit2)/WebProcess/GPU/media \
+    $(WebKit2)/WebProcess/GPU/webrtc \
     $(WebKit2)/WebProcess/IconDatabase \
     $(WebKit2)/WebProcess/MediaCache \
     $(WebKit2)/WebProcess/MediaStream \
@@ -109,12 +111,14 @@ MESSAGE_RECEIVERS = \
     DrawingAreaProxy \
     EditableImageController \
     EventDispatcher \
-    LegacyCustomProtocolManager \
-    LegacyCustomProtocolManagerProxy \
     GPUProcess \
     GPUProcessProxy \
     GPUProcessConnection \
     GPUConnectionToWebProcess \
+    LegacyCustomProtocolManager \
+    LegacyCustomProtocolManagerProxy \
+    LibWebRTCCodecs \
+    LibWebRTCCodecsProxy \
     NPObjectMessageReceiver \
     NetworkConnectionToWebProcess \
     NetworkContentRuleListManager \
index 54803ca..8d6e78f 100644 (file)
@@ -33,6 +33,8 @@
 #include "GPUProcess.h"
 #include "GPUProcessMessages.h"
 #include "GPUProcessProxyMessages.h"
+#include "LibWebRTCCodecsProxy.h"
+#include "LibWebRTCCodecsProxyMessages.h"
 #include "Logging.h"
 #include "RemoteLayerTreeDrawingAreaProxyMessages.h"
 #include "RemoteMediaPlayerManagerProxy.h"
@@ -127,6 +129,16 @@ UserMediaCaptureManagerProxy& GPUConnectionToWebProcess::userMediaCaptureManager
 }
 #endif
 
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+LibWebRTCCodecsProxy& GPUConnectionToWebProcess::libWebRTCCodecsProxy()
+{
+    if (!m_libWebRTCCodecsProxy)
+        m_libWebRTCCodecsProxy = makeUnique<LibWebRTCCodecsProxy>(*this);
+
+    return *m_libWebRTCCodecsProxy;
+}
+#endif
+
 void GPUConnectionToWebProcess::didReceiveMessage(IPC::Connection& connection, IPC::Decoder& decoder)
 {
     if (decoder.messageReceiverName() == Messages::RemoteMediaPlayerManagerProxy::messageReceiverName()) {
@@ -139,6 +151,12 @@ void GPUConnectionToWebProcess::didReceiveMessage(IPC::Connection& connection, I
         return;
     }
 #endif
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+    if (decoder.messageReceiverName() == Messages::LibWebRTCCodecsProxy::messageReceiverName()) {
+        libWebRTCCodecsProxy().didReceiveMessageFromWebProcess(connection, decoder);
+        return;
+    }
+#endif
 }
 
 void GPUConnectionToWebProcess::didReceiveSyncMessage(IPC::Connection& connection, IPC::Decoder& decoder, std::unique_ptr<IPC::Encoder>& replyEncoder)
index 873dcea..8fb20f2 100644 (file)
@@ -39,6 +39,7 @@
 namespace WebKit {
 
 class GPUProcess;
+class LibWebRTCCodecsProxy;
 class RemoteMediaPlayerManagerProxy;
 class UserMediaCaptureManagerProxy;
 
@@ -63,6 +64,9 @@ private:
     GPUConnectionToWebProcess(GPUProcess&, WebCore::ProcessIdentifier, IPC::Connection::Identifier, PAL::SessionID);
 
     RemoteMediaPlayerManagerProxy& remoteMediaPlayerManagerProxy();
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+    LibWebRTCCodecsProxy& libWebRTCCodecsProxy();
+#endif
 #if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
     UserMediaCaptureManagerProxy& userMediaCaptureManagerProxy();
 #endif
@@ -83,6 +87,9 @@ private:
 #if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM)
     std::unique_ptr<UserMediaCaptureManagerProxy> m_userMediaCaptureManagerProxy;
 #endif
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+    std::unique_ptr<LibWebRTCCodecsProxy> m_libWebRTCCodecsProxy;
+#endif
 };
 
 } // namespace WebKit
diff --git a/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.h b/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.h
new file mode 100644 (file)
index 0000000..f988039
--- /dev/null
@@ -0,0 +1,68 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+
+#include "MessageReceiver.h"
+#include "RTCDecoderIdentifier.h"
+
+namespace IPC {
+class Connection;
+class Decoder;
+class DataReference;
+}
+
+namespace webrtc {
+using LocalDecoder = void*;
+}
+
+namespace WebKit {
+
+class GPUConnectionToWebProcess;
+
+class LibWebRTCCodecsProxy : private IPC::MessageReceiver {
+    WTF_MAKE_FAST_ALLOCATED;
+public:
+    explicit LibWebRTCCodecsProxy(GPUConnectionToWebProcess&);
+    ~LibWebRTCCodecsProxy();
+
+    void didReceiveMessageFromWebProcess(IPC::Connection& connection, IPC::Decoder& decoder) { didReceiveMessage(connection, decoder); }
+
+private:
+    // IPC::MessageReceiver
+    void didReceiveMessage(IPC::Connection&, IPC::Decoder&) final;
+    void createDecoder(RTCDecoderIdentifier);
+    void releaseDecoder(RTCDecoderIdentifier);
+    void decodeFrame(RTCDecoderIdentifier, uint32_t timeStamp, const IPC::DataReference&);
+
+    GPUConnectionToWebProcess& m_gpuConnectionToWebProcess;
+    HashMap<RTCDecoderIdentifier, webrtc::LocalDecoder> m_decoders;
+};
+
+}
+
+#endif
diff --git a/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in b/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in
new file mode 100644 (file)
index 0000000..c7a1334
--- /dev/null
@@ -0,0 +1,32 @@
+# Copyright (C) 2019 Apple Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+# THE POSSIBILITY OF SUCH DAMAGE.
+
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+
+messages -> LibWebRTCCodecsProxy NotRefCounted {
+    CreateDecoder(WebKit::RTCDecoderIdentifier id)
+    ReleaseDecoder(WebKit::RTCDecoderIdentifier id)
+    DecodeFrame(WebKit::RTCDecoderIdentifier id, uint32_t timeStamp, IPC::DataReference data)
+}
+
+#endif
diff --git a/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.mm b/Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.mm
new file mode 100644 (file)
index 0000000..3afab6a
--- /dev/null
@@ -0,0 +1,82 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "LibWebRTCCodecsProxy.h"
+
+#include "DataReference.h"
+#include "GPUConnectionToWebProcess.h"
+#include "LibWebRTCCodecsMessages.h"
+#include "WebCoreArgumentCoders.h"
+#include <WebCore/LibWebRTCMacros.h>
+#include <WebCore/RemoteVideoSample.h>
+#include <webrtc/sdk/WebKit/WebKitUtilities.h>
+#include <wtf/MediaTime.h>
+
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+
+namespace WebKit {
+
+LibWebRTCCodecsProxy::LibWebRTCCodecsProxy(GPUConnectionToWebProcess& connection)
+    : m_gpuConnectionToWebProcess(connection)
+{
+}
+
+LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy()
+{
+    for (auto decoder : m_decoders.values())
+        webrtc::releaseLocalDecoder(decoder);
+}
+
+void LibWebRTCCodecsProxy::createDecoder(RTCDecoderIdentifier identifier)
+{
+    ASSERT(!m_decoders.contains(identifier));
+    m_decoders.add(identifier, webrtc::createLocalDecoder(^(CVPixelBufferRef pixelBuffer, uint32_t timeStampNs, uint32_t timeStamp) {
+        if (auto sample = WebCore::RemoteVideoSample::create(pixelBuffer, MediaTime(timeStampNs, 1)))
+            m_gpuConnectionToWebProcess.connection().send(Messages::LibWebRTCCodecs::CompletedDecoding { identifier, timeStamp, *sample }, 0);
+    }));
+}
+
+void LibWebRTCCodecsProxy::releaseDecoder(RTCDecoderIdentifier identifier)
+{
+    ASSERT(m_decoders.contains(identifier));
+    if (auto decoder = m_decoders.take(identifier))
+        webrtc::releaseLocalDecoder(decoder);
+}
+
+void LibWebRTCCodecsProxy::decodeFrame(RTCDecoderIdentifier identifier, uint32_t timeStamp, const IPC::DataReference& data)
+{
+    ASSERT(m_decoders.contains(identifier));
+    auto decoder = m_decoders.get(identifier);
+    if (!decoder)
+        return;
+
+    if (webrtc::decodeFrame(decoder, timeStamp, data.data(), data.size()))
+        m_gpuConnectionToWebProcess.connection().send(Messages::LibWebRTCCodecs::FailedDecoding { identifier }, 0);
+}
+
+}
+
+#endif
index f2e9ed7..753ea97 100644 (file)
@@ -220,6 +220,7 @@ def types_that_cannot_be_forward_declared():
         'WebKit::ActivityStateChangeID',
         'WebKit::LayerHostingContextID',
         'WebKit::MediaPlayerPrivateRemoteIdentifier',
+        'WebKit::RTCDecoderIdentifier',
         'WebKit::StorageAreaIdentifier',
         'WebKit::StorageAreaImplIdentifier',
         'WebKit::StorageNamespaceIdentifier',
index 1916db3..acf49c7 100644 (file)
@@ -504,12 +504,13 @@ WebProcess/MediaStream/UserMediaPermissionRequestManager.cpp
 WebProcess/GPU/GPUProcessConnection.cpp
 WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp
 WebProcess/GPU/media/RemoteMediaPlayerManager.cpp
+WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp
 
 WebProcess/Network/NetworkProcessConnection.cpp
 WebProcess/Network/WebLoaderStrategy.cpp
 WebProcess/Network/WebResourceInterceptController.cpp
 WebProcess/Network/WebResourceLoader.cpp
-WebProcess/Network/WebSocketChannel.cpp
+WebProcess/Network/WebSocketChannel.cpp @no-unify
 WebProcess/Network/WebSocketChannelManager.cpp
 WebProcess/Network/WebSocketProvider.cpp
 WebProcess/Network/WebSocketStream.cpp
index a4aa55e..c713bae 100644 (file)
@@ -61,6 +61,7 @@ GPUProcess/ios/GPUProcessIOS.mm
 GPUProcess/mac/GPUProcessMac.mm
 GPUProcess/media/RemoteMediaPlayerProxy.cpp
 GPUProcess/media/RemoteMediaPlayerManagerProxy.cpp
+GPUProcess/webrtc/LibWebRTCCodecsProxy.mm
 GPUProcess/EntryPoint/Cocoa/XPCService/GPUServiceEntryPoint.mm
 
 Platform/cf/ModuleCF.cpp
@@ -633,7 +634,7 @@ EditableImageControllerMessageReceiver.cpp
 GPUConnectionToWebProcessMessageReceiver.cpp
 GPUProcessProxyMessageReceiver.cpp
 GPUProcessMessageReceiver.cpp
-//RemoteMediaPlayerManagerMessageReceiver.cpp
-//RemoteMediaPlayerManagerProxyMessageReceiver.cpp
+LibWebRTCCodecsProxyMessageReceiver.cpp
+LibWebRTCCodecsMessageReceiver.cpp
 ServiceWorkerFetchTaskMessageReceiver.cpp
 TextCheckingControllerProxyMessageReceiver.cpp
index e3b9192..7a2e81e 100644 (file)
                41DC459C1E3DBB2800B11F51 /* LibWebRTCSocketClient.h in Headers */ = {isa = PBXBuildFile; fileRef = 41DC459A1E3DBB2400B11F51 /* LibWebRTCSocketClient.h */; };
                41DC459F1E3DBDA500B11F51 /* WebRTCSocket.h in Headers */ = {isa = PBXBuildFile; fileRef = 41FAF5F31E3BFE7F001AE678 /* WebRTCSocket.h */; };
                41DE7C6C22278F1E00532B65 /* ServiceWorkerFetchTask.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 41518536222704F6005430C6 /* ServiceWorkerFetchTask.cpp */; };
+               41E0A7CB23B645CD00561060 /* WebSocketChannel.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 417915AC2256BB7400D6F97E /* WebSocketChannel.cpp */; };
                41F060E11654318500F3281C /* WebSocketChannelMessageReceiver.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 41F060DD1654317500F3281C /* WebSocketChannelMessageReceiver.cpp */; };
                41FABD2A1F4DE001006A6C97 /* CacheStorageEngineCache.h in Headers */ = {isa = PBXBuildFile; fileRef = 41FABD281F4DDFDC006A6C97 /* CacheStorageEngineCache.h */; };
                41FAF5F51E3C0649001AE678 /* WebRTCResolver.h in Headers */ = {isa = PBXBuildFile; fileRef = 41FAF5F41E3C0641001AE678 /* WebRTCResolver.h */; };
                4151E5C31FBB90A900E47E2D /* FormDataReference.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FormDataReference.h; sourceTree = "<group>"; };
                4157853021276B6F00DD3800 /* copy-webcontent-resources-to-private-headers.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = "copy-webcontent-resources-to-private-headers.sh"; sourceTree = "<group>"; };
                4157E4AF20E2EC9800A6C0D7 /* com.google.o1dbrowserplugin.sb */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = com.google.o1dbrowserplugin.sb; sourceTree = "<group>"; };
+               4172198923B6128200AE5686 /* LibWebRTCCodecs.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LibWebRTCCodecs.h; sourceTree = "<group>"; };
+               4172198A23B6128200AE5686 /* LibWebRTCCodecs.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LibWebRTCCodecs.cpp; sourceTree = "<group>"; };
+               4172198C23B612E800AE5686 /* RTCDecoderIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RTCDecoderIdentifier.h; sourceTree = "<group>"; };
+               4172198D23B62C7C00AE5686 /* LibWebRTCCodecs.messages.in */ = {isa = PBXFileReference; lastKnownFileType = text; path = LibWebRTCCodecs.messages.in; sourceTree = "<group>"; };
                4176901322FDD41B00B1576D /* NetworkRTCProvider.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = NetworkRTCProvider.mm; sourceTree = "<group>"; };
                417915AC2256BB7400D6F97E /* WebSocketChannel.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = WebSocketChannel.cpp; path = Network/WebSocketChannel.cpp; sourceTree = "<group>"; };
                417915AD2256BB7400D6F97E /* WebSocketChannel.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = WebSocketChannel.h; path = Network/WebSocketChannel.h; sourceTree = "<group>"; };
                41DC459A1E3DBB2400B11F51 /* LibWebRTCSocketClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LibWebRTCSocketClient.h; sourceTree = "<group>"; };
                41DC459D1E3DBCF000B11F51 /* WebRTCSocket.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = WebRTCSocket.cpp; path = Network/webrtc/WebRTCSocket.cpp; sourceTree = "<group>"; };
                41DC45A01E3DC53C00B11F51 /* WebRTCResolver.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = WebRTCResolver.cpp; path = Network/webrtc/WebRTCResolver.cpp; sourceTree = "<group>"; };
+               41E0A7C623B6397800561060 /* LibWebRTCCodecsProxy.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = LibWebRTCCodecsProxy.h; sourceTree = "<group>"; };
+               41E0A7C723B6397900561060 /* LibWebRTCCodecsProxy.messages.in */ = {isa = PBXFileReference; lastKnownFileType = text; path = LibWebRTCCodecsProxy.messages.in; sourceTree = "<group>"; };
+               41E0A7C823B6397900561060 /* LibWebRTCCodecsProxy.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = LibWebRTCCodecsProxy.mm; sourceTree = "<group>"; };
                41F060DD1654317500F3281C /* WebSocketChannelMessageReceiver.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = WebSocketChannelMessageReceiver.cpp; path = DerivedSources/WebKit2/WebSocketChannelMessageReceiver.cpp; sourceTree = BUILT_PRODUCTS_DIR; };
                41FABD281F4DDFDC006A6C97 /* CacheStorageEngineCache.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CacheStorageEngineCache.h; sourceTree = "<group>"; };
                41FAF5F31E3BFE7F001AE678 /* WebRTCSocket.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = WebRTCSocket.h; path = Network/webrtc/WebRTCSocket.h; sourceTree = "<group>"; };
                        isa = PBXGroup;
                        children = (
                                0792312F239B3B0C009598E2 /* media */,
+                               4172198823B6126100AE5686 /* webrtc */,
                                2D7E43C023752CD900EA5CA0 /* GPUProcessConnection.cpp */,
                                2D7E43C323752CD900EA5CA0 /* GPUProcessConnection.h */,
                                2D7E43C223752CD900EA5CA0 /* GPUProcessConnection.messages.in */,
                                2D9FB2042375209D0049F936 /* ios */,
                                2D9FB2062375209D0049F936 /* mac */,
                                07C75C172399A3DB0088E65B /* media */,
+                               4172198E23B633CA00AE5686 /* webrtc */,
                                2D9FB20C2375209D0049F936 /* GPUConnectionToWebProcess.cpp */,
                                2D9FB2002375209D0049F936 /* GPUConnectionToWebProcess.h */,
                                2D9FB2092375209D0049F936 /* GPUConnectionToWebProcess.messages.in */,
                        name = webrtc;
                        sourceTree = "<group>";
                };
+               4172198823B6126100AE5686 /* webrtc */ = {
+                       isa = PBXGroup;
+                       children = (
+                               4172198A23B6128200AE5686 /* LibWebRTCCodecs.cpp */,
+                               4172198923B6128200AE5686 /* LibWebRTCCodecs.h */,
+                               4172198D23B62C7C00AE5686 /* LibWebRTCCodecs.messages.in */,
+                               4172198C23B612E800AE5686 /* RTCDecoderIdentifier.h */,
+                       );
+                       path = webrtc;
+                       sourceTree = "<group>";
+               };
+               4172198E23B633CA00AE5686 /* webrtc */ = {
+                       isa = PBXGroup;
+                       children = (
+                               41E0A7C623B6397800561060 /* LibWebRTCCodecsProxy.h */,
+                               41E0A7C723B6397900561060 /* LibWebRTCCodecsProxy.messages.in */,
+                               41E0A7C823B6397900561060 /* LibWebRTCCodecsProxy.mm */,
+                       );
+                       path = webrtc;
+                       sourceTree = "<group>";
+               };
                41D129D81F3D101400D15E47 /* Cache */ = {
                        isa = PBXGroup;
                        children = (
                                51F060E11654318500F3281F /* WebRTCMonitorMessageReceiver.cpp in Sources */,
                                51F060E11654318500F3282C /* WebRTCResolverMessageReceiver.cpp in Sources */,
                                51F060E11654318500F3281C /* WebRTCSocketMessageReceiver.cpp in Sources */,
+                               41E0A7CB23B645CD00561060 /* WebSocketChannel.cpp in Sources */,
                                41F060E11654318500F3281C /* WebSocketChannelMessageReceiver.cpp in Sources */,
                                5C0B17791E7C882100E9123C /* WebSocketStreamMessageReceiver.cpp in Sources */,
                                41287D4E225D1ECB009A3E26 /* WebSocketTaskCocoa.mm in Sources */,
index e8daae0..64a4b4f 100644 (file)
@@ -30,6 +30,8 @@
 
 #include "DataReference.h"
 #include "GPUConnectionToWebProcessMessages.h"
+#include "LibWebRTCCodecs.h"
+#include "LibWebRTCCodecsMessages.h"
 #include "RemoteMediaPlayerManager.h"
 #include "RemoteMediaPlayerManagerMessages.h"
 #include "UserMediaCaptureManager.h"
@@ -75,6 +77,12 @@ void GPUProcessConnection::didReceiveMessage(IPC::Connection& connection, IPC::D
         return;
     }
 #endif
+#if USE(LIBWEBRTC) && PLATFORM(COCOA)
+    if (decoder.messageReceiverName() == Messages::LibWebRTCCodecs::messageReceiverName()) {
+        WebProcess::singleton().libWebRTCCodecs().didReceiveMessage(connection, decoder);
+        return;
+    }
+#endif
 }
 
 } // namespace WebKit
diff --git a/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp b/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp
new file mode 100644 (file)
index 0000000..b271d8e
--- /dev/null
@@ -0,0 +1,172 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "LibWebRTCCodecs.h"
+
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+
+#include "GPUProcessConnection.h"
+#include "LibWebRTCCodecsProxyMessages.h"
+#include "SharedBufferDataReference.h"
+#include "WebProcess.h"
+#include <WebCore/LibWebRTCMacros.h>
+#include <WebCore/RemoteVideoSample.h>
+#include <pal/cf/CoreMediaSoftLink.h>
+#include <webrtc/sdk/WebKit/WebKitUtilities.h>
+#include <wtf/MainThread.h>
+
+namespace WebKit {
+
+static webrtc::WebKitVideoDecoder createVideoDecoder(const webrtc::SdpVideoFormat& format)
+{
+    if (format.name != "H264")
+        return nullptr;
+
+    return WebProcess::singleton().libWebRTCCodecs().createDecoder();
+}
+
+static int32_t releaseVideoDecoder(webrtc::WebKitVideoDecoder decoder)
+{
+    return WebProcess::singleton().libWebRTCCodecs().releaseDecoder(*static_cast<LibWebRTCCodecs::Decoder*>(decoder));
+}
+
+static int32_t decodeVideoFrame(webrtc::WebKitVideoDecoder decoder, uint32_t timeStamp, const uint8_t* data, size_t size)
+{
+    return WebProcess::singleton().libWebRTCCodecs().decodeFrame(*static_cast<LibWebRTCCodecs::Decoder*>(decoder), timeStamp, data, size);
+}
+
+static int32_t registerDecodeCompleteCallback(webrtc::WebKitVideoDecoder decoder, void* decodedImageCallback)
+{
+    WebProcess::singleton().libWebRTCCodecs().registerDecodeFrameCallback(*static_cast<LibWebRTCCodecs::Decoder*>(decoder), decodedImageCallback);
+    return 0;
+}
+
+void LibWebRTCCodecs::setVideoDecoderCallbacks(bool useGPUProcess)
+{
+    ASSERT(isMainThread());
+
+    if (!useGPUProcess) {
+        webrtc::setVideoDecoderCallbacks(nullptr, nullptr, nullptr, nullptr);
+        return;
+    }
+    // Let's create WebProcess libWebRTCCodecs since it may be called from various threads.
+    WebProcess::singleton().libWebRTCCodecs();
+    webrtc::setVideoDecoderCallbacks(createVideoDecoder, releaseVideoDecoder, decodeVideoFrame, registerDecodeCompleteCallback);
+}
+
+LibWebRTCCodecs::Decoder* LibWebRTCCodecs::createDecoder()
+{
+    auto decoder = makeUnique<Decoder>();
+    auto* result = decoder.get();
+    decoder->identifier = RTCDecoderIdentifier::generateThreadSafe();
+
+    callOnMainRunLoop([this, decoder = WTFMove(decoder)]() mutable {
+        decoder->connection = &WebProcess::singleton().ensureGPUProcessConnection().connection();
+
+        auto decoderIdentifier = decoder->identifier;
+        decoder->connection->send(Messages::LibWebRTCCodecsProxy::CreateDecoder { decoderIdentifier }, 0);
+
+        ASSERT(!m_decodeCallbacks.contains(decoderIdentifier));
+        m_decodeCallbacks.add(decoderIdentifier, WTFMove(decoder));
+    });
+    return result;
+}
+
+int32_t LibWebRTCCodecs::releaseDecoder(Decoder& decoder)
+{
+    LockHolder holder(decoder.decodedImageCallbackLock);
+    decoder.decodedImageCallback = nullptr;
+
+    callOnMainRunLoop([this, decoderIdentifier = decoder.identifier] {
+        ASSERT(m_decodeCallbacks.contains(decoderIdentifier));
+
+        m_decodeCallbacks.remove(decoderIdentifier);
+        WebProcess::singleton().ensureGPUProcessConnection().connection().send(Messages::LibWebRTCCodecsProxy::ReleaseDecoder { decoderIdentifier }, 0);
+    });
+    return 0;
+}
+
+int32_t LibWebRTCCodecs::decodeFrame(Decoder& decoder, uint32_t timeStamp, const uint8_t* data, size_t size)
+{
+    if (!decoder.connection || decoder.hasError) {
+        decoder.hasError = false;
+        return WEBRTC_VIDEO_CODEC_ERROR;
+    }
+
+    decoder.connection->send(Messages::LibWebRTCCodecsProxy::DecodeFrame { decoder.identifier, timeStamp, IPC::DataReference { data, size } }, 0);
+    return WEBRTC_VIDEO_CODEC_OK;
+}
+
+void LibWebRTCCodecs::registerDecodeFrameCallback(Decoder& decoder, void* decodedImageCallback)
+{
+    LockHolder holder(decoder.decodedImageCallbackLock);
+    decoder.decodedImageCallback = decodedImageCallback;
+}
+
+void LibWebRTCCodecs::failedDecoding(RTCDecoderIdentifier decoderIdentifier)
+{
+    ASSERT(isMainThread());
+
+    if (auto* decoder = m_decodeCallbacks.get(decoderIdentifier))
+        decoder->hasError = true;
+}
+
+void LibWebRTCCodecs::completedDecoding(RTCDecoderIdentifier decoderIdentifier, uint32_t timeStamp, WebCore::RemoteVideoSample&& remoteSample)
+{
+    ASSERT(isMainThread());
+
+    // FIXME: Do error logging.
+    auto* decoder = m_decodeCallbacks.get(decoderIdentifier);
+    if (!decoder)
+        return;
+
+    auto locker = tryHoldLock(decoder->decodedImageCallbackLock);
+    if (!locker)
+        return;
+
+    if (!decoder->decodedImageCallback)
+        return;
+
+    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != remoteSample.videoFormat())
+        m_imageTransferSession = WebCore::ImageTransferSessionVT::create(remoteSample.videoFormat());
+
+    if (!m_imageTransferSession) {
+        ASSERT_NOT_REACHED();
+        return;
+    }
+
+    auto pixelBuffer = m_imageTransferSession->createPixelBuffer(remoteSample.surface(), remoteSample.size());
+    if (!pixelBuffer) {
+        ASSERT_NOT_REACHED();
+        return;
+    }
+
+    webrtc::RemoteVideoDecoder::decodeComplete(decoder->decodedImageCallback, timeStamp, pixelBuffer.get(), remoteSample.time().toDouble());
+}
+
+}
+
+#endif
diff --git a/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.h b/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.h
new file mode 100644 (file)
index 0000000..e3cd8a9
--- /dev/null
@@ -0,0 +1,84 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+
+#include "MessageReceiver.h"
+#include "RTCDecoderIdentifier.h"
+#include <WebCore/ImageTransferSessionVT.h>
+#include <wtf/HashMap.h>
+#include <wtf/HashSet.h>
+#include <wtf/Lock.h>
+
+namespace IPC {
+class Connection;
+class Decoder;
+}
+
+namespace WebCore {
+class RemoteVideoSample;
+}
+
+namespace WebKit {
+
+class LibWebRTCCodecs : private IPC::MessageReceiver {
+    WTF_MAKE_FAST_ALLOCATED;
+public:
+    LibWebRTCCodecs() = default;
+
+    struct Decoder {
+        WTF_MAKE_FAST_ALLOCATED;
+    public:
+        RTCDecoderIdentifier identifier;
+        void* decodedImageCallback { nullptr };
+        Lock decodedImageCallbackLock;
+        bool hasError { false };
+        RefPtr<IPC::Connection> connection;
+    };
+
+    static void setVideoDecoderCallbacks(bool useGPUProcess);
+
+    Decoder* createDecoder();
+    int32_t releaseDecoder(Decoder&);
+    int32_t decodeFrame(Decoder&, uint32_t timeStamp, const uint8_t*, size_t);
+    void registerDecodeFrameCallback(Decoder&, void* decodedImageCallback);
+
+    void didReceiveMessage(IPC::Connection&, IPC::Decoder&) final;
+
+private:
+    void failedDecoding(RTCDecoderIdentifier);
+    void completedDecoding(RTCDecoderIdentifier, uint32_t timeStamp, WebCore::RemoteVideoSample&&);
+
+    HashMap<RTCDecoderIdentifier, std::unique_ptr<Decoder>> m_decodeCallbacks;
+    HashSet<RTCDecoderIdentifier> m_decodingErrors;
+
+    std::unique_ptr<WebCore::ImageTransferSessionVT> m_imageTransferSession;
+};
+
+} // namespace WebKit
+
+#endif
diff --git a/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in b/Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in
new file mode 100644 (file)
index 0000000..2aae978
--- /dev/null
@@ -0,0 +1,30 @@
+# Copyright (C) 2020 Apple Inc. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1.  Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+# 2.  Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in the
+#     documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+
+messages -> LibWebRTCCodecs NotRefCounted {
+    FailedDecoding(WebKit::RTCDecoderIdentifier id)
+    CompletedDecoding(WebKit::RTCDecoderIdentifier id, uint32_t timeStamp, WebCore::RemoteVideoSample sample)
+}
+
+#endif // USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
diff --git a/Source/WebKit/WebProcess/GPU/webrtc/RTCDecoderIdentifier.h b/Source/WebKit/WebProcess/GPU/webrtc/RTCDecoderIdentifier.h
new file mode 100644 (file)
index 0000000..cdaa3bc
--- /dev/null
@@ -0,0 +1,35 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include <wtf/ObjectIdentifier.h>
+
+namespace WebKit {
+
+enum RTCDecoderIdentifierType { };
+using RTCDecoderIdentifier = ObjectIdentifier<RTCDecoderIdentifierType>;
+
+} // namespace WebKit
index 468df6c..f983622 100644 (file)
 
 #if USE(LIBWEBRTC)
 
+#if ENABLE(GPU_PROCESS) && PLATFORM(COCOA)
+#include "LibWebRTCCodecs.h"
+#endif
+
 #include "LibWebRTCNetwork.h"
 #include "WebProcess.h"
 #include <webrtc/api/async_resolver_factory.h>
@@ -131,6 +135,19 @@ std::unique_ptr<LibWebRTCProvider::SuspendableSocketFactory> LibWebRTCProvider::
     return makeUnique<RTCSocketFactory>(WTFMove(userAgent));
 }
 
+#if PLATFORM(COCOA)
+std::unique_ptr<webrtc::VideoDecoderFactory> LibWebRTCProvider::createDecoderFactory()
+{
+#if ENABLE(GPU_PROCESS)
+    // We only support efficient sending of video frames with IOSURFACE
+#if HAVE(IOSURFACE) && !PLATFORM(MACCATALYST)
+    LibWebRTCCodecs::setVideoDecoderCallbacks(m_useGPUProcess);
+#endif
+#endif
+    return LibWebRTCProviderCocoa::createDecoderFactory();
+}
+#endif
+
 } // namespace WebKit
 
 #endif // USE(LIBWEBRTC)
index 0248eed..80f7525 100644 (file)
@@ -57,6 +57,10 @@ private:
     void unregisterMDNSNames(uint64_t documentIdentifier) final;
     void registerMDNSName(uint64_t documentIdentifier, const String& ipAddress, CompletionHandler<void(MDNSNameOrError&&)>&&) final;
     void disableNonLocalhostConnections() final;
+
+#if PLATFORM(COCOA)
+    std::unique_ptr<webrtc::VideoDecoderFactory> createDecoderFactory() final;
+#endif
 };
 #else
 using LibWebRTCProvider = WebCore::LibWebRTCProvider;
index 1513271..c4ad39f 100644 (file)
 #include "RemoteMediaPlayerManager.h"
 #endif
 
+#if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
+#include "LibWebRTCCodecs.h"
+#endif
+
 // This should be less than plugInAutoStartExpirationTimeThreshold in PlugInAutoStartProvider.
 static const Seconds plugInAutoStartExpirationTimeUpdateThreshold { 29 * 24 * 60 * 60 };
 
@@ -1310,6 +1314,15 @@ void WebProcess::gpuProcessConnectionClosed(GPUProcessConnection* connection)
     m_gpuProcessConnection = nullptr;
 }
 
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+LibWebRTCCodecs& WebProcess::libWebRTCCodecs()
+{
+    if (!m_libWebRTCCodecs)
+        m_libWebRTCCodecs = makeUnique<LibWebRTCCodecs>();
+    return *m_libWebRTCCodecs;
+}
+#endif
+
 #endif // ENABLE(GPU_PROCESS)
 
 void WebProcess::setEnhancedAccessibility(bool flag)
index c2ce88b..8f9bfe5 100644 (file)
@@ -109,6 +109,7 @@ class EventDispatcher;
 class GamepadData;
 class GPUProcessConnection;
 class InjectedBundle;
+class LibWebRTCCodecs;
 class LibWebRTCNetwork;
 class NetworkProcessConnection;
 class ObjCObjectGraph;
@@ -218,6 +219,11 @@ public:
     GPUProcessConnection& ensureGPUProcessConnection();
     void gpuProcessConnectionClosed(GPUProcessConnection*);
     GPUProcessConnection* existingGPUProcessConnection() { return m_gpuProcessConnection.get(); }
+
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+    LibWebRTCCodecs& libWebRTCCodecs();
+#endif
+
 #endif // ENABLE(GPU_PROCESS)
 
     LibWebRTCNetwork& libWebRTCNetwork();
@@ -523,6 +529,9 @@ private:
 
 #if ENABLE(GPU_PROCESS)
     RefPtr<GPUProcessConnection> m_gpuProcessConnection;
+#if PLATFORM(COCOA) && USE(LIBWEBRTC)
+    std::unique_ptr<LibWebRTCCodecs> m_libWebRTCCodecs;
+#endif
 #endif
 
     Ref<WebCacheStorageProvider> m_cacheStorageProvider;