Implement RTC VTB encoders in GPUProcess
authoryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 6 Jan 2020 09:47:37 +0000 (09:47 +0000)
committeryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 6 Jan 2020 09:47:37 +0000 (09:47 +0000)
https://bugs.webkit.org/show_bug.cgi?id=205713

Reviewed by Eric Carlson.

Source/ThirdParty/libwebrtc:

Add support for remote video encoders created and used through simple routines.
Add factory to create video encoders implemented elsewhere than in WebRTC backend.
This is used for H264 encoders.

* Configurations/libwebrtc.iOS.exp:
* Configurations/libwebrtc.iOSsim.exp:
* Configurations/libwebrtc.mac.exp:
* Source/webrtc/sdk/WebKit/WebKitEncoder.h: Added.
(webrtc::WebKitRTPFragmentationHeader::value):
(webrtc::WebKitEncodedFrameInfo::decode):
(webrtc::WebKitEncodedFrameInfo::encode const):
(webrtc::WebKitRTPFragmentationHeader::WebKitRTPFragmentationHeader):
(webrtc::WebKitRTPFragmentationHeader::encode const):
(webrtc::WebKitRTPFragmentationHeader::decode):
* Source/webrtc/sdk/WebKit/WebKitEncoder.mm: Added.
(webrtc::VideoEncoderFactoryWithSimulcast::CreateVideoEncoder):
(webrtc::videoEncoderCallbacks):
(webrtc::setVideoEncoderCallbacks):
(webrtc::RemoteVideoEncoder::RemoteVideoEncoder):
(webrtc::RemoteVideoEncoder::InitEncode):
(webrtc::RemoteVideoEncoder::Release):
(webrtc::RemoteVideoEncoder::Encode):
(webrtc::RemoteVideoEncoder::SetRates):
(webrtc::RemoteVideoEncoder::GetEncoderInfo const):
(webrtc::RemoteVideoEncoder::RegisterEncodeCompleteCallback):
(webrtc::RemoteVideoEncoder::encodeComplete):
(webrtc::createLocalEncoder):
(webrtc::releaseLocalEncoder):
(webrtc::initializeLocalEncoder):
(webrtc::encodeLocalEncoderFrame):
(webrtc::setLocalEncoderRates):
* Source/webrtc/sdk/WebKit/WebKitUtilities.h:
* Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
* libwebrtc.xcodeproj/project.pbxproj:

Source/WebCore:

Refactoring to expose a createPixelBuffer routine, to set/get RemoteVideoSample and to create a CVPixelBuffer from an IOSurfaceRef.

Covered by existing tests, in particular webrtc/video-gpuProcess.html.

* SourcesCocoa.txt:
* WebCore.xcodeproj/project.pbxproj:
* platform/graphics/RemoteVideoSample.cpp:
(WebCore::RemoteVideoSample::create):
* platform/graphics/RemoteVideoSample.h:
(WebCore::RemoteVideoSample::rotation const):
* platform/graphics/cv/ImageTransferSessionVT.h:
* platform/graphics/cv/ImageTransferSessionVT.mm:
(WebCore::ImageTransferSessionVT::createPixelBuffer):
* platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
(WebCore::RealtimeIncomingVideoSourceCocoa::pixelBufferPool):
* platform/mediastream/mac/RealtimeVideoUtilities.h:
* platform/mediastream/mac/RealtimeVideoUtilities.mm: Added.
(WebCore::createPixelBufferPool):

Source/WebKit:

Add support for remote video encoders in GPUProcess based on libwebrtc exposed routines.
WebProcess sets callbacks in the webrtc backend to implement H264 encoding as a remote video encoder.
In addition to video frames, we also need to send rate setting orders.

* Configurations/WebKit.xcconfig:
* GPUProcess/webrtc/LibWebRTCCodecsProxy.h:
* GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in:
* GPUProcess/webrtc/LibWebRTCCodecsProxy.mm:
(WebKit::LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy):
(WebKit::LibWebRTCCodecsProxy::createEncoder):
(WebKit::LibWebRTCCodecsProxy::releaseEncoder):
(WebKit::LibWebRTCCodecsProxy::initializeEncoder):
(WebKit::toWebRTCVideoRotation):
(WebKit::LibWebRTCCodecsProxy::encodeFrame):
(WebKit::LibWebRTCCodecsProxy::setEncodeRates):
* Scripts/webkit/messages.py:
* WebKit.xcodeproj/project.pbxproj:
* WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp:
(WebKit::createVideoEncoder):
(WebKit::releaseVideoEncoder):
(WebKit::initializeVideoEncoder):
(WebKit::toMediaSampleVideoRotation):
(WebKit::encodeVideoFrame):
(WebKit::registerEncodeCompleteCallback):
(WebKit::setEncodeRatesCallback):
(WebKit::LibWebRTCCodecs::setCallbacks):
(WebKit::LibWebRTCCodecs::createDecoder):
(WebKit::LibWebRTCCodecs::releaseDecoder):
(WebKit::LibWebRTCCodecs::failedDecoding):
(WebKit::LibWebRTCCodecs::completedDecoding):
(WebKit::LibWebRTCCodecs::createEncoder):
(WebKit::LibWebRTCCodecs::releaseEncoder):
(WebKit::LibWebRTCCodecs::initializeEncoder):
(WebKit::LibWebRTCCodecs::encodeFrame):
(WebKit::LibWebRTCCodecs::registerEncodeFrameCallback):
(WebKit::LibWebRTCCodecs::setEncodeRates):
(WebKit::LibWebRTCCodecs::completedEncoding):
(WebKit::LibWebRTCCodecs::pixelBufferPool):
* WebProcess/GPU/webrtc/LibWebRTCCodecs.h:
* WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in:
* WebProcess/GPU/webrtc/RTCEncoderIdentifier.h: Added.
* WebProcess/Network/webrtc/LibWebRTCProvider.cpp:
(WebKit::LibWebRTCProvider::createDecoderFactory):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@254050 268f45cc-cd09-0410-ab3c-d52691b4dbfc

30 files changed:
Source/ThirdParty/libwebrtc/ChangeLog
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOS.exp
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOSsim.exp
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.mac.exp
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitEncoder.h [new file with mode: 0644]
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitEncoder.mm [new file with mode: 0644]
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.h
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitUtilities.mm
Source/ThirdParty/libwebrtc/libwebrtc.xcodeproj/project.pbxproj
Source/WebCore/ChangeLog
Source/WebCore/SourcesCocoa.txt
Source/WebCore/WebCore.xcodeproj/project.pbxproj
Source/WebCore/platform/graphics/RemoteVideoSample.cpp
Source/WebCore/platform/graphics/RemoteVideoSample.h
Source/WebCore/platform/graphics/cv/ImageTransferSessionVT.h
Source/WebCore/platform/graphics/cv/ImageTransferSessionVT.mm
Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm
Source/WebCore/platform/mediastream/mac/RealtimeVideoUtilities.h
Source/WebCore/platform/mediastream/mac/RealtimeVideoUtilities.mm [new file with mode: 0644]
Source/WebKit/ChangeLog
Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.h
Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in
Source/WebKit/GPUProcess/webrtc/LibWebRTCCodecsProxy.mm
Source/WebKit/Scripts/webkit/messages.py
Source/WebKit/WebKit.xcodeproj/project.pbxproj
Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp
Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.h
Source/WebKit/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in
Source/WebKit/WebProcess/GPU/webrtc/RTCEncoderIdentifier.h [new file with mode: 0644]
Source/WebKit/WebProcess/Network/webrtc/LibWebRTCProvider.cpp

index 1d825c4..d202b42 100644 (file)
@@ -1,3 +1,45 @@
+2020-01-06  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB encoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205713
+
+        Reviewed by Eric Carlson.
+
+        Add support for remote video encoders created and used through simple routines.
+        Add factory to create video encoders implemented elsewhere than in WebRTC backend.
+        This is used for H264 encoders.
+
+        * Configurations/libwebrtc.iOS.exp:
+        * Configurations/libwebrtc.iOSsim.exp:
+        * Configurations/libwebrtc.mac.exp:
+        * Source/webrtc/sdk/WebKit/WebKitEncoder.h: Added.
+        (webrtc::WebKitRTPFragmentationHeader::value):
+        (webrtc::WebKitEncodedFrameInfo::decode):
+        (webrtc::WebKitEncodedFrameInfo::encode const):
+        (webrtc::WebKitRTPFragmentationHeader::WebKitRTPFragmentationHeader):
+        (webrtc::WebKitRTPFragmentationHeader::encode const):
+        (webrtc::WebKitRTPFragmentationHeader::decode):
+        * Source/webrtc/sdk/WebKit/WebKitEncoder.mm: Added.
+        (webrtc::VideoEncoderFactoryWithSimulcast::CreateVideoEncoder):
+        (webrtc::videoEncoderCallbacks):
+        (webrtc::setVideoEncoderCallbacks):
+        (webrtc::RemoteVideoEncoder::RemoteVideoEncoder):
+        (webrtc::RemoteVideoEncoder::InitEncode):
+        (webrtc::RemoteVideoEncoder::Release):
+        (webrtc::RemoteVideoEncoder::Encode):
+        (webrtc::RemoteVideoEncoder::SetRates):
+        (webrtc::RemoteVideoEncoder::GetEncoderInfo const):
+        (webrtc::RemoteVideoEncoder::RegisterEncodeCompleteCallback):
+        (webrtc::RemoteVideoEncoder::encodeComplete):
+        (webrtc::createLocalEncoder):
+        (webrtc::releaseLocalEncoder):
+        (webrtc::initializeLocalEncoder):
+        (webrtc::encodeLocalEncoderFrame):
+        (webrtc::setLocalEncoderRates):
+        * Source/webrtc/sdk/WebKit/WebKitUtilities.h:
+        * Source/webrtc/sdk/WebKit/WebKitUtilities.mm:
+        * libwebrtc.xcodeproj/project.pbxproj:
+
 2020-01-01  youenn fablet  <youenn@apple.com>
 
         Implement transceiver setCodecPreferences
index ea789e2..53e3547 100644 (file)
@@ -268,3 +268,17 @@ __ZN6webrtc11decodeFrameEPvjPKhm
 __ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
 __ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
 __ZN6webrtc18RtpCodecCapabilityC1Ev
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
+__ZN6webrtc18RtpCodecCapabilityC1Ev
+__ZN6webrtc22RTPFragmentationHeaderD1Ev
+__ZN6webrtc22RTPFragmentationHeaderC1Ev
+__ZN6webrtc18RemoteVideoEncoder14encodeCompleteEPvPhmRKNS_22WebKitEncodedFrameInfoEPKNS_22RTPFragmentationHeaderE
+__ZN6webrtc22RTPFragmentationHeader6ResizeEm
+__ZN6webrtc24setVideoEncoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_RKNS_10VideoCodecEEPFiS0_RKNS_10VideoFrameEbEPFiS0_S0_EPFvS0_RKNS_12VideoEncoder21RateControlParametersEE
+__ZN6webrtc20setLocalEncoderRatesEPvjj
+__ZN6webrtc22initializeLocalEncoderEPvttjjjj
+__ZN6webrtc14SdpVideoFormatC1ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEE
+__ZN6webrtc18createLocalEncoderERKNS_14SdpVideoFormatEU13block_pointerFvPKhmRKNS_22WebKitEncodedFrameInfoEPNS_22RTPFragmentationHeaderEE
+__ZN6webrtc14SdpVideoFormatD1Ev
+__ZN6webrtc19releaseLocalEncoderEPv
+__ZN6webrtc23encodeLocalEncoderFrameEPvP10__CVBufferxNS_13VideoRotationEb
index ea789e2..53e3547 100644 (file)
@@ -268,3 +268,17 @@ __ZN6webrtc11decodeFrameEPvjPKhm
 __ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
 __ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
 __ZN6webrtc18RtpCodecCapabilityC1Ev
+__ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
+__ZN6webrtc18RtpCodecCapabilityC1Ev
+__ZN6webrtc22RTPFragmentationHeaderD1Ev
+__ZN6webrtc22RTPFragmentationHeaderC1Ev
+__ZN6webrtc18RemoteVideoEncoder14encodeCompleteEPvPhmRKNS_22WebKitEncodedFrameInfoEPKNS_22RTPFragmentationHeaderE
+__ZN6webrtc22RTPFragmentationHeader6ResizeEm
+__ZN6webrtc24setVideoEncoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_RKNS_10VideoCodecEEPFiS0_RKNS_10VideoFrameEbEPFiS0_S0_EPFvS0_RKNS_12VideoEncoder21RateControlParametersEE
+__ZN6webrtc20setLocalEncoderRatesEPvjj
+__ZN6webrtc22initializeLocalEncoderEPvttjjjj
+__ZN6webrtc14SdpVideoFormatC1ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEE
+__ZN6webrtc18createLocalEncoderERKNS_14SdpVideoFormatEU13block_pointerFvPKhmRKNS_22WebKitEncodedFrameInfoEPNS_22RTPFragmentationHeaderEE
+__ZN6webrtc14SdpVideoFormatD1Ev
+__ZN6webrtc19releaseLocalEncoderEPv
+__ZN6webrtc23encodeLocalEncoderFrameEPvP10__CVBufferxNS_13VideoRotationEb
index ea789e2..1aaa600 100644 (file)
@@ -268,3 +268,15 @@ __ZN6webrtc11decodeFrameEPvjPKhm
 __ZN6webrtc18RemoteVideoDecoder14decodeCompleteEPvjP10__CVBufferj
 __ZN6webrtc24setVideoDecoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_jPKhmEPFiS0_S0_E
 __ZN6webrtc18RtpCodecCapabilityC1Ev
+__ZN6webrtc22RTPFragmentationHeaderD1Ev
+__ZN6webrtc22RTPFragmentationHeaderC1Ev
+__ZN6webrtc18RemoteVideoEncoder14encodeCompleteEPvPhmRKNS_22WebKitEncodedFrameInfoEPKNS_22RTPFragmentationHeaderE
+__ZN6webrtc22RTPFragmentationHeader6ResizeEm
+__ZN6webrtc24setVideoEncoderCallbacksEPFPvRKNS_14SdpVideoFormatEEPFiS0_EPFiS0_RKNS_10VideoCodecEEPFiS0_RKNS_10VideoFrameEbEPFiS0_S0_EPFvS0_RKNS_12VideoEncoder21RateControlParametersEE
+__ZN6webrtc20setLocalEncoderRatesEPvjj
+__ZN6webrtc22initializeLocalEncoderEPvttjjjj
+__ZN6webrtc14SdpVideoFormatC1ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS1_3mapIS7_S7_NS1_4lessIS7_EENS5_INS1_4pairIS8_S7_EEEEEE
+__ZN6webrtc18createLocalEncoderERKNS_14SdpVideoFormatEU13block_pointerFvPKhmRKNS_22WebKitEncodedFrameInfoEPNS_22RTPFragmentationHeaderEE
+__ZN6webrtc14SdpVideoFormatD1Ev
+__ZN6webrtc19releaseLocalEncoderEPv
+__ZN6webrtc23encodeLocalEncoderFrameEPvP10__CVBufferxNS_13VideoRotationEb
diff --git a/Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitEncoder.h b/Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitEncoder.h
new file mode 100644 (file)
index 0000000..a20a254
--- /dev/null
@@ -0,0 +1,256 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "api/video_codecs/video_encoder.h"
+#include "api/video_codecs/video_encoder_factory.h"
+#include "api/video/encoded_image.h"
+#include "modules/include/module_common_types.h"
+
+using CVPixelBufferRef = struct __CVBuffer*;
+
+namespace webrtc {
+
+class EncodedImageCallback;
+class VideoCodec;
+class VideoFrame;
+struct SdpVideoFormat;
+class Settings;
+
+class VideoEncoderFactoryWithSimulcast final : public VideoEncoderFactory {
+public:
+    explicit VideoEncoderFactoryWithSimulcast(std::unique_ptr<VideoEncoderFactory>&& factory)
+        : m_internalEncoderFactory(std::move(factory))
+    {
+    }
+
+    VideoEncoderFactory::CodecInfo QueryVideoEncoder(const SdpVideoFormat& format) const final { return m_internalEncoderFactory->QueryVideoEncoder(format); }
+
+    std::unique_ptr<VideoEncoder> CreateVideoEncoder(const SdpVideoFormat& format) final;
+    std::vector<SdpVideoFormat> GetSupportedFormats() const final { return m_internalEncoderFactory->GetSupportedFormats(); }
+
+private:
+    const std::unique_ptr<VideoEncoderFactory> m_internalEncoderFactory;
+};
+
+using WebKitVideoEncoder = void*;
+using VideoEncoderCreateCallback = WebKitVideoEncoder(*)(const SdpVideoFormat& format);
+using VideoEncoderReleaseCallback = int32_t(*)(WebKitVideoEncoder);
+using VideoEncoderInitializeCallback = int32_t(*)(WebKitVideoEncoder, const VideoCodec&);
+using VideoEncoderEncodeCallback = int32_t(*)(WebKitVideoEncoder, const VideoFrame&, bool shouldEncodeAsKeyFrame);
+using VideoEncoderRegisterEncodeCompleteCallback = int32_t(*)(WebKitVideoEncoder, void* encodedImageCallback);
+using VideoEncoderSetRatesCallback = void(*)(WebKitVideoEncoder, const VideoEncoder::RateControlParameters&);
+
+void setVideoEncoderCallbacks(VideoEncoderCreateCallback, VideoEncoderReleaseCallback, VideoEncoderInitializeCallback, VideoEncoderEncodeCallback, VideoEncoderRegisterEncodeCompleteCallback, VideoEncoderSetRatesCallback);
+
+using WebKitEncodedFrameTiming = EncodedImage::Timing;
+struct WebKitEncodedFrameInfo {
+    uint32_t width { 0 };
+    uint32_t height { 0 };
+    int64_t timeStamp { 0 };
+    int64_t ntpTimeMS { 0 };
+    int64_t captureTimeMS { 0 };
+    VideoFrameType frameType { VideoFrameType::kVideoFrameDelta };
+    VideoRotation rotation { kVideoRotation_0 };
+    VideoContentType contentType { VideoContentType::UNSPECIFIED };
+    bool completeFrame = false;
+    int qp { -1 };
+    WebKitEncodedFrameTiming timing;
+
+    template<class Encoder> void encode(Encoder&) const;
+    template<class Decoder> static bool decode(Decoder&, WebKitEncodedFrameInfo&);
+};
+
+class WebKitRTPFragmentationHeader {
+public:
+    explicit WebKitRTPFragmentationHeader(webrtc::RTPFragmentationHeader* = nullptr);
+
+    template<class Encoder> void encode(Encoder&) const;
+    template<class Decoder> static bool decode(Decoder&, WebKitRTPFragmentationHeader&);
+
+    webrtc::RTPFragmentationHeader* value() { return m_value; };
+
+private:
+    webrtc::RTPFragmentationHeader* m_value { nullptr };
+    std::unique_ptr<webrtc::RTPFragmentationHeader> m_ownedHeader;
+};
+
+class RemoteVideoEncoder final : public webrtc::VideoEncoder {
+public:
+    explicit RemoteVideoEncoder(WebKitVideoEncoder);
+    ~RemoteVideoEncoder() = default;
+
+    static void encodeComplete(void* callback, uint8_t* buffer, size_t length, const WebKitEncodedFrameInfo&, const webrtc::RTPFragmentationHeader*);
+
+private:
+    int32_t InitEncode(const VideoCodec*, const Settings&) final;
+    int32_t RegisterEncodeCompleteCallback(EncodedImageCallback*) final;
+    int32_t Release() final;
+    int32_t Encode(const VideoFrame&, const std::vector<VideoFrameType>*) final;
+    void SetRates(const RateControlParameters&) final;
+    EncoderInfo GetEncoderInfo() const final;
+
+    WebKitVideoEncoder m_internalEncoder;
+};
+
+using LocalEncoder = void*;
+using LocalEncoderCallback = void (^)(const uint8_t* buffer, size_t size, const webrtc::WebKitEncodedFrameInfo&, webrtc::RTPFragmentationHeader*);
+void* createLocalEncoder(const webrtc::SdpVideoFormat&, LocalEncoderCallback);
+void releaseLocalEncoder(LocalEncoder);
+void initializeLocalEncoder(LocalEncoder, uint16_t width, uint16_t height, unsigned int startBitrate, unsigned int maxBitrate, unsigned int minBitrate, uint32_t maxFramerate);
+void encodeLocalEncoderFrame(LocalEncoder, CVPixelBufferRef, int64_t timeStamp, webrtc::VideoRotation, bool isKeyframeRequired);
+void setLocalEncoderRates(LocalEncoder, uint32_t bitRate, uint32_t frameRate);
+
+template<class Decoder>
+bool WebKitEncodedFrameInfo::decode(Decoder& decoder, WebKitEncodedFrameInfo& info)
+{
+    if (!decoder.decode(info.width))
+        return false;
+    if (!decoder.decode(info.height))
+        return false;
+    if (!decoder.decode(info.timeStamp))
+        return false;
+    if (!decoder.decode(info.ntpTimeMS))
+        return false;
+    if (!decoder.decode(info.captureTimeMS))
+        return false;
+    if (!decoder.decodeEnum(info.frameType))
+        return false;
+    if (!decoder.decodeEnum(info.rotation))
+        return false;
+    if (!decoder.decodeEnum(info.contentType))
+        return false;
+    if (!decoder.decode(info.completeFrame))
+        return false;
+    if (!decoder.decode(info.qp))
+        return false;
+
+    if (!decoder.decode(info.timing.flags))
+        return false;
+
+    if (!decoder.decode(info.timing.encode_start_ms))
+        return false;
+
+    if (!decoder.decode(info.timing.encode_finish_ms))
+        return false;
+
+    if (!decoder.decode(info.timing.packetization_finish_ms))
+        return false;
+
+    if (!decoder.decode(info.timing.pacer_exit_ms))
+        return false;
+
+    if (!decoder.decode(info.timing.network_timestamp_ms))
+        return false;
+
+    if (!decoder.decode(info.timing.network2_timestamp_ms))
+        return false;
+
+    if (!decoder.decode(info.timing.receive_start_ms))
+        return false;
+
+    return true;
+}
+
+template<class Encoder>
+void WebKitEncodedFrameInfo::encode(Encoder& encoder) const
+{
+    encoder << width;
+    encoder << height;
+    encoder << timeStamp;
+    encoder << ntpTimeMS;
+    encoder << captureTimeMS;
+    encoder.encodeEnum(frameType);
+    encoder.encodeEnum(rotation);
+    encoder.encodeEnum(contentType);
+    encoder << completeFrame;
+    encoder << qp;
+
+    encoder << timing.flags;
+    encoder << timing.encode_start_ms;
+    encoder << timing.encode_finish_ms;
+    encoder << timing.packetization_finish_ms;
+    encoder << timing.pacer_exit_ms;
+    encoder << timing.network_timestamp_ms;
+    encoder << timing.network2_timestamp_ms;
+    encoder << timing.receive_start_ms;
+    encoder << timing.receive_finish_ms;
+}
+
+inline WebKitRTPFragmentationHeader::WebKitRTPFragmentationHeader(webrtc::RTPFragmentationHeader* header)
+    : m_value(header)
+{
+}
+
+template<class Encoder>
+void WebKitRTPFragmentationHeader::encode(Encoder& encoder) const
+{
+    encoder << !!m_value;
+    if (!m_value)
+        return;
+    encoder << static_cast<unsigned>(m_value->Size());
+    for (unsigned i = 0; i < m_value->Size(); ++i) {
+        encoder << static_cast<unsigned>(m_value->Offset(i));
+        encoder << static_cast<unsigned>(m_value->Length(i));
+    }
+}
+
+template<class Decoder>
+bool WebKitRTPFragmentationHeader::decode(Decoder& decoder, WebKitRTPFragmentationHeader& header)
+{
+    bool isNull;
+    if (!decoder.decode(isNull))
+        return false;
+    
+    if (isNull) {
+        header.m_value = nullptr;
+        return true;
+    }
+
+    unsigned size;
+    if (!decoder.decode(size))
+        return false;
+
+    header.m_ownedHeader = std::make_unique<webrtc::RTPFragmentationHeader>();
+    header.m_value = header.m_ownedHeader.get();
+
+    header.m_value->VerifyAndAllocateFragmentationHeader(size);
+    for (size_t i = 0; i < header.m_value->Size(); ++i) {
+        unsigned offset, length;
+        if (!decoder.decode(offset))
+            return false;
+        if (!decoder.decode(length))
+            return false;
+
+        header.m_value->fragmentationOffset[i] = offset;
+        header.m_value->fragmentationLength[i] = length;
+    }
+
+    return true;
+}
+
+}
diff --git a/Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitEncoder.mm b/Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/WebKitEncoder.mm
new file mode 100644 (file)
index 0000000..27616a0
--- /dev/null
@@ -0,0 +1,216 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "WebKitEncoder.h"
+
+#include "WebKitUtilities.h"
+#include "api/video/video_frame.h"
+#include "components/video_codec/RTCCodecSpecificInfoH264+Private.h"
+#include "modules/video_coding/utility/simulcast_utility.h"
+#include "sdk/objc/api/peerconnection/RTCEncodedImage+Private.h"
+#include "sdk/objc/api/peerconnection/RTCRtpFragmentationHeader+Private.h"
+#include "sdk/objc/api/peerconnection/RTCVideoCodecInfo+Private.h"
+#include "sdk/objc/api/peerconnection/RTCVideoEncoderSettings+Private.h"
+#include "sdk/objc/components/video_codec/RTCVideoEncoderH264.h"
+#include "sdk/objc/native/src/objc_frame_buffer.h"
+
+namespace webrtc {
+
+std::unique_ptr<VideoEncoder> VideoEncoderFactoryWithSimulcast::CreateVideoEncoder(const SdpVideoFormat& format)
+{
+    return std::make_unique<EncoderSimulcastProxy>(m_internalEncoderFactory.get(), format);
+}
+
+struct VideoEncoderCallbacks {
+    VideoEncoderCreateCallback createCallback;
+    VideoEncoderReleaseCallback releaseCallback;
+    VideoEncoderInitializeCallback initializeCallback;
+    VideoEncoderEncodeCallback encodeCallback;
+    VideoEncoderRegisterEncodeCompleteCallback registerEncodeCompleteCallback;
+    VideoEncoderSetRatesCallback setRatesCallback;
+};
+
+static VideoEncoderCallbacks& videoEncoderCallbacks() {
+    static VideoEncoderCallbacks callbacks;
+    return callbacks;
+}
+
+void setVideoEncoderCallbacks(VideoEncoderCreateCallback createCallback, VideoEncoderReleaseCallback releaseCallback, VideoEncoderInitializeCallback initializeCallback, VideoEncoderEncodeCallback encodeCallback, VideoEncoderRegisterEncodeCompleteCallback registerEncodeCompleteCallback, VideoEncoderSetRatesCallback setRatesCallback)
+{
+    auto& callbacks = videoEncoderCallbacks();
+    callbacks.createCallback = createCallback;
+    callbacks.releaseCallback = releaseCallback;
+    callbacks.initializeCallback = initializeCallback;
+    callbacks.encodeCallback = encodeCallback;
+    callbacks.registerEncodeCompleteCallback = registerEncodeCompleteCallback;
+    callbacks.setRatesCallback = setRatesCallback;
+}
+
+RemoteVideoEncoder::RemoteVideoEncoder(WebKitVideoEncoder internalEncoder)
+    : m_internalEncoder(internalEncoder)
+{
+}
+
+int32_t RemoteVideoEncoder::InitEncode(const VideoCodec* codec, const Settings&)
+{
+    if (SimulcastUtility::NumberOfSimulcastStreams(*codec) > 1)
+        return WEBRTC_VIDEO_CODEC_ERR_SIMULCAST_PARAMETERS_NOT_SUPPORTED;
+
+    return videoEncoderCallbacks().initializeCallback(m_internalEncoder, *codec);
+}
+
+int32_t RemoteVideoEncoder::Release()
+{
+    return videoEncoderCallbacks().releaseCallback(m_internalEncoder);
+}
+
+int32_t RemoteVideoEncoder::Encode(const VideoFrame& frame, const std::vector<VideoFrameType>* frameTypes)
+{
+    bool shouldEncodeKeyFrame = false;
+    for (size_t i = 0; i < frameTypes->size(); ++i) {
+        if (frameTypes->at(i) == VideoFrameType::kVideoFrameKey) {
+            shouldEncodeKeyFrame = true;
+            break;
+        }
+    }
+
+    return videoEncoderCallbacks().encodeCallback(m_internalEncoder, frame, shouldEncodeKeyFrame);
+}
+
+void RemoteVideoEncoder::SetRates(const RateControlParameters& rates)
+{
+    videoEncoderCallbacks().setRatesCallback(m_internalEncoder, rates);
+}
+
+RemoteVideoEncoder::EncoderInfo RemoteVideoEncoder::GetEncoderInfo() const
+{
+    EncoderInfo info;
+    info.supports_native_handle = true;
+    info.implementation_name = "RemoteVideoToolBox";
+    info.is_hardware_accelerated = true;
+    info.has_internal_source = false;
+
+    // Values taken from RTCVideoEncoderH264.mm
+    const int kLowH264QpThreshold = 28;
+    const int kHighH264QpThreshold = 39;
+    info.scaling_settings = ScalingSettings(kLowH264QpThreshold, kHighH264QpThreshold);
+
+    return info;
+}
+
+int32_t RemoteVideoEncoder::RegisterEncodeCompleteCallback(EncodedImageCallback* callback)
+{
+    return videoEncoderCallbacks().registerEncodeCompleteCallback(m_internalEncoder, callback);
+}
+
+void RemoteVideoEncoder::encodeComplete(void* callback, uint8_t* buffer, size_t length, const WebKitEncodedFrameInfo& info, const webrtc::RTPFragmentationHeader* header)
+{
+    webrtc::EncodedImage encodedImage(buffer, length, length);
+    encodedImage._encodedWidth = info.width;
+    encodedImage._encodedHeight = info.height;
+    encodedImage.SetTimestamp(info.timeStamp);
+    encodedImage.capture_time_ms_ = info.captureTimeMS;
+    encodedImage.ntp_time_ms_ = info.ntpTimeMS;
+    encodedImage.timing_ = info.timing;
+    encodedImage._frameType = info.frameType;
+    encodedImage.rotation_ = info.rotation;
+    encodedImage._completeFrame = info.completeFrame;
+    encodedImage.qp_ = info.qp;
+    encodedImage.content_type_ = info.contentType;
+
+    CodecSpecificInfo codecSpecificInfo;
+    codecSpecificInfo.codecType = webrtc::kVideoCodecH264;
+    codecSpecificInfo.codecSpecific.H264.packetization_mode = H264PacketizationMode::NonInterleaved;
+
+    static_cast<EncodedImageCallback*>(callback)->OnEncodedImage(encodedImage, &codecSpecificInfo, header);
+}
+
+void* createLocalEncoder(const webrtc::SdpVideoFormat& format, LocalEncoderCallback callback)
+{
+    auto *codecInfo = [[RTCVideoCodecInfo alloc] initWithNativeSdpVideoFormat: format];
+    auto *encoder = [[RTCVideoEncoderH264 alloc] initWithCodecInfo:codecInfo];
+    
+    [encoder setCallback:^BOOL(RTCEncodedImage *_Nonnull frame, id<RTCCodecSpecificInfo> _Nonnull codecSpecificInfo,  RTCRtpFragmentationHeader *_Nonnull header) {
+        EncodedImage encodedImage = [frame nativeEncodedImage];
+
+        WebKitEncodedFrameInfo info;
+        info.width = encodedImage._encodedWidth;
+        info.height = encodedImage._encodedHeight;
+        info.timeStamp = encodedImage.Timestamp();
+        info.ntpTimeMS = encodedImage.ntp_time_ms_;
+        info.captureTimeMS = encodedImage.capture_time_ms_;
+        info.frameType = encodedImage._frameType;
+        info.rotation = encodedImage.rotation_;
+        info.contentType = encodedImage.content_type_;
+        info.completeFrame = encodedImage._completeFrame;
+        info.qp = encodedImage.qp_;
+        info.timing = encodedImage.timing_;
+
+        callback(encodedImage.data(), encodedImage.size(), info, [header createNativeFragmentationHeader].get());
+        return YES;
+    }];
+
+    return (__bridge_retained void*)encoder;
+
+}
+
+void releaseLocalEncoder(LocalEncoder localEncoder)
+{
+    auto *encoder = (__bridge_transfer RTCVideoEncoderH264 *)(localEncoder);
+    [encoder releaseEncoder];
+}
+
+void initializeLocalEncoder(LocalEncoder localEncoder, uint16_t width, uint16_t height, unsigned int startBitrate, unsigned int maxBitrate, unsigned int minBitrate, uint32_t maxFramerate)
+{
+    webrtc::VideoCodec codecSettings;
+    codecSettings.width = width;
+    codecSettings.height = height;
+    codecSettings.startBitrate = startBitrate;
+    codecSettings.maxBitrate = maxBitrate;
+    codecSettings.minBitrate = minBitrate;
+    codecSettings.maxFramerate = maxFramerate;
+
+    auto *encoder = (__bridge RTCVideoEncoderH264 *)(localEncoder);
+    [encoder startEncodeWithSettings:[[RTCVideoEncoderSettings alloc] initWithNativeVideoCodec:&codecSettings] numberOfCores:1];
+}
+
+void encodeLocalEncoderFrame(LocalEncoder localEncoder, CVPixelBufferRef pixelBuffer, int64_t timeStamp, webrtc::VideoRotation rotation, bool isKeyframeRequired)
+{
+    NSMutableArray<NSNumber *> *rtcFrameTypes = [NSMutableArray array];
+    if (isKeyframeRequired)
+        [rtcFrameTypes addObject:@(RTCFrameType(RTCFrameTypeVideoFrameKey))];
+
+    auto *videoFrame = [[RTCVideoFrame alloc] initWithBuffer:ToObjCVideoFrameBuffer(pixelBufferToFrame(pixelBuffer)) rotation:RTCVideoRotation(rotation) timeStampNs:timeStamp];
+    auto *encoder = (__bridge RTCVideoEncoderH264 *)(localEncoder);
+    [encoder encode:videoFrame codecSpecificInfo:nil frameTypes:rtcFrameTypes];
+}
+
+void setLocalEncoderRates(LocalEncoder localEncoder, uint32_t bitRate, uint32_t frameRate)
+{
+    auto *encoder = (__bridge RTCVideoEncoderH264 *)(localEncoder);
+    [encoder setBitrate:bitRate framerate:frameRate];
+}
+
+}
index 6a58ba1..ab77942 100644 (file)
@@ -52,23 +52,6 @@ bool isH264HardwareEncoderAllowed();
 CVPixelBufferRef pixelBufferFromFrame(const VideoFrame&, const std::function<CVPixelBufferRef(size_t, size_t)>&);
 rtc::scoped_refptr<webrtc::VideoFrameBuffer> pixelBufferToFrame(CVPixelBufferRef);
 
-class VideoEncoderFactoryWithSimulcast final : public VideoEncoderFactory {
-public:
-    explicit VideoEncoderFactoryWithSimulcast(std::unique_ptr<VideoEncoderFactory>&& factory)
-        : m_internalEncoderFactory(std::move(factory))
-    {
-    }
-
-    VideoEncoderFactory::CodecInfo QueryVideoEncoder(const SdpVideoFormat& format) const final { return m_internalEncoderFactory->QueryVideoEncoder(format); }
-
-    std::unique_ptr<VideoEncoder> CreateVideoEncoder(const SdpVideoFormat& format) final { return std::make_unique<EncoderSimulcastProxy>(m_internalEncoderFactory.get(), format); }
-
-    std::vector<SdpVideoFormat> GetSupportedFormats() const final { return m_internalEncoderFactory->GetSupportedFormats(); }
-
-private:
-    const std::unique_ptr<VideoEncoderFactory> m_internalEncoderFactory;
-};
-
 using WebKitVideoDecoder = void*;
 using VideoDecoderCreateCallback = WebKitVideoDecoder(*)(const SdpVideoFormat& format);
 using VideoDecoderReleaseCallback = int32_t(*)(WebKitVideoDecoder);
index 121eda9..1402c2b 100644 (file)
@@ -41,6 +41,7 @@
 #include "Framework/Native/api/video_decoder_factory.h"
 #include "Framework/Native/api/video_encoder_factory.h"
 #include "VideoProcessingSoftLink.h"
+#include "WebKitEncoder.h"
 
 #include <mutex>
 
index 18401da..0e8a045 100644 (file)
                415449B621CAC4CE001C0A55 /* util.c in Sources */ = {isa = PBXBuildFile; fileRef = 415449B521CAC4CE001C0A55 /* util.c */; };
                415449B821CAC4EC001C0A55 /* p256.c in Sources */ = {isa = PBXBuildFile; fileRef = 415449B721CAC4EC001C0A55 /* p256.c */; };
                415449BB21CAC5B4001C0A55 /* ecdh_extra.c in Sources */ = {isa = PBXBuildFile; fileRef = 415449BA21CAC5B3001C0A55 /* ecdh_extra.c */; };
+               4158649823BE05E900A0A61E /* WebKitEncoder.h in Headers */ = {isa = PBXBuildFile; fileRef = 4158649623BE05E800A0A61E /* WebKitEncoder.h */; };
+               4158649923BE05E900A0A61E /* WebKitEncoder.mm in Sources */ = {isa = PBXBuildFile; fileRef = 4158649723BE05E900A0A61E /* WebKitEncoder.mm */; };
                415F1FA621272FBA00064CBF /* adaptive_digital_gain_applier.cc in Sources */ = {isa = PBXBuildFile; fileRef = 41A08BE421272F85001D5D7B /* adaptive_digital_gain_applier.cc */; };
                415F1FA721272FBA00064CBF /* adaptive_digital_gain_applier.h in Headers */ = {isa = PBXBuildFile; fileRef = 41A08BDB21272F84001D5D7B /* adaptive_digital_gain_applier.h */; };
                415F1FA821272FBA00064CBF /* adaptive_mode_level_estimator_agc.cc in Sources */ = {isa = PBXBuildFile; fileRef = 41A08BE921272F86001D5D7B /* adaptive_mode_level_estimator_agc.cc */; };
                415667A1215BE7750032296A /* dbgfmt.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = dbgfmt.h; sourceTree = "<group>"; };
                415667A2215BE7750032296A /* bc-align.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = "bc-align.c"; sourceTree = "<group>"; };
                415667A3215BE7760032296A /* symrec.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = symrec.c; sourceTree = "<group>"; };
+               4158649623BE05E800A0A61E /* WebKitEncoder.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebKitEncoder.h; sourceTree = "<group>"; };
+               4158649723BE05E900A0A61E /* WebKitEncoder.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebKitEncoder.mm; sourceTree = "<group>"; };
                415F1FC52127308E00064CBF /* audio_state.cc */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = audio_state.cc; path = call/audio_state.cc; sourceTree = "<group>"; };
                415F1FD02127313D00064CBF /* receive_time_calculator.cc */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = receive_time_calculator.cc; path = call/receive_time_calculator.cc; sourceTree = "<group>"; };
                415F1FD12127313E00064CBF /* rtp_payload_params.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = rtp_payload_params.h; path = call/rtp_payload_params.h; sourceTree = "<group>"; };
                                41ECEAF920646663009D5141 /* EncoderUtilities.h */,
                                41ECEAFA20646663009D5141 /* VideoProcessingSoftLink.cpp */,
                                41ECEAFB20646664009D5141 /* VideoProcessingSoftLink.h */,
+                               4158649623BE05E800A0A61E /* WebKitEncoder.h */,
+                               4158649723BE05E900A0A61E /* WebKitEncoder.mm */,
                                41ECEABB206403C1009D5141 /* WebKitUtilities.h */,
                                41ECEABD20640498009D5141 /* WebKitUtilities.mm */,
                        );
                                5C4B4C811E431F9C002651C8 /* wav_file.h in Headers */,
                                5C4B4C841E431F9C002651C8 /* wav_header.h in Headers */,
                                4131C163234B89E20028A615 /* weak_ptr.h in Headers */,
+                               4158649823BE05E900A0A61E /* WebKitEncoder.h in Headers */,
                                41ECEABC206403C2009D5141 /* WebKitUtilities.h in Headers */,
                                5CDD8CAB1E43C6F700621E92 /* webrtc_cng.h in Headers */,
                                4145E4911EF88EF700FCF6E6 /* webrtc_libyuv.h in Headers */,
                                41BE71CA215C464800A7B196 /* hamt.c in Sources */,
                                41BE71CB215C464800A7B196 /* insn.c in Sources */,
                                41BE717B215C459000A7B196 /* intnum.c in Sources */,
-                               417C037323B210C400DE5D8B /* main_noop.c in Sources */,
                                41BE717C215C459000A7B196 /* inttree.c in Sources */,
                                41BE71CC215C464800A7B196 /* linemap.c in Sources */,
                                419EA1F3215C48C70082BFD2 /* macho-objfmt.c in Sources */,
+                               417C037323B210C400DE5D8B /* main_noop.c in Sources */,
                                41BE717D215C459000A7B196 /* md5.c in Sources */,
                                41BE717E215C459000A7B196 /* mergesort.c in Sources */,
                                419EA1F9215C4A9E0082BFD2 /* module.c in Sources */,
                                5C4B4C801E431F9C002651C8 /* wav_file.cc in Sources */,
                                5C4B4C831E431F9C002651C8 /* wav_header.cc in Sources */,
                                4131C0F9234B89E20028A615 /* weak_ptr.cc in Sources */,
+                               4158649923BE05E900A0A61E /* WebKitEncoder.mm in Sources */,
                                41ECEABE20640498009D5141 /* WebKitUtilities.mm in Sources */,
                                5CDD8CAA1E43C6F700621E92 /* webrtc_cng.cc in Sources */,
                                5CDD83671E4325D500621E92 /* webrtc_libyuv.cc in Sources */,
index 75fe8af..4addc82 100644 (file)
@@ -1,3 +1,29 @@
+2020-01-06  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB encoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205713
+
+        Reviewed by Eric Carlson.
+
+        Refactoring to expose a createPixelBuffer routine, to set/get RemoteVideoSample and to create a CVPixelBuffer from an IOSurfaceRef.
+
+        Covered by existing tests, in particular webrtc/video-gpuProcess.html.
+
+        * SourcesCocoa.txt:
+        * WebCore.xcodeproj/project.pbxproj:
+        * platform/graphics/RemoteVideoSample.cpp:
+        (WebCore::RemoteVideoSample::create):
+        * platform/graphics/RemoteVideoSample.h:
+        (WebCore::RemoteVideoSample::rotation const):
+        * platform/graphics/cv/ImageTransferSessionVT.h:
+        * platform/graphics/cv/ImageTransferSessionVT.mm:
+        (WebCore::ImageTransferSessionVT::createPixelBuffer):
+        * platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.mm:
+        (WebCore::RealtimeIncomingVideoSourceCocoa::pixelBufferPool):
+        * platform/mediastream/mac/RealtimeVideoUtilities.h:
+        * platform/mediastream/mac/RealtimeVideoUtilities.mm: Added.
+        (WebCore::createPixelBufferPool):
+
 2020-01-05  Eric Carlson  <eric.carlson@apple.com>
 
         [Media in GPU process] Add remote MIME type cache
index 22e0dff..24f5c05 100644 (file)
@@ -558,6 +558,7 @@ platform/mediastream/mac/RealtimeMediaSourceCenterMac.cpp
 platform/mediastream/mac/RealtimeMediaSourceCenterMac.mm
 platform/mediastream/mac/RealtimeOutgoingAudioSourceCocoa.cpp
 platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp
+platform/mediastream/mac/RealtimeVideoUtilities.mm
 platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm
 platform/mediastream/mac/WindowDisplayCaptureSourceMac.mm
 
index 079e946..2f93b8b 100644 (file)
                415747481E3869A700E914D8 /* LibWebRTCProvider.h in Headers */ = {isa = PBXBuildFile; fileRef = 415747441E38699E00E914D8 /* LibWebRTCProvider.h */; settings = {ATTRIBUTES = (Private, ); }; };
                4157EBFA1E3AB67900AC9FE9 /* MockLibWebRTCPeerConnection.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 4157EBF91E3AB06800AC9FE9 /* MockLibWebRTCPeerConnection.cpp */; };
                4157EBFB1E3AB67F00AC9FE9 /* MockLibWebRTCPeerConnection.h in Headers */ = {isa = PBXBuildFile; fileRef = 4157EBF81E3AB06800AC9FE9 /* MockLibWebRTCPeerConnection.h */; };
+               415864A023BF7CBF00A0A61E /* RealtimeVideoUtilities.h in Headers */ = {isa = PBXBuildFile; fileRef = 41D1938F2152C561006F14CA /* RealtimeVideoUtilities.h */; settings = {ATTRIBUTES = (Private, ); }; };
                415CDAF51E6B8F8B004F11EE /* CanvasCaptureMediaStreamTrack.h in Headers */ = {isa = PBXBuildFile; fileRef = 41C7E1061E6A54360027B4DE /* CanvasCaptureMediaStreamTrack.h */; };
                41614A791DA64241004AD06F /* HTTPHeaderValues.h in Headers */ = {isa = PBXBuildFile; fileRef = 41614A771DA64236004AD06F /* HTTPHeaderValues.h */; };
                4161E2D51FE48DC500EC2E96 /* FetchLoader.h in Headers */ = {isa = PBXBuildFile; fileRef = 4147E2B51C89912600A7E715 /* FetchLoader.h */; settings = {ATTRIBUTES = (Private, ); }; };
                415747441E38699E00E914D8 /* LibWebRTCProvider.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LibWebRTCProvider.h; path = libwebrtc/LibWebRTCProvider.h; sourceTree = "<group>"; };
                4157EBF81E3AB06800AC9FE9 /* MockLibWebRTCPeerConnection.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MockLibWebRTCPeerConnection.h; sourceTree = "<group>"; };
                4157EBF91E3AB06800AC9FE9 /* MockLibWebRTCPeerConnection.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MockLibWebRTCPeerConnection.cpp; sourceTree = "<group>"; };
+               4158649F23BF7B9300A0A61E /* RealtimeVideoUtilities.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = RealtimeVideoUtilities.mm; sourceTree = "<group>"; };
                415A3B732138E264001B4BAA /* LibWebRTCObservers.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LibWebRTCObservers.h; path = libwebrtc/LibWebRTCObservers.h; sourceTree = "<group>"; };
                415CDAF61E6CE0D3004F11EE /* JSMediaStreamTrackCustom.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSMediaStreamTrackCustom.cpp; sourceTree = "<group>"; };
                415E1BB3215010800022DA96 /* RTCRtpContributingSource.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RTCRtpContributingSource.h; sourceTree = "<group>"; };
                                5CDD833C1E4324BB00621B83 /* RealtimeOutgoingVideoSourceCocoa.h */,
                                419242472127B7CC00634FCF /* RealtimeOutgoingVideoSourceCocoa.mm */,
                                41D1938F2152C561006F14CA /* RealtimeVideoUtilities.h */,
+                               4158649F23BF7B9300A0A61E /* RealtimeVideoUtilities.mm */,
                                070A9F5E1FFECC70003DF649 /* ScreenDisplayCaptureSourceMac.h */,
                                070A9F601FFECC71003DF649 /* ScreenDisplayCaptureSourceMac.mm */,
                                07D6373E1BB0B11300256CE9 /* WebAudioSourceProviderAVFObjC.h */,
                                41103AAC1E39791000769F14 /* RealtimeOutgoingAudioSourceCocoa.h in Headers */,
                                072880D12010F1F60071B255 /* RealtimeVideoCaptureSource.h in Headers */,
                                41BF204922BA7BE80004F812 /* RealtimeVideoSource.h in Headers */,
+                               415864A023BF7CBF00A0A61E /* RealtimeVideoUtilities.h in Headers */,
                                91B952241F58A58F00931DC2 /* RecordingSwizzleTypes.h in Headers */,
                                BC4368E80C226E32005EFB5F /* Rect.h in Headers */,
                                FD45A958175D414C00C21EC8 /* RectangleShape.h in Headers */,
index 9fe824b..aa6446d 100644 (file)
@@ -61,7 +61,7 @@ std::unique_ptr<RemoteVideoSample> RemoteVideoSample::create(MediaSample&& sampl
     return std::unique_ptr<RemoteVideoSample>(new RemoteVideoSample(surface, sRGBColorSpaceRef(), sample.presentationTime(), sample.videoRotation(), sample.videoMirrored()));
 }
 
-std::unique_ptr<RemoteVideoSample> RemoteVideoSample::create(CVPixelBufferRef imageBuffer, MediaTime&& presentationTime)
+std::unique_ptr<RemoteVideoSample> RemoteVideoSample::create(CVPixelBufferRef imageBuffer, MediaTime&& presentationTime, MediaSample::VideoRotation rotation)
 {
     auto surface = CVPixelBufferGetIOSurface(imageBuffer);
     if (!surface) {
@@ -69,7 +69,7 @@ std::unique_ptr<RemoteVideoSample> RemoteVideoSample::create(CVPixelBufferRef im
         return nullptr;
     }
 
-    return std::unique_ptr<RemoteVideoSample>(new RemoteVideoSample(surface, sRGBColorSpaceRef(), WTFMove(presentationTime), MediaSample::VideoRotation::None, false));
+    return std::unique_ptr<RemoteVideoSample>(new RemoteVideoSample(surface, sRGBColorSpaceRef(), WTFMove(presentationTime), rotation, false));
 }
 
 RemoteVideoSample::RemoteVideoSample(IOSurfaceRef surface, CGColorSpaceRef colorSpace, MediaTime&& time, MediaSample::VideoRotation rotation, bool mirrored)
index bd255f7..72ebc69 100644 (file)
@@ -49,13 +49,14 @@ public:
 
 #if HAVE(IOSURFACE)
     WEBCORE_EXPORT static std::unique_ptr<RemoteVideoSample> create(MediaSample&&);
-    WEBCORE_EXPORT static std::unique_ptr<RemoteVideoSample> create(CVPixelBufferRef, MediaTime&& presentationTime);
+    WEBCORE_EXPORT static std::unique_ptr<RemoteVideoSample> create(CVPixelBufferRef, MediaTime&& presentationTime, MediaSample::VideoRotation = MediaSample::VideoRotation::None);
     WEBCORE_EXPORT IOSurfaceRef surface();
 #endif
 
     const MediaTime& time() const { return m_time; }
     uint32_t videoFormat() const { return m_videoFormat; }
     IntSize size() const { return m_size; }
+    MediaSample::VideoRotation rotation() const { return m_rotation; }
 
     template<class Encoder> void encode(Encoder& encoder) const
     {
index 3b7430f..7380f04 100644 (file)
@@ -56,6 +56,7 @@ public:
 #endif
 
 #if HAVE(IOSURFACE) && !PLATFORM(MACCATALYST)
+    WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> createPixelBuffer(IOSurfaceRef);
     WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> createPixelBuffer(IOSurfaceRef, const IntSize&);
 #endif
 
index 861ac1d..15397ab 100644 (file)
@@ -307,6 +307,20 @@ CFDictionaryRef ImageTransferSessionVT::ioSurfacePixelBufferCreationOptions(IOSu
     return m_ioSurfaceBufferAttributes.get();
 }
 
+RetainPtr<CVPixelBufferRef> ImageTransferSessionVT::createPixelBuffer(IOSurfaceRef surface)
+{
+    if (!surface)
+        return nullptr;
+
+    CVPixelBufferRef pixelBuffer;
+    auto status = CVPixelBufferCreateWithIOSurface(kCFAllocatorDefault, surface, ioSurfacePixelBufferCreationOptions(surface), &pixelBuffer);
+    if (status) {
+        RELEASE_LOG(Media, "CVPixelBufferCreateWithIOSurface failed with error code: %d", static_cast<int>(status));
+        return nullptr;
+    }
+    return adoptCF(pixelBuffer);
+}
+
 RetainPtr<CVPixelBufferRef> ImageTransferSessionVT::createPixelBuffer(IOSurfaceRef surface, const IntSize& size)
 {
     if (!surface || !setSize(size))
index c66f185..06b84da 100644 (file)
@@ -43,8 +43,6 @@ ALLOW_UNUSED_PARAMETERS_END
 
 #include <wtf/cf/TypeCastsCF.h>
 
-#include "CoreVideoSoftLink.h"
-
 namespace WebCore {
 using namespace PAL;
 
@@ -95,28 +93,7 @@ RetainPtr<CVPixelBufferRef> createBlackPixelBuffer(size_t width, size_t height)
 CVPixelBufferPoolRef RealtimeIncomingVideoSourceCocoa::pixelBufferPool(size_t width, size_t height)
 {
     if (!m_pixelBufferPool || m_pixelBufferPoolWidth != width || m_pixelBufferPoolHeight != height) {
-        const OSType videoCaptureFormat = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
-        auto pixelAttributes = @{
-            (__bridge NSString *)kCVPixelBufferWidthKey: @(width),
-            (__bridge NSString *)kCVPixelBufferHeightKey: @(height),
-            (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(videoCaptureFormat),
-            (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: @NO,
-#if PLATFORM(IOS_FAMILY)
-            (__bridge NSString *)kCVPixelFormatOpenGLESCompatibility : @YES,
-#else
-            (__bridge NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES,
-#endif
-            (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ }
-        };
-
-        CVPixelBufferPoolRef pool = nullptr;
-        auto status = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &pool);
-
-        if (status != kCVReturnSuccess) {
-            ERROR_LOG_IF(loggerPtr(), LOGIDENTIFIER, "Failed creating a pixel buffer pool with error ", status);
-            return nullptr;
-        }
-        m_pixelBufferPool = adoptCF(pool);
+        m_pixelBufferPool = createPixelBufferPool(width, height);
         m_pixelBufferPoolWidth = width;
         m_pixelBufferPoolHeight = height;
     }
index 380d373..add4d71 100644 (file)
@@ -27,6 +27,9 @@
 
 #include <CoreVideo/CoreVideo.h>
 
+using CVPixelBufferPoolRef = struct __CVPixelBufferPool*;
+using CVPixelBufferRef = struct __CVBuffer*;
+
 namespace WebCore {
 
 static inline OSType preferedPixelBufferFormat()
@@ -38,4 +41,7 @@ static inline OSType preferedPixelBufferFormat()
 #endif
 }
 
+WEBCORE_EXPORT RetainPtr<CVPixelBufferPoolRef> createPixelBufferPool(size_t width, size_t height);
+WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> createPixelBufferFromPool(CVPixelBufferPoolRef);
+
 }
diff --git a/Source/WebCore/platform/mediastream/mac/RealtimeVideoUtilities.mm b/Source/WebCore/platform/mediastream/mac/RealtimeVideoUtilities.mm
new file mode 100644 (file)
index 0000000..9af48fb
--- /dev/null
@@ -0,0 +1,68 @@
+/*
+ * Copyright (C) 2020 Apple Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted, provided that the following conditions
+ * are required to be met:
+ *
+ * 1.  Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
+ * 2.  Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS "AS IS" AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED. IN NO EVENT SHALL APPLE INC. AND ITS CONTRIBUTORS BE LIABLE FOR
+ * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "RealtimeVideoUtilities.h"
+
+#include "CoreVideoSoftLink.h"
+
+namespace WebCore {
+
+RetainPtr<CVPixelBufferPoolRef> createPixelBufferPool(size_t width, size_t height)
+{
+    const OSType videoCaptureFormat = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
+    auto pixelAttributes = @{
+        (__bridge NSString *)kCVPixelBufferWidthKey: @(width),
+        (__bridge NSString *)kCVPixelBufferHeightKey: @(height),
+        (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(videoCaptureFormat),
+        (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: @NO,
+#if PLATFORM(IOS_FAMILY)
+        (__bridge NSString *)kCVPixelFormatOpenGLESCompatibility : @YES,
+#else
+        (__bridge NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES,
+#endif
+        (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ }
+    };
+
+    CVPixelBufferPoolRef pool = nullptr;
+    auto status = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &pool);
+
+    if (status != kCVReturnSuccess)
+        return nullptr;
+
+    return adoptCF(pool);
+}
+
+RetainPtr<CVPixelBufferRef> createPixelBufferFromPool(CVPixelBufferPoolRef pixelBufferPool)
+{
+    CVPixelBufferRef pixelBuffer = nullptr;
+    auto status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferPool, &pixelBuffer);
+
+    if (status != kCVReturnSuccess)
+        return nullptr;
+    return adoptCF(pixelBuffer);
+}
+
+}
index 447ed30..a192f88 100644 (file)
@@ -1,3 +1,54 @@
+2020-01-06  youenn fablet  <youenn@apple.com>
+
+        Implement RTC VTB encoders in GPUProcess
+        https://bugs.webkit.org/show_bug.cgi?id=205713
+
+        Reviewed by Eric Carlson.
+
+        Add support for remote video encoders in GPUProcess based on libwebrtc exposed routines.
+        WebProcess sets callbacks in the webrtc backend to implement H264 encoding as a remote video encoder.
+        In addition to video frames, we also need to send rate setting orders.
+
+        * Configurations/WebKit.xcconfig:
+        * GPUProcess/webrtc/LibWebRTCCodecsProxy.h:
+        * GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in:
+        * GPUProcess/webrtc/LibWebRTCCodecsProxy.mm:
+        (WebKit::LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy):
+        (WebKit::LibWebRTCCodecsProxy::createEncoder):
+        (WebKit::LibWebRTCCodecsProxy::releaseEncoder):
+        (WebKit::LibWebRTCCodecsProxy::initializeEncoder):
+        (WebKit::toWebRTCVideoRotation):
+        (WebKit::LibWebRTCCodecsProxy::encodeFrame):
+        (WebKit::LibWebRTCCodecsProxy::setEncodeRates):
+        * Scripts/webkit/messages.py:
+        * WebKit.xcodeproj/project.pbxproj:
+        * WebProcess/GPU/webrtc/LibWebRTCCodecs.cpp:
+        (WebKit::createVideoEncoder):
+        (WebKit::releaseVideoEncoder):
+        (WebKit::initializeVideoEncoder):
+        (WebKit::toMediaSampleVideoRotation):
+        (WebKit::encodeVideoFrame):
+        (WebKit::registerEncodeCompleteCallback):
+        (WebKit::setEncodeRatesCallback):
+        (WebKit::LibWebRTCCodecs::setCallbacks):
+        (WebKit::LibWebRTCCodecs::createDecoder):
+        (WebKit::LibWebRTCCodecs::releaseDecoder):
+        (WebKit::LibWebRTCCodecs::failedDecoding):
+        (WebKit::LibWebRTCCodecs::completedDecoding):
+        (WebKit::LibWebRTCCodecs::createEncoder):
+        (WebKit::LibWebRTCCodecs::releaseEncoder):
+        (WebKit::LibWebRTCCodecs::initializeEncoder):
+        (WebKit::LibWebRTCCodecs::encodeFrame):
+        (WebKit::LibWebRTCCodecs::registerEncodeFrameCallback):
+        (WebKit::LibWebRTCCodecs::setEncodeRates):
+        (WebKit::LibWebRTCCodecs::completedEncoding):
+        (WebKit::LibWebRTCCodecs::pixelBufferPool):
+        * WebProcess/GPU/webrtc/LibWebRTCCodecs.h:
+        * WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in:
+        * WebProcess/GPU/webrtc/RTCEncoderIdentifier.h: Added.
+        * WebProcess/Network/webrtc/LibWebRTCProvider.cpp:
+        (WebKit::LibWebRTCProvider::createDecoderFactory):
+
 2020-01-05  Eric Carlson  <eric.carlson@apple.com>
 
         [Media in GPU process] Add remote MIME type cache
index f988039..0f74bb6 100644 (file)
@@ -29,6 +29,8 @@
 
 #include "MessageReceiver.h"
 #include "RTCDecoderIdentifier.h"
+#include "RTCEncoderIdentifier.h"
+#include <WebCore/ImageTransferSessionVT.h>
 
 namespace IPC {
 class Connection;
@@ -36,8 +38,13 @@ class Decoder;
 class DataReference;
 }
 
+namespace WebCore {
+class RemoteVideoSample;
+}
+
 namespace webrtc {
 using LocalDecoder = void*;
+using LocalEncoder = void*;
 }
 
 namespace WebKit {
@@ -59,8 +66,19 @@ private:
     void releaseDecoder(RTCDecoderIdentifier);
     void decodeFrame(RTCDecoderIdentifier, uint32_t timeStamp, const IPC::DataReference&);
 
+    void createEncoder(RTCEncoderIdentifier, const Vector<std::pair<String, String>>&);
+    void releaseEncoder(RTCEncoderIdentifier);
+    void initializeEncoder(RTCEncoderIdentifier, uint16_t width, uint16_t height, unsigned startBitrate, unsigned maxBitrate, unsigned minBitrate, uint32_t maxFramerate);
+    void encodeFrame(RTCEncoderIdentifier, WebCore::RemoteVideoSample&&, bool shouldEncodeAsKeyFrame);
+    void setEncodeRates(RTCEncoderIdentifier, uint32_t bitRate, uint32_t frameRate);
+
+    CFDictionaryRef ioSurfacePixelBufferCreationOptions(IOSurfaceRef);
+
     GPUConnectionToWebProcess& m_gpuConnectionToWebProcess;
     HashMap<RTCDecoderIdentifier, webrtc::LocalDecoder> m_decoders;
+    HashMap<RTCEncoderIdentifier, webrtc::LocalEncoder> m_encoders;
+
+    std::unique_ptr<WebCore::ImageTransferSessionVT> m_imageTransferSession;
 };
 
 }
index c7a1334..2b34e56 100644 (file)
@@ -27,6 +27,12 @@ messages -> LibWebRTCCodecsProxy NotRefCounted {
     CreateDecoder(WebKit::RTCDecoderIdentifier id)
     ReleaseDecoder(WebKit::RTCDecoderIdentifier id)
     DecodeFrame(WebKit::RTCDecoderIdentifier id, uint32_t timeStamp, IPC::DataReference data)
+
+    CreateEncoder(WebKit::RTCEncoderIdentifier id, Vector<std::pair<String, String>> parameters);
+    ReleaseEncoder(WebKit::RTCEncoderIdentifier id)
+    InitializeEncoder(WebKit::RTCEncoderIdentifier id, uint16_t width, uint16_t height, unsigned startBitrate, unsigned maxBitrate, unsigned minBitrate, uint32_t maxFramerate)
+    EncodeFrame(WebKit::RTCEncoderIdentifier id, WebCore::RemoteVideoSample sample, bool shouldEncodeAsKeyFrame)
+    SetEncodeRates(WebKit::RTCEncoderIdentifier id, uint32_t bitRate, uint32_t frameRate)
 }
 
 #endif
index 53c0317..18cc5d7 100644 (file)
@@ -34,6 +34,7 @@
 #include "WebCoreArgumentCoders.h"
 #include <WebCore/LibWebRTCMacros.h>
 #include <WebCore/RemoteVideoSample.h>
+#include <webrtc/sdk/WebKit/WebKitEncoder.h>
 #include <webrtc/sdk/WebKit/WebKitUtilities.h>
 #include <wtf/MediaTime.h>
 
@@ -48,6 +49,8 @@ LibWebRTCCodecsProxy::~LibWebRTCCodecsProxy()
 {
     for (auto decoder : m_decoders.values())
         webrtc::releaseLocalDecoder(decoder);
+    for (auto encoder : m_encoders.values())
+        webrtc::releaseLocalEncoder(encoder);
 }
 
 void LibWebRTCCodecsProxy::createDecoder(RTCDecoderIdentifier identifier)
@@ -77,6 +80,79 @@ void LibWebRTCCodecsProxy::decodeFrame(RTCDecoderIdentifier identifier, uint32_t
         m_gpuConnectionToWebProcess.connection().send(Messages::LibWebRTCCodecs::FailedDecoding { identifier }, 0);
 }
 
+void LibWebRTCCodecsProxy::createEncoder(RTCEncoderIdentifier identifier, const Vector<std::pair<String, String>>& parameters)
+{
+    ASSERT(!m_encoders.contains(identifier));
+
+    std::map<std::string, std::string> rtcParameters;
+    for (auto& parameter : parameters)
+        rtcParameters.emplace(parameter.first.utf8().data(), parameter.second.utf8().data());
+    
+    m_encoders.add(identifier, webrtc::createLocalEncoder(webrtc::SdpVideoFormat { "H264", rtcParameters }, ^(const uint8_t* buffer, size_t size, const webrtc::WebKitEncodedFrameInfo& info, webrtc::RTPFragmentationHeader* header) {
+        
+        m_gpuConnectionToWebProcess.connection().send(Messages::LibWebRTCCodecs::CompletedEncoding { identifier, IPC::DataReference { buffer, size }, info, webrtc::WebKitRTPFragmentationHeader { header } }, 0);
+    }));
+}
+
+void LibWebRTCCodecsProxy::releaseEncoder(RTCEncoderIdentifier identifier)
+{
+    ASSERT(m_encoders.contains(identifier));
+    if (auto encoder = m_encoders.take(identifier))
+        webrtc::releaseLocalEncoder(encoder);
+}
+
+void LibWebRTCCodecsProxy::initializeEncoder(RTCEncoderIdentifier identifier, uint16_t width, uint16_t height, unsigned startBitrate, unsigned maxBitrate, unsigned minBitrate, uint32_t maxFramerate)
+{
+    ASSERT(m_encoders.contains(identifier));
+    auto encoder = m_encoders.get(identifier);
+    if (!encoder)
+        return;
+
+    webrtc::initializeLocalEncoder(encoder, width, height, startBitrate, maxBitrate, minBitrate, maxFramerate);
+}
+
+static inline webrtc::VideoRotation toWebRTCVideoRotation(MediaSample::VideoRotation rotation)
+{
+    switch (rotation) {
+    case MediaSample::VideoRotation::None:
+        return webrtc::kVideoRotation_0;
+    case MediaSample::VideoRotation::UpsideDown:
+        return webrtc::kVideoRotation_180;
+    case MediaSample::VideoRotation::Right:
+        return webrtc::kVideoRotation_90;
+    case MediaSample::VideoRotation::Left:
+        return webrtc::kVideoRotation_270;
+    }
+    ASSERT_NOT_REACHED();
+    return webrtc::kVideoRotation_0;
+}
+
+void LibWebRTCCodecsProxy::encodeFrame(RTCEncoderIdentifier identifier, WebCore::RemoteVideoSample&& sample, bool shouldEncodeAsKeyFrame)
+{
+    ASSERT(m_encoders.contains(identifier));
+    auto encoder = m_encoders.get(identifier);
+    if (!encoder)
+        return;
+
+    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoFormat())
+        m_imageTransferSession = WebCore::ImageTransferSessionVT::create(sample.videoFormat());
+
+#if HAVE(IOSURFACE) && !PLATFORM(MACCATALYST)
+    auto pixelBuffer = m_imageTransferSession->createPixelBuffer(sample.surface());
+    webrtc::encodeLocalEncoderFrame(encoder, pixelBuffer.get(), sample.time().toTimeScale(1000000).timeValue(), toWebRTCVideoRotation(sample.rotation()), shouldEncodeAsKeyFrame);
+#endif
+}
+
+void LibWebRTCCodecsProxy::setEncodeRates(RTCEncoderIdentifier identifier, uint32_t bitRate, uint32_t frameRate)
+{
+    ASSERT(m_encoders.contains(identifier));
+    auto encoder = m_encoders.get(identifier);
+    if (!encoder)
+        return;
+
+    webrtc::setLocalEncoderRates(encoder, bitRate, frameRate);
+}
+
 }
 
 #endif
index 6906310..c9d0e7c 100644 (file)
@@ -222,6 +222,7 @@ def types_that_cannot_be_forward_declared():
         'WebKit::MediaPlayerPrivateRemoteIdentifier',
         'WebKit::RemoteMediaResourceIdentifier',
         'WebKit::RTCDecoderIdentifier',
+        'WebKit::RTCEncoderIdentifier',
         'WebKit::StorageAreaIdentifier',
         'WebKit::StorageAreaImplIdentifier',
         'WebKit::StorageNamespaceIdentifier',
@@ -611,6 +612,8 @@ def headers_for_type(type):
         'struct WebKit::WebUserScriptData': ['"WebUserContentControllerDataTypes.h"'],
         'struct WebKit::WebUserStyleSheetData': ['"WebUserContentControllerDataTypes.h"'],
         'struct WebKit::WebScriptMessageHandlerData': ['"WebUserContentControllerDataTypes.h"'],
+        'webrtc::WebKitEncodedFrameInfo': ['<webrtc/sdk/WebKit/WebKitEncoder.h>'],
+        'webrtc::WebKitRTPFragmentationHeader': ['<webrtc/sdk/WebKit/WebKitEncoder.h>'],
     }
 
     headers = []
index 1b58868..50b2188 100644 (file)
                4151E5C31FBB90A900E47E2D /* FormDataReference.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FormDataReference.h; sourceTree = "<group>"; };
                4157853021276B6F00DD3800 /* copy-webcontent-resources-to-private-headers.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = "copy-webcontent-resources-to-private-headers.sh"; sourceTree = "<group>"; };
                4157E4AF20E2EC9800A6C0D7 /* com.google.o1dbrowserplugin.sb */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = com.google.o1dbrowserplugin.sb; sourceTree = "<group>"; };
+               4158649A23BE092400A0A61E /* RTCEncoderIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RTCEncoderIdentifier.h; sourceTree = "<group>"; };
                4172198923B6128200AE5686 /* LibWebRTCCodecs.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LibWebRTCCodecs.h; sourceTree = "<group>"; };
                4172198A23B6128200AE5686 /* LibWebRTCCodecs.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LibWebRTCCodecs.cpp; sourceTree = "<group>"; };
                4172198C23B612E800AE5686 /* RTCDecoderIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RTCDecoderIdentifier.h; sourceTree = "<group>"; };
                                4172198923B6128200AE5686 /* LibWebRTCCodecs.h */,
                                4172198D23B62C7C00AE5686 /* LibWebRTCCodecs.messages.in */,
                                4172198C23B612E800AE5686 /* RTCDecoderIdentifier.h */,
+                               4158649A23BE092400A0A61E /* RTCEncoderIdentifier.h */,
                        );
                        path = webrtc;
                        sourceTree = "<group>";
index b271d8e..46d1198 100644 (file)
 #include "SharedBufferDataReference.h"
 #include "WebProcess.h"
 #include <WebCore/LibWebRTCMacros.h>
+#include <WebCore/RealtimeVideoUtilities.h>
 #include <WebCore/RemoteVideoSample.h>
 #include <pal/cf/CoreMediaSoftLink.h>
+#include <webrtc/sdk/WebKit/WebKitEncoder.h>
 #include <webrtc/sdk/WebKit/WebKitUtilities.h>
 #include <wtf/MainThread.h>
 
@@ -64,17 +66,85 @@ static int32_t registerDecodeCompleteCallback(webrtc::WebKitVideoDecoder decoder
     return 0;
 }
 
-void LibWebRTCCodecs::setVideoDecoderCallbacks(bool useGPUProcess)
+static webrtc::WebKitVideoEncoder createVideoEncoder(const webrtc::SdpVideoFormat& format)
+{
+    if (format.name != "H264")
+        return nullptr;
+
+    return WebProcess::singleton().libWebRTCCodecs().createEncoder(format.parameters);
+}
+
+static int32_t releaseVideoEncoder(webrtc::WebKitVideoEncoder encoder)
+{
+    return WebProcess::singleton().libWebRTCCodecs().releaseEncoder(*static_cast<LibWebRTCCodecs::Encoder*>(encoder));
+}
+
+static int32_t initializeVideoEncoder(webrtc::WebKitVideoEncoder encoder, const webrtc::VideoCodec& codec)
+{
+    return WebProcess::singleton().libWebRTCCodecs().initializeEncoder(*static_cast<LibWebRTCCodecs::Encoder*>(encoder), codec.width, codec.height, codec.startBitrate, codec.maxBitrate, codec.minBitrate, codec.maxFramerate);
+}
+
+static inline MediaSample::VideoRotation toMediaSampleVideoRotation(webrtc::VideoRotation rotation)
+{
+    switch (rotation) {
+    case webrtc::kVideoRotation_0:
+        return MediaSample::VideoRotation::None;
+    case webrtc::kVideoRotation_180:
+        return MediaSample::VideoRotation::UpsideDown;
+    case webrtc::kVideoRotation_90:
+        return MediaSample::VideoRotation::Right;
+    case webrtc::kVideoRotation_270:
+        return MediaSample::VideoRotation::Left;
+    }
+    ASSERT_NOT_REACHED();
+    return MediaSample::VideoRotation::None;
+}
+
+static int32_t encodeVideoFrame(webrtc::WebKitVideoEncoder encoder, const webrtc::VideoFrame& frame, bool shouldEncodeAsKeyFrame)
+{
+    RetainPtr<CVPixelBufferRef> newPixelBuffer;
+    auto pixelBuffer = webrtc::pixelBufferFromFrame(frame, [&newPixelBuffer](size_t width, size_t height) -> CVPixelBufferRef {
+        auto pixelBufferPool = WebProcess::singleton().libWebRTCCodecs().pixelBufferPool(width, height);
+        if (!pixelBufferPool)
+            return nullptr;
+
+        newPixelBuffer = WebCore::createPixelBufferFromPool(pixelBufferPool);
+        return newPixelBuffer.get();
+    });
+
+    if (!pixelBuffer)
+        return WEBRTC_VIDEO_CODEC_ERROR;
+
+    auto sample = RemoteVideoSample::create(pixelBuffer, MediaTime(frame.timestamp_us(), 1000000), toMediaSampleVideoRotation(frame.rotation()));
+    return WebProcess::singleton().libWebRTCCodecs().encodeFrame(*static_cast<LibWebRTCCodecs::Encoder*>(encoder), *sample, shouldEncodeAsKeyFrame);
+}
+
+static int32_t registerEncodeCompleteCallback(webrtc::WebKitVideoEncoder encoder, void* encodedImageCallback)
+{
+    WebProcess::singleton().libWebRTCCodecs().registerEncodeFrameCallback(*static_cast<LibWebRTCCodecs::Encoder*>(encoder), encodedImageCallback);
+    return 0;
+}
+
+static void setEncodeRatesCallback(webrtc::WebKitVideoEncoder encoder, const webrtc::VideoEncoder::RateControlParameters& parameters)
+{
+    uint32_t bitRate = parameters.bitrate.get_sum_kbps();
+    uint32_t frameRate = static_cast<uint32_t>(parameters.framerate_fps + 0.5);
+    WebProcess::singleton().libWebRTCCodecs().setEncodeRates(*static_cast<LibWebRTCCodecs::Encoder*>(encoder), bitRate, frameRate);
+}
+
+void LibWebRTCCodecs::setCallbacks(bool useGPUProcess)
 {
     ASSERT(isMainThread());
 
     if (!useGPUProcess) {
         webrtc::setVideoDecoderCallbacks(nullptr, nullptr, nullptr, nullptr);
+        webrtc::setVideoEncoderCallbacks(nullptr, nullptr, nullptr, nullptr, nullptr, nullptr);
         return;
     }
     // Let's create WebProcess libWebRTCCodecs since it may be called from various threads.
     WebProcess::singleton().libWebRTCCodecs();
     webrtc::setVideoDecoderCallbacks(createVideoDecoder, releaseVideoDecoder, decodeVideoFrame, registerDecodeCompleteCallback);
+    webrtc::setVideoEncoderCallbacks(createVideoEncoder, releaseVideoEncoder, initializeVideoEncoder, encodeVideoFrame, registerEncodeCompleteCallback, setEncodeRatesCallback);
 }
 
 LibWebRTCCodecs::Decoder* LibWebRTCCodecs::createDecoder()
@@ -89,8 +159,8 @@ LibWebRTCCodecs::Decoder* LibWebRTCCodecs::createDecoder()
         auto decoderIdentifier = decoder->identifier;
         decoder->connection->send(Messages::LibWebRTCCodecsProxy::CreateDecoder { decoderIdentifier }, 0);
 
-        ASSERT(!m_decodeCallbacks.contains(decoderIdentifier));
-        m_decodeCallbacks.add(decoderIdentifier, WTFMove(decoder));
+        ASSERT(!m_decoders.contains(decoderIdentifier));
+        m_decoders.add(decoderIdentifier, WTFMove(decoder));
     });
     return result;
 }
@@ -101,9 +171,9 @@ int32_t LibWebRTCCodecs::releaseDecoder(Decoder& decoder)
     decoder.decodedImageCallback = nullptr;
 
     callOnMainRunLoop([this, decoderIdentifier = decoder.identifier] {
-        ASSERT(m_decodeCallbacks.contains(decoderIdentifier));
+        ASSERT(m_decoders.contains(decoderIdentifier));
 
-        m_decodeCallbacks.remove(decoderIdentifier);
+        m_decoders.remove(decoderIdentifier);
         WebProcess::singleton().ensureGPUProcessConnection().connection().send(Messages::LibWebRTCCodecsProxy::ReleaseDecoder { decoderIdentifier }, 0);
     });
     return 0;
@@ -130,7 +200,7 @@ void LibWebRTCCodecs::failedDecoding(RTCDecoderIdentifier decoderIdentifier)
 {
     ASSERT(isMainThread());
 
-    if (auto* decoder = m_decodeCallbacks.get(decoderIdentifier))
+    if (auto* decoder = m_decoders.get(decoderIdentifier))
         decoder->hasError = true;
 }
 
@@ -139,7 +209,7 @@ void LibWebRTCCodecs::completedDecoding(RTCDecoderIdentifier decoderIdentifier,
     ASSERT(isMainThread());
 
     // FIXME: Do error logging.
-    auto* decoder = m_decodeCallbacks.get(decoderIdentifier);
+    auto* decoder = m_decoders.get(decoderIdentifier);
     if (!decoder)
         return;
 
@@ -167,6 +237,111 @@ void LibWebRTCCodecs::completedDecoding(RTCDecoderIdentifier decoderIdentifier,
     webrtc::RemoteVideoDecoder::decodeComplete(decoder->decodedImageCallback, timeStamp, pixelBuffer.get(), remoteSample.time().toDouble());
 }
 
+LibWebRTCCodecs::Encoder* LibWebRTCCodecs::createEncoder(const std::map<std::string, std::string>& formatParameters)
+{
+    auto encoder = makeUnique<Encoder>();
+    auto* result = encoder.get();
+    encoder->identifier = RTCEncoderIdentifier::generateThreadSafe();
+
+    Vector<std::pair<String, String>> parameters;
+    for (auto& keyValue : formatParameters)
+        parameters.append(std::make_pair(String::fromUTF8(keyValue.first.data(), keyValue.first.length()), String::fromUTF8(keyValue.second.data(), keyValue.second.length())));
+
+    callOnMainRunLoop([this, encoder = WTFMove(encoder), parameters = WTFMove(parameters)]() mutable {
+        auto encoderIdentifier = encoder->identifier;
+        ASSERT(!m_encoders.contains(encoderIdentifier));
+
+        WebProcess::singleton().ensureGPUProcessConnection().connection().send(Messages::LibWebRTCCodecsProxy::CreateEncoder { encoderIdentifier, parameters }, 0);
+        m_encoders.add(encoderIdentifier, WTFMove(encoder));
+    });
+    return result;
+}
+
+int32_t LibWebRTCCodecs::releaseEncoder(Encoder& encoder)
+{
+    LockHolder holder(encoder.encodedImageCallbackLock);
+    encoder.encodedImageCallback = nullptr;
+
+    callOnMainRunLoop([this, encoderIdentifier = encoder.identifier] {
+        ASSERT(m_encoders.contains(encoderIdentifier));
+
+        m_encoders.remove(encoderIdentifier);
+        WebProcess::singleton().ensureGPUProcessConnection().connection().send(Messages::LibWebRTCCodecsProxy::ReleaseEncoder { encoderIdentifier }, 0);
+    });
+    return 0;
+}
+
+int32_t LibWebRTCCodecs::initializeEncoder(Encoder& encoder, uint16_t width, uint16_t height, unsigned startBitRate, unsigned maxBitRate, unsigned minBitRate, uint32_t maxFrameRate)
+{
+    callOnMainRunLoop([this, encoderIdentifier = encoder.identifier, width, height, startBitRate, maxBitRate, minBitRate, maxFrameRate] {
+        if (auto* encoder = m_encoders.get(encoderIdentifier)) {
+            auto& connection = WebProcess::singleton().ensureGPUProcessConnection().connection();
+            connection.send(Messages::LibWebRTCCodecsProxy::InitializeEncoder { encoderIdentifier, width, height, startBitRate, maxBitRate, minBitRate, maxFrameRate }, 0);
+            // We set encoder->connection here so that InitializeEncoder is sent before any EncodeFrame message.
+            encoder->connection = &connection;
+        }
+    });
+    return 0;
+}
+
+int32_t LibWebRTCCodecs::encodeFrame(Encoder& encoder, const WebCore::RemoteVideoSample& frame, bool shouldEncodeAsKeyFrame)
+{
+    if (!encoder.connection)
+        return WEBRTC_VIDEO_CODEC_ERROR;
+
+    encoder.connection->send(Messages::LibWebRTCCodecsProxy::EncodeFrame { encoder.identifier, frame, shouldEncodeAsKeyFrame }, 0);
+    return WEBRTC_VIDEO_CODEC_OK;
+}
+
+void LibWebRTCCodecs::registerEncodeFrameCallback(Encoder& encoder, void* encodedImageCallback)
+{
+    LockHolder holder(encoder.encodedImageCallbackLock);
+    encoder.encodedImageCallback = encodedImageCallback;
+}
+
+void LibWebRTCCodecs::setEncodeRates(Encoder& encoder, uint32_t bitRate, uint32_t frameRate)
+{
+    if (!encoder.connection) {
+        callOnMainRunLoop([this, encoderIdentifier = encoder.identifier, bitRate, frameRate] {
+            UNUSED_PARAM(this);
+            ASSERT(m_encoders.contains(encoderIdentifier));
+            ASSERT(m_encoders.get(encoderIdentifier)->connection);
+            WebProcess::singleton().ensureGPUProcessConnection().connection().send(Messages::LibWebRTCCodecsProxy::SetEncodeRates { encoderIdentifier, bitRate, frameRate }, 0);
+        });
+        return;
+    }
+    encoder.connection->send(Messages::LibWebRTCCodecsProxy::SetEncodeRates { encoder.identifier, bitRate, frameRate }, 0);
+}
+
+void LibWebRTCCodecs::completedEncoding(RTCEncoderIdentifier identifier, IPC::DataReference&& data, const webrtc::WebKitEncodedFrameInfo& info, webrtc::WebKitRTPFragmentationHeader&& fragmentationHeader)
+{
+    ASSERT(isMainThread());
+
+    // FIXME: Do error logging.
+    auto* encoder = m_encoders.get(identifier);
+    if (!encoder)
+        return;
+
+    auto locker = tryHoldLock(encoder->encodedImageCallbackLock);
+    if (!locker)
+        return;
+
+    if (!encoder->encodedImageCallback)
+        return;
+
+    webrtc::RemoteVideoEncoder::encodeComplete(encoder->encodedImageCallback, const_cast<uint8_t*>(data.data()), data.size(), info, fragmentationHeader.value());
+}
+
+CVPixelBufferPoolRef LibWebRTCCodecs::pixelBufferPool(size_t width, size_t height)
+{
+    if (!m_pixelBufferPool || m_pixelBufferPoolWidth != width || m_pixelBufferPoolHeight != height) {
+        m_pixelBufferPool = createPixelBufferPool(width, height);
+        m_pixelBufferPoolWidth = width;
+        m_pixelBufferPoolHeight = height;
+    }
+    return m_pixelBufferPool.get();
+}
+
 }
 
 #endif
index e3cd8a9..893e198 100644 (file)
 
 #include "MessageReceiver.h"
 #include "RTCDecoderIdentifier.h"
+#include "RTCEncoderIdentifier.h"
 #include <WebCore/ImageTransferSessionVT.h>
+#include <map>
 #include <wtf/HashMap.h>
 #include <wtf/HashSet.h>
 #include <wtf/Lock.h>
 
+using CVPixelBufferPoolRef = struct __CVPixelBufferPool*;
+
 namespace IPC {
 class Connection;
+class DataReference;
 class Decoder;
 }
 
@@ -43,6 +48,11 @@ namespace WebCore {
 class RemoteVideoSample;
 }
 
+namespace webrtc {
+class WebKitRTPFragmentationHeader;
+struct WebKitEncodedFrameInfo;
+}
+
 namespace WebKit {
 
 class LibWebRTCCodecs : private IPC::MessageReceiver {
@@ -50,6 +60,8 @@ class LibWebRTCCodecs : private IPC::MessageReceiver {
 public:
     LibWebRTCCodecs() = default;
 
+    static void setCallbacks(bool useGPUProcess);
+
     struct Decoder {
         WTF_MAKE_FAST_ALLOCATED;
     public:
@@ -60,23 +72,46 @@ public:
         RefPtr<IPC::Connection> connection;
     };
 
-    static void setVideoDecoderCallbacks(bool useGPUProcess);
-
     Decoder* createDecoder();
     int32_t releaseDecoder(Decoder&);
     int32_t decodeFrame(Decoder&, uint32_t timeStamp, const uint8_t*, size_t);
     void registerDecodeFrameCallback(Decoder&, void* decodedImageCallback);
 
+    struct Encoder {
+        WTF_MAKE_FAST_ALLOCATED;
+    public:
+        RTCEncoderIdentifier identifier;
+        void* encodedImageCallback { nullptr };
+        Lock encodedImageCallbackLock;
+        RefPtr<IPC::Connection> connection;
+    };
+
+    Encoder* createEncoder(const std::map<std::string, std::string>&);
+    int32_t releaseEncoder(Encoder&);
+    int32_t initializeEncoder(Encoder&, uint16_t width, uint16_t height, unsigned startBitrate, unsigned maxBitrate, unsigned minBitrate, uint32_t maxFramerate);
+    int32_t encodeFrame(Encoder&, const WebCore::RemoteVideoSample&, bool shouldEncodeAsKeyFrame);
+    void registerEncodeFrameCallback(Encoder&, void* encodedImageCallback);
+    void setEncodeRates(Encoder&, uint32_t bitRate, uint32_t frameRate);
+    
+    CVPixelBufferPoolRef pixelBufferPool(size_t width, size_t height);
+
     void didReceiveMessage(IPC::Connection&, IPC::Decoder&) final;
 
 private:
     void failedDecoding(RTCDecoderIdentifier);
     void completedDecoding(RTCDecoderIdentifier, uint32_t timeStamp, WebCore::RemoteVideoSample&&);
+    void completedEncoding(RTCEncoderIdentifier, IPC::DataReference&&, const webrtc::WebKitEncodedFrameInfo&, webrtc::WebKitRTPFragmentationHeader&&);
 
-    HashMap<RTCDecoderIdentifier, std::unique_ptr<Decoder>> m_decodeCallbacks;
+private:
+    HashMap<RTCDecoderIdentifier, std::unique_ptr<Decoder>> m_decoders;
     HashSet<RTCDecoderIdentifier> m_decodingErrors;
 
+    HashMap<RTCEncoderIdentifier, std::unique_ptr<Encoder>> m_encoders;
+
     std::unique_ptr<WebCore::ImageTransferSessionVT> m_imageTransferSession;
+    RetainPtr<CVPixelBufferPoolRef> m_pixelBufferPool;
+    size_t m_pixelBufferPoolWidth { 0 };
+    size_t m_pixelBufferPoolHeight { 0 };
 };
 
 } // namespace WebKit
index 2aae978..f2bf5b8 100644 (file)
 #if USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
 
 messages -> LibWebRTCCodecs NotRefCounted {
-    FailedDecoding(WebKit::RTCDecoderIdentifier id)
-    CompletedDecoding(WebKit::RTCDecoderIdentifier id, uint32_t timeStamp, WebCore::RemoteVideoSample sample)
+    FailedDecoding(WebKit::RTCDecoderIdentifier identifier)
+    CompletedDecoding(WebKit::RTCDecoderIdentifier identifier, uint32_t timeStamp, WebCore::RemoteVideoSample sample)
+
+    CompletedEncoding(WebKit::RTCEncoderIdentifier identifier, IPC::DataReference data, struct webrtc::WebKitEncodedFrameInfo info, webrtc::WebKitRTPFragmentationHeader fragmentationHeader);
 }
 
 #endif // USE(LIBWEBRTC) && PLATFORM(COCOA) && ENABLE(GPU_PROCESS)
diff --git a/Source/WebKit/WebProcess/GPU/webrtc/RTCEncoderIdentifier.h b/Source/WebKit/WebProcess/GPU/webrtc/RTCEncoderIdentifier.h
new file mode 100644 (file)
index 0000000..e0a7f3c
--- /dev/null
@@ -0,0 +1,35 @@
+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include <wtf/ObjectIdentifier.h>
+
+namespace WebKit {
+
+enum RTCEncoderIdentifierType { };
+using RTCEncoderIdentifier = ObjectIdentifier<RTCEncoderIdentifierType>;
+
+} // namespace WebKit
index f983622..d39e399 100644 (file)
@@ -141,7 +141,7 @@ std::unique_ptr<webrtc::VideoDecoderFactory> LibWebRTCProvider::createDecoderFac
 #if ENABLE(GPU_PROCESS)
     // We only support efficient sending of video frames with IOSURFACE
 #if HAVE(IOSURFACE) && !PLATFORM(MACCATALYST)
-    LibWebRTCCodecs::setVideoDecoderCallbacks(m_useGPUProcess);
+    LibWebRTCCodecs::setCallbacks(m_useGPUProcess);
 #endif
 #endif
     return LibWebRTCProviderCocoa::createDecoderFactory();