Enable H264 simulcast
authoryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 3 Oct 2018 12:11:23 +0000 (12:11 +0000)
committeryouenn@apple.com <youenn@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 3 Oct 2018 12:11:23 +0000 (12:11 +0000)
https://bugs.webkit.org/show_bug.cgi?id=190167

Reviewed by Eric Carlson.

Source/ThirdParty/libwebrtc:

Rename .m files to .mm to enable C++ compilation of included header files.
Rename RTCH264VideoEncoder to RTCSingleH264Encoder.
Implement a new RTCH264VideoEncoder that spawns as many RTCSingleH264Encoder as needed for simulcast.
Update ObjC API to allow passing simulcast parameters to/from RTCH264VideoEncoder.

* Configurations/libwebrtc.iOS.exp:
* Configurations/libwebrtc.iOSsim.exp:
* Configurations/libwebrtc.mac.exp:
* Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoDecoderFactory.mm: Renamed from Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoDecoderFactory.m.
* Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoEncoderFactory.mm: Renamed from Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoEncoderFactory.m.
* Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoCodec+Private.h:
* Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoCodecH264.mm:
(-[RTCCodecSpecificInfoH264 nativeCodecSpecificInfo]):
* Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoEncoderSettings.mm:
(-[RTCVideoEncoderSettings initWithNativeVideoCodec:]):
* Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCWrappedNativeVideoEncoder.mm:
(-[RTCWrappedNativeVideoEncoder setBitrate:framerate:]):
(-[RTCWrappedNativeVideoEncoder setRateAllocation:framerate:]):
* Source/webrtc/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm:
(-[RTCSingleVideoEncoderH264 initWithCodecInfo:simulcastIndex:]):
(-[RTCSingleVideoEncoderH264 startEncodeWithSettings:numberOfCores:]):
(-[RTCSingleVideoEncoderH264 encode:codecSpecificInfo:frameTypes:]):
(-[RTCSingleVideoEncoderH264 resetCompressionSessionWithPixelFormat:]):
(-[RTCSingleVideoEncoderH264 scalingSettings]):
(-[RTCSingleVideoEncoderH264 setRateAllocation:framerate:]):
(-[RTCVideoEncoderH264 initWithCodecInfo:]):
(-[RTCVideoEncoderH264 setCallback:]):
(-[RTCVideoEncoderH264 startEncodeWithSettings:numberOfCores:]):
(-[RTCVideoEncoderH264 releaseEncoder]):
(-[RTCVideoEncoderH264 encode:codecSpecificInfo:frameTypes:]):
(-[RTCVideoEncoderH264 setRateAllocation:framerate:]):
(-[RTCVideoEncoderH264 implementationName]):
(-[RTCVideoEncoderH264 scalingSettings]):
(-[RTCVideoEncoderH264 setBitrate:framerate:]):
* Source/webrtc/sdk/objc/Framework/Headers/WebRTC/RTCVideoCodec.h:
* Source/webrtc/sdk/objc/Framework/Headers/WebRTC/RTCVideoCodecH264.h:
* Source/webrtc/sdk/objc/Framework/Native/src/objc_video_encoder_factory.mm:
* libwebrtc.xcodeproj/project.pbxproj:

Source/WebCore:

Activate H264 simulcast trial field.
Make track.getSettings() expose width and height for incoming tracks.

Test: webrtc/simulcast-h264.html

* Configurations/WebCore.xcconfig:
* Modules/mediastream/libwebrtc/LibWebRTCMediaEndpoint.cpp:
(WebCore::LibWebRTCMediaEndpoint::LibWebRTCMediaEndpoint):
* platform/mediastream/RealtimeIncomingVideoSource.cpp:
(WebCore::RealtimeIncomingVideoSource::RealtimeIncomingVideoSource):

LayoutTests:

* TestExpectations:
* webrtc/simulcast-h264-expected.txt: Added.
* webrtc/simulcast-h264.html: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@236792 268f45cc-cd09-0410-ab3c-d52691b4dbfc

22 files changed:
LayoutTests/ChangeLog
LayoutTests/TestExpectations
LayoutTests/webrtc/simulcast-h264-expected.txt [new file with mode: 0644]
LayoutTests/webrtc/simulcast-h264.html [new file with mode: 0644]
Source/ThirdParty/libwebrtc/ChangeLog
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOS.exp
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.iOSsim.exp
Source/ThirdParty/libwebrtc/Configurations/libwebrtc.mac.exp
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoDecoderFactory.mm [moved from Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoDecoderFactory.m with 97% similarity]
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoEncoderFactory.mm [moved from Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoEncoderFactory.m with 96% similarity]
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoCodec+Private.h
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoCodecH264.mm
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoEncoderSettings.mm
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCWrappedNativeVideoEncoder.mm
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Headers/WebRTC/RTCVideoCodec.h
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Headers/WebRTC/RTCVideoCodecH264.h
Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Native/src/objc_video_encoder_factory.mm
Source/ThirdParty/libwebrtc/libwebrtc.xcodeproj/project.pbxproj
Source/WebCore/ChangeLog
Source/WebCore/Modules/mediastream/libwebrtc/LibWebRTCMediaEndpoint.cpp
Source/WebCore/platform/mediastream/RealtimeIncomingVideoSource.cpp

index b902b15..04e5d4c 100644 (file)
@@ -1,3 +1,14 @@
+2018-10-03  Youenn Fablet  <youenn@apple.com>
+
+        Enable H264 simulcast
+        https://bugs.webkit.org/show_bug.cgi?id=190167
+
+        Reviewed by Eric Carlson.
+
+        * TestExpectations:
+        * webrtc/simulcast-h264-expected.txt: Added.
+        * webrtc/simulcast-h264.html: Added.
+
 2018-10-03  Ryosuke Niwa  <rniwa@webkit.org>
 
         Enable selectionAcrossShadowBoundariesEnabled by default in WebKitLegacy
index ffc2cf6..4441f39 100644 (file)
@@ -1211,6 +1211,7 @@ webkit.org/b/146182 editing/selection/leak-document-with-selection-inside.html [
 # Media Sessions is not yet enabled by default: ENABLE(MEDIA_SESSION)
 media/session [ Skip ]
 
+webrtc/simulcast-h264.html [ Slow ]
 webrtc/datachannel/multiple-connections.html [ Slow ]
 webkit.org/b/187180 webrtc/datachannel/mdns-ice-candidates.html [ Pass Failure ]
 webkit.org/b/171094 imported/w3c/web-platform-tests/webrtc/rtcpeerconnection/rtcpeerconnection-idl.html [ Failure ]
diff --git a/LayoutTests/webrtc/simulcast-h264-expected.txt b/LayoutTests/webrtc/simulcast-h264-expected.txt
new file mode 100644 (file)
index 0000000..621fe18
--- /dev/null
@@ -0,0 +1,4 @@
+   
+
+PASS Testing simulcast 
+
diff --git a/LayoutTests/webrtc/simulcast-h264.html b/LayoutTests/webrtc/simulcast-h264.html
new file mode 100644 (file)
index 0000000..16f2f84
--- /dev/null
@@ -0,0 +1,126 @@
+<!doctype html>
+<html>
+<head>
+<script src="../resources/testharness.js"></script>
+<script src="../resources/testharnessreport.js"></script>
+</head>
+<body>
+<div>
+    <video id="low" playsinline autoplay width="320"></video>
+    <video id="mid" playsinline autoplay width="320"></video>
+    <video id="high" playsinline autoplay width="320"></video>
+</div>
+<script>
+if (window.testRunner)
+    testRunner.setWebRTCUnifiedPlanEnabled(false);
+var state;
+var finished = false;
+// This test is largely inspired from Chrome/Firefox/Simulcast playground tests.
+promise_test(async (test) => {
+    if (window.testRunner && testRunner.timeout) {
+        setTimeout(() => {
+            if (!finished)
+                throw "test stucked in state: " + state;
+        }, testRunner.timeout * 0.9);
+    }
+
+    state = "start";
+
+    const pc1 = new RTCPeerConnection();
+    const pc2 = new RTCPeerConnection();
+    pc1.onicecandidate = (e) => pc2.addIceCandidate(e.candidate);
+    pc2.onicecandidate = (e) => pc1.addIceCandidate(e.candidate);
+
+    let counter = 0;
+    pc2.ontrack = (e) => {
+        if (counter == 0)
+            low.srcObject = new MediaStream([e.track]);
+        else if (counter == 1)
+            mid.srcObject = new MediaStream([e.track]);
+        else
+            high.srcObject = new MediaStream([e.track]);
+        ++counter;
+    }
+
+    const localStream = await navigator.mediaDevices.getUserMedia({ video: { width: 640, height: 480 } });
+    pc1.addTrack(localStream.getVideoTracks()[0], localStream);
+    const offer = await pc1.createOffer();
+
+    state = "got offer";
+
+    match = offer.sdp.match(/a=ssrc:(\d+) cname:(.*)\r\n/);
+    msid = offer.sdp.match(/a=ssrc:(\d+) msid:(.*)\r\n/);
+    var lines = offer.sdp.trim().split('\r\n');
+    var removed = lines.splice(lines.length - 4, 4);
+    var videoSSRC1 = parseInt(match[1]);
+    rtxSSRC1 = offer.sdp.split('\r\n').filter((line) => { return line.startsWith('a=ssrc-group:FID ')})[0].split(' ')[2];
+    var videoSSRC2 = videoSSRC1 + 1;
+    var rtxSSRC2 = videoSSRC1 + 2;
+    var videoSSRC3 = videoSSRC1 + 3;
+    var rtxSSRC3 = videoSSRC1 + 4;
+    lines.push(removed[0]);
+    lines.push(removed[1]);
+    lines.push('a=ssrc:' + videoSSRC2 + ' cname:' + match[2]);
+    lines.push('a=ssrc:' + videoSSRC2 + ' msid:' + msid[2]);
+    lines.push('a=ssrc:' + rtxSSRC2 + ' cname:' + match[2]);
+    lines.push('a=ssrc:' + rtxSSRC2 + ' msid:' + msid[2]);
+
+    lines.push('a=ssrc:' + videoSSRC3 + ' cname:' + match[2]);
+    lines.push('a=ssrc:' + videoSSRC3 + ' msid:' + msid[2]);
+    lines.push('a=ssrc:' + rtxSSRC3 + ' cname:' + match[2]);
+    lines.push('a=ssrc:' + rtxSSRC3 + ' msid:' + msid[2]);
+
+    lines.push('a=ssrc-group:FID ' + videoSSRC2 + ' ' + rtxSSRC2);
+    lines.push('a=ssrc-group:FID ' + videoSSRC3 + ' ' + rtxSSRC3);
+    lines.push('a=ssrc-group:SIM ' + videoSSRC1 + ' ' + videoSSRC2 + ' ' + videoSSRC3);
+    offer.sdp = lines.join('\r\n') + '\r\n';
+
+    var offer2 = {
+        type: 'offer',
+        sdp: offer.sdp,
+    };
+    offer2.sdp = offer2.sdp.replace('a=ssrc-group:SIM ' + videoSSRC1 + ' ' + videoSSRC2 + ' ' + videoSSRC3 + '\r\n', '');
+
+    offer2.sdp = offer2.sdp.replace('a=ssrc:' + videoSSRC1 + ' msid:' + msid[2], 'a=ssrc:' + videoSSRC1 + ' msid:low low');
+    offer2.sdp = offer2.sdp.replace('a=ssrc:' + rtxSSRC1 + ' msid:' + msid[2], 'a=ssrc:' + rtxSSRC1 + ' msid:low low');
+
+    offer2.sdp = offer2.sdp.replace('a=ssrc:' + videoSSRC2 + ' msid:' + msid[2], 'a=ssrc:' + videoSSRC2 + ' msid:mid mid');
+    offer2.sdp = offer2.sdp.replace('a=ssrc:' + rtxSSRC2 + ' msid:' + msid[2], 'a=ssrc:' + rtxSSRC2 + ' msid:mid mid');
+
+    offer2.sdp = offer2.sdp.replace('a=ssrc:' + videoSSRC3 + ' msid:' + msid[2], 'a=ssrc:' + videoSSRC3 + ' msid:hi hi');
+    offer2.sdp = offer2.sdp.replace('a=ssrc:' + rtxSSRC3 + ' msid:' + msid[2], 'a=ssrc:' + rtxSSRC3 + ' msid:hi hi');
+    await Promise.all([
+        pc1.setLocalDescription(offer),
+        pc2.setRemoteDescription(offer2),
+    ]);
+
+    state = "set description";
+
+    const answer = await pc2.createAnswer();
+
+    state = "got answer";
+
+    await Promise.all([
+        pc2.setLocalDescription(answer),
+        pc1.setRemoteDescription(answer),
+    ]);
+
+    state = "set description 2";
+
+    await low.play();
+    state = "video low plays";
+
+    assert_equals(low.srcObject.getVideoTracks()[0].getSettings().height, 240);
+    assert_equals(low.srcObject.getVideoTracks()[0].getSettings().width, 320);
+
+    await mid.play();
+    state = "video mid plays";
+
+    assert_equals(mid.srcObject.getVideoTracks()[0].getSettings().height, 480);
+    assert_equals(mid.srcObject.getVideoTracks()[0].getSettings().width, 640);
+
+    finished = true;
+}, "Testing simulcast");
+</script>
+</body>
+</html>
index 26a1185..641fae7 100644 (file)
@@ -1,3 +1,49 @@
+2018-10-03  Youenn Fablet  <youenn@apple.com>
+
+        Enable H264 simulcast
+        https://bugs.webkit.org/show_bug.cgi?id=190167
+
+        Reviewed by Eric Carlson.
+
+        Rename .m files to .mm to enable C++ compilation of included header files.
+        Rename RTCH264VideoEncoder to RTCSingleH264Encoder.
+        Implement a new RTCH264VideoEncoder that spawns as many RTCSingleH264Encoder as needed for simulcast.
+        Update ObjC API to allow passing simulcast parameters to/from RTCH264VideoEncoder.
+
+        * Configurations/libwebrtc.iOS.exp:
+        * Configurations/libwebrtc.iOSsim.exp:
+        * Configurations/libwebrtc.mac.exp:
+        * Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoDecoderFactory.mm: Renamed from Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoDecoderFactory.m.
+        * Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoEncoderFactory.mm: Renamed from Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCDefaultVideoEncoderFactory.m.
+        * Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoCodec+Private.h:
+        * Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoCodecH264.mm:
+        (-[RTCCodecSpecificInfoH264 nativeCodecSpecificInfo]):
+        * Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCVideoEncoderSettings.mm:
+        (-[RTCVideoEncoderSettings initWithNativeVideoCodec:]):
+        * Source/webrtc/sdk/objc/Framework/Classes/PeerConnection/RTCWrappedNativeVideoEncoder.mm:
+        (-[RTCWrappedNativeVideoEncoder setBitrate:framerate:]):
+        (-[RTCWrappedNativeVideoEncoder setRateAllocation:framerate:]):
+        * Source/webrtc/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm:
+        (-[RTCSingleVideoEncoderH264 initWithCodecInfo:simulcastIndex:]):
+        (-[RTCSingleVideoEncoderH264 startEncodeWithSettings:numberOfCores:]):
+        (-[RTCSingleVideoEncoderH264 encode:codecSpecificInfo:frameTypes:]):
+        (-[RTCSingleVideoEncoderH264 resetCompressionSessionWithPixelFormat:]):
+        (-[RTCSingleVideoEncoderH264 scalingSettings]):
+        (-[RTCSingleVideoEncoderH264 setRateAllocation:framerate:]):
+        (-[RTCVideoEncoderH264 initWithCodecInfo:]):
+        (-[RTCVideoEncoderH264 setCallback:]):
+        (-[RTCVideoEncoderH264 startEncodeWithSettings:numberOfCores:]):
+        (-[RTCVideoEncoderH264 releaseEncoder]):
+        (-[RTCVideoEncoderH264 encode:codecSpecificInfo:frameTypes:]):
+        (-[RTCVideoEncoderH264 setRateAllocation:framerate:]):
+        (-[RTCVideoEncoderH264 implementationName]):
+        (-[RTCVideoEncoderH264 scalingSettings]):
+        (-[RTCVideoEncoderH264 setBitrate:framerate:]):
+        * Source/webrtc/sdk/objc/Framework/Headers/WebRTC/RTCVideoCodec.h:
+        * Source/webrtc/sdk/objc/Framework/Headers/WebRTC/RTCVideoCodecH264.h:
+        * Source/webrtc/sdk/objc/Framework/Native/src/objc_video_encoder_factory.mm:
+        * libwebrtc.xcodeproj/project.pbxproj:
+
 2018-09-29  Youenn Fablet  <youenn@apple.com>
 
         Add yasm as third party tool for libwebrtc compilation
index 99c56c3..11e940a 100644 (file)
@@ -215,3 +215,4 @@ __ZN6webrtc21RtpEncodingParametersD1Ev
 __ZN6webrtc18RtpTransceiverInitC1Ev
 __ZN6webrtc18RtpTransceiverInitD1Ev
 __ZN6webrtc9RtpSourceD1Ev
+__ZN6webrtc11field_trial25InitFieldTrialsFromStringEPKc
index 34e55b7..c80c7c0 100644 (file)
@@ -216,3 +216,4 @@ __ZN6webrtc21RtpEncodingParametersD1Ev
 __ZN6webrtc18RtpTransceiverInitC1Ev
 __ZN6webrtc18RtpTransceiverInitD1Ev
 __ZN6webrtc9RtpSourceD1Ev
+__ZN6webrtc11field_trial25InitFieldTrialsFromStringEPKc
index 34e55b7..c80c7c0 100644 (file)
@@ -216,3 +216,4 @@ __ZN6webrtc21RtpEncodingParametersD1Ev
 __ZN6webrtc18RtpTransceiverInitC1Ev
 __ZN6webrtc18RtpTransceiverInitD1Ev
 __ZN6webrtc9RtpSourceD1Ev
+__ZN6webrtc11field_trial25InitFieldTrialsFromStringEPKc
 - (id<RTCVideoDecoder>)createDecoder:(RTCVideoCodecInfo *)info {
   if ([info.name isEqualToString:kRTCVideoCodecH264Name]) {
     return [[RTCVideoDecoderH264 alloc] init];
+#if !defined(RTC_DISABLE_VP8)
   } else if ([info.name isEqualToString:kRTCVideoCodecVp8Name]) {
     return [RTCVideoDecoderVP8 vp8Decoder];
+#endif
 #if !defined(RTC_DISABLE_VP9)
   } else if ([info.name isEqualToString:kRTCVideoCodecVp9Name]) {
     return [RTCVideoDecoderVP9 vp9Decoder];
@@ -40,7 +40,9 @@
       [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecH264Name
                                    parameters:constrainedBaselineParams];
 
+#if !defined(RTC_DISABLE_VP8)
   RTCVideoCodecInfo *vp8Info = [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecVp8Name];
+#endif
 
 #if !defined(RTC_DISABLE_VP9)
   RTCVideoCodecInfo *vp9Info = [[RTCVideoCodecInfo alloc] initWithName:kRTCVideoCodecVp9Name];
@@ -49,7 +51,9 @@
   return @[
     constrainedHighInfo,
     constrainedBaselineInfo,
+#if !defined(RTC_DISABLE_VP8)
     vp8Info,
+#endif
 #if !defined(RTC_DISABLE_VP9)
     vp9Info,
 #endif
 - (id<RTCVideoEncoder>)createEncoder:(RTCVideoCodecInfo *)info {
   if ([info.name isEqualToString:kRTCVideoCodecH264Name]) {
     return [[RTCVideoEncoderH264 alloc] initWithCodecInfo:info];
+#if !defined(RTC_DISABLE_VP8)
   } else if ([info.name isEqualToString:kRTCVideoCodecVp8Name]) {
     return [RTCVideoEncoderVP8 vp8Encoder];
+#endif
 #if !defined(RTC_DISABLE_VP9)
   } else if ([info.name isEqualToString:kRTCVideoCodecVp9Name]) {
     return [RTCVideoEncoderVP9 vp9Encoder];
index 4a2cf52..0d8563c 100644 (file)
@@ -30,7 +30,8 @@ NS_ASSUME_NONNULL_BEGIN
 @interface RTCVideoEncoderSettings ()
 
 - (instancetype)initWithNativeVideoCodec:(const webrtc::VideoCodec *__nullable)videoCodec;
-- (webrtc::VideoCodec)nativeVideoCodec;
+
+@property(nonatomic, assign) webrtc::VideoCodec nativeVideoCodec;
 
 @end
 
index 51583d0..97d759a 100644 (file)
 #include "rtc_base/timeutils.h"
 
 // H264 specific settings.
-@implementation RTCCodecSpecificInfoH264
+@implementation RTCCodecSpecificInfoH264 { }
 
 @synthesize packetizationMode = _packetizationMode;
+@synthesize simulcastIndex = _simulcastIndex;
 
 - (webrtc::CodecSpecificInfo)nativeCodecSpecificInfo {
   webrtc::CodecSpecificInfo codecSpecificInfo;
@@ -28,6 +29,7 @@
   codecSpecificInfo.codec_name = [kRTCVideoCodecH264Name cStringUsingEncoding:NSUTF8StringEncoding];
   codecSpecificInfo.codecSpecific.H264.packetization_mode =
       (webrtc::H264PacketizationMode)_packetizationMode;
+  codecSpecificInfo.codecSpecific.H264.simulcast_idx = _simulcastIndex;
 
   return codecSpecificInfo;
 }
index 95233ec..69079d5 100644 (file)
@@ -26,6 +26,7 @@
 @synthesize maxFramerate = _maxFramerate;
 @synthesize qpMax = _qpMax;
 @synthesize mode = _mode;
+@synthesize nativeVideoCodec = _nativeVideoCodec;
 
 - (instancetype)initWithNativeVideoCodec:(const webrtc::VideoCodec *)videoCodec {
   if (self = [super init]) {
@@ -33,6 +34,8 @@
       const char *codecName = CodecTypeToPayloadString(videoCodec->codecType);
       _name = [NSString stringWithUTF8String:codecName];
 
+      _nativeVideoCodec = *videoCodec;
+
       _width = videoCodec->width;
       _height = videoCodec->height;
       _startBitrate = videoCodec->startBitrate;
   return self;
 }
 
-- (webrtc::VideoCodec)nativeVideoCodec {
-  webrtc::VideoCodec videoCodec;
-  videoCodec.width = _width;
-  videoCodec.height = _height;
-  videoCodec.startBitrate = _startBitrate;
-  videoCodec.maxBitrate = _maxBitrate;
-  videoCodec.minBitrate = _minBitrate;
-  videoCodec.targetBitrate = _targetBitrate;
-  videoCodec.maxBitrate = _maxBitrate;
-  videoCodec.qpMax = _qpMax;
-  videoCodec.mode = (webrtc::VideoCodecMode)_mode;
-
-  return videoCodec;
-}
-
 @end
index 8988e6d..574fcd1 100644 (file)
 }
 
 - (int)setBitrate:(uint32_t)bitrateKbit framerate:(uint32_t)framerate {
+    RTC_NOTREACHED();
+    return 0;
+}
+
+- (int)setRateAllocation:(const webrtc::VideoBitrateAllocation *)bitRateAllocation framerate:(uint32_t) framerate {
   RTC_NOTREACHED();
   return 0;
 }
index ae10956..7a21a4d 100644 (file)
@@ -72,8 +72,9 @@ static inline bool isStandardFrameSize(int32_t width, int32_t height)
 }
 #endif
 
-@interface RTCVideoEncoderH264 ()
-
+__attribute__((objc_runtime_name("WK_RTCSingleVideoEncoderH264")))
+@interface RTCSingleVideoEncoderH264 : NSObject <RTCVideoEncoder>
+- (instancetype)initWithCodecInfo:(RTCVideoCodecInfo *)codecInfo simulcastIndex: (int)index;
 - (void)frameWasEncoded:(OSStatus)status
                   flags:(VTEncodeInfoFlags)infoFlags
            sampleBuffer:(CMSampleBufferRef)sampleBuffer
@@ -83,7 +84,6 @@ static inline bool isStandardFrameSize(int32_t width, int32_t height)
            renderTimeMs:(int64_t)renderTimeMs
               timestamp:(uint32_t)timestamp
                rotation:(RTCVideoRotation)rotation;
-
 @end
 
 namespace {  // anonymous namespace
@@ -102,7 +102,7 @@ const OSType kNV12PixelFormat = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
 // Struct that we pass to the encoder per frame to encode. We receive it again
 // in the encoder callback.
 struct RTCFrameEncodeParams {
-  RTCFrameEncodeParams(RTCVideoEncoderH264 *e,
+  RTCFrameEncodeParams(RTCSingleVideoEncoderH264 *e,
                        RTCCodecSpecificInfoH264 *csi,
                        int32_t w,
                        int32_t h,
@@ -117,7 +117,7 @@ struct RTCFrameEncodeParams {
     }
   }
 
-  RTCVideoEncoderH264 *encoder;
+  RTCSingleVideoEncoderH264 *encoder;
   RTCCodecSpecificInfoH264 *codecSpecificInfo;
   int32_t width;
   int32_t height;
@@ -312,7 +312,7 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
 }
 }  // namespace
 
-@implementation RTCVideoEncoderH264 {
+@implementation RTCSingleVideoEncoderH264 {
   RTCVideoCodecInfo *_codecInfo;
   std::unique_ptr<webrtc::BitrateAdjuster> _bitrateAdjuster;
   uint32_t _targetBitrateBps;
@@ -327,6 +327,9 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
 
   webrtc::H264BitstreamParser _h264BitstreamParser;
   std::vector<uint8_t> _frameScaleBuffer;
+
+  webrtc::VideoCodec _nativeVideoCodec;
+  int _simulcastIndex;
 }
 
 // .5 is set as a mininum to prevent overcompensating for large temporary
@@ -336,12 +339,13 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
 // drastically reduced bitrate, so we want to avoid that. In steady state
 // conditions, 0.95 seems to give us better overall bitrate over long periods
 // of time.
-- (instancetype)initWithCodecInfo:(RTCVideoCodecInfo *)codecInfo {
+- (instancetype)initWithCodecInfo:(RTCVideoCodecInfo *)codecInfo simulcastIndex:(int)index {
   if (self = [super init]) {
     _codecInfo = codecInfo;
     _bitrateAdjuster.reset(new webrtc::BitrateAdjuster(.5, .95));
     _packetizationMode = RTCH264PacketizationModeNonInterleaved;
     _profile = ExtractProfile([codecInfo nativeSdpVideoFormat]);
+    _simulcastIndex = index;
     RTC_LOG(LS_INFO) << "Using profile " << CFStringToString(_profile);
     RTC_CHECK([codecInfo.name isEqualToString:kRTCVideoCodecH264Name]);
 
@@ -364,6 +368,9 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
   _width = settings.width;
   _height = settings.height;
   _mode = settings.mode;
+  _nativeVideoCodec = settings.nativeVideoCodec;
+
+  RTC_DCHECK(_nativeVideoCodec.numberOfSimulcastStreams != 1);
 
   // We can only set average bitrate on the HW encoder.
   _targetBitrateBps = settings.startBitrate;
@@ -475,6 +482,7 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
                                               frame.timeStamp,
                                               frame.rotation));
   encodeParams->codecSpecificInfo.packetizationMode = _packetizationMode;
+  encodeParams->codecSpecificInfo.simulcastIndex = _simulcastIndex;
 
   // Update the bitrate if needed.
   [self setBitrateBps:_bitrateAdjuster->GetAdjustedBitrateBps()];
@@ -503,8 +511,8 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
   _callback = callback;
 }
 
-- (int)setBitrate:(uint32_t)bitrateKbit framerate:(uint32_t)framerate {
-  _targetBitrateBps = 1000 * bitrateKbit;
+- (int)setBitrate:(uint32_t)bitrateBps framerate:(uint32_t)framerate {
+  _targetBitrateBps = bitrateBps;
   _bitrateAdjuster->SetTargetBitrateBps(_targetBitrateBps);
   [self setBitrateBps:_bitrateAdjuster->GetAdjustedBitrateBps()];
   return WEBRTC_VIDEO_CODEC_OK;
@@ -662,7 +670,7 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
     }
 
     if (!getkVTVideoEncoderSpecification_Usage()) {
-      RTC_LOG(LS_ERROR) << "RTCVideoEncoderH264 cannot create a H264 software encoder";
+      RTC_LOG(LS_ERROR) << "RTCSingleVideoEncoderH264 cannot create a H264 software encoder";
       return WEBRTC_VIDEO_CODEC_ERROR;
     }
 
@@ -879,8 +887,86 @@ CFStringRef ExtractProfile(webrtc::SdpVideoFormat videoFormat) {
 }
 
 - (RTCVideoEncoderQpThresholds *)scalingSettings {
-  return [[RTCVideoEncoderQpThresholds alloc] initWithThresholdsLow:kLowH264QpThreshold
-                                                               high:kHighH264QpThreshold];
+  return [[RTCVideoEncoderQpThresholds alloc] initWithThresholdsLow:kLowH264QpThreshold high:kHighH264QpThreshold];
+}
+
+- (int)setRateAllocation:(nonnull const webrtc::VideoBitrateAllocation *)allocation framerate:(uint32_t)framerate {
+  return 0;
+}
+
+@end
+
+@implementation RTCVideoEncoderH264 {
+    NSMutableArray<RTCSingleVideoEncoderH264*> *_codecs;
+    RTCVideoCodecInfo *_codecInfo;
+}
+- (instancetype)initWithCodecInfo:(RTCVideoCodecInfo *)codecInfo {
+    if (self = [super init]) {
+        _codecInfo = codecInfo;
+    }
+    return self;
+}
+
+- (void)setCallback:(RTCVideoEncoderCallback)callback {
+    for (RTCSingleVideoEncoderH264 *codec : _codecs)
+        [codec setCallback:callback];
+}
+
+- (NSInteger)startEncodeWithSettings:(RTCVideoEncoderSettings *)settings numberOfCores:(int)numberOfCores {
+    auto nativeCodecSettings = settings.nativeVideoCodec;
+
+    _codecs = [[NSMutableArray alloc] init];
+    for (unsigned index = 0 ; index < nativeCodecSettings.numberOfSimulcastStreams; ++index) {
+        auto codec = [[RTCSingleVideoEncoderH264 alloc] initWithCodecInfo:_codecInfo simulcastIndex:index];
+        [_codecs addObject:codec];
+
+        auto codecSettings = nativeCodecSettings;
+        codecSettings.width = nativeCodecSettings.simulcastStream[index].width;
+        codecSettings.height = nativeCodecSettings.simulcastStream[index].height;
+        codecSettings.maxBitrate = nativeCodecSettings.simulcastStream[index].maxBitrate;
+        codecSettings.targetBitrate = nativeCodecSettings.simulcastStream[index].targetBitrate;
+        codecSettings.minBitrate = nativeCodecSettings.simulcastStream[index].minBitrate;
+        codecSettings.qpMax = nativeCodecSettings.simulcastStream[index].qpMax;
+        codecSettings.active = true;
+
+        auto *settings = [[RTCVideoEncoderSettings alloc] initWithNativeVideoCodec:&codecSettings];
+        [codec startEncodeWithSettings:settings numberOfCores:numberOfCores];
+    }
+    return 0;
+}
+
+- (NSInteger)releaseEncoder {
+    for (RTCSingleVideoEncoderH264 *codec : _codecs)
+        [codec releaseEncoder];
+    _codecs = nil;
+    return 0;
+}
+
+- (NSInteger)encode:(RTCVideoFrame *)frame codecSpecificInfo:(nullable id<RTCCodecSpecificInfo>)info frameTypes:(NSArray<NSNumber *> *)frameTypes {
+    int result = 0;
+    for (RTCSingleVideoEncoderH264 *codec : _codecs)
+        result |= [codec encode:frame codecSpecificInfo:info frameTypes:frameTypes];
+    return result;
+}
+
+- (int)setRateAllocation:(const webrtc::VideoBitrateAllocation *)bitRateAllocation framerate:(uint32_t) framerate {
+    int result = 0;
+    unsigned counter = 0;
+    for (RTCSingleVideoEncoderH264 *codec : _codecs)
+        result |= [codec setBitrate:bitRateAllocation->GetSpatialLayerSum(counter++) framerate:framerate];
+    return result;
+}
+
+- (NSString *)implementationName {
+    return @"VideoToolbox";
+}
+
+- (RTCVideoEncoderQpThresholds *)scalingSettings {
+    return [[RTCVideoEncoderQpThresholds alloc] initWithThresholdsLow:kLowH264QpThreshold high:kHighH264QpThreshold];
+}
+
+- (int)setBitrate:(uint32_t)bitrateKbit framerate:(uint32_t)framerate {
+    return 0;
 }
 
 @end
index 6fccf44..8b9cb30 100644 (file)
@@ -37,6 +37,10 @@ typedef NS_ENUM(NSUInteger, RTCVideoContentType) {
   RTCVideoContentTypeScreenshare,
 };
 
+namespace webrtc {
+class VideoBitrateAllocation;
+};
+
 /** Represents an encoded frame. Corresponds to webrtc::EncodedImage. */
 RTC_EXPORT
 __attribute__((objc_runtime_name("WK_RTCEncodedImage")))
@@ -158,6 +162,8 @@ RTC_EXPORT
     codecSpecificInfo:(nullable id<RTCCodecSpecificInfo>)info
            frameTypes:(NSArray<NSNumber *> *)frameTypes;
 - (int)setBitrate:(uint32_t)bitrateKbit framerate:(uint32_t)framerate;
+- (int)setRateAllocation: (const webrtc::VideoBitrateAllocation *)allocation framerate:(uint32_t)framerate;
+
 - (NSString *)implementationName;
 
 /** Returns QP scaling settings for encoder. The quality scaler adjusts the resolution in order to
index 84bf9b7..f01a8ad 100644 (file)
@@ -24,6 +24,7 @@ __attribute__((objc_runtime_name("WK_RTCCodecSpecificInfoH264")))
 @interface RTCCodecSpecificInfoH264 : NSObject <RTCCodecSpecificInfo>
 
 @property(nonatomic, assign) RTCH264PacketizationMode packetizationMode;
+@property(nonatomic, assign) int simulcastIndex;
 
 @end
 
index 3794da6..28c7ea7 100644 (file)
@@ -90,6 +90,10 @@ class ObjCVideoEncoder : public VideoEncoder {
     return [encoder_ setBitrate:bitrate framerate:framerate];
   }
 
+  int32_t SetRateAllocation(const VideoBitrateAllocation& allocation, uint32_t framerate) {
+    return [encoder_ setRateAllocation: &allocation framerate:framerate];
+  }
+
   bool SupportsNativeHandle() const { return true; }
 
   VideoEncoder::ScalingSettings GetScalingSettings() const {
index 79ad534..0aef111 100644 (file)
@@ -70,6 +70,7 @@
                41109AAF1E5FA19200C0955A /* frame_callback.h in Headers */ = {isa = PBXBuildFile; fileRef = 41109AA81E5FA19200C0955A /* frame_callback.h */; };
                41109AB01E5FA19200C0955A /* bitrate_adjuster.h in Headers */ = {isa = PBXBuildFile; fileRef = 41109AA91E5FA19200C0955A /* bitrate_adjuster.h */; };
                41109AB11E5FA19200C0955A /* incoming_video_stream.h in Headers */ = {isa = PBXBuildFile; fileRef = 41109AAA1E5FA19200C0955A /* incoming_video_stream.h */; };
+               41239B3E21476DC000396F81 /* RTCDefaultVideoDecoderFactory.mm in Sources */ = {isa = PBXBuildFile; fileRef = 41EAF1B2212E29C1009F73EC /* RTCDefaultVideoDecoderFactory.mm */; };
                41239B3F21476DC400396F81 /* screenshare_layers.cc in Sources */ = {isa = PBXBuildFile; fileRef = 419C84301FE24E7F0040C30F /* screenshare_layers.cc */; };
                41239B4C2147716D00396F81 /* video_stream_encoder_create.h in Headers */ = {isa = PBXBuildFile; fileRef = 41F263C02126818900274F59 /* video_stream_encoder_create.h */; };
                41239B4D2147743A00396F81 /* video_stream_encoder_create.cc in Sources */ = {isa = PBXBuildFile; fileRef = 4102F6D521273415006AE8D7 /* video_stream_encoder_create.cc */; };
                41EA54111EFC2D1B002FF04C /* sqrt.c in Sources */ = {isa = PBXBuildFile; fileRef = 41EA53FB1EFC2D1B002FF04C /* sqrt.c */; };
                41EA54191EFC2D53002FF04C /* md4.c in Sources */ = {isa = PBXBuildFile; fileRef = 41EA54131EFC2D4F002FF04C /* md4.c */; };
                41EA541A1EFC2D53002FF04C /* md5.c in Sources */ = {isa = PBXBuildFile; fileRef = 41EA54181EFC2D53002FF04C /* md5.c */; };
+               41EAF1B6212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.mm in Sources */ = {isa = PBXBuildFile; fileRef = 41EAF1B3212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.mm */; };
                41ECEAB620630108009D5141 /* RTCVideoCodec+Private.h in Headers */ = {isa = PBXBuildFile; fileRef = 41ECEAB320630107009D5141 /* RTCVideoCodec+Private.h */; };
                41ECEAB720630108009D5141 /* RTCVideoCodec.mm in Sources */ = {isa = PBXBuildFile; fileRef = 41ECEAB420630107009D5141 /* RTCVideoCodec.mm */; };
                41ECEABC206403C2009D5141 /* WebKitUtilities.h in Headers */ = {isa = PBXBuildFile; fileRef = 41ECEABB206403C1009D5141 /* WebKitUtilities.h */; };
                41EA54161EFC2D53002FF04C /* md5-586.pl */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.script.perl; path = "md5-586.pl"; sourceTree = "<group>"; };
                41EA54171EFC2D53002FF04C /* md5-x86_64.pl */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.script.perl; path = "md5-x86_64.pl"; sourceTree = "<group>"; };
                41EA54181EFC2D53002FF04C /* md5.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = md5.c; sourceTree = "<group>"; };
-               41EAF1B2212E29C1009F73EC /* RTCDefaultVideoDecoderFactory.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; name = RTCDefaultVideoDecoderFactory.m; path = PeerConnection/RTCDefaultVideoDecoderFactory.m; sourceTree = "<group>"; };
-               41EAF1B3212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; name = RTCDefaultVideoEncoderFactory.m; path = PeerConnection/RTCDefaultVideoEncoderFactory.m; sourceTree = "<group>"; };
+               41EAF1B2212E29C1009F73EC /* RTCDefaultVideoDecoderFactory.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = RTCDefaultVideoDecoderFactory.mm; path = PeerConnection/RTCDefaultVideoDecoderFactory.mm; sourceTree = "<group>"; };
+               41EAF1B3212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = RTCDefaultVideoEncoderFactory.mm; path = PeerConnection/RTCDefaultVideoEncoderFactory.mm; sourceTree = "<group>"; };
                41EAF1B4212E29C1009F73EC /* RTCVideoCodecVP8.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = RTCVideoCodecVP8.mm; path = PeerConnection/RTCVideoCodecVP8.mm; sourceTree = "<group>"; };
                41EAF1B9212E2AAD009F73EC /* vpx_codec.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = vpx_codec.c; sourceTree = "<group>"; };
                41EAF1BA212E2AAD009F73EC /* vpx_encoder.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = vpx_encoder.c; sourceTree = "<group>"; };
                41ECEAB2206300E3009D5141 /* PeerConnection */ = {
                        isa = PBXGroup;
                        children = (
-                               41EAF1B2212E29C1009F73EC /* RTCDefaultVideoDecoderFactory.m */,
-                               41EAF1B3212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.m */,
+                               41EAF1B2212E29C1009F73EC /* RTCDefaultVideoDecoderFactory.mm */,
+                               41EAF1B3212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.mm */,
                                41ECEAC720641328009D5141 /* RTCEncodedImage.mm */,
                                4192423F21276D4700634FCF /* RTCPeerConnectionFactory.mm */,
                                41ECEAD9206414A6009D5141 /* RTCRtpFragmentationHeader.mm */,
                                413A230F1FE18E0700373E99 /* rtccertificate.cc in Sources */,
                                413A22BE1FE18E0700373E99 /* rtccertificategenerator.cc in Sources */,
                                41ECEAC620641303009D5141 /* RTCCVPixelBuffer.mm in Sources */,
+                               41239B3E21476DC000396F81 /* RTCDefaultVideoDecoderFactory.mm in Sources */,
+                               41EAF1B6212E29C1009F73EC /* RTCDefaultVideoEncoderFactory.mm in Sources */,
                                41ECEAC820641328009D5141 /* RTCEncodedImage.mm in Sources */,
                                41F411B01EF8DA0100343C26 /* rtcerror.cc in Sources */,
                                41ECEACF20641370009D5141 /* RTCI420Buffer.mm in Sources */,
index 0708190..29c0dfe 100644 (file)
@@ -1,3 +1,21 @@
+2018-10-03  Youenn Fablet  <youenn@apple.com>
+
+        Enable H264 simulcast
+        https://bugs.webkit.org/show_bug.cgi?id=190167
+
+        Reviewed by Eric Carlson.
+
+        Activate H264 simulcast trial field.
+        Make track.getSettings() expose width and height for incoming tracks.
+
+        Test: webrtc/simulcast-h264.html
+
+        * Configurations/WebCore.xcconfig:
+        * Modules/mediastream/libwebrtc/LibWebRTCMediaEndpoint.cpp:
+        (WebCore::LibWebRTCMediaEndpoint::LibWebRTCMediaEndpoint):
+        * platform/mediastream/RealtimeIncomingVideoSource.cpp:
+        (WebCore::RealtimeIncomingVideoSource::RealtimeIncomingVideoSource):
+
 2018-10-03  Michael Catanzaro  <mcatanzaro@igalia.com>
 
         -Wunused-variable in RenderLayer::updateScrollableAreaSet
index 44f651b..91e9334 100644 (file)
@@ -57,6 +57,7 @@
 #include <webrtc/p2p/base/basicpacketsocketfactory.h>
 #include <webrtc/p2p/client/basicportallocator.h>
 #include <webrtc/pc/peerconnectionfactory.h>
+#include <webrtc/system_wrappers/include/field_trial_default.h>
 #include <wtf/MainThread.h>
 
 namespace WebCore {
@@ -75,6 +76,8 @@ LibWebRTCMediaEndpoint::LibWebRTCMediaEndpoint(LibWebRTCPeerConnectionBackend& p
 {
     ASSERT(isMainThread());
     ASSERT(client.factory());
+
+    webrtc::field_trial::InitFieldTrialsFromString("WebRTC-H264Simulcast/Enabled/");
 }
 
 bool LibWebRTCMediaEndpoint::setConfiguration(LibWebRTCProvider& client, webrtc::PeerConnectionInterface::RTCConfiguration&& configuration)
index d5b4964..b678280 100644 (file)
@@ -42,6 +42,12 @@ RealtimeIncomingVideoSource::RealtimeIncomingVideoSource(rtc::scoped_refptr<webr
     , m_videoTrack(WTFMove(videoTrack))
 {
     notifyMutedChange(!m_videoTrack);
+
+    RealtimeMediaSourceSupportedConstraints constraints;
+    constraints.setSupportsWidth(true);
+    constraints.setSupportsHeight(true);
+    m_currentSettings = RealtimeMediaSourceSettings { };
+    m_currentSettings->setSupportedConstraints(WTFMove(constraints));
 }
 
 void RealtimeIncomingVideoSource::startProducingData()