[MediaStream] Add Mac screen capture source
authoreric.carlson@apple.com <eric.carlson@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 5 Jan 2018 23:34:30 +0000 (23:34 +0000)
committereric.carlson@apple.com <eric.carlson@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 5 Jan 2018 23:34:30 +0000 (23:34 +0000)
https://bugs.webkit.org/show_bug.cgi?id=181333
<rdar://problem/36323219>

Reviewed by Dean Jackson.

Source/WebCore:

* SourcesCocoa.txt: Add ScreenDisplayCaptureSourceMac.mm.

* WebCore.xcodeproj/project.pbxproj: Ditto.

* platform/cocoa/CoreVideoSoftLink.cpp: Declare new constants used.
* platform/cocoa/CoreVideoSoftLink.h:

* platform/mediastream/mac/DisplayCaptureManagerCocoa.cpp:
(WebCore::displayReconfigurationCallBack): Call refreshCaptureDevices.
(WebCore::DisplayCaptureManagerCocoa::~DisplayCaptureManagerCocoa): Unregister for display
reconfiguration callbacks.
(WebCore::DisplayCaptureManagerCocoa::captureDevices): Register for display reconfigrations.
(WebCore::DisplayCaptureManagerCocoa::refreshCaptureDevices): Use CGActiveDisplayList to
get list of active screens.
(WebCore::DisplayCaptureManagerCocoa::screenCaptureDeviceWithPersistentID): Validate screen
ID, return CaptureDevice.
* platform/mediastream/mac/DisplayCaptureManagerCocoa.h:

* platform/mediastream/mac/RealtimeMediaSourceCenterMac.cpp:
(WebCore::VideoCaptureSourceFactoryMac::createVideoCaptureSource): Deal with screen capture
on macOS.

Implement Mac screen capture with CGDisplayStream.
* platform/mediastream/mac/ScreenDisplayCaptureSourceMac.h: Added.
(WebCore::ScreenDisplayCaptureSourceMac::DisplaySurface::~DisplaySurface):
(WebCore::ScreenDisplayCaptureSourceMac::DisplaySurface::operator=):
(WebCore::ScreenDisplayCaptureSourceMac::DisplaySurface::ioSurface const):
* platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm: Added.
(WebCore::roundUpToMacroblockMultiple):
(WebCore::ScreenDisplayCaptureSourceMac::updateDisplayID):
(WebCore::ScreenDisplayCaptureSourceMac::create):
(WebCore::ScreenDisplayCaptureSourceMac::ScreenDisplayCaptureSourceMac):
(WebCore::ScreenDisplayCaptureSourceMac::~ScreenDisplayCaptureSourceMac):
(WebCore::ScreenDisplayCaptureSourceMac::createDisplayStream):
(WebCore::ScreenDisplayCaptureSourceMac::startProducingData):
(WebCore::ScreenDisplayCaptureSourceMac::stopProducingData):
(WebCore::ScreenDisplayCaptureSourceMac::sampleBufferFromPixelBuffer):
(WebCore::ScreenDisplayCaptureSourceMac::pixelBufferFromIOSurface):
(WebCore::ScreenDisplayCaptureSourceMac::generateFrame):
(WebCore::ScreenDisplayCaptureSourceMac::startDisplayStream):
(WebCore::ScreenDisplayCaptureSourceMac::applySize):
(WebCore::ScreenDisplayCaptureSourceMac::applyFrameRate):
(WebCore::ScreenDisplayCaptureSourceMac::commitConfiguration):
(WebCore::ScreenDisplayCaptureSourceMac::displayWasReconfigured):
(WebCore::ScreenDisplayCaptureSourceMac::displayReconfigurationCallBack):
(WebCore::ScreenDisplayCaptureSourceMac::frameAvailable):

Source/WebCore/PAL:

* pal/spi/cg/CoreGraphicsSPI.h: Declare some CGDisplayMode SPI.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@226468 268f45cc-cd09-0410-ab3c-d52691b4dbfc

12 files changed:
Source/WebCore/ChangeLog
Source/WebCore/PAL/ChangeLog
Source/WebCore/PAL/pal/spi/cg/CoreGraphicsSPI.h
Source/WebCore/SourcesCocoa.txt
Source/WebCore/WebCore.xcodeproj/project.pbxproj
Source/WebCore/platform/cocoa/CoreVideoSoftLink.cpp
Source/WebCore/platform/cocoa/CoreVideoSoftLink.h
Source/WebCore/platform/mediastream/mac/DisplayCaptureManagerCocoa.cpp
Source/WebCore/platform/mediastream/mac/DisplayCaptureManagerCocoa.h
Source/WebCore/platform/mediastream/mac/RealtimeMediaSourceCenterMac.cpp
Source/WebCore/platform/mediastream/mac/ScreenDisplayCaptureSourceMac.h [new file with mode: 0644]
Source/WebCore/platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm [new file with mode: 0644]

index 1a5a0779b78741f1b686e17be9b544ac03616c8b..dbd5b09a30968efaa068ec3305234665747946e2 100644 (file)
@@ -1,3 +1,58 @@
+2018-01-05  Eric Carlson  <eric.carlson@apple.com>
+
+        [MediaStream] Add Mac screen capture source
+        https://bugs.webkit.org/show_bug.cgi?id=181333
+        <rdar://problem/36323219>
+
+        Reviewed by Dean Jackson.
+
+        * SourcesCocoa.txt: Add ScreenDisplayCaptureSourceMac.mm.
+
+        * WebCore.xcodeproj/project.pbxproj: Ditto.
+
+        * platform/cocoa/CoreVideoSoftLink.cpp: Declare new constants used.
+        * platform/cocoa/CoreVideoSoftLink.h:
+
+        * platform/mediastream/mac/DisplayCaptureManagerCocoa.cpp:
+        (WebCore::displayReconfigurationCallBack): Call refreshCaptureDevices.
+        (WebCore::DisplayCaptureManagerCocoa::~DisplayCaptureManagerCocoa): Unregister for display
+        reconfiguration callbacks.
+        (WebCore::DisplayCaptureManagerCocoa::captureDevices): Register for display reconfigrations.
+        (WebCore::DisplayCaptureManagerCocoa::refreshCaptureDevices): Use CGActiveDisplayList to
+        get list of active screens.
+        (WebCore::DisplayCaptureManagerCocoa::screenCaptureDeviceWithPersistentID): Validate screen
+        ID, return CaptureDevice.
+        * platform/mediastream/mac/DisplayCaptureManagerCocoa.h:
+
+        * platform/mediastream/mac/RealtimeMediaSourceCenterMac.cpp:
+        (WebCore::VideoCaptureSourceFactoryMac::createVideoCaptureSource): Deal with screen capture
+        on macOS.
+
+        Implement Mac screen capture with CGDisplayStream.
+        * platform/mediastream/mac/ScreenDisplayCaptureSourceMac.h: Added.
+        (WebCore::ScreenDisplayCaptureSourceMac::DisplaySurface::~DisplaySurface):
+        (WebCore::ScreenDisplayCaptureSourceMac::DisplaySurface::operator=):
+        (WebCore::ScreenDisplayCaptureSourceMac::DisplaySurface::ioSurface const):
+        * platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm: Added.
+        (WebCore::roundUpToMacroblockMultiple):
+        (WebCore::ScreenDisplayCaptureSourceMac::updateDisplayID):
+        (WebCore::ScreenDisplayCaptureSourceMac::create):
+        (WebCore::ScreenDisplayCaptureSourceMac::ScreenDisplayCaptureSourceMac):
+        (WebCore::ScreenDisplayCaptureSourceMac::~ScreenDisplayCaptureSourceMac):
+        (WebCore::ScreenDisplayCaptureSourceMac::createDisplayStream):
+        (WebCore::ScreenDisplayCaptureSourceMac::startProducingData):
+        (WebCore::ScreenDisplayCaptureSourceMac::stopProducingData):
+        (WebCore::ScreenDisplayCaptureSourceMac::sampleBufferFromPixelBuffer):
+        (WebCore::ScreenDisplayCaptureSourceMac::pixelBufferFromIOSurface):
+        (WebCore::ScreenDisplayCaptureSourceMac::generateFrame):
+        (WebCore::ScreenDisplayCaptureSourceMac::startDisplayStream):
+        (WebCore::ScreenDisplayCaptureSourceMac::applySize):
+        (WebCore::ScreenDisplayCaptureSourceMac::applyFrameRate):
+        (WebCore::ScreenDisplayCaptureSourceMac::commitConfiguration):
+        (WebCore::ScreenDisplayCaptureSourceMac::displayWasReconfigured):
+        (WebCore::ScreenDisplayCaptureSourceMac::displayReconfigurationCallBack):
+        (WebCore::ScreenDisplayCaptureSourceMac::frameAvailable):
+
 2018-01-05  Don Olmstead  <don.olmstead@sony.com>
 
         [curl] Can't load file:// URL with a URL fragment identifier
index 2294aa9f9adf986f537d4d846f55af4a7f1c20f2..ecd69796cfd37be55389fbff9a430c1b9ae4c518 100644 (file)
@@ -1,3 +1,13 @@
+2018-01-05  Eric Carlson  <eric.carlson@apple.com>
+
+        [MediaStream] Add Mac screen capture source
+        https://bugs.webkit.org/show_bug.cgi?id=181333
+        <rdar://problem/36323219>
+
+        Reviewed by Dean Jackson.
+
+        * pal/spi/cg/CoreGraphicsSPI.h: Declare some CGDisplayMode SPI.
+
 2018-01-03  Ting-Wei Lan  <lantw44@gmail.com>
 
         Replace hard-coded paths in shebangs with #!/usr/bin/env
index 1a12d83a6d37ea85217af50cdaf07de58819446e..8841d6dc6a7ad2bdac17a14fb2c98f257dbbe33e 100644 (file)
@@ -296,6 +296,10 @@ CGError CGSGetScreenRectForWindow(CGSConnectionID, CGSWindowID, CGRect *);
 CGError CGSRegisterConnectionNotifyProc(CGSConnectionID, CGSNotifyConnectionProcPtr, CGSNotificationType, void* arg);
 CGError CGSRegisterNotifyProc(CGSNotifyProcPtr, CGSNotificationType, void* arg);
 bool ColorSyncProfileIsWideGamut(ColorSyncProfileRef);
+
+size_t CGDisplayModeGetPixelsWide(CGDisplayModeRef);
+size_t CGDisplayModeGetPixelsHigh(CGDisplayModeRef);
+
 #endif
 
 WTF_EXTERN_C_END
index ec11692847889ac4fa19edfffad39c715b23ea82..6aab4c91a31baaa31438868467bb0ee296c7d533 100644 (file)
@@ -361,6 +361,7 @@ platform/mediastream/mac/RealtimeIncomingVideoSourceCocoa.cpp
 platform/mediastream/mac/RealtimeMediaSourceCenterMac.cpp
 platform/mediastream/mac/RealtimeOutgoingAudioSourceCocoa.cpp
 platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp
+platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm
 
 platform/audio/mac/AudioSampleDataSource.mm
 
index de190a12f1e0309ec25bbaeba356120c9a3f7b7a..7339fc317422fd635fe2ba1f633a25a53a6d367f 100644 (file)
                0709D7911AE5557E004E42F8 /* WebMediaSessionManagerMac.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebMediaSessionManagerMac.h; sourceTree = "<group>"; };
                0709D7941AE55A29004E42F8 /* WebMediaSessionManagerClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebMediaSessionManagerClient.h; sourceTree = "<group>"; };
                0709FC4D1025DEE30059CDBA /* AccessibilitySlider.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AccessibilitySlider.h; sourceTree = "<group>"; };
+               070A9F5E1FFECC70003DF649 /* ScreenDisplayCaptureSourceMac.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ScreenDisplayCaptureSourceMac.h; sourceTree = "<group>"; };
+               070A9F601FFECC71003DF649 /* ScreenDisplayCaptureSourceMac.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = ScreenDisplayCaptureSourceMac.mm; sourceTree = "<group>"; };
                070DD8F50F01868000727DEB /* mediaControls.css */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = mediaControls.css; sourceTree = "<group>"; };
                070E09181875ED93003A1D3C /* PlatformMediaSession.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = PlatformMediaSession.h; sourceTree = "<group>"; };
                070E091A1875EF71003A1D3C /* PlatformMediaSession.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = PlatformMediaSession.cpp; sourceTree = "<group>"; };
                                41103AA81E39790A00769F14 /* RealtimeOutgoingAudioSourceCocoa.h */,
                                5CDD833B1E4324BB00621B83 /* RealtimeOutgoingVideoSourceCocoa.cpp */,
                                5CDD833C1E4324BB00621B83 /* RealtimeOutgoingVideoSourceCocoa.h */,
+                               070A9F5E1FFECC70003DF649 /* ScreenDisplayCaptureSourceMac.h */,
+                               070A9F601FFECC71003DF649 /* ScreenDisplayCaptureSourceMac.mm */,
                                07D6373E1BB0B11300256CE9 /* WebAudioSourceProviderAVFObjC.h */,
                                07D6373F1BB0B11300256CE9 /* WebAudioSourceProviderAVFObjC.mm */,
                        );
index ed81cc93a9a90ca345718cbb836496ca8bf67a02..7af100c5a525de27e2cc832be798a8790aed3983 100644 (file)
@@ -82,3 +82,10 @@ SOFT_LINK_FUNCTION_FOR_SOURCE(WebCore, CoreVideo, CVOpenGLTextureGetName, GLuint
 SOFT_LINK_FUNCTION_FOR_SOURCE(WebCore, CoreVideo, CVOpenGLTextureGetCleanTexCoords, void, (CVOpenGLTextureRef image, GLfloat lowerLeft[2], GLfloat lowerRight[2], GLfloat upperLeft[2], GLfloat upperRight[2]), (image, lowerLeft, lowerRight, upperLeft, upperRight))
 SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferIOSurfaceOpenGLFBOCompatibilityKey, CFStringRef)
 #endif
+
+#if PLATFORM(MAC)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferExtendedPixelsRightKey, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferExtendedPixelsBottomKey, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferOpenGLCompatibilityKey, CFStringRef)
+#endif
+
index b0fe99e21fe12ea58eace804e782eac7f5ab0568..f1f01e30e3d6f799d5b3562a54b6a9c771769e41 100644 (file)
@@ -133,4 +133,13 @@ SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferIOSurfaceOpenGLF
 #define kCVPixelBufferIOSurfaceOpenGLFBOCompatibilityKey get_CoreVideo_kCVPixelBufferIOSurfaceOpenGLFBOCompatibilityKey()
 #endif
 
+#if PLATFORM(MAC)
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferExtendedPixelsRightKey, CFStringRef)
+#define kCVPixelBufferExtendedPixelsRightKey get_CoreVideo_kCVPixelBufferExtendedPixelsRightKey()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferExtendedPixelsBottomKey, CFStringRef)
+#define kCVPixelBufferExtendedPixelsBottomKey get_CoreVideo_kCVPixelBufferExtendedPixelsBottomKey()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferOpenGLCompatibilityKey, CFStringRef)
+#define kCVPixelBufferOpenGLCompatibilityKey get_CoreVideo_kCVPixelBufferOpenGLCompatibilityKey()
+#endif
+
 #endif // CoreVideoSoftLink_h
index c820d30b9afb0cfb93e6aec6cb90ae3d2fedbbe5..ceb2a8b5dffb6bc4820a4a534f63433cd3590ba0 100644 (file)
 #include "Logging.h"
 #include <wtf/NeverDestroyed.h>
 
+#if PLATFORM(MAC)
+#include "ScreenDisplayCaptureSourceMac.h"
+#include <CoreGraphics/CGDirectDisplay.h>
+#endif
+
 namespace WebCore {
 
+#if PLATFORM(MAC)
+static void displayReconfigurationCallBack(CGDirectDisplayID, CGDisplayChangeSummaryFlags, void* userInfo)
+{
+    if (userInfo)
+        reinterpret_cast<DisplayCaptureManagerCocoa*>(userInfo)->refreshCaptureDevices();
+}
+#endif
+
 DisplayCaptureManagerCocoa& DisplayCaptureManagerCocoa::singleton()
 {
     static NeverDestroyed<DisplayCaptureManagerCocoa> manager;
@@ -41,17 +54,102 @@ DisplayCaptureManagerCocoa& DisplayCaptureManagerCocoa::singleton()
 
 DisplayCaptureManagerCocoa::~DisplayCaptureManagerCocoa()
 {
+#if PLATFORM(MAC)
+    if (m_observingDisplayChanges)
+        CGDisplayRemoveReconfigurationCallback(displayReconfigurationCallBack, this);
+#endif
 }
 
 const Vector<CaptureDevice>& DisplayCaptureManagerCocoa::captureDevices()
 {
+    static bool initialized;
+    if (!initialized) {
+        refreshCaptureDevices();
+
+#if PLATFORM(MAC)
+        CGDisplayRegisterReconfigurationCallback(displayReconfigurationCallBack, this);
+#endif
+
+        m_observingDisplayChanges = true;
+        initialized = true;
+    };
+    
     return m_displays;
 }
 
+void DisplayCaptureManagerCocoa::refreshCaptureDevices()
+{
+#if PLATFORM(MAC)
+    uint32_t displayCount = 0;
+    auto err = CGGetActiveDisplayList(0, nullptr, &displayCount);
+    if (err) {
+        RELEASE_LOG(Media, "CGGetActiveDisplayList() returned error %d when trying to get display count", (int)err);
+        return;
+    }
+
+    if (!displayCount) {
+        RELEASE_LOG(Media, "CGGetActiveDisplayList() returned a display count of 0");
+        return;
+    }
+
+    CGDirectDisplayID activeDisplays[displayCount];
+    err = CGGetActiveDisplayList(displayCount, &(activeDisplays[0]), &displayCount);
+    if (err) {
+        RELEASE_LOG(Media, "CGGetActiveDisplayList() returned error %d when trying to get the active display list", (int)err);
+        return;
+    }
+
+    bool haveDeviceChanges = false;
+    for (auto displayID : activeDisplays) {
+        if (std::any_of(m_displaysInternal.begin(), m_displaysInternal.end(), [displayID](auto& device) { return device.cgDirectDisplayID == displayID; }))
+            continue;
+        haveDeviceChanges = true;
+        m_displaysInternal.append({ displayID, CGDisplayIDToOpenGLDisplayMask(displayID) });
+    }
+
+    for (auto& display : m_displaysInternal) {
+        auto displayMask = CGDisplayIDToOpenGLDisplayMask(display.cgDirectDisplayID);
+        if (display.cgOpenGLDisplayMask != displayMask) {
+            display.cgOpenGLDisplayMask = displayMask;
+            haveDeviceChanges = true;
+        }
+    }
+
+    if (!haveDeviceChanges)
+        return;
+
+    int count = 0;
+    m_displays = Vector<CaptureDevice>();
+    for (auto& device : m_displaysInternal) {
+        CaptureDevice displayDevice(String::number(device.cgDirectDisplayID), CaptureDevice::DeviceType::Screen, makeString("Screen ", String::number(count++)));
+        displayDevice.setEnabled(device.cgOpenGLDisplayMask);
+        m_displays.append(WTFMove(displayDevice));
+    }
+#endif
+}
+
 std::optional<CaptureDevice> DisplayCaptureManagerCocoa::screenCaptureDeviceWithPersistentID(const String& deviceID)
 {
+#if PLATFORM(MAC)
+    bool ok;
+    auto displayID = deviceID.toUIntStrict(&ok);
+    if (!ok) {
+        RELEASE_LOG(Media, "Display ID does not convert to 32-bit integer");
+        return std::nullopt;
+    }
+
+    auto actualDisplayID = ScreenDisplayCaptureSourceMac::updateDisplayID(displayID);
+    if (!actualDisplayID)
+        return std::nullopt;
+
+    auto device = CaptureDevice(String::number(actualDisplayID.value()), CaptureDevice::DeviceType::Screen, ASCIILiteral("ScreenCaptureDevice"));
+    device.setEnabled(true);
+
+    return device;
+#else
     UNUSED_PARAM(deviceID);
     return std::nullopt;
+#endif
 }
 
 std::optional<CaptureDevice> DisplayCaptureManagerCocoa::captureDeviceWithPersistentID(CaptureDevice::DeviceType type, const String& id)
index 84962f72748d6c7d713694a498eaa25b7bbb0470..d762864f2e424f7ff59c8b5e25fa63c4c6a4e54f 100644 (file)
@@ -36,6 +36,8 @@ public:
     static DisplayCaptureManagerCocoa& singleton();
     DisplayCaptureManagerCocoa() = default;
 
+    void refreshCaptureDevices() final;
+
 private:
     virtual ~DisplayCaptureManagerCocoa();
 
@@ -43,7 +45,13 @@ private:
     std::optional<CaptureDevice> captureDeviceWithPersistentID(CaptureDevice::DeviceType, const String&) final;
     std::optional<CaptureDevice> screenCaptureDeviceWithPersistentID(const String&);
 
+    struct CGDisplayCaptureDevice {
+        uint32_t cgDirectDisplayID;
+        uint32_t cgOpenGLDisplayMask;
+    };
+    Vector<CGDisplayCaptureDevice> m_displaysInternal;
     Vector<CaptureDevice> m_displays;
+    bool m_observingDisplayChanges { false };
 };
 
 } // namespace WebCore
index 735e2639984d1cfcbdc57191526a99cd43ea2c56..5875149108d3f462258e0c20e42992d491a2b68e 100644 (file)
@@ -41,6 +41,7 @@
 #include "DisplayCaptureManagerCocoa.h"
 #include "Logging.h"
 #include "MediaStreamPrivate.h"
+#include "ScreenDisplayCaptureSourceMac.h"
 #include <wtf/MainThread.h>
 
 namespace WebCore {
@@ -55,6 +56,10 @@ public:
             return AVVideoCaptureSource::create(device.persistentId(), constraints);
             break;
         case CaptureDevice::DeviceType::Screen:
+#if PLATFORM(MAC)
+            return ScreenDisplayCaptureSourceMac::create(device.persistentId(), constraints);
+            break;
+#endif
         case CaptureDevice::DeviceType::Application:
         case CaptureDevice::DeviceType::Window:
         case CaptureDevice::DeviceType::Browser:
diff --git a/Source/WebCore/platform/mediastream/mac/ScreenDisplayCaptureSourceMac.h b/Source/WebCore/platform/mediastream/mac/ScreenDisplayCaptureSourceMac.h
new file mode 100644 (file)
index 0000000..5e18748
--- /dev/null
@@ -0,0 +1,116 @@
+/*
+ * Copyright (C) 2017 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if ENABLE(MEDIA_STREAM) && PLATFORM(MAC)
+
+#include "DisplayCaptureSourceCocoa.h"
+#include <CoreGraphics/CGDisplayConfiguration.h>
+#include <CoreGraphics/CGDisplayStream.h>
+#include <wtf/Lock.h>
+#include <wtf/OSObjectPtr.h>
+#include <wtf/WeakPtr.h>
+
+typedef struct __CVBuffer *CVPixelBufferRef;
+typedef struct opaqueCMSampleBuffer *CMSampleBufferRef;
+
+namespace WebCore {
+
+class ScreenDisplayCaptureSourceMac : public DisplayCaptureSourceCocoa {
+public:
+    static CaptureSourceOrError create(const String&, const MediaConstraints*);
+
+    WEBCORE_EXPORT static VideoCaptureFactory& factory();
+
+    WEBCORE_EXPORT static std::optional<CGDirectDisplayID> updateDisplayID(CGDirectDisplayID);
+
+private:
+    ScreenDisplayCaptureSourceMac(uint32_t);
+    virtual ~ScreenDisplayCaptureSourceMac();
+
+    static void displayReconfigurationCallBack(CGDirectDisplayID, CGDisplayChangeSummaryFlags, void*);
+
+    void displayWasReconfigured(CGDirectDisplayID, CGDisplayChangeSummaryFlags);
+
+    void frameAvailable(CGDisplayStreamFrameStatus, uint64_t, IOSurfaceRef, CGDisplayStreamUpdateRef);
+
+    void generateFrame() final;
+    void startProducingData() final;
+    void stopProducingData() final;
+    bool applySize(const IntSize&) final;
+    bool applyFrameRate(double) final;
+    void commitConfiguration() final;
+
+    RetainPtr<CMSampleBufferRef> sampleBufferFromPixelBuffer(CVPixelBufferRef);
+    RetainPtr<CVPixelBufferRef> pixelBufferFromIOSurface(IOSurfaceRef);
+    bool createDisplayStream();
+    void startDisplayStream();
+    
+    class DisplaySurface {
+    public:
+        DisplaySurface() = default;
+        ~DisplaySurface()
+        {
+            if (m_surface)
+                IOSurfaceDecrementUseCount(m_surface.get());
+        }
+
+        DisplaySurface& operator=(IOSurfaceRef surface)
+        {
+            if (m_surface)
+                IOSurfaceDecrementUseCount(m_surface.get());
+            if (surface)
+                IOSurfaceIncrementUseCount(surface);
+            m_surface = surface;
+            return *this;
+        }
+
+        IOSurfaceRef ioSurface() const { return m_surface.get(); }
+
+    private:
+        RetainPtr<IOSurfaceRef> m_surface;
+    };
+
+    mutable Lock m_currentFrameMutex;
+    DisplaySurface m_currentFrame;
+    RetainPtr<CGDisplayStreamRef> m_displayStream;
+    RetainPtr<CFMutableDictionaryRef> m_bufferAttributes;
+    CGDisplayStreamFrameAvailableHandler m_frameAvailableBlock;
+    WeakPtrFactory<ScreenDisplayCaptureSourceMac> m_weakFactory;
+    MediaTime m_presentationTimeStamp;
+    MediaTime m_frameDuration;
+
+    OSObjectPtr<dispatch_queue_t> m_captureQueue;
+
+    double m_lastFrameTime { NAN };
+    uint32_t m_displayID { 0 };
+    bool m_isRunning { false };
+    bool m_observingDisplayChanges { false };
+};
+
+} // namespace WebCore
+
+#endif // ENABLE(MEDIA_STREAM) && PLATFORM(MAC)
diff --git a/Source/WebCore/platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm b/Source/WebCore/platform/mediastream/mac/ScreenDisplayCaptureSourceMac.mm
new file mode 100644 (file)
index 0000000..314a172
--- /dev/null
@@ -0,0 +1,397 @@
+/*
+ * Copyright (C) 2017 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "ScreenDisplayCaptureSourceMac.h"
+
+#if ENABLE(MEDIA_STREAM) && PLATFORM(MAC)
+
+#include "GraphicsContextCG.h"
+#include "ImageBuffer.h"
+#include "Logging.h"
+#include "MediaConstraints.h"
+#include "MediaSampleAVFObjC.h"
+#include "NotImplemented.h"
+#include "PlatformLayer.h"
+#include "RealtimeMediaSourceSettings.h"
+
+#include "CoreVideoSoftLink.h"
+
+extern "C" {
+size_t CGDisplayModeGetPixelsWide(CGDisplayModeRef);
+size_t CGDisplayModeGetPixelsHigh(CGDisplayModeRef);
+}
+
+namespace WebCore {
+
+static int32_t roundUpToMacroblockMultiple(int32_t size)
+{
+    return (size + 15) & ~15;
+}
+
+std::optional<CGDirectDisplayID> ScreenDisplayCaptureSourceMac::updateDisplayID(CGDirectDisplayID displayID)
+{
+    uint32_t displayCount = 0;
+    auto err = CGGetActiveDisplayList(0, nullptr, &displayCount);
+    if (err) {
+        RELEASE_LOG(Media, "CGGetActiveDisplayList() returned error %d when trying to get display count", static_cast<int>(err));
+        return std::nullopt;
+    }
+
+    if (!displayCount) {
+        RELEASE_LOG(Media, "CGGetActiveDisplayList() returned a display count of 0");
+        return std::nullopt;
+    }
+
+    CGDirectDisplayID activeDisplays[displayCount];
+    err = CGGetActiveDisplayList(displayCount, &(activeDisplays[0]), &displayCount);
+    if (err) {
+        RELEASE_LOG(Media, "CGGetActiveDisplayList() returned error %d when trying to get the active display list", static_cast<int>(err));
+        return std::nullopt;
+    }
+
+    auto displayMask = CGDisplayIDToOpenGLDisplayMask(displayID);
+    for (auto display : activeDisplays) {
+        if (displayMask == CGDisplayIDToOpenGLDisplayMask(display))
+            return display;
+    }
+
+    return std::nullopt;
+}
+
+CaptureSourceOrError ScreenDisplayCaptureSourceMac::create(const String& deviceID, const MediaConstraints* constraints)
+{
+    bool ok;
+    auto displayID = deviceID.toUIntStrict(&ok);
+    if (!ok) {
+        RELEASE_LOG(Media, "Display ID does not convert to 32-bit integer");
+        return { };
+    }
+
+    auto actualDisplayID = updateDisplayID(displayID);
+    if (!actualDisplayID)
+        return { };
+
+    auto source = adoptRef(*new ScreenDisplayCaptureSourceMac(actualDisplayID.value()));
+    if (constraints && source->applyConstraints(*constraints))
+        return { };
+
+    return CaptureSourceOrError(WTFMove(source));
+}
+
+ScreenDisplayCaptureSourceMac::ScreenDisplayCaptureSourceMac(uint32_t displayID)
+    : DisplayCaptureSourceCocoa("Screen")
+    , m_displayID(displayID)
+{
+}
+
+ScreenDisplayCaptureSourceMac::~ScreenDisplayCaptureSourceMac()
+{
+    if (m_observingDisplayChanges)
+        CGDisplayRemoveReconfigurationCallback(displayReconfigurationCallBack, this);
+
+    m_currentFrame = nullptr;
+}
+
+bool ScreenDisplayCaptureSourceMac::createDisplayStream()
+{
+    static const int screenQueueMaximumLength = 6;
+
+    auto actualDisplayID = updateDisplayID(m_displayID);
+    if (!actualDisplayID) {
+        captureFailed();
+        return false;
+    }
+
+    if (m_displayID != actualDisplayID.value()) {
+        m_displayID = actualDisplayID.value();
+        RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::create(%p), display ID changed to %d", this, static_cast<int>(m_displayID));
+    }
+
+    if (!m_displayStream) {
+
+        if (size().isEmpty()) {
+            CGDisplayModeRef displayMode = CGDisplayCopyDisplayMode(m_displayID);
+            auto screenWidth = CGDisplayModeGetPixelsWide(displayMode);
+            auto screenHeight = CGDisplayModeGetPixelsHigh(displayMode);
+            if (!screenWidth || !screenHeight) {
+                RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::createDisplayStream(%p), unable to get screen width/height", this);
+                captureFailed();
+                return false;
+            }
+            setWidth(screenWidth);
+            setHeight(screenHeight);
+            CGDisplayModeRelease(displayMode);
+        }
+
+        if (!m_captureQueue)
+            m_captureQueue = adoptOSObject(dispatch_queue_create("ScreenDisplayCaptureSourceMac Capture Queue", DISPATCH_QUEUE_SERIAL));
+
+        static CGColorSpaceRef deviceRGBColorSpace = CGColorSpaceCreateDeviceRGB();
+        double frameTime = 1 / frameRate();
+        auto frameTimeCF = adoptCF(CFNumberCreate(nullptr,  kCFNumberDoubleType,  &frameTime));
+        int depth = screenQueueMaximumLength;
+        auto depthCF = adoptCF(CFNumberCreate(nullptr,  kCFNumberIntType,  &depth));
+        CFTypeRef keys[] = {
+            kCGDisplayStreamMinimumFrameTime,
+            kCGDisplayStreamQueueDepth,
+            kCGDisplayStreamColorSpace,
+            kCGDisplayStreamShowCursor,
+        };
+        CFTypeRef values[] = {
+            frameTimeCF.get(),
+            depthCF.get(),
+            deviceRGBColorSpace,
+            kCFBooleanTrue,
+        };
+        auto streamOptions = adoptCF(CFDictionaryCreate(kCFAllocatorDefault, keys, values, WTF_ARRAY_LENGTH(keys), &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks));
+
+        auto weakThis = m_weakFactory.createWeakPtr(*this);
+        m_frameAvailableBlock = Block_copy(^(CGDisplayStreamFrameStatus status, uint64_t displayTime, IOSurfaceRef frameSurface, CGDisplayStreamUpdateRef updateRef) {
+            if (!weakThis)
+                return;
+
+            weakThis->frameAvailable(status, displayTime, frameSurface, updateRef);
+        });
+
+        m_displayStream = adoptCF(CGDisplayStreamCreateWithDispatchQueue(m_displayID, size().width(), size().height(), kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, streamOptions.get(), m_captureQueue.get(), m_frameAvailableBlock));
+        if (!m_displayStream) {
+            RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::createDisplayStream(%p), CGDisplayStreamCreate failed", this);
+            captureFailed();
+            return false;
+        }
+    }
+
+    if (!m_observingDisplayChanges) {
+        CGDisplayRegisterReconfigurationCallback(displayReconfigurationCallBack, this);
+        m_observingDisplayChanges = true;
+    }
+
+    return true;
+}
+
+void ScreenDisplayCaptureSourceMac::startProducingData()
+{
+    DisplayCaptureSourceCocoa::startProducingData();
+
+    if (m_isRunning)
+        return;
+
+    startDisplayStream();
+}
+
+void ScreenDisplayCaptureSourceMac::stopProducingData()
+{
+    DisplayCaptureSourceCocoa::stopProducingData();
+
+    if (!m_isRunning)
+        return;
+
+    if (m_displayStream)
+        CGDisplayStreamStop(m_displayStream.get());
+
+    m_isRunning = false;
+}
+
+RetainPtr<CMSampleBufferRef> ScreenDisplayCaptureSourceMac::sampleBufferFromPixelBuffer(CVPixelBufferRef pixelBuffer)
+{
+    if (!pixelBuffer)
+        return nullptr;
+
+    CMTime sampleTime = CMTimeMake((elapsedTime() + .1) * 100, 100);
+    CMSampleTimingInfo timingInfo = { kCMTimeInvalid, sampleTime, sampleTime };
+
+    CMVideoFormatDescriptionRef formatDescription = nullptr;
+    auto status = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, (CVImageBufferRef)pixelBuffer, &formatDescription);
+    if (status) {
+        RELEASE_LOG(Media, "Failed to initialize CMVideoFormatDescription with error code: %d", static_cast<int>(status));
+        return nullptr;
+    }
+
+    CMSampleBufferRef sampleBuffer;
+    status = CMSampleBufferCreateReadyWithImageBuffer(kCFAllocatorDefault, (CVImageBufferRef)pixelBuffer, formatDescription, &timingInfo, &sampleBuffer);
+    CFRelease(formatDescription);
+    if (status) {
+        RELEASE_LOG(Media, "Failed to initialize CMSampleBuffer with error code: %d", static_cast<int>(status));
+        return nullptr;
+    }
+
+    return adoptCF(sampleBuffer);
+}
+
+RetainPtr<CVPixelBufferRef> ScreenDisplayCaptureSourceMac::pixelBufferFromIOSurface(IOSurfaceRef surface)
+{
+    if (!m_bufferAttributes) {
+        m_bufferAttributes = adoptCF(CFDictionaryCreateMutable(nullptr, 0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks));
+
+        auto format = IOSurfaceGetPixelFormat(surface);
+        if (format == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange || format == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
+
+            // If the width x height isn't a multiple of 16 x 16 and the surface has extra memory in the planes, set pixel buffer attributes to reflect it.
+            auto width = IOSurfaceGetWidth(surface);
+            auto height = IOSurfaceGetHeight(surface);
+            int32_t extendedRight = roundUpToMacroblockMultiple(width) - width;
+            int32_t extendedBottom = roundUpToMacroblockMultiple(height) - height;
+
+            if ((IOSurfaceGetBytesPerRowOfPlane(surface, 0) >= width + extendedRight)
+                && (IOSurfaceGetBytesPerRowOfPlane(surface, 1) >= width + extendedRight)
+                && (IOSurfaceGetAllocSize(surface) >= (height + extendedBottom) * IOSurfaceGetBytesPerRowOfPlane(surface, 0) * 3 / 2)) {
+                    auto cfInt = adoptCF(CFNumberCreate(nullptr,  kCFNumberIntType,  &extendedRight));
+                    CFDictionarySetValue(m_bufferAttributes.get(), kCVPixelBufferExtendedPixelsRightKey, cfInt.get());
+                    cfInt = adoptCF(CFNumberCreate(nullptr,  kCFNumberIntType,  &extendedBottom));
+                    CFDictionarySetValue(m_bufferAttributes.get(), kCVPixelBufferExtendedPixelsBottomKey, cfInt.get());
+            }
+        }
+
+        CFDictionarySetValue(m_bufferAttributes.get(), kCVPixelBufferOpenGLCompatibilityKey, kCFBooleanTrue);
+    }
+
+    CVPixelBufferRef pixelBuffer;
+    auto status = CVPixelBufferCreateWithIOSurface(kCFAllocatorDefault, surface, m_bufferAttributes.get(), &pixelBuffer);
+    if (status) {
+        RELEASE_LOG(Media, "Failed to initialize CMVideoFormatDescription with error code: %d", static_cast<int>(status));
+        return nullptr;
+    }
+
+    return adoptCF(pixelBuffer);
+}
+
+void ScreenDisplayCaptureSourceMac::generateFrame()
+{
+    if (!m_currentFrame.ioSurface())
+        return;
+
+    DisplaySurface currentFrame;
+    {
+        LockHolder lock(m_currentFrameMutex);
+        currentFrame = m_currentFrame.ioSurface();
+    }
+
+    auto pixelBuffer = pixelBufferFromIOSurface(currentFrame.ioSurface());
+    if (!pixelBuffer)
+        return;
+
+    auto sampleBuffer = sampleBufferFromPixelBuffer(pixelBuffer.get());
+    if (!sampleBuffer)
+        return;
+
+    videoSampleAvailable(MediaSampleAVFObjC::create(sampleBuffer.get()));
+}
+
+void ScreenDisplayCaptureSourceMac::startDisplayStream()
+{
+    auto actualDisplayID = updateDisplayID(m_displayID);
+    if (!actualDisplayID)
+        return;
+
+    if (m_displayID != actualDisplayID.value()) {
+        m_displayID = actualDisplayID.value();
+        RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::create(%p), display ID changed to %d", this, static_cast<int>(m_displayID));
+    }
+
+    if (!m_displayStream && !createDisplayStream())
+        return;
+
+    auto err = CGDisplayStreamStart(m_displayStream.get());
+    if (err) {
+        RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::startProducingData(%p), CGDisplayStreamStart failed with error %d", this, static_cast<int>(err));
+        captureFailed();
+        return;
+    }
+
+    m_isRunning = true;
+}
+
+bool ScreenDisplayCaptureSourceMac::applySize(const IntSize& newSize)
+{
+    if (size() == newSize)
+        return true;
+
+    m_bufferAttributes = nullptr;
+    m_displayStream = nullptr;
+    return true;
+}
+
+bool ScreenDisplayCaptureSourceMac::applyFrameRate(double rate)
+{
+    if (frameRate() != rate) {
+        m_bufferAttributes = nullptr;
+        m_displayStream = nullptr;
+    }
+
+    return DisplayCaptureSourceCocoa::applyFrameRate(rate);
+}
+
+void ScreenDisplayCaptureSourceMac::commitConfiguration()
+{
+    if (m_isRunning && !m_displayStream)
+        startDisplayStream();
+}
+
+void ScreenDisplayCaptureSourceMac::displayWasReconfigured(CGDirectDisplayID, CGDisplayChangeSummaryFlags)
+{
+    // FIXME: implement!
+}
+
+void ScreenDisplayCaptureSourceMac::displayReconfigurationCallBack(CGDirectDisplayID display, CGDisplayChangeSummaryFlags flags, void *userInfo)
+{
+    if (userInfo)
+        reinterpret_cast<ScreenDisplayCaptureSourceMac *>(userInfo)->displayWasReconfigured(display, flags);
+}
+
+void ScreenDisplayCaptureSourceMac::frameAvailable(CGDisplayStreamFrameStatus status, uint64_t displayTime, IOSurfaceRef frameSurface, CGDisplayStreamUpdateRef updateRef)
+{
+    switch (status) {
+    case kCGDisplayStreamFrameStatusFrameComplete:
+        break;
+
+    case kCGDisplayStreamFrameStatusFrameIdle:
+        break;
+
+    case kCGDisplayStreamFrameStatusFrameBlank:
+        RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::frameAvailable(%p), kCGDisplayStreamFrameStatusFrameBlank", this);
+        break;
+
+    case kCGDisplayStreamFrameStatusStopped:
+        RELEASE_LOG(Media, "ScreenDisplayCaptureSourceMac::frameAvailable(%p), kCGDisplayStreamFrameStatusStopped", this);
+        break;
+    }
+
+    if (!frameSurface || !displayTime)
+        return;
+
+    size_t count;
+    auto* rects = CGDisplayStreamUpdateGetRects(updateRef, kCGDisplayStreamUpdateDirtyRects, &count);
+    if (!rects || !count)
+        return;
+
+    LockHolder lock(m_currentFrameMutex);
+    m_lastFrameTime = monotonicallyIncreasingTime();
+    m_currentFrame = frameSurface;
+}
+
+} // namespace WebCore
+
+#endif // ENABLE(MEDIA_STREAM) && PLATFORM(MAC)