Performance: do pixel conformance and texturing in a single step.
authorjer.noble@apple.com <jer.noble@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 13 Oct 2017 16:32:38 +0000 (16:32 +0000)
committerjer.noble@apple.com <jer.noble@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 13 Oct 2017 16:32:38 +0000 (16:32 +0000)
https://bugs.webkit.org/show_bug.cgi?id=178219
<rdar://problem/34937237>

Reviewed by Dean Jackson.

No new tests; performance improvements should have no behavior change.

Rather than asking the VTDecompressionSession to conform the output CVPixelBuffer into a
pixel format compatible with OpenGL (& ES), don't constrain the output at all, and only do a
conformance step if the output is not already compatible with OpenGL. This eliminates one
copy (in hardware) operation.

Move the TextureCacheCV object into VideoTextureCopierCV; it will be conditionally used to
create the texture if the pixel buffer is compatible.

Refactor copyVideoTextureToPlatformTexture(CVOpenGLTextureRef) in VideoTextureCopierCV. The
new entry point, copyImageToPlatformTexture(), will attempt to use the texture cache first,
and call a new common copyVideoTextureToPlatformTexture(Platform3DObject) with the result.

The new copyImageToPlatformTexture() will pull planar YUV frames into two textures, and combine
the two with a color transfer function when drawing to the output texture.

* platform/graphics/GraphicsContext3D.h:
* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::copyVideoTextureToPlatformTexture):
* platform/graphics/cocoa/GraphicsContext3DCocoa.mm:
(WebCore::GraphicsContext3D::texImageIOSurface2D):
* platform/graphics/cocoa/WebCoreDecompressionSession.mm:
(WebCore::WebCoreDecompressionSession::ensureDecompressionSessionForSample):
* platform/graphics/cv/TextureCacheCV.h:
* platform/graphics/cv/TextureCacheCV.mm:
(WebCore::TextureCacheCV::textureFromImage):
* platform/graphics/cv/VideoTextureCopierCV.cpp:
(WebCore::pixelRangeFromPixelFormat):
(WebCore::transferFunctionFromString):
(WebCore::YCbCrToRGBMatrixForRangeAndTransferFunction):
(WebCore::VideoTextureCopierCV::~VideoTextureCopierCV):
(WebCore::VideoTextureCopierCV::initializeUVContextObjects):
(WebCore::VideoTextureCopierCV::copyImageToPlatformTexture):
(WebCore::VideoTextureCopierCV::copyVideoTextureToPlatformTexture):
* platform/graphics/cv/VideoTextureCopierCV.h:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@223280 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/WebCore/ChangeLog
Source/WebCore/platform/cocoa/CoreVideoSoftLink.cpp
Source/WebCore/platform/cocoa/CoreVideoSoftLink.h
Source/WebCore/platform/graphics/GraphicsContext3D.h
Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm
Source/WebCore/platform/graphics/cocoa/GraphicsContext3DCocoa.mm
Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm
Source/WebCore/platform/graphics/cv/TextureCacheCV.h
Source/WebCore/platform/graphics/cv/TextureCacheCV.mm
Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.cpp
Source/WebCore/platform/graphics/cv/VideoTextureCopierCV.h

index f04b65c..41eb6f7 100644 (file)
@@ -1,3 +1,48 @@
+2017-10-13  Jer Noble  <jer.noble@apple.com>
+
+        Performance: do pixel conformance and texturing in a single step.
+        https://bugs.webkit.org/show_bug.cgi?id=178219
+        <rdar://problem/34937237>
+
+        Reviewed by Dean Jackson.
+
+        No new tests; performance improvements should have no behavior change.
+
+        Rather than asking the VTDecompressionSession to conform the output CVPixelBuffer into a
+        pixel format compatible with OpenGL (& ES), don't constrain the output at all, and only do a
+        conformance step if the output is not already compatible with OpenGL. This eliminates one
+        copy (in hardware) operation.
+
+        Move the TextureCacheCV object into VideoTextureCopierCV; it will be conditionally used to
+        create the texture if the pixel buffer is compatible.
+
+        Refactor copyVideoTextureToPlatformTexture(CVOpenGLTextureRef) in VideoTextureCopierCV. The
+        new entry point, copyImageToPlatformTexture(), will attempt to use the texture cache first,
+        and call a new common copyVideoTextureToPlatformTexture(Platform3DObject) with the result.
+
+        The new copyImageToPlatformTexture() will pull planar YUV frames into two textures, and combine
+        the two with a color transfer function when drawing to the output texture.
+
+        * platform/graphics/GraphicsContext3D.h:
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
+        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::copyVideoTextureToPlatformTexture):
+        * platform/graphics/cocoa/GraphicsContext3DCocoa.mm:
+        (WebCore::GraphicsContext3D::texImageIOSurface2D):
+        * platform/graphics/cocoa/WebCoreDecompressionSession.mm:
+        (WebCore::WebCoreDecompressionSession::ensureDecompressionSessionForSample):
+        * platform/graphics/cv/TextureCacheCV.h:
+        * platform/graphics/cv/TextureCacheCV.mm:
+        (WebCore::TextureCacheCV::textureFromImage):
+        * platform/graphics/cv/VideoTextureCopierCV.cpp:
+        (WebCore::pixelRangeFromPixelFormat):
+        (WebCore::transferFunctionFromString):
+        (WebCore::YCbCrToRGBMatrixForRangeAndTransferFunction):
+        (WebCore::VideoTextureCopierCV::~VideoTextureCopierCV):
+        (WebCore::VideoTextureCopierCV::initializeUVContextObjects):
+        (WebCore::VideoTextureCopierCV::copyImageToPlatformTexture):
+        (WebCore::VideoTextureCopierCV::copyVideoTextureToPlatformTexture):
+        * platform/graphics/cv/VideoTextureCopierCV.h:
+
 2017-10-13  Romain Bellessort  <romain.bellessort@crf.canon.fr>
 
         [Readable Streams API] Align queue with spec for ReadableStreamDefaultController
index dc496e8..343ab37 100644 (file)
@@ -52,6 +52,13 @@ SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelFormatOpenGLCompatibil
 SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelFormatOpenGLESCompatibility, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferIOSurfacePropertiesKey, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVPixelBufferPoolMinimumBufferCountKey, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrixKey, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_709_2, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_601_4, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_SMPTE_240M_1995, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_DCI_P3, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_P3_D65, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_2020, CFStringRef)
 
 #if PLATFORM(IOS)
 SOFT_LINK_FUNCTION_FOR_SOURCE(WebCore, CoreVideo, CVOpenGLESTextureCacheCreate, CVReturn, (CFAllocatorRef allocator, CFDictionaryRef cacheAttributes, CVEAGLContext eaglContext, CFDictionaryRef textureAttributes, CVOpenGLESTextureCacheRef* cacheOut), (allocator, cacheAttributes, eaglContext, textureAttributes, cacheOut))
index 766db71..94d12f1 100644 (file)
@@ -70,6 +70,20 @@ SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferIOSurfacePropert
 #define kCVPixelBufferIOSurfacePropertiesKey get_CoreVideo_kCVPixelBufferIOSurfacePropertiesKey()
 SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVPixelBufferPoolMinimumBufferCountKey, CFStringRef)
 #define kCVPixelBufferPoolMinimumBufferCountKey get_CoreVideo_kCVPixelBufferPoolMinimumBufferCountKey()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrixKey, CFStringRef)
+#define kCVImageBufferYCbCrMatrixKey get_CoreVideo_kCVImageBufferYCbCrMatrixKey()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_709_2, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_ITU_R_709_2 get_CoreVideo_kCVImageBufferYCbCrMatrix_ITU_R_709_2()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_601_4, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_ITU_R_601_4 get_CoreVideo_kCVImageBufferYCbCrMatrix_ITU_R_601_4()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_SMPTE_240M_1995, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_SMPTE_240M_1995 get_CoreVideo_kCVImageBufferYCbCrMatrix_SMPTE_240M_1995()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_DCI_P3, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_DCI_P3 get_CoreVideo_kCVImageBufferYCbCrMatrix_DCI_P3()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_P3_D65, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_P3_D65 get_CoreVideo_kCVImageBufferYCbCrMatrix_P3_D65()
+SOFT_LINK_CONSTANT_FOR_HEADER(WebCore, CoreVideo, kCVImageBufferYCbCrMatrix_ITU_R_2020, CFStringRef)
+#define kCVImageBufferYCbCrMatrix_ITU_R_2020 get_CoreVideo_kCVImageBufferYCbCrMatrix_ITU_R_2020()
 
 #if PLATFORM(IOS)
 SOFT_LINK_FUNCTION_FOR_HEADER(WebCore, CoreVideo, CVOpenGLESTextureCacheCreate, CVReturn, (CFAllocatorRef allocator, CFDictionaryRef cacheAttributes, CVEAGLContext eaglContext, CFDictionaryRef textureAttributes, CVOpenGLESTextureCacheRef* cacheOut), (allocator, cacheAttributes, eaglContext, textureAttributes, cacheOut))
index d901010..9854e63 100644 (file)
@@ -59,6 +59,7 @@
 #include <wtf/RetainPtr.h>
 OBJC_CLASS CALayer;
 OBJC_CLASS WebGLLayer;
+typedef struct __IOSurface* IOSurfaceRef;
 #elif PLATFORM(GTK) || PLATFORM(WIN_CAIRO) || PLATFORM(WPE)
 typedef unsigned int GLuint;
 #endif
@@ -1149,6 +1150,7 @@ public:
 
 #if PLATFORM(COCOA)
     void endPaint();
+    bool texImageIOSurface2D(GC3Denum target, GC3Denum internalFormat, GC3Dsizei width, GC3Dsizei height, GC3Denum format, GC3Denum type, IOSurfaceRef, GC3Duint plane);
 #endif
 
 #if PLATFORM(MAC)
index 41b637a..4938d62 100644 (file)
@@ -608,14 +608,6 @@ bool MediaPlayerPrivateMediaSourceAVFObjC::copyVideoTextureToPlatformTexture(Gra
     if (updateLastPixelBuffer()) {
         if (!m_lastPixelBuffer)
             return false;
-
-        if (!m_textureCache) {
-            m_textureCache = TextureCacheCV::create(*context);
-            if (!m_textureCache)
-                return false;
-        }
-
-        m_lastTexture = m_textureCache->textureFromImage(m_lastPixelBuffer.get(), outputTarget, level, internalFormat, format, type);
     }
 
     size_t width = CVPixelBufferGetWidth(m_lastPixelBuffer.get());
@@ -624,7 +616,7 @@ bool MediaPlayerPrivateMediaSourceAVFObjC::copyVideoTextureToPlatformTexture(Gra
     if (!m_videoTextureCopier)
         m_videoTextureCopier = std::make_unique<VideoTextureCopierCV>(*context);
 
-    return m_videoTextureCopier->copyVideoTextureToPlatformTexture(m_lastTexture.get(), width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
+    return m_videoTextureCopier->copyImageToPlatformTexture(m_lastPixelBuffer.get(), width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
 }
 
 bool MediaPlayerPrivateMediaSourceAVFObjC::hasAvailableVideoFrame() const
index c0f84bf..57bbe31 100644 (file)
@@ -50,6 +50,7 @@
 #if PLATFORM(IOS)
 #import <OpenGLES/EAGL.h>
 #import <OpenGLES/EAGLDrawable.h>
+#import <OpenGLES/EAGLIOSurface.h>
 #import <OpenGLES/ES2/glext.h>
 #import <QuartzCore/QuartzCore.h>
 #import <pal/spi/ios/OpenGLESSPI.h>
@@ -672,6 +673,17 @@ void GraphicsContext3D::endPaint()
 #endif
 }
 
+bool GraphicsContext3D::texImageIOSurface2D(GC3Denum target, GC3Denum internalFormat, GC3Dsizei width, GC3Dsizei height, GC3Denum format, GC3Denum type, IOSurfaceRef surface, GC3Duint plane)
+{
+#if PLATFORM(MAC)
+    return kCGLNoError == CGLTexImageIOSurface2D(platformGraphicsContext3D(), target, internalFormat, width, height, format, type, surface, plane);
+#elif PLATFORM(IOS) && !PLATFORM(IOS_SIMULATOR)
+    return [platformGraphicsContext3D() texImageIOSurface:surface target:target internalFormat:internalFormat width:width height:height format:format type:type plane:plane];
+#else
+    return false;
+#endif
+}
+
 #if PLATFORM(MAC)
 void GraphicsContext3D::allocateIOSurfaceBackingStore(IntSize size)
 {
index 6d9db62..a22cc87 100644 (file)
@@ -219,11 +219,7 @@ void WebCoreDecompressionSession::ensureDecompressionSessionForSample(CMSampleBu
 
         NSDictionary *attributes;
         if (m_mode == OpenGL) {
-#if PLATFORM(IOS)
-            attributes = @{(NSString *)kCVPixelBufferIOSurfaceOpenGLESFBOCompatibilityKey: @YES};
-#else
-            attributes = @{(NSString *)kCVPixelBufferIOSurfaceOpenGLFBOCompatibilityKey: @YES};
-#endif
+            attributes = nil;
         } else {
             ASSERT(m_mode == RGB);
             attributes = @{(NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA)};
index d796dd7..84ef7c1 100644 (file)
@@ -31,7 +31,8 @@
 #include <wtf/RetainPtr.h>
 #include <wtf/WeakPtr.h>
 
-typedef struct  __CVBuffer* CVImageBufferRef;
+typedef struct __CVBuffer* CVImageBufferRef;
+typedef CVImageBufferRef CVPixelBufferRef;
 typedef CVImageBufferRef CVOpenGLTextureRef;
 typedef CVImageBufferRef CVOpenGLESTextureRef;
 typedef struct __CVOpenGLTextureCache *CVOpenGLTextureCacheRef;
@@ -55,7 +56,7 @@ public:
 
     TextureCacheCV(GraphicsContext3D&, RetainPtr<TextureCacheType>&&);
 
-    RetainPtr<TextureType> textureFromImage(CVImageBufferRef, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type);
+    RetainPtr<TextureType> textureFromImage(CVPixelBufferRef, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type);
     GraphicsContext3D& context() { return m_context.get(); }
 
 private:
index 3da0164..ef1f359 100644 (file)
@@ -53,7 +53,7 @@ TextureCacheCV::TextureCacheCV(GraphicsContext3D& context, RetainPtr<TextureCach
 {
 }
 
-RetainPtr<TextureCacheCV::TextureType> TextureCacheCV::textureFromImage(CVImageBufferRef image, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type)
+RetainPtr<TextureCacheCV::TextureType> TextureCacheCV::textureFromImage(CVPixelBufferRef image, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type)
 {
     TextureType bareVideoTexture = nullptr;
 #if PLATFORM(IOS)
index 18e9928..0919055 100644 (file)
@@ -26,7 +26,9 @@
 #include "config.h"
 #include "VideoTextureCopierCV.h"
 
+#include "FourCC.h"
 #include "Logging.h"
+#include "TextureCacheCV.h"
 #include <wtf/NeverDestroyed.h>
 #include <wtf/text/StringBuilder.h>
 
 
 namespace WebCore {
 
+enum class PixelRange {
+    Unknown,
+    Video,
+    Full,
+};
+
+enum class TransferFunction {
+    Unknown,
+    kITU_R_709_2,
+    kITU_R_601_4,
+    kSMPTE_240M_1995,
+    kDCI_P3,
+    kP3_D65,
+    kITU_R_2020,
+};
+
+#if PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 101300
+enum {
+    kCVPixelFormatType_ARGB2101010LEPacked = 'l10r',
+    kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange = 'x420',
+    kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange = 'x422',
+    kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange = 'x444',
+    kCVPixelFormatType_420YpCbCr10BiPlanarFullRange  = 'xf20',
+    kCVPixelFormatType_422YpCbCr10BiPlanarFullRange  = 'xf22',
+    kCVPixelFormatType_444YpCbCr10BiPlanarFullRange  = 'xf44',
+};
+#endif
+
+static PixelRange pixelRangeFromPixelFormat(OSType pixelFormat)
+{
+    switch (pixelFormat) {
+    case kCVPixelFormatType_4444AYpCbCr8:
+    case kCVPixelFormatType_4444AYpCbCr16:
+    case kCVPixelFormatType_422YpCbCr_4A_8BiPlanar:
+    case kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange:
+    case kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange:
+    case kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange:
+    case kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange:
+        return PixelRange::Video;
+    case kCVPixelFormatType_420YpCbCr8PlanarFullRange:
+    case kCVPixelFormatType_420YpCbCr8BiPlanarFullRange:
+    case kCVPixelFormatType_422YpCbCr8FullRange:
+    case kCVPixelFormatType_ARGB2101010LEPacked:
+    case kCVPixelFormatType_420YpCbCr10BiPlanarFullRange:
+    case kCVPixelFormatType_422YpCbCr10BiPlanarFullRange:
+    case kCVPixelFormatType_444YpCbCr10BiPlanarFullRange:
+        return PixelRange::Full;
+    default:
+        return PixelRange::Unknown;
+    }
+}
+
+static TransferFunction transferFunctionFromString(CFStringRef string)
+{
+    if (string == kCVImageBufferYCbCrMatrix_ITU_R_709_2)
+        return TransferFunction::kITU_R_709_2;
+    if (string == kCVImageBufferYCbCrMatrix_ITU_R_601_4)
+        return TransferFunction::kITU_R_601_4;
+    if (string == kCVImageBufferYCbCrMatrix_SMPTE_240M_1995)
+        return TransferFunction::kSMPTE_240M_1995;
+    if (string == kCVImageBufferYCbCrMatrix_DCI_P3)
+        return TransferFunction::kDCI_P3;
+    if (string == kCVImageBufferYCbCrMatrix_P3_D65)
+        return TransferFunction::kP3_D65;
+    if (string == kCVImageBufferYCbCrMatrix_ITU_R_2020)
+        return TransferFunction::kITU_R_2020;
+    return TransferFunction::Unknown;
+}
+
+static const Vector<GLfloat> YCbCrToRGBMatrixForRangeAndTransferFunction(PixelRange range, TransferFunction transferFunction)
+{
+    using MapKey = std::pair<PixelRange, TransferFunction>;
+    using MatrixMap = std::map<MapKey, Vector<GLfloat>>;
+
+    static NeverDestroyed<MatrixMap> matrices;
+    static dispatch_once_t onceToken;
+    dispatch_once(&onceToken, ^{
+        // Matrices are derived from the components in the ITU R.601 rev 4 specification
+        // https://www.itu.int/rec/R-REC-BT.601
+        matrices.get().emplace(MapKey(PixelRange::Video, TransferFunction::kITU_R_601_4), Vector<GLfloat>({
+            1.164383562f,  0.0f,           1.596026786f,
+            1.164383562f, -0.3917622901f, -0.8129676472f,
+            1.164383562f,  2.017232143f,   0.0f,
+        }));
+        matrices.get().emplace(MapKey({PixelRange::Full, TransferFunction::kITU_R_601_4}), Vector<GLfloat>({
+            1.000000000f,  0.0f,           1.4075196850f,
+            1.000000000f, -0.3454911535f, -0.7169478464f,
+            1.000000000f,  1.7789763780f,  0.0f,
+        }));
+        // Matrices are derived from the components in the ITU R.709 rev 2 specification
+        // https://www.itu.int/rec/R-REC-BT.709-2-199510-S
+        matrices.get().emplace(MapKey({PixelRange::Video, TransferFunction::kITU_R_709_2}), Vector<GLfloat>({
+            1.164383562f,  0.0f,           1.792741071f,
+            1.164383562f, -0.2132486143f, -0.5329093286f,
+            1.164383562f,  2.112401786f,   0.0f,
+        }));
+        matrices.get().emplace(MapKey({PixelRange::Full, TransferFunction::kITU_R_709_2}), Vector<GLfloat>({
+            1.000000000f,  0.0f,           1.5810000000f,
+            1.000000000f, -0.1880617701f, -0.4699672819f,
+            1.000000000f,  1.8629055118f,  0.0f,
+        }));
+    });
+
+    // We should never be asked to handle a Pixel Format whose range value is unknown.
+    ASSERT(range != PixelRange::Unknown);
+    if (range == PixelRange::Unknown)
+        range = PixelRange::Full;
+
+    auto iterator = matrices.get().find({range, transferFunction});
+
+    // Assume unknown transfer functions are r.601:
+    if (iterator == matrices.get().end())
+        iterator = matrices.get().find({range, TransferFunction::kITU_R_601_4});
+
+    ASSERT(iterator != matrices.get().end());
+    return iterator->second;
+}
+
 VideoTextureCopierCV::VideoTextureCopierCV(GraphicsContext3D& context)
     : m_context(context)
     , m_framebuffer(context.createFramebuffer())
@@ -50,6 +170,10 @@ VideoTextureCopierCV::~VideoTextureCopierCV()
         m_context->deleteProgram(m_vertexBuffer);
     if (m_program)
         m_context->deleteProgram(m_program);
+    if (m_yuvVertexBuffer)
+        m_context->deleteProgram(m_yuvVertexBuffer);
+    if (m_yuvProgram)
+        m_context->deleteProgram(m_yuvProgram);
     m_context->deleteFramebuffer(m_framebuffer);
 }
 
@@ -258,21 +382,233 @@ bool VideoTextureCopierCV::initializeContextObjects()
     return true;
 }
 
-bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(TextureType inputVideoTexture, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+bool VideoTextureCopierCV::initializeUVContextObjects()
 {
-    if (!inputVideoTexture)
+    String vertexShaderSource = ASCIILiteral(
+        "attribute vec2 a_position;\n"
+        "uniform vec2 u_yTextureSize;\n"
+        "uniform vec2 u_uvTextureSize;\n"
+        "uniform int u_flipY;\n"
+        "varying vec2 v_yTextureCoordinate;\n"
+        "varying vec2 v_uvTextureCoordinate;\n"
+        "void main() {\n"
+        "   gl_Position = vec4(a_position, 0, 1.0);\n"
+        "   if (u_flipY == 1) {\n"
+        "       gl_Position.y = -gl_Position.y;\n"
+        "   }\n"
+        "   vec2 normalizedPosition = a_position * .5 + .5;\n"
+#if PLATFORM(IOS)
+        "   v_yTextureCoordinate = normalizedPosition;\n"
+        "   v_uvTextureCoordinate = normalizedPosition;\n"
+#else
+        "   v_yTextureCoordinate = normalizedPosition * u_yTextureSize;\n"
+        "   v_uvTextureCoordinate = normalizedPosition * u_uvTextureSize;\n"
+#endif
+        "}\n"
+    );
+
+    Platform3DObject vertexShader = m_context->createShader(GraphicsContext3D::VERTEX_SHADER);
+    m_context->shaderSource(vertexShader, vertexShaderSource);
+    m_context->compileShaderDirect(vertexShader);
+
+    GC3Dint status = 0;
+    m_context->getShaderiv(vertexShader, GraphicsContext3D::COMPILE_STATUS, &status);
+    if (!status) {
+        LOG(WebGL, "VideoTextureCopierCV::initializeUVContextObjects(%p) - Vertex shader failed to compile.", this);
+        m_context->deleteShader(vertexShader);
+        return false;
+    }
+
+    String fragmentShaderSource = ASCIILiteral(
+#if PLATFORM(IOS)
+        "precision mediump float;\n"
+        "#define SAMPLERTYPE sampler2D\n"
+        "#define TEXTUREFUNC texture2D\n"
+#else
+        "#define SAMPLERTYPE sampler2DRect\n"
+        "#define TEXTUREFUNC texture2DRect\n"
+#endif
+        "uniform SAMPLERTYPE u_yTexture;\n"
+        "uniform SAMPLERTYPE u_uvTexture;\n"
+        "uniform mat3 u_colorMatrix;\n"
+        "varying vec2 v_yTextureCoordinate;\n"
+        "varying vec2 v_uvTextureCoordinate;\n"
+        "void main() {\n"
+        "    vec3 yuv;\n"
+        "    yuv.r = TEXTUREFUNC(u_yTexture, v_yTextureCoordinate).r;\n"
+        "    yuv.gb = TEXTUREFUNC(u_uvTexture, v_uvTextureCoordinate).rg - vec2(0.5, 0.5);\n"
+        "    gl_FragColor = vec4(yuv * u_colorMatrix, 1);\n"
+        "}\n"
+    );
+
+    Platform3DObject fragmentShader = m_context->createShader(GraphicsContext3D::FRAGMENT_SHADER);
+    m_context->shaderSource(fragmentShader, fragmentShaderSource);
+    m_context->compileShaderDirect(fragmentShader);
+
+    m_context->getShaderiv(fragmentShader, GraphicsContext3D::COMPILE_STATUS, &status);
+    if (!status) {
+        LOG(WebGL, "VideoTextureCopierCV::initializeUVContextObjects(%p) - Fragment shader failed to compile.", this);
+        m_context->deleteShader(vertexShader);
+        m_context->deleteShader(fragmentShader);
+        return false;
+    }
+
+    m_yuvProgram = m_context->createProgram();
+    m_context->attachShader(m_yuvProgram, vertexShader);
+    m_context->attachShader(m_yuvProgram, fragmentShader);
+    m_context->linkProgram(m_yuvProgram);
+
+    m_context->getProgramiv(m_yuvProgram, GraphicsContext3D::LINK_STATUS, &status);
+    if (!status) {
+        LOG(WebGL, "VideoTextureCopierCV::initializeUVContextObjects(%p) - Program failed to link.", this);
+        m_context->deleteShader(vertexShader);
+        m_context->deleteShader(fragmentShader);
+        m_context->deleteProgram(m_yuvProgram);
+        m_yuvProgram = 0;
+        return false;
+    }
+
+    m_yTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_yTexture"));
+    m_uvTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_uvTexture"));
+    m_colorMatrixUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_colorMatrix"));
+    m_yuvFlipYUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_flipY"));
+    m_yTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_yTextureSize"));
+    m_uvTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, ASCIILiteral("u_uvTextureSize"));
+    m_yuvPositionAttributeLocation = m_context->getAttribLocationDirect(m_yuvProgram, ASCIILiteral("a_position"));
+
+    m_context->detachShader(m_yuvProgram, vertexShader);
+    m_context->detachShader(m_yuvProgram, fragmentShader);
+    m_context->deleteShader(vertexShader);
+    m_context->deleteShader(fragmentShader);
+
+    m_yuvVertexBuffer = m_context->createBuffer();
+    float vertices[12] = { -1, -1, 1, -1, 1, 1, 1, 1, -1, 1, -1, -1 };
+
+    m_context->bindBuffer(GraphicsContext3D::ARRAY_BUFFER, m_yuvVertexBuffer);
+    m_context->bufferData(GraphicsContext3D::ARRAY_BUFFER, sizeof(vertices), vertices, GraphicsContext3D::STATIC_DRAW);
+
+    return true;
+}
+
+bool VideoTextureCopierCV::copyImageToPlatformTexture(CVPixelBufferRef image, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+{
+    if (!m_textureCache) {
+        m_textureCache = TextureCacheCV::create(m_context);
+        if (!m_textureCache)
+            return false;
+    }
+
+    if (auto texture = m_textureCache->textureFromImage(image, outputTarget, level, internalFormat, format, type))
+        return copyVideoTextureToPlatformTexture(texture.get(), width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
+
+    // FIXME: This currently only supports '420v' and '420f' pixel formats. Investigate supporting more pixel formats.
+    OSType pixelFormat = CVPixelBufferGetPixelFormatType(image);
+    if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange && pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
+        LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Asked to copy an unsupported pixel format ('%s').", this, FourCC(pixelFormat).toString().utf8().data());
+        return false;
+    }
+
+    IOSurfaceRef surface = CVPixelBufferGetIOSurface(image);
+    if (!surface)
         return false;
 
     GC3DStateSaver stateSaver(m_context.get());
 
-    if (!m_program) {
-        if (!initializeContextObjects()) {
+    if (!m_yuvProgram) {
+        if (!initializeUVContextObjects()) {
             LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);
             return false;
         }
     }
 
-    stateSaver.saveVertexAttribState(m_positionAttributeLocation);
+    stateSaver.saveVertexAttribState(m_yuvPositionAttributeLocation);
+
+    m_context->bindFramebuffer(GraphicsContext3D::FRAMEBUFFER, m_framebuffer);
+
+    // Allocate memory for the output texture.
+    m_context->bindTexture(GraphicsContext3D::TEXTURE_2D, outputTexture);
+    m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_MAG_FILTER, GraphicsContext3D::LINEAR);
+    m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_MIN_FILTER, GraphicsContext3D::LINEAR);
+    m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_WRAP_S, GraphicsContext3D::CLAMP_TO_EDGE);
+    m_context->texParameteri(GraphicsContext3D::TEXTURE_2D, GraphicsContext3D::TEXTURE_WRAP_T, GraphicsContext3D::CLAMP_TO_EDGE);
+    m_context->texImage2DDirect(GraphicsContext3D::TEXTURE_2D, level, internalFormat, width, height, 0, format, type, nullptr);
+
+    m_context->framebufferTexture2D(GraphicsContext3D::FRAMEBUFFER, GraphicsContext3D::COLOR_ATTACHMENT0, GraphicsContext3D::TEXTURE_2D, outputTexture, level);
+    GC3Denum status = m_context->checkFramebufferStatus(GraphicsContext3D::FRAMEBUFFER);
+    if (status != GraphicsContext3D::FRAMEBUFFER_COMPLETE) {
+        LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Unable to create framebuffer for outputTexture.", this);
+        return false;
+    }
+
+    m_context->useProgram(m_yuvProgram);
+    m_context->viewport(0, 0, width, height);
+
+    // Bind and set up the textures for the video source.
+    auto yPlaneWidth = IOSurfaceGetWidthOfPlane(surface, 0);
+    auto yPlaneHeight = IOSurfaceGetHeightOfPlane(surface, 0);
+    auto uvPlaneWidth = IOSurfaceGetWidthOfPlane(surface, 1);
+    auto uvPlaneHeight = IOSurfaceGetHeightOfPlane(surface, 1);
+
+#if PLATFORM(IOS)
+    GC3Denum videoTextureTarget = GraphicsContext3D::TEXTURE_2D;
+#else
+    GC3Denum videoTextureTarget = GL_TEXTURE_RECTANGLE_ARB;
+#endif
+    auto uvTexture = m_context->createTexture();
+    m_context->activeTexture(GraphicsContext3D::TEXTURE1);
+    m_context->bindTexture(videoTextureTarget, uvTexture);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MAG_FILTER, GraphicsContext3D::LINEAR);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MIN_FILTER, GraphicsContext3D::LINEAR);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_S, GraphicsContext3D::CLAMP_TO_EDGE);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_T, GraphicsContext3D::CLAMP_TO_EDGE);
+    if (!m_context->texImageIOSurface2D(videoTextureTarget, GL_RG, uvPlaneWidth, uvPlaneHeight, GL_RG, GL_UNSIGNED_BYTE, surface, 1)) {
+        m_context->deleteTexture(uvTexture);
+        return false;
+    }
+
+    auto yTexture = m_context->createTexture();
+    m_context->activeTexture(GraphicsContext3D::TEXTURE0);
+    m_context->bindTexture(videoTextureTarget, yTexture);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MAG_FILTER, GraphicsContext3D::LINEAR);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_MIN_FILTER, GraphicsContext3D::LINEAR);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_S, GraphicsContext3D::CLAMP_TO_EDGE);
+    m_context->texParameteri(videoTextureTarget, GraphicsContext3D::TEXTURE_WRAP_T, GraphicsContext3D::CLAMP_TO_EDGE);
+    if (!m_context->texImageIOSurface2D(videoTextureTarget, GL_LUMINANCE, yPlaneWidth, yPlaneHeight, GL_LUMINANCE, GL_UNSIGNED_BYTE, surface, 0)) {
+        m_context->deleteTexture(yTexture);
+        m_context->deleteTexture(uvTexture);
+        return false;
+    }
+
+    // Configure the drawing parameters.
+    m_context->uniform1i(m_yTextureUniformLocation, 0);
+    m_context->uniform1i(m_uvTextureUniformLocation, 1);
+    m_context->uniform1i(m_yuvFlipYUniformLocation, flipY);
+    m_context->uniform2f(m_yTextureSizeUniformLocation, yPlaneWidth, yPlaneHeight);
+    m_context->uniform2f(m_uvTextureSizeUniformLocation, uvPlaneWidth, uvPlaneHeight);
+
+    auto range = pixelRangeFromPixelFormat(pixelFormat);
+    auto transferFunction = transferFunctionFromString((CFStringRef)CVBufferGetAttachment(image, kCVImageBufferYCbCrMatrixKey, nil));
+    auto& colorMatrix = YCbCrToRGBMatrixForRangeAndTransferFunction(range, transferFunction);
+    m_context->uniformMatrix3fv(m_colorMatrixUniformLocation, 1, GL_FALSE, colorMatrix.data());
+
+    // Do the actual drawing.
+    m_context->bindBuffer(GraphicsContext3D::ARRAY_BUFFER, m_yuvVertexBuffer);
+    m_context->enableVertexAttribArray(m_yuvPositionAttributeLocation);
+    m_context->vertexAttribPointer(m_yuvPositionAttributeLocation, 2, GraphicsContext3D::FLOAT, false, 0, 0);
+    m_context->drawArrays(GraphicsContext3D::TRIANGLES, 0, 6);
+
+    // Clean-up.
+    m_context->deleteTexture(yTexture);
+    m_context->deleteTexture(uvTexture);
+    m_context->bindTexture(videoTextureTarget, 0);
+
+    return true;
+}
+
+bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(TextureType inputVideoTexture, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+{
+    if (!inputVideoTexture)
+        return false;
 
     GLfloat lowerLeft[2] = { 0, 0 };
     GLfloat lowerRight[2] = { 0, 0 };
@@ -288,8 +624,27 @@ bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(TextureType inputVi
     CVOpenGLTextureGetCleanTexCoords(inputVideoTexture, lowerLeft, lowerRight, upperRight, upperLeft);
 #endif
 
+    if (lowerLeft[1] < upperRight[1])
+        flipY = !flipY;
+
+    return copyVideoTextureToPlatformTexture(videoTextureName, videoTextureTarget, width, height, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
+}
+
+bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(Platform3DObject videoTextureName, GC3Denum videoTextureTarget, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY)
+{
     LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - internalFormat: %s, format: %s, type: %s flipY: %s, premultiplyAlpha: %s", this, enumToStringMap()[internalFormat], enumToStringMap()[format], enumToStringMap()[type], flipY ? "true" : "false", premultiplyAlpha ? "true" : "false");
 
+    GC3DStateSaver stateSaver(m_context.get());
+
+    if (!m_program) {
+        if (!initializeContextObjects()) {
+            LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);
+            return false;
+        }
+    }
+
+    stateSaver.saveVertexAttribState(m_positionAttributeLocation);
+
     m_context->bindFramebuffer(GraphicsContext3D::FRAMEBUFFER, m_framebuffer);
     
     // Allocate memory for the output texture.
@@ -326,9 +681,6 @@ bool VideoTextureCopierCV::copyVideoTextureToPlatformTexture(TextureType inputVi
     m_context->uniform2f(m_textureDimensionsUniformLocation, width, height);
 #endif
 
-    if (lowerLeft[1] < upperRight[1])
-        flipY = !flipY;
-
     m_context->uniform1i(m_flipYUniformLocation, flipY);
     m_context->uniform1i(m_premultiplyUniformLocation, premultiplyAlpha);
 
index 511766d..29e4c51 100644 (file)
 #import "GraphicsContext3D.h"
 
 typedef struct __CVBuffer* CVImageBufferRef;
+typedef struct __CVBuffer* CVPixelBufferRef;
 typedef CVImageBufferRef CVOpenGLTextureRef;
 typedef CVImageBufferRef CVOpenGLESTextureRef;
 
 namespace WebCore {
 
+class TextureCacheCV;
+
 class VideoTextureCopierCV {
 public:
     VideoTextureCopierCV(GraphicsContext3D&);
@@ -45,11 +48,14 @@ public:
     typedef CVOpenGLTextureRef TextureType;
 #endif
 
+    bool copyImageToPlatformTexture(CVPixelBufferRef, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY);
     bool copyVideoTextureToPlatformTexture(TextureType, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY);
 
     GraphicsContext3D& context() { return m_context.get(); }
 
 private:
+    bool copyVideoTextureToPlatformTexture(Platform3DObject inputTexture, GC3Denum inputTarget, size_t width, size_t height, Platform3DObject outputTexture, GC3Denum outputTarget, GC3Dint level, GC3Denum internalFormat, GC3Denum format, GC3Denum type, bool premultiplyAlpha, bool flipY);
+
     class GC3DStateSaver {
     public:
         GC3DStateSaver(GraphicsContext3D&);
@@ -75,8 +81,10 @@ private:
     };
 
     bool initializeContextObjects();
+    bool initializeUVContextObjects();
 
     Ref<GraphicsContext3D> m_context;
+    std::unique_ptr<TextureCacheCV> m_textureCache;
     Platform3DObject m_framebuffer { 0 };
     Platform3DObject m_program { 0 };
     Platform3DObject m_vertexBuffer { 0 };
@@ -85,6 +93,15 @@ private:
     GC3Dint m_flipYUniformLocation { -1 };
     GC3Dint m_premultiplyUniformLocation { -1 };
     GC3Dint m_positionAttributeLocation { -1 };
+    Platform3DObject m_yuvProgram { 0 };
+    Platform3DObject m_yuvVertexBuffer { 0 };
+    GC3Dint m_yTextureUniformLocation { -1 };
+    GC3Dint m_uvTextureUniformLocation { -1 };
+    GC3Dint m_yuvFlipYUniformLocation { -1 };
+    GC3Dint m_colorMatrixUniformLocation { -1 };
+    GC3Dint m_yuvPositionAttributeLocation { -1 };
+    GC3Dint m_yTextureSizeUniformLocation { -1 };
+    GC3Dint m_uvTextureSizeUniformLocation { -1 };
 };
 
 }