Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples...
authorjer.noble@apple.com <jer.noble@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 4 Jan 2019 21:32:48 +0000 (21:32 +0000)
committerjer.noble@apple.com <jer.noble@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 4 Jan 2019 21:32:48 +0000 (21:32 +0000)
https://bugs.webkit.org/show_bug.cgi?id=191806
<rdar://problem/46151477>

Reviewed by Dean Jackson.

Source/WebCore:

Test: http/tests/images/mp4-partial-load.html

Rather than use an AVAssetReaderTrackOutput, which will load both sample metadata and sample data
synchronously when a sample is requested, use AVAssetReaderSampleReferenceOutput, which only loads
sample metadata, including the byte offset and byte length of the sample data. By waiting until the
AVAsset signals that it's own metadata is loaded, we can safely parse all the sample metadata without
blocking on network loads. Once enough data is loaded, we can replace the byte reference and offset
attachements in the sample with actual data, and mark the sample as "complete".

Because the existing ImageSource assumes that image data parsing will occur synchronously, and that
synchronous parsing could cause a hang if the metadata is not loaded, add a new callback method which
allows the ImageSource to be notified when the encodedDataStatus changes. The ImageSource notifies the
CacheImage, which notifies the RenderImage, and thus the asynchronous parsing will kick off the
renderer's animation loop.

* loader/cache/CachedImage.cpp:
(WebCore::CachedImage::CachedImageObserver::encodedDataStatusChanged):
(WebCore::CachedImage::encodedDataStatusChanged):
* loader/cache/CachedImage.h:
* platform/graphics/ImageDecoder.h:
(WebCore::ImageDecoder::setEncodedDataStatusChangeCallback):
* platform/graphics/ImageObserver.h:
(WebCore::ImageObserver::encodedDataStatusChanged):
* platform/graphics/ImageSource.cpp:
(WebCore::ImageSource::ensureDecoderAvailable):
(WebCore::ImageSource::encodedDataStatusChanged):
(WebCore::ImageSource::frameDecodingStatusAtIndex):
* platform/graphics/ImageSource.h:
* platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h:
* platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:
(-[WebCoreSharedBufferResourceLoaderDelegate data]):
(WebCore::ImageDecoderAVFObjCSample::byteRange const):
(WebCore::ImageDecoderAVFObjC::readSamples):
(WebCore::ImageDecoderAVFObjC::setEncodedDataStatusChangeCallback):
(WebCore::ImageDecoderAVFObjC::encodedDataStatus const):
(WebCore::ImageDecoderAVFObjC::frameIsCompleteAtIndex const):
(WebCore::ImageDecoderAVFObjC::createFrameImageAtIndex):
(WebCore::ImageDecoderAVFObjC::sampleIsComplete const):

Source/WebCore/PAL:

* pal/cf/CoreMediaSoftLink.cpp:
* pal/cf/CoreMediaSoftLink.h:

LayoutTests:

* http/tests/images/mp4-partial-load-expected.txt: Added.
* http/tests/images/mp4-partial-load.html: Added.
* platform/win/http/tests/mp4-partial-load-expected.txt: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@239636 268f45cc-cd09-0410-ab3c-d52691b4dbfc

16 files changed:
LayoutTests/ChangeLog
LayoutTests/http/tests/images/mp4-partial-load-expected.txt [new file with mode: 0644]
LayoutTests/http/tests/images/mp4-partial-load.html [new file with mode: 0644]
LayoutTests/platform/win/http/tests/images/mp4-partial-load-expected.txt [new file with mode: 0644]
Source/WebCore/ChangeLog
Source/WebCore/PAL/ChangeLog
Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp
Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h
Source/WebCore/loader/cache/CachedImage.cpp
Source/WebCore/loader/cache/CachedImage.h
Source/WebCore/platform/graphics/ImageDecoder.h
Source/WebCore/platform/graphics/ImageObserver.h
Source/WebCore/platform/graphics/ImageSource.cpp
Source/WebCore/platform/graphics/ImageSource.h
Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h
Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm

index 7720269..162bd0b 100644 (file)
@@ -1,3 +1,15 @@
+2019-01-04  Jer Noble  <jer.noble@apple.com>
+
+        Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples for many seconds on imgur.com
+        https://bugs.webkit.org/show_bug.cgi?id=191806
+        <rdar://problem/46151477>
+
+        Reviewed by Dean Jackson.
+
+        * http/tests/images/mp4-partial-load-expected.txt: Added.
+        * http/tests/images/mp4-partial-load.html: Added.
+        * platform/win/http/tests/mp4-partial-load-expected.txt: Added.
+
 2019-01-04  Youenn Fablet  <youenn@apple.com>
 
         CSP violation reports should bypass CSP checks
diff --git a/LayoutTests/http/tests/images/mp4-partial-load-expected.txt b/LayoutTests/http/tests/images/mp4-partial-load-expected.txt
new file mode 100644 (file)
index 0000000..afc08ad
--- /dev/null
@@ -0,0 +1,10 @@
+This tests that an mp4 loaded in a video tag does not cause a hang when the load stalls.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS img.naturalWidth became 320
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
diff --git a/LayoutTests/http/tests/images/mp4-partial-load.html b/LayoutTests/http/tests/images/mp4-partial-load.html
new file mode 100644 (file)
index 0000000..3eb746a
--- /dev/null
@@ -0,0 +1,63 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>mp4-partial-load</title>
+    <script src="../../resources/js-test-pre.js"></script>
+    <script>
+    jsTestIsAsync = true;
+    if (window.internals && window.testRunner) {
+        testRunner.waitUntilDone();
+        testRunner.dumpAsText();
+        internals.clearMemoryCache();
+    }
+
+    function loadAndStall()
+    {
+        return "http://127.0.0.1:8000/resources/load-and-stall.php";
+    }
+
+    function mp4Image()
+    {
+        return "?name=../../../media/content/test.mp4&mimeType=video%2Fmp4";
+    }
+
+    function endTest()
+    {
+        if (window.timeoutID)
+            clearTimeout(timeoutID);
+        finishJSTest();
+    }
+
+    function errorReceived()
+    {
+        testFailed('Received an erorr loading image');
+        endTest();
+    }
+
+    function timedOut()
+    {
+        testFailed('Took too long');
+        endTest();
+    }
+
+    function runTest()
+    {
+        description('This tests that an mp4 loaded in a video tag does not cause a hang when the load stalls.')
+
+        setTimeout(() => {
+            img = document.querySelector('img');
+            img.addEventListener("error", errorReceived);
+            img.src = loadAndStall() + mp4Image() + "&stallAt=15799&stallFor=60";
+            window.timeoutID = setTimeout(timedOut, 5000);
+            shouldBecomeEqual('img.naturalWidth', '320', endTest);
+        });
+    }
+
+    window.addEventListener('load', runTest, {once: true});
+    </script>
+    <script src="../../resources/js-test-post.js"></script>
+</head>
+<body>
+    <img>
+</body>
+</html>
diff --git a/LayoutTests/platform/win/http/tests/images/mp4-partial-load-expected.txt b/LayoutTests/platform/win/http/tests/images/mp4-partial-load-expected.txt
new file mode 100644 (file)
index 0000000..db52d86
--- /dev/null
@@ -0,0 +1,10 @@
+This tests that an mp4 loaded in a video tag does not cause a hang when the load stalls.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+FAIL Took too long
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
index e53c249..8d9dfc6 100644 (file)
@@ -1,3 +1,50 @@
+2019-01-04  Jer Noble  <jer.noble@apple.com>
+
+        Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples for many seconds on imgur.com
+        https://bugs.webkit.org/show_bug.cgi?id=191806
+        <rdar://problem/46151477>
+
+        Reviewed by Dean Jackson.
+
+        Test: http/tests/images/mp4-partial-load.html
+
+        Rather than use an AVAssetReaderTrackOutput, which will load both sample metadata and sample data
+        synchronously when a sample is requested, use AVAssetReaderSampleReferenceOutput, which only loads
+        sample metadata, including the byte offset and byte length of the sample data. By waiting until the
+        AVAsset signals that it's own metadata is loaded, we can safely parse all the sample metadata without
+        blocking on network loads. Once enough data is loaded, we can replace the byte reference and offset
+        attachements in the sample with actual data, and mark the sample as "complete".
+
+        Because the existing ImageSource assumes that image data parsing will occur synchronously, and that
+        synchronous parsing could cause a hang if the metadata is not loaded, add a new callback method which
+        allows the ImageSource to be notified when the encodedDataStatus changes. The ImageSource notifies the
+        CacheImage, which notifies the RenderImage, and thus the asynchronous parsing will kick off the
+        renderer's animation loop.
+
+        * loader/cache/CachedImage.cpp:
+        (WebCore::CachedImage::CachedImageObserver::encodedDataStatusChanged):
+        (WebCore::CachedImage::encodedDataStatusChanged):
+        * loader/cache/CachedImage.h:
+        * platform/graphics/ImageDecoder.h:
+        (WebCore::ImageDecoder::setEncodedDataStatusChangeCallback):
+        * platform/graphics/ImageObserver.h:
+        (WebCore::ImageObserver::encodedDataStatusChanged):
+        * platform/graphics/ImageSource.cpp:
+        (WebCore::ImageSource::ensureDecoderAvailable):
+        (WebCore::ImageSource::encodedDataStatusChanged):
+        (WebCore::ImageSource::frameDecodingStatusAtIndex):
+        * platform/graphics/ImageSource.h:
+        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h:
+        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:
+        (-[WebCoreSharedBufferResourceLoaderDelegate data]):
+        (WebCore::ImageDecoderAVFObjCSample::byteRange const):
+        (WebCore::ImageDecoderAVFObjC::readSamples):
+        (WebCore::ImageDecoderAVFObjC::setEncodedDataStatusChangeCallback):
+        (WebCore::ImageDecoderAVFObjC::encodedDataStatus const):
+        (WebCore::ImageDecoderAVFObjC::frameIsCompleteAtIndex const):
+        (WebCore::ImageDecoderAVFObjC::createFrameImageAtIndex):
+        (WebCore::ImageDecoderAVFObjC::sampleIsComplete const):
+
 2019-01-04  Youenn Fablet  <youenn@apple.com>
 
         CSP violation reports should bypass CSP checks
index 89a382d..c8d74df 100644 (file)
@@ -1,3 +1,14 @@
+2019-01-04  Jer Noble  <jer.noble@apple.com>
+
+        Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples for many seconds on imgur.com
+        https://bugs.webkit.org/show_bug.cgi?id=191806
+        <rdar://problem/46151477>
+
+        Reviewed by Dean Jackson.
+
+        * pal/cf/CoreMediaSoftLink.cpp:
+        * pal/cf/CoreMediaSoftLink.h:
+
 2018-12-27  Alex Christensen  <achristensen@webkit.org>
 
         Resurrect Mac CMake build
index 548bdfb..c127bf3 100644 (file)
@@ -108,7 +108,10 @@ SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetSampleAttachments
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetSampleTimingInfoArray, OSStatus, (CMSampleBufferRef sbuf, CMItemCount timingArrayEntries, CMSampleTimingInfo *timingArrayOut, CMItemCount *timingArrayEntriesNeededOut), (sbuf, timingArrayEntries, timingArrayOut, timingArrayEntriesNeededOut))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimeConvertScale, CMTime, (CMTime time, int32_t newTimescale, CMTimeRoundingMethod method), (time, newTimescale, method))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetTotalSampleSize, size_t, (CMSampleBufferRef sbuf), (sbuf))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferSetDataBuffer, OSStatus, (CMSampleBufferRef sbuf, CMBlockBufferRef buffer), (sbuf, buffer))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMGetAttachment, CFTypeRef, (CMAttachmentBearerRef target, CFStringRef key, CMAttachmentMode* attachmentModeOut), (target, key, attachmentModeOut))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSetAttachment, void, (CMAttachmentBearerRef target, CFStringRef key, CFTypeRef value, CMAttachmentMode attachmentMode), (target, key, value, attachmentMode))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMRemoveAttachment, void, (CMAttachmentBearerRef target, CFStringRef key), (target, key))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimebaseCreateWithMasterClock, OSStatus, (CFAllocatorRef allocator, CMClockRef masterClock, CMTimebaseRef *timebaseOut), (allocator, masterClock, timebaseOut))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimebaseGetTime, CMTime, (CMTimebaseRef timebase), (timebase))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimebaseGetRate, Float64, (CMTimebaseRef timebase), (timebase))
@@ -141,6 +144,8 @@ SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_Drain
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_EmptyMedia, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_PostNotificationWhenConsumed, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceByteOffset, CFStringRef)
+SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceURL, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleAttachmentKey_DisplayImmediately, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleAttachmentKey_IsDependedOnByOthers, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferConsumerNotification_BufferConsumed, CFStringRef)
@@ -148,6 +153,7 @@ SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferConsumerNotificatio
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMTimebaseNotification_EffectiveRateChanged, CFStringRef)
 SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMTimebaseNotification_TimeJumped, CFStringRef)
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMAudioFormatDescriptionGetStreamBasicDescription, const AudioStreamBasicDescription *, (CMAudioFormatDescriptionRef desc), (desc))
+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMBlockBufferCreateWithMemoryBlock, OSStatus, (CFAllocatorRef structureAllocator, void* memoryBlock, size_t blockLength, CFAllocatorRef blockAllocator, const CMBlockBufferCustomBlockSource* customBlockSource, size_t offsetToData, size_t dataLength, CMBlockBufferFlags flags, CMBlockBufferRef* blockBufferOut), (structureAllocator, memoryBlock, blockLength, blockAllocator, customBlockSource, offsetToData, dataLength, flags, blockBufferOut))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer, OSStatus, (CMSampleBufferRef sbuf, size_t *bufferListSizeNeededOut, AudioBufferList *bufferListOut, size_t bufferListSize, CFAllocatorRef bbufStructAllocator, CFAllocatorRef bbufMemoryAllocator, uint32_t flags, CMBlockBufferRef *blockBufferOut), (sbuf, bufferListSizeNeededOut, bufferListOut, bufferListSize, bbufStructAllocator, bbufMemoryAllocator, flags, blockBufferOut))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetNumSamples, CMItemCount, (CMSampleBufferRef sbuf), (sbuf))
 SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferCopySampleBufferForRange, OSStatus, (CFAllocatorRef allocator, CMSampleBufferRef sbuf, CFRange sampleRange, CMSampleBufferRef* sBufOut), (allocator, sbuf, sampleRange, sBufOut))
index ffb1347..c951049 100644 (file)
@@ -171,8 +171,14 @@ SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetSampleTimingInfoA
 #define CMSampleBufferGetSampleTimingInfoArray softLink_CoreMedia_CMSampleBufferGetSampleTimingInfoArray
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetTotalSampleSize, size_t, (CMSampleBufferRef sbuf), (sbuf))
 #define CMSampleBufferGetTotalSampleSize softLink_CoreMedia_CMSampleBufferGetTotalSampleSize
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferSetDataBuffer, OSStatus, (CMSampleBufferRef sbuf, CMBlockBufferRef buffer), (sbuf, buffer))
+#define CMSampleBufferSetDataBuffer softLink_CoreMedia_CMSampleBufferSetDataBuffer
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMGetAttachment, CFTypeRef, (CMAttachmentBearerRef target, CFStringRef key, CMAttachmentMode* attachmentModeOut), (target, key, attachmentModeOut))
+#define CMGetAttachment softLink_CoreMedia_CMGetAttachment
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSetAttachment, void, (CMAttachmentBearerRef target, CFStringRef key, CFTypeRef value, CMAttachmentMode attachmentMode), (target, key, value, attachmentMode))
 #define CMSetAttachment softLink_CoreMedia_CMSetAttachment
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMRemoveAttachment, void, (CMAttachmentBearerRef target, CFStringRef key), (target, key))
+#define CMRemoveAttachment softLink_CoreMedia_CMRemoveAttachment
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMTimebaseCreateWithMasterClock, OSStatus, (CFAllocatorRef allocator, CMClockRef masterClock, CMTimebaseRef *timebaseOut), (allocator, masterClock, timebaseOut))
 #define CMTimebaseCreateWithMasterClock softLink_CoreMedia_CMTimebaseCreateWithMasterClock
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMTimebaseGetTime, CMTime, (CMTimebaseRef timebase), (timebase))
@@ -240,6 +246,10 @@ SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_PostN
 #define kCMSampleBufferAttachmentKey_PostNotificationWhenConsumed get_CoreMedia_kCMSampleBufferAttachmentKey_PostNotificationWhenConsumed()
 SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding, CFStringRef)
 #define kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding get_CoreMedia_kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding()
+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceByteOffset, CFStringRef)
+#define kCMSampleBufferAttachmentKey_SampleReferenceByteOffset get_CoreMedia_kCMSampleBufferAttachmentKey_SampleReferenceByteOffset()
+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceURL, CFStringRef)
+#define kCMSampleBufferAttachmentKey_SampleReferenceURL get_CoreMedia_kCMSampleBufferAttachmentKey_SampleReferenceURL()
 SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMTimebaseNotification_EffectiveRateChanged, CFStringRef)
 #define kCMTimebaseNotification_EffectiveRateChanged get_CoreMedia_kCMTimebaseNotification_EffectiveRateChanged()
 SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMTimebaseNotification_TimeJumped, CFStringRef)
@@ -248,6 +258,8 @@ SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferConsumerNotificatio
 #define kCMSampleBufferConsumerNotification_BufferConsumed get_CoreMedia_kCMSampleBufferConsumerNotification_BufferConsumed()
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMAudioFormatDescriptionGetStreamBasicDescription, const AudioStreamBasicDescription *, (CMAudioFormatDescriptionRef desc), (desc))
 #define CMAudioFormatDescriptionGetStreamBasicDescription softLink_CoreMedia_CMAudioFormatDescriptionGetStreamBasicDescription
+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMBlockBufferCreateWithMemoryBlock, OSStatus, (CFAllocatorRef structureAllocator, void* memoryBlock, size_t blockLength, CFAllocatorRef blockAllocator, const CMBlockBufferCustomBlockSource* customBlockSource, size_t offsetToData, size_t dataLength, CMBlockBufferFlags flags, CMBlockBufferRef* blockBufferOut), (structureAllocator, memoryBlock, blockLength, blockAllocator, customBlockSource, offsetToData, dataLength, flags, blockBufferOut))
+#define CMBlockBufferCreateWithMemoryBlock softLink_CoreMedia_CMBlockBufferCreateWithMemoryBlock
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer, OSStatus, (CMSampleBufferRef sbuf, size_t *bufferListSizeNeededOut, AudioBufferList *bufferListOut, size_t bufferListSize, CFAllocatorRef bbufStructAllocator, CFAllocatorRef bbufMemoryAllocator, uint32_t flags, CMBlockBufferRef *blockBufferOut), (sbuf, bufferListSizeNeededOut, bufferListOut, bufferListSize, bbufStructAllocator, bbufMemoryAllocator, flags, blockBufferOut))
 #define CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer softLink_CoreMedia_CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer
 SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetNumSamples, CMItemCount, (CMSampleBufferRef sbuf), (sbuf))
index 18fdae2..77a0ba5 100644 (file)
@@ -371,6 +371,12 @@ CachedImage::CachedImageObserver::CachedImageObserver(CachedImage& image)
     m_cachedImages.add(&image);
 }
 
+void CachedImage::CachedImageObserver::encodedDataStatusChanged(const Image& image, EncodedDataStatus status)
+{
+    for (auto cachedImage : m_cachedImages)
+        cachedImage->encodedDataStatusChanged(image, status);
+}
+
 void CachedImage::CachedImageObserver::decodedSizeChanged(const Image& image, long long delta)
 {
     for (auto cachedImage : m_cachedImages)
@@ -582,6 +588,14 @@ void CachedImage::destroyDecodedData()
         m_image->destroyDecodedData();
 }
 
+void CachedImage::encodedDataStatusChanged(const Image& image, EncodedDataStatus)
+{
+    if (&image != m_image)
+        return;
+
+    notifyObservers();
+}
+
 void CachedImage::decodedSizeChanged(const Image& image, long long delta)
 {
     if (&image != m_image)
index b23957c..9113c9c 100644 (file)
@@ -143,6 +143,7 @@ private:
         String mimeType() const override { return !m_cachedImages.isEmpty() ? (*m_cachedImages.begin())->mimeType() : emptyString(); }
         long long expectedContentLength() const override { return !m_cachedImages.isEmpty() ? (*m_cachedImages.begin())->expectedContentLength() : 0; }
 
+        void encodedDataStatusChanged(const Image&, EncodedDataStatus) final;
         void decodedSizeChanged(const Image&, long long delta) final;
         void didDraw(const Image&) final;
 
@@ -153,6 +154,7 @@ private:
         HashSet<CachedImage*> m_cachedImages;
     };
 
+    void encodedDataStatusChanged(const Image&, EncodedDataStatus);
     void decodedSizeChanged(const Image&, long long delta);
     void didDraw(const Image&);
     bool canDestroyDecodedData(const Image&);
index dc46d11..727f1ed 100644 (file)
@@ -55,6 +55,7 @@ public:
     virtual size_t bytesDecodedToDetermineProperties() const = 0;
 
     virtual EncodedDataStatus encodedDataStatus() const = 0;
+    virtual void setEncodedDataStatusChangeCallback(WTF::Function<void(EncodedDataStatus)>&&) { }
     virtual bool isSizeAvailable() const { return encodedDataStatus() >= EncodedDataStatus::SizeAvailable; }
     virtual IntSize size() const = 0;
     virtual size_t frameCount() const = 0;
index f28323c..7fd2740 100644 (file)
@@ -42,6 +42,7 @@ public:
     virtual String mimeType() const = 0;
     virtual long long expectedContentLength() const = 0;
 
+    virtual void encodedDataStatusChanged(const Image&, EncodedDataStatus) { };
     virtual void decodedSizeChanged(const Image&, long long delta) = 0;
 
     virtual void didDraw(const Image&) = 0;
index 68dff96..5b015c8 100644 (file)
@@ -76,6 +76,10 @@ bool ImageSource::ensureDecoderAvailable(SharedBuffer* data)
         return true;
 
     m_decoder = ImageDecoder::create(*data, mimeType(), m_alphaOption, m_gammaAndColorProfileOption);
+    m_decoder->setEncodedDataStatusChangeCallback([weakThis = makeWeakPtr(this)] (auto status) {
+        if (weakThis)
+            weakThis->encodedDataStatusChanged(status);
+    });
     if (!isDecoderAvailable())
         return false;
 
@@ -154,6 +158,20 @@ void ImageSource::clearFrameBufferCache(size_t beforeFrame)
     m_decoder->clearFrameBufferCache(beforeFrame);
 }
 
+void ImageSource::encodedDataStatusChanged(EncodedDataStatus status)
+{
+    if (status == m_encodedDataStatus)
+        return;
+
+    m_encodedDataStatus = status;
+
+    if (status >= EncodedDataStatus::SizeAvailable)
+        growFrames();
+
+    if (m_image && m_image->imageObserver())
+        m_image->imageObserver()->encodedDataStatusChanged(*m_image, status);
+}
+
 void ImageSource::decodedSizeChanged(long long decodedSize)
 {
     if (!decodedSize || !m_image || !m_image->imageObserver())
@@ -595,7 +613,7 @@ bool ImageSource::frameIsBeingDecodedAndIsCompatibleWithOptionsAtIndex(size_t in
 
 DecodingStatus ImageSource::frameDecodingStatusAtIndex(size_t index)
 {
-    return frameMetadataAtIndex<DecodingStatus>(index, (&ImageFrame::decodingStatus));
+    return frameMetadataAtIndexCacheIfNeeded<DecodingStatus>(index, (&ImageFrame::decodingStatus), nullptr, ImageFrame::Caching::Metadata);
 }
 
 bool ImageSource::frameHasAlphaAtIndex(size_t index)
index d46cd9b..52f1954 100644 (file)
@@ -30,6 +30,7 @@
 #include <wtf/Forward.h>
 #include <wtf/Optional.h>
 #include <wtf/SynchronizedFixedQueue.h>
+#include <wtf/WeakPtr.h>
 #include <wtf/WorkQueue.h>
 #include <wtf/text/TextStream.h>
 
@@ -39,7 +40,7 @@ class BitmapImage;
 class GraphicsContext;
 class ImageDecoder;
 
-class ImageSource : public ThreadSafeRefCounted<ImageSource> {
+class ImageSource : public ThreadSafeRefCounted<ImageSource>, public CanMakeWeakPtr<ImageSource> {
     friend class BitmapImage;
 public:
     ~ImageSource();
@@ -142,6 +143,7 @@ private:
     void decodedSizeIncreased(unsigned decodedSize);
     void decodedSizeDecreased(unsigned decodedSize);
     void decodedSizeReset(unsigned decodedSize);
+    void encodedDataStatusChanged(EncodedDataStatus);
 
     void setNativeImage(NativeImagePtr&&);
     void cacheMetadataAtIndex(size_t, SubsamplingLevel, DecodingStatus = DecodingStatus::Invalid);
index 60ac341..4a2dc49 100644 (file)
@@ -35,7 +35,6 @@
 
 OBJC_CLASS AVAssetTrack;
 OBJC_CLASS AVAssetReader;
-OBJC_CLASS AVAssetReaderTrackOutput;
 OBJC_CLASS AVURLAsset;
 OBJC_CLASS WebCoreSharedBufferResourceLoaderDelegate;
 typedef struct opaqueCMSampleBuffer* CMSampleBufferRef;
@@ -66,6 +65,7 @@ public:
 
     const String& mimeType() const { return m_mimeType; }
 
+    void setEncodedDataStatusChangeCallback(WTF::Function<void(EncodedDataStatus)>&&) final;
     EncodedDataStatus encodedDataStatus() const final;
     IntSize size() const final;
     size_t frameCount() const final;
@@ -109,6 +109,7 @@ private:
     void setTrack(AVAssetTrack *);
 
     const ImageDecoderAVFObjCSample* sampleAtIndex(size_t) const;
+    bool sampleIsComplete(const ImageDecoderAVFObjCSample&) const;
 
     String m_mimeType;
     String m_uti;
@@ -118,6 +119,7 @@ private:
     RetainPtr<VTImageRotationSessionRef> m_rotationSession;
     RetainPtr<CVPixelBufferPoolRef> m_rotationPool;
     Ref<WebCoreDecompressionSession> m_decompressionSession;
+    WTF::Function<void(EncodedDataStatus)> m_encodedDataStatusChangedCallback;
 
     SampleMap m_sampleData;
     DecodeOrderSampleMap::iterator m_cursor;
index ce8dab2..17dfd7a 100644 (file)
@@ -51,6 +51,7 @@
 #import <wtf/MainThread.h>
 #import <wtf/MediaTime.h>
 #import <wtf/NeverDestroyed.h>
+#import <wtf/Optional.h>
 #import <wtf/SoftLinking.h>
 #import <wtf/Vector.h>
 
@@ -63,7 +64,7 @@
 SOFT_LINK_FRAMEWORK_OPTIONAL(AVFoundation)
 SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVURLAsset)
 SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVAssetReader)
-SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVAssetReaderTrackOutput)
+SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVAssetReaderSampleReferenceOutput)
 SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVMediaCharacteristicVisual, NSString *)
 SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVURLAssetReferenceRestrictionsKey, NSString *)
 SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVURLAssetUsesNoPersistentCacheKey, NSString *)
@@ -81,6 +82,7 @@ SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVURLAssetUsesNoPersistentCacheKey, NS
     Vector<RetainPtr<AVAssetResourceLoadingRequest>> _requests;
     Lock _dataLock;
 }
+@property (readonly) NSData* data;
 - (id)initWithParent:(WebCore::ImageDecoderAVFObjC*)parent;
 - (void)setExpectedContentSize:(long long)expectedContentSize;
 - (void)updateData:(NSData *)data complete:(BOOL)complete;
@@ -100,6 +102,11 @@ SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVURLAssetUsesNoPersistentCacheKey, NS
     return self;
 }
 
+- (NSData*)data
+{
+    return _data.get();
+}
+
 - (void)setExpectedContentSize:(long long)expectedContentSize
 {
     LockHolder holder { _dataLock };
@@ -285,6 +292,36 @@ public:
         m_hasAlpha = alphaInfo != kCGImageAlphaNone && alphaInfo != kCGImageAlphaNoneSkipLast && alphaInfo != kCGImageAlphaNoneSkipFirst;
     }
 
+    struct ByteRange {
+        size_t byteOffset { 0 };
+        size_t byteLength { 0 };
+    };
+
+    Optional<ByteRange> byteRange() const
+    {
+        if (PAL::CMSampleBufferGetDataBuffer(m_sample.get())
+            || PAL::CMSampleBufferGetImageBuffer(m_sample.get())
+            || !PAL::CMSampleBufferDataIsReady(m_sample.get()))
+            return WTF::nullopt;
+
+        CFNumberRef byteOffsetCF = (CFNumberRef)PAL::CMGetAttachment(m_sample.get(), PAL::kCMSampleBufferAttachmentKey_SampleReferenceByteOffset, nullptr);
+        if (!byteOffsetCF || CFGetTypeID(byteOffsetCF) != CFNumberGetTypeID())
+            return WTF::nullopt;
+
+        int64_t byteOffset { 0 };
+        if (!CFNumberGetValue(byteOffsetCF, kCFNumberSInt64Type, &byteOffset))
+            return WTF::nullopt;
+
+        CMItemCount sizeArrayEntries = 0;
+        PAL::CMSampleBufferGetSampleSizeArray(m_sample.get(), 0, nullptr, &sizeArrayEntries);
+        if (sizeArrayEntries != 1)
+            return WTF::nullopt;
+
+        size_t singleSizeEntry;
+        PAL::CMSampleBufferGetSampleSizeArray(m_sample.get(), 1, &singleSizeEntry, nullptr);
+        return {{static_cast<size_t>(byteOffset), singleSizeEntry}};
+    }
+
     SampleFlags flags() const override
     {
         return (SampleFlags)(MediaSampleAVFObjC::flags() | (m_hasAlpha ? HasAlpha : 0));
@@ -387,19 +424,22 @@ void ImageDecoderAVFObjC::readSamples()
         return;
 
     auto assetReader = adoptNS([allocAVAssetReaderInstance() initWithAsset:m_asset.get() error:nil]);
-    auto assetReaderOutput = adoptNS([allocAVAssetReaderTrackOutputInstance() initWithTrack:m_track.get() outputSettings:nil]);
+    auto referenceOutput = adoptNS([allocAVAssetReaderSampleReferenceOutputInstance() initWithTrack:m_track.get()]);
 
-    assetReaderOutput.get().alwaysCopiesSampleData = NO;
-    [assetReader addOutput:assetReaderOutput.get()];
+    referenceOutput.get().alwaysCopiesSampleData = NO;
+    [assetReader addOutput:referenceOutput.get()];
     [assetReader startReading];
 
-    while (auto sampleBuffer = adoptCF([assetReaderOutput copyNextSampleBuffer])) {
+    while (auto sampleBuffer = adoptCF([referenceOutput copyNextSampleBuffer])) {
         // NOTE: Some samples emitted by the AVAssetReader simply denote the boundary of edits
         // and do not carry media data.
         if (!(PAL::CMSampleBufferGetNumSamples(sampleBuffer.get())))
             continue;
         m_sampleData.addSample(ImageDecoderAVFObjCSample::create(WTFMove(sampleBuffer)).get());
     }
+
+    if (m_encodedDataStatusChangedCallback)
+        m_encodedDataStatusChangedCallback(encodedDataStatus());
 }
 
 void ImageDecoderAVFObjC::readTrackMetadata()
@@ -500,11 +540,20 @@ void ImageDecoderAVFObjC::setTrack(AVAssetTrack *track)
     }];
 }
 
+void ImageDecoderAVFObjC::setEncodedDataStatusChangeCallback(WTF::Function<void(EncodedDataStatus)>&& callback)
+{
+    m_encodedDataStatusChangedCallback = WTFMove(callback);
+}
+
 EncodedDataStatus ImageDecoderAVFObjC::encodedDataStatus() const
 {
-    if (m_sampleData.empty())
-        return EncodedDataStatus::Unknown;
-    return EncodedDataStatus::Complete;
+    if (!m_sampleData.empty())
+        return EncodedDataStatus::Complete;
+    if (m_size)
+        return EncodedDataStatus::SizeAvailable;
+    if (m_track)
+        return EncodedDataStatus::TypeAvailable;
+    return EncodedDataStatus::Unknown;
 }
 
 IntSize ImageDecoderAVFObjC::size() const
@@ -548,7 +597,7 @@ bool ImageDecoderAVFObjC::frameIsCompleteAtIndex(size_t index) const
     if (!sampleData)
         return false;
 
-    return PAL::CMSampleBufferDataIsReady(sampleData->sampleBuffer());
+    return sampleIsComplete(*sampleData);
 }
 
 ImageOrientation ImageDecoderAVFObjC::frameOrientationAtIndex(size_t) const
@@ -615,11 +664,44 @@ NativeImagePtr ImageDecoderAVFObjC::createFrameImageAtIndex(size_t index, Subsam
         if (decodeTime < m_cursor->second->decodeTime())
             return nullptr;
 
-        auto cursorSample = toSample(m_cursor)->sampleBuffer();
+        auto cursorSample = toSample(m_cursor);
         if (!cursorSample)
+            return nullptr;
+
+        if (!sampleIsComplete(*cursorSample))
+            return nullptr;
+
+        if (auto byteRange = cursorSample->byteRange()) {
+            auto& byteRangeValue = byteRange.value();
+            auto* data = m_loader.get().data;
+            CMBlockBufferCustomBlockSource source {
+                0,
+                nullptr,
+                [](void* refcon, void*, size_t) {
+                    [(id)refcon release];
+                },
+                [data retain]
+            };
+            CMBlockBufferRef rawBlockBuffer = nullptr;
+            if (noErr != PAL::CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, const_cast<void*>(data.bytes), data.length, nullptr, &source, byteRangeValue.byteOffset, byteRangeValue.byteLength, 0, &rawBlockBuffer))
+                return nullptr;
+
+            if (!rawBlockBuffer)
+                return nullptr;
+
+            if (noErr != PAL::CMSampleBufferSetDataBuffer(cursorSample->sampleBuffer(), rawBlockBuffer))
+                return nullptr;
+            CFRelease(rawBlockBuffer);
+
+            PAL::CMRemoveAttachment(cursorSample->sampleBuffer(), PAL::kCMSampleBufferAttachmentKey_SampleReferenceByteOffset);
+            PAL::CMRemoveAttachment(cursorSample->sampleBuffer(), PAL::kCMSampleBufferAttachmentKey_SampleReferenceURL);
+        }
+
+        auto cursorSampleBuffer = cursorSample->sampleBuffer();
+        if (!cursorSampleBuffer)
             break;
 
-        if (!storeSampleBuffer(cursorSample))
+        if (!storeSampleBuffer(cursorSampleBuffer))
             break;
 
         advanceCursor();
@@ -679,6 +761,16 @@ const ImageDecoderAVFObjCSample* ImageDecoderAVFObjC::sampleAtIndex(size_t index
     return toSample(iter);
 }
 
+bool ImageDecoderAVFObjC::sampleIsComplete(const ImageDecoderAVFObjCSample& sample) const
+{
+    if (auto byteRange = sample.byteRange()) {
+        auto& byteRangeValue = byteRange.value();
+        return byteRangeValue.byteOffset + byteRangeValue.byteLength <= m_loader.get().data.length;
+    }
+
+    return PAL::CMSampleBufferDataIsReady(sample.sampleBuffer());
+}
+
 }
 
 #endif