[MSE] YouTube videos fail to play
authorjer.noble@apple.com <jer.noble@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 3 Mar 2014 20:47:52 +0000 (20:47 +0000)
committerjer.noble@apple.com <jer.noble@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 3 Mar 2014 20:47:52 +0000 (20:47 +0000)
https://bugs.webkit.org/show_bug.cgi?id=129525

Reviewed by Darin Adler.

Source/WebCore:

Test: media/media-source/media-source-fudge-factor.html

Add a very simple playability metric to SourceBuffer. Track the number of seconds buffered
and use that metric to determine whether the MediaSource has buffered enough data to play
through.

* Modules/mediasource/MediaSource.cpp:
(WebCore::MediaSource::currentTime): Added simple accessor.
(WebCore::MediaSource::monitorSourceBuffers): Replace the functor-based iterators with lambdas.
(WebCore::MediaSource::addSourceBuffer): Drive-by fix; only add new source buffers to the
    activeSourceBuffers list if those buffers are actually active.
* Modules/mediasource/MediaSource.h:
* Modules/mediasource/SourceBuffer.cpp:
(WebCore::SourceBuffer::SourceBuffer): Initialize new ivars.
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveSample): Measure the number of seconds buffered.
(WebCore::SourceBuffer::monitorBufferingRate): Use a simple Exponential moving average to
    track the buffering rate.
(WebCore::SourceBuffer::hasCurrentTime): Return true if the current time is within 1/24 s
    of a buffered range.
(WebCore::SourceBuffer::hasFutureTime): Moved from MediaSource.
(WebCore::SourceBuffer::canPlayThrough): Return true if the buffering rate is > 1s per
    second, or if the rate is sufficient to keep up over the remaining time.
* Modules/mediasource/SourceBuffer.h:

LayoutTests:

* media/media-source/media-source-addsourcebuffer.html: Updated.
* media/media-source/media-source-addsourcebuffer-expected.txt: Updated.
* media/media-source/media-source-fudge-factor-expected.txt: Added.
* media/media-source/media-source-fudge-factor.html: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@164999 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/ChangeLog
LayoutTests/media/media-source/media-source-addsourcebuffer-expected.txt
LayoutTests/media/media-source/media-source-addsourcebuffer.html
LayoutTests/media/media-source/media-source-fudge-factor-expected.txt [new file with mode: 0644]
LayoutTests/media/media-source/media-source-fudge-factor.html [new file with mode: 0644]
Source/WebCore/ChangeLog
Source/WebCore/Modules/mediasource/MediaSource.cpp
Source/WebCore/Modules/mediasource/MediaSource.h
Source/WebCore/Modules/mediasource/SourceBuffer.cpp
Source/WebCore/Modules/mediasource/SourceBuffer.h

index d878438..1b880c0 100644 (file)
@@ -1,3 +1,15 @@
+2014-03-03  Jer Noble  <jer.noble@apple.com>
+
+        [MSE] YouTube videos fail to play
+        https://bugs.webkit.org/show_bug.cgi?id=129525
+
+        Reviewed by Darin Adler.
+
+        * media/media-source/media-source-addsourcebuffer.html: Updated.
+        * media/media-source/media-source-addsourcebuffer-expected.txt: Updated.
+        * media/media-source/media-source-fudge-factor-expected.txt: Added.
+        * media/media-source/media-source-fudge-factor.html: Added.
+
 2014-03-03  Bear Travis  <betravis@adobe.com>
 
         [CSS Shapes] Serialize circle positions
index bb3964e..defbfee 100644 (file)
@@ -7,6 +7,5 @@ RUN(video.src = URL.createObjectURL(source))
 EVENT(sourceopen)
 RUN(sourceBuffer = source.addSourceBuffer("video/mock; codecs=mock"))
 EXPECTED (source.sourceBuffers[0] == '[object SourceBuffer]') OK
-EXPECTED (source.activeSourceBuffers[0] == '[object SourceBuffer]') OK
 END OF TEST
 
index 396c53c..9ca8c72 100644 (file)
@@ -27,7 +27,6 @@
     function sourceOpen() {
         run('sourceBuffer = source.addSourceBuffer("video/mock; codecs=mock")');
         testExpected('source.sourceBuffers[0]', sourceBuffer);
-        testExpected('source.activeSourceBuffers[0]', sourceBuffer);
         endTest();
     }
     
@@ -36,4 +35,4 @@
 <body onload="runTest()">
     <video></video>
 </body>
-</html>
\ No newline at end of file
+</html>
diff --git a/LayoutTests/media/media-source/media-source-fudge-factor-expected.txt b/LayoutTests/media/media-source/media-source-fudge-factor-expected.txt
new file mode 100644 (file)
index 0000000..afaddaa
--- /dev/null
@@ -0,0 +1,15 @@
+
+RUN(video.src = URL.createObjectURL(source))
+EVENT(loadedmetadata)
+Samples with presentation times after currentTime should not cause loadedData.
+EXPECTED (video.readyState == '1') OK
+Samples with presentation times very close to currentTime should cause loadedData.
+EVENT(loadeddata)
+EXPECTED (video.readyState == '2') OK
+Samples with presentation end times very close to currentTime should not cause canPlay.
+EXPECTED (video.readyState == '2') OK
+Continuous samples with presentation end times after currentTime should cause canPlay.
+EVENT(canplay)
+EXPECTED (video.readyState >= '3') OK
+END OF TEST
+
diff --git a/LayoutTests/media/media-source/media-source-fudge-factor.html b/LayoutTests/media/media-source/media-source-fudge-factor.html
new file mode 100644 (file)
index 0000000..621fce3
--- /dev/null
@@ -0,0 +1,73 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>media-source-fudge-factor</title>
+    <script src="mock-media-source.js"></script>
+    <script src="../video-test.js"></script>
+    <script>
+    var source;
+    var sourceBuffer;
+
+    var requestLength = 200000;
+    var nextRequest = 0;
+    var totalLength = 100;
+
+    if (window.internals)
+        internals.initializeMockMediaSource();
+
+    function runTest() {
+        findMediaElement();
+        source = new MediaSource();
+        source.addEventListener('sourceopen', startLoad);
+        run('video.src = URL.createObjectURL(source)');
+    }
+
+    function startLoad() {
+        sourceBuffer = source.addSourceBuffer('video/mock; codecs="mock"');
+
+        // Make an init segment with 1 video track
+        var init = makeAInit(100, [makeATrack(1, 'mock', TRACK_KIND.VIDEO)]);
+        waitForEventOnce('loadedmetadata', loadedMetadata);
+        sourceBuffer.appendBuffer(init)
+    }
+
+    function loadedMetadata() {
+        consoleWrite('Samples with presentation times after currentTime should not cause loadedData.');
+        sourceBuffer.appendBuffer(makeASample(0.5, 0.5, 0.5, 1, SAMPLE_FLAG.SYNC));
+        setTimeout(notLoadedData, 50);
+    }
+
+    function notLoadedData() {
+        testExpected('video.readyState', HTMLMediaElement.HAVE_METADATA);
+
+        consoleWrite('Samples with presentation times very close to currentTime should cause loadedData.');
+        sourceBuffer.appendBuffer(makeASample(0.01, 0.01, 0.01, 1, SAMPLE_FLAG.SYNC));
+        waitForEvent('loadeddata', loadedData);
+    }
+
+    function loadedData() {
+        testExpected('video.readyState', HTMLMediaElement.HAVE_CURRENT_DATA);
+
+        consoleWrite('Samples with presentation end times very close to currentTime should not cause canPlay.');
+        sourceBuffer.appendBuffer(makeASample(0.02, 0.02, 0.01, 1, SAMPLE_FLAG.SYNC));
+        setTimeout(notCanPlay, 50);
+    }
+
+    function notCanPlay() {
+        testExpected('video.readyState', HTMLMediaElement.HAVE_CURRENT_DATA);
+
+        consoleWrite('Continuous samples with presentation end times after currentTime should cause canPlay.');
+        sourceBuffer.appendBuffer(makeASample(0.03, 0.03, 0.47, 1, SAMPLE_FLAG.SYNC));
+        waitForEvent('canplay', canPlay);
+    }
+
+    function canPlay() {
+        testExpected('video.readyState', HTMLMediaElement.HAVE_FUTURE_DATA, '>=');
+        endTest();
+    }
+    </script>
+</head>
+<body onload="runTest()">
+    <video controls width=960 height=510></video>
+</body>
+</html>
index 91e9126..b08f2ba 100644 (file)
@@ -1,3 +1,34 @@
+2014-02-28  Jer Noble  <jer.noble@apple.com>
+
+        [MSE] YouTube videos fail to play
+        https://bugs.webkit.org/show_bug.cgi?id=129525
+
+        Reviewed by Darin Adler.
+
+        Test: media/media-source/media-source-fudge-factor.html
+
+        Add a very simple playability metric to SourceBuffer. Track the number of seconds buffered
+        and use that metric to determine whether the MediaSource has buffered enough data to play
+        through.
+
+        * Modules/mediasource/MediaSource.cpp:
+        (WebCore::MediaSource::currentTime): Added simple accessor.
+        (WebCore::MediaSource::monitorSourceBuffers): Replace the functor-based iterators with lambdas.
+        (WebCore::MediaSource::addSourceBuffer): Drive-by fix; only add new source buffers to the
+            activeSourceBuffers list if those buffers are actually active.
+        * Modules/mediasource/MediaSource.h:
+        * Modules/mediasource/SourceBuffer.cpp:
+        (WebCore::SourceBuffer::SourceBuffer): Initialize new ivars.
+        (WebCore::SourceBuffer::sourceBufferPrivateDidReceiveSample): Measure the number of seconds buffered.
+        (WebCore::SourceBuffer::monitorBufferingRate): Use a simple Exponential moving average to
+            track the buffering rate.
+        (WebCore::SourceBuffer::hasCurrentTime): Return true if the current time is within 1/24 s
+            of a buffered range.
+        (WebCore::SourceBuffer::hasFutureTime): Moved from MediaSource.
+        (WebCore::SourceBuffer::canPlayThrough): Return true if the buffering rate is > 1s per
+            second, or if the rate is sufficient to keep up over the remaining time.
+        * Modules/mediasource/SourceBuffer.h:
+
 2014-03-03  Bear Travis  <betravis@adobe.com>
 
         [CSS Shapes] Serialize circle positions
index 9cb36e7..806d206 100644 (file)
@@ -131,6 +131,11 @@ double MediaSource::duration() const
     return isClosed() ? std::numeric_limits<float>::quiet_NaN() : m_private->duration();
 }
 
+double MediaSource::currentTime() const
+{
+    return m_mediaElement ? m_mediaElement->currentTime() : 0;
+}
+
 std::unique_ptr<PlatformTimeRanges> MediaSource::buffered() const
 {
     // Implements MediaSource algorithm for HTMLMediaElement.buffered.
@@ -175,61 +180,17 @@ std::unique_ptr<PlatformTimeRanges> MediaSource::buffered() const
     return PlatformTimeRanges::create(intersectionRanges->ranges());
 }
 
-class SourceBufferBufferedDoesNotContainTime {
-public:
-    SourceBufferBufferedDoesNotContainTime(double time) : m_time(time) { }
-    bool operator()(RefPtr<SourceBuffer> sourceBuffer)
-    {
-        return !sourceBuffer->buffered()->contain(m_time);
-    }
-
-    double m_time;
-};
-
-class SourceBufferBufferedHasEnough {
-public:
-    SourceBufferBufferedHasEnough(double time, double duration) : m_time(time), m_duration(duration) { }
-    bool operator()(RefPtr<SourceBuffer> sourceBuffer)
-    {
-        size_t rangePos = sourceBuffer->buffered()->find(m_time);
-        if (rangePos == notFound)
-            return false;
-
-        double endTime = sourceBuffer->buffered()->end(rangePos, IGNORE_EXCEPTION);
-        return m_duration - endTime < 1;
-    }
-
-    double m_time;
-    double m_duration;
-};
-
-class SourceBufferBufferedHasFuture {
-public:
-    SourceBufferBufferedHasFuture(double time) : m_time(time) { }
-    bool operator()(RefPtr<SourceBuffer> sourceBuffer)
-    {
-        size_t rangePos = sourceBuffer->buffered()->find(m_time);
-        if (rangePos == notFound)
-            return false;
-
-        double endTime = sourceBuffer->buffered()->end(rangePos, IGNORE_EXCEPTION);
-        return endTime - m_time > 1;
-    }
-
-    double m_time;
-};
-
 void MediaSource::monitorSourceBuffers()
 {
-    double currentTime = mediaElement()->currentTime();
-
     // 2.4.4 SourceBuffer Monitoring
     // https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media-source.html#buffer-monitoring
     // ↳ If buffered for all objects in activeSourceBuffers do not contain TimeRanges for the current
     // playback position:
     auto begin = m_activeSourceBuffers->begin();
     auto end = m_activeSourceBuffers->end();
-    if (std::all_of(begin, end, SourceBufferBufferedDoesNotContainTime(currentTime))) {
+    if (std::all_of(begin, end, [](RefPtr<SourceBuffer>& sourceBuffer) {
+        return !sourceBuffer->hasCurrentTime();
+    })) {
         // 1. Set the HTMLMediaElement.readyState attribute to HAVE_METADATA.
         // 2. If this is the first transition to HAVE_METADATA, then queue a task to fire a simple event
         // named loadedmetadata at the media element.
@@ -241,7 +202,9 @@ void MediaSource::monitorSourceBuffers()
 
     // ↳ If buffered for all objects in activeSourceBuffers contain TimeRanges that include the current
     // playback position and enough data to ensure uninterrupted playback:
-    if (std::all_of(begin, end, SourceBufferBufferedHasEnough(currentTime, mediaElement()->duration()))) {
+    if (std::all_of(begin, end, [](RefPtr<SourceBuffer>& sourceBuffer) {
+        return sourceBuffer->hasFutureTime() && sourceBuffer->canPlayThrough();
+    })) {
         // 1. Set the HTMLMediaElement.readyState attribute to HAVE_ENOUGH_DATA.
         // 2. Queue a task to fire a simple event named canplaythrough at the media element.
         // 3. Playback may resume at this point if it was previously suspended by a transition to HAVE_CURRENT_DATA.
@@ -253,7 +216,9 @@ void MediaSource::monitorSourceBuffers()
 
     // ↳ If buffered for at least one object in activeSourceBuffers contains a TimeRange that includes
     // the current playback position but not enough data to ensure uninterrupted playback:
-    if (std::any_of(begin, end, SourceBufferBufferedHasFuture(currentTime))) {
+    if (std::any_of(begin, end, [](RefPtr<SourceBuffer>& sourceBuffer) {
+        return sourceBuffer->hasFutureTime();
+    })) {
         // 1. Set the HTMLMediaElement.readyState attribute to HAVE_FUTURE_DATA.
         // 2. If the previous value of HTMLMediaElement.readyState was less than HAVE_FUTURE_DATA, then queue a task to fire a simple event named canplay at the media element.
         // 3. Playback may resume at this point if it was previously suspended by a transition to HAVE_CURRENT_DATA.
@@ -438,7 +403,9 @@ SourceBuffer* MediaSource::addSourceBuffer(const String& type, ExceptionCode& ec
     RefPtr<SourceBuffer> buffer = SourceBuffer::create(sourceBufferPrivate.releaseNonNull(), this);
     // 6. Add the new object to sourceBuffers and fire a addsourcebuffer on that object.
     m_sourceBuffers->add(buffer);
-    m_activeSourceBuffers->add(buffer);
+
+    if (buffer->active())
+        m_activeSourceBuffers->add(buffer);
     // 7. Return the new object to the caller.
     return buffer.get();
 }
index 70fe309..dc591c0 100644 (file)
@@ -80,6 +80,7 @@ public:
     void monitorSourceBuffers();
 
     void setDuration(double, ExceptionCode&);
+    double currentTime() const;
     const AtomicString& readyState() const { return m_readyState; }
     void setReadyState(const AtomicString&);
     void endOfStream(const AtomicString& error, ExceptionCode&);
index be9ceb3..a9ac0a3 100644 (file)
 #include "TimeRanges.h"
 #include "VideoTrackList.h"
 #include <map>
+#include <wtf/CurrentTime.h>
 #include <wtf/NeverDestroyed.h>
 
 namespace WebCore {
 
+static double ExponentialMovingAverageCoefficient = 0.1;
+
+// Allow hasCurrentTime() to be off by as much as the length of a 24fps video frame
+static double CurrentTimeFudgeFactor = 1. / 24;
+
 struct SourceBuffer::TrackBuffer {
     MediaTime lastDecodeTimestamp;
     MediaTime lastFrameDuration;
@@ -94,6 +100,9 @@ SourceBuffer::SourceBuffer(PassRef<SourceBufferPrivate> sourceBufferPrivate, Med
     , m_buffered(TimeRanges::create())
     , m_active(false)
     , m_appendState(WaitingForSegment)
+    , m_timeOfBufferingMonitor(monotonicallyIncreasingTime())
+    , m_bufferedSinceLastMonitor(0)
+    , m_averageBufferRate(0)
 {
     ASSERT(m_private);
     ASSERT(m_source);
@@ -999,6 +1008,7 @@ void SourceBuffer::sourceBufferPrivateDidReceiveSample(SourceBufferPrivate*, Pas
             m_highestPresentationEndTimestamp = frameEndTimestamp;
 
         m_buffered->add(presentationTimestamp.toDouble(), (presentationTimestamp + frameDuration + microsecond).toDouble());
+        m_bufferedSinceLastMonitor += frameDuration.toDouble();
 
         break;
     } while (1);
@@ -1166,6 +1176,73 @@ void SourceBuffer::didDropSample()
         m_source->mediaElement()->incrementDroppedFrameCount();
 }
 
+void SourceBuffer::monitorBufferingRate()
+{
+    if (!m_bufferedSinceLastMonitor)
+        return;
+
+    double now = monotonicallyIncreasingTime();
+    double interval = now - m_timeOfBufferingMonitor;
+    double rateSinceLastMonitor = m_bufferedSinceLastMonitor / interval;
+
+    m_timeOfBufferingMonitor = now;
+    m_bufferedSinceLastMonitor = 0;
+
+    m_averageBufferRate = m_averageBufferRate * (1 - ExponentialMovingAverageCoefficient) + rateSinceLastMonitor * ExponentialMovingAverageCoefficient;
+
+    LOG(Media, "SourceBuffer::monitorBufferingRate(%p) - m_avegareBufferRate: %lf", this, m_averageBufferRate);
+}
+
+bool SourceBuffer::hasCurrentTime() const
+{
+    if (!m_buffered->length())
+        return false;
+
+    double currentTime = m_source->currentTime();
+    return fabs(m_buffered->nearest(m_source->currentTime()) - currentTime) <= CurrentTimeFudgeFactor;
+}
+
+bool SourceBuffer::hasFutureTime() const
+{
+    double currentTime = m_source->currentTime();
+    const PlatformTimeRanges& ranges = m_buffered->ranges();
+    double nearest = m_buffered->nearest(m_source->currentTime());
+    if (fabs(m_buffered->nearest(m_source->currentTime()) - currentTime) > CurrentTimeFudgeFactor)
+        return false;
+
+    size_t found = ranges.find(nearest);
+    ASSERT(found != notFound);
+
+    bool ignoredValid = false;
+    return ranges.end(found, ignoredValid) - currentTime > CurrentTimeFudgeFactor;
+}
+
+bool SourceBuffer::canPlayThrough()
+{
+    monitorBufferingRate();
+
+    // Assuming no fluctuations in the buffering rate, loading 1 second per second or greater
+    // means indefinite playback. This could be improved by taking jitter into account.
+    if (m_averageBufferRate > 1)
+        return true;
+
+    // Add up all the time yet to be buffered.
+    double unbufferedTime = 0;
+    double currentTime = m_source->currentTime();
+    double duration = m_source->duration();
+
+    PlatformTimeRanges unbufferedRanges = m_buffered->ranges();
+    unbufferedRanges.invert();
+    unbufferedRanges.intersectWith(PlatformTimeRanges(currentTime, std::max(currentTime, duration)));
+    bool valid = true;
+
+    for (size_t i = 0, end = unbufferedRanges.length(); i < end; ++i)
+        unbufferedTime += unbufferedRanges.end(i, valid) - unbufferedRanges.start(i, valid);
+
+    double timeRemaining = duration - currentTime;
+    return unbufferedTime / m_averageBufferRate < timeRemaining;
+}
+
 } // namespace WebCore
 
 #endif
index 1634050..37414b3 100644 (file)
@@ -83,6 +83,12 @@ public:
     TextTrackList* textTracks();
 #endif
 
+    bool hasCurrentTime() const;
+    bool hasFutureTime() const;
+    bool canPlayThrough();
+
+    bool active() const { return m_active; }
+
     // ActiveDOMObject interface
     virtual bool hasPendingActivity() const override;
     virtual void stop() override;
@@ -144,6 +150,8 @@ private:
     void provideMediaData(TrackBuffer&, AtomicString trackID);
     void didDropSample();
 
+    void monitorBufferingRate();
+
     RefPtr<SourceBufferPrivate> m_private;
     MediaSource* m_source;
     GenericEventQueue m_asyncEventQueue;
@@ -172,6 +180,9 @@ private:
     enum AppendStateType { WaitingForSegment, ParsingInitSegment, ParsingMediaSegment };
     AppendStateType m_appendState;
 
+    double m_timeOfBufferingMonitor;
+    double m_bufferedSinceLastMonitor;
+    double m_averageBufferRate;
 };
 
 } // namespace WebCore