[MSE][GStreamer] Fix video sometimes not appearing
authoraboya@igalia.com <aboya@igalia.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 9 Oct 2019 19:40:05 +0000 (19:40 +0000)
committeraboya@igalia.com <aboya@igalia.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 9 Oct 2019 19:40:05 +0000 (19:40 +0000)
https://bugs.webkit.org/show_bug.cgi?id=201401

Reviewed by Xabier Rodriguez-Calvar.

LayoutTests/imported/w3c:

Added a test that reproduces the sequence of operations that was
causing the video to not appear in WebKitGTK and checks that this time
there is visible output.

* web-platform-tests/lint.whitelist:
* web-platform-tests/media-source/mediasource-video-is-visible-expected.html: Added.
* web-platform-tests/media-source/mediasource-video-is-visible.html: Added.
* web-platform-tests/media-source/mp4/test-a-1s.mp4: Added.
* web-platform-tests/media-source/mp4/test-a-1s.mp4-manifest.json: Added.
* web-platform-tests/media-source/mp4/test-v-1s-blue.mp4: Added.
* web-platform-tests/media-source/mp4/test-v-1s-blue.mp4-manifest.json: Added.
* web-platform-tests/media-source/webm/test-a-1s.webm: Added.
* web-platform-tests/media-source/webm/test-a-1s.webm-manifest.json: Added.
* web-platform-tests/media-source/webm/test-v-1s-blue.webm: Added.
* web-platform-tests/media-source/webm/test-v-1s-blue.webm-manifest.json: Added.

Source/WebCore:

The code in MediaPlayerPrivateGStreamer::changePipelineState() was
supposed to call `ensureGLVideoSinkContext()` on upwards transitions
to PAUSED but the code did not take into account non-step-by-step
state transitions, which happens frequently with playbin3 in the MSE
case.

Before the patch, when playbin3 transitioned from READY to PLAYING
without stopping for preroll this call would not be made and the
texture IDs received at the sink would not correspond to the
compositor GL context, leading to artifacts (often the player controls
or a blank screen).

Test: imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible.html

* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
(WebCore::MediaPlayerPrivateGStreamer::changePipelineState):

LayoutTests:

Added an exact expectation picture to avoid errors in the test runner
(only a fuzzy match is expected, and different platforms render video
with slightly different colors).

* platform/mac/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html: Added.
* platform/mac/TestExpectations:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@250922 268f45cc-cd09-0410-ab3c-d52691b4dbfc

17 files changed:
LayoutTests/ChangeLog
LayoutTests/imported/w3c/ChangeLog
LayoutTests/imported/w3c/web-platform-tests/lint.whitelist
LayoutTests/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible.html [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4 [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4-manifest.json [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4 [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4-manifest.json [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm-manifest.json [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm [new file with mode: 0644]
LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm-manifest.json [new file with mode: 0644]
LayoutTests/platform/mac/TestExpectations
LayoutTests/platform/mac/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html [new file with mode: 0644]
Source/WebCore/ChangeLog
Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp

index 162b504..6d250b7 100644 (file)
@@ -1,3 +1,17 @@
+2019-10-09  Alicia Boya García  <aboya@igalia.com>
+
+        [MSE][GStreamer] Fix video sometimes not appearing
+        https://bugs.webkit.org/show_bug.cgi?id=201401
+
+        Reviewed by Xabier Rodriguez-Calvar.
+
+        Added an exact expectation picture to avoid errors in the test runner
+        (only a fuzzy match is expected, and different platforms render video
+        with slightly different colors).
+
+        * platform/mac/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html: Added.
+        * platform/mac/TestExpectations:
+
 2019-10-09  Dean Jackson  <dino@apple.com>
 
         REGRESSION (r250755): fast/events/ios/ipad/fast-click-not-always.html is Failing
index 2b43ce2..1afaae6 100644 (file)
@@ -1,3 +1,26 @@
+2019-10-09  Alicia Boya García  <aboya@igalia.com>
+
+        [MSE][GStreamer] Fix video sometimes not appearing
+        https://bugs.webkit.org/show_bug.cgi?id=201401
+
+        Reviewed by Xabier Rodriguez-Calvar.
+
+        Added a test that reproduces the sequence of operations that was
+        causing the video to not appear in WebKitGTK and checks that this time
+        there is visible output.
+
+        * web-platform-tests/lint.whitelist:
+        * web-platform-tests/media-source/mediasource-video-is-visible-expected.html: Added.
+        * web-platform-tests/media-source/mediasource-video-is-visible.html: Added.
+        * web-platform-tests/media-source/mp4/test-a-1s.mp4: Added.
+        * web-platform-tests/media-source/mp4/test-a-1s.mp4-manifest.json: Added.
+        * web-platform-tests/media-source/mp4/test-v-1s-blue.mp4: Added.
+        * web-platform-tests/media-source/mp4/test-v-1s-blue.mp4-manifest.json: Added.
+        * web-platform-tests/media-source/webm/test-a-1s.webm: Added.
+        * web-platform-tests/media-source/webm/test-a-1s.webm-manifest.json: Added.
+        * web-platform-tests/media-source/webm/test-v-1s-blue.webm: Added.
+        * web-platform-tests/media-source/webm/test-v-1s-blue.webm-manifest.json: Added.
+
 2019-10-08  Wenson Hsieh  <wenson_hsieh@apple.com>
 
         [Clipboard API] Import web-platform-tests/clipboard-apis
index 287acd6..e4b0d7b 100644 (file)
@@ -879,3 +879,4 @@ CONSOLE:payment-request/payment-request-response-id.html
 LAYOUTTESTS APIS: css/css-regions-1/interactivity/*
 LAYOUTTESTS APIS: css/work-in-progress/opera/css-device-adapt/template.blink.html
 LAYOUTTESTS APIS: webgl/conformance-1.0.3/*
+LAYOUTTESTS APIS: media-source/mediasource-video-is-visible.html
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html b/LayoutTests/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html
new file mode 100644 (file)
index 0000000..a961700
--- /dev/null
@@ -0,0 +1,36 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <title>HTML MSE test: video is shown</title>
+    <link rel="author" title="Alicia Boya García" href="mailto:aboya@igalia.com">
+    <style>
+        html, body {
+            margin: 0;
+            padding: 0;
+            overflow: hidden;
+            background: white;
+        }
+        body {
+            position: absolute;
+            top: 0;
+            bottom: 0;
+            left: 0;
+            right: 0;
+        }
+        #video {
+            width: 320px;
+            height: 240px;
+            background: blue;
+        }
+        #explanation {
+            position: absolute;
+            top: 260px;
+        }
+    </style>
+</head>
+<body>
+<div id="video"></div>
+<div id="explanation">A video containing a blue static image should be visible above.</div>
+</body>
+</html>
\ No newline at end of file
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible.html b/LayoutTests/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible.html
new file mode 100644 (file)
index 0000000..9f38cfd
--- /dev/null
@@ -0,0 +1,159 @@
+<!DOCTYPE html>
+<html lang="en" class="reftest-wait">
+<head>
+    <meta charset="UTF-8">
+    <title>HTML MSE test: video is shown</title>
+    <link rel="author" title="Alicia Boya García" href="mailto:aboya@igalia.com">
+    <link rel="match" href="mediasource-video-is-visible-expected.html">
+    <meta name="fuzzy" content="maxDifference=30;totalPixels=76800">
+    <meta name="assert" content="Video is rendered in the MSE player.">
+    <style>
+        html, body {
+            margin: 0;
+            padding: 0;
+            overflow: hidden;
+            background: white;
+        }
+        body {
+            position: absolute;
+            top: 0;
+            bottom: 0;
+            left: 0;
+            right: 0;
+        }
+        #video {
+            width: 320px;
+            height: 240px;
+        }
+        #explanation {
+            position: absolute;
+            top: 260px;
+        }
+    </style>
+</head>
+<body>
+<video id="video"></video>
+<div id="explanation">A video containing a blue static image should be visible above.</div>
+<script>
+    if ("testRunner" in window)
+        // WebKit reftests use a different mechanism for delaying the test finish.
+        testRunner.waitUntilDone();
+
+    const mediaElement = document.getElementById("video");
+
+    class NetworkError extends Error {
+        constructor(message) {
+            super(message);
+        }
+    }
+
+    function requestJSON(url) {
+        return new Promise((resolve, reject) => {
+            const xhr = new XMLHttpRequest();
+            xhr.open("GET", url, true);
+            xhr.responseType = "text";
+            xhr.onreadystatechange = () => {
+                if (xhr.readyState === 4) {
+                    if (xhr.status >= 200 && xhr.status < 300) {
+                        resolve(JSON.parse(xhr.responseText));
+                    } else {
+                        reject(new NetworkError(`Error ${xhr.status}`));
+                    }
+                }
+            };
+            xhr.send(null);
+        });
+    }
+
+    function requestBinaryMedia(url) {
+        return new Promise((resolve, reject) => {
+            const xhr = new XMLHttpRequest();
+            xhr.open("GET", url, true);
+            xhr.responseType = "arraybuffer";
+            xhr.onreadystatechange = () => {
+                if (xhr.readyState === 4) {
+                    if (xhr.status >= 200 && xhr.status < 300) {
+                        resolve(xhr.response);
+                    } else {
+                        reject(new NetworkError(`Error ${xhr.status}`));
+                    }
+                }
+            };
+            xhr.send(null);
+        })
+    }
+
+    function waitForEvent(element, eventName) {
+        return new Promise(resolve => {
+            element.addEventListener(eventName, function handler() {
+                element.removeEventListener(eventName, handler);
+                resolve();
+            })
+        });
+    }
+
+    function waitForAppend(sourceBuffer, data) {
+        return new Promise(resolve => {
+            const listener = () => {
+                sourceBuffer.removeEventListener("update", listener);
+                resolve();
+            };
+            sourceBuffer.addEventListener("update", listener);
+            sourceBuffer.appendBuffer(data);
+        });
+    }
+
+    let mediaSource;
+    let audioSB;
+    let videoSB;
+
+    async function main(audioType, audioUrl, videoType, videoUrl) {
+        const audioManifest = await requestJSON(audioUrl);
+        const videoManifest = await requestJSON(videoUrl);
+        const audioData = await requestBinaryMedia(audioManifest.url);
+        const videoData = await requestBinaryMedia(videoManifest.url);
+
+        mediaSource = new MediaSource();
+        mediaElement.src = window.URL.createObjectURL(mediaSource);
+        mediaElement.volume = 0.1;
+        await waitForEvent(mediaSource, "sourceopen");
+
+        audioSB = mediaSource.addSourceBuffer(audioType);
+        videoSB = mediaSource.addSourceBuffer(videoType);
+
+        await waitForAppend(audioSB, audioData.slice(0, audioManifest.init.size));
+        await waitForAppend(videoSB, videoData.slice(0, videoManifest.init.size));
+
+        await waitForAppend(audioSB, audioData.slice(audioManifest.init.size));
+        await waitForAppend(videoSB, videoData.slice(videoManifest.init.size));
+
+        mediaSource.endOfStream();
+        mediaElement.play();
+        await waitForEvent(mediaElement, "ended");
+
+        if ("testRunner" in window)
+            testRunner.notifyDone();
+        document.documentElement.classList.remove("reftest-wait");
+    }
+
+    function selectAlternative(alternatives) {
+        for (let alternative of alternatives) {
+            if (MediaSource.isTypeSupported(alternative.mimeType))
+                return [alternative.mimeType, alternative.url];
+        }
+        throw new Error("Could not find suitable alternative");
+    }
+
+    const [audioType, audioUrl] = selectAlternative([
+        {mimeType: 'audio/mp4; codecs="mp4a.40.2"', url: "mp4/test-a-1s.mp4-manifest.json"},
+        {mimeType: 'audio/webm; codecs="opus"', url: "webm/test-a-1s.webm-manifest.json"},
+    ]);
+    const [videoType, videoUrl] = selectAlternative([
+        {mimeType: 'video/mp4; codecs="avc1.42C015"', url: "mp4/test-v-1s-blue.mp4-manifest.json"},
+        {mimeType: 'video/webm; codecs="vp9"', url: "webm/test-v-1s-blue.webm-manifest.json"},
+    ]);
+
+    main(audioType, audioUrl, videoType, videoUrl);
+</script>
+</body>
+</html>
\ No newline at end of file
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4 b/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4
new file mode 100644 (file)
index 0000000..3479719
Binary files /dev/null and b/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4 differ
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4-manifest.json b/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-a-1s.mp4-manifest.json
new file mode 100644 (file)
index 0000000..43785c0
--- /dev/null
@@ -0,0 +1,24 @@
+{
+    "url": "mp4/test-a-1s.mp4",
+    "init": {
+        "offset": 0,
+        "size": 942
+    },
+    "media": [
+        {
+            "offset": 942,
+            "size": 4067,
+            "timestamp": 0.0
+        },
+        {
+            "offset": 5009,
+            "size": 3624,
+            "timestamp": 0.5108390022675737
+        },
+        {
+            "offset": 8633,
+            "size": 570,
+            "timestamp": 0.9984580498866213
+        }
+    ]
+}
\ No newline at end of file
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4 b/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4
new file mode 100644 (file)
index 0000000..569692b
Binary files /dev/null and b/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4 differ
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4-manifest.json b/LayoutTests/imported/w3c/web-platform-tests/media-source/mp4/test-v-1s-blue.mp4-manifest.json
new file mode 100644 (file)
index 0000000..7c3635f
--- /dev/null
@@ -0,0 +1,19 @@
+{
+    "url": "mp4/test-v-1s-blue.mp4",
+    "init": {
+        "offset": 0,
+        "size": 1053
+    },
+    "media": [
+        {
+            "offset": 1053,
+            "size": 1392,
+            "timestamp": 0.0
+        },
+        {
+            "offset": 2445,
+            "size": 351,
+            "timestamp": 0.5
+        }
+    ]
+}
\ No newline at end of file
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm b/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm
new file mode 100644 (file)
index 0000000..8e05081
Binary files /dev/null and b/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm differ
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm-manifest.json b/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-a-1s.webm-manifest.json
new file mode 100644 (file)
index 0000000..801246e
--- /dev/null
@@ -0,0 +1,14 @@
+{
+    "url": "webm/test-a-1s.webm",
+    "init": {
+        "offset": 0,
+        "size": 600
+    },
+    "media": [
+        {
+            "offset": 600,
+            "size": 7880,
+            "timestamp": 0.0
+        }
+    ]
+}
\ No newline at end of file
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm b/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm
new file mode 100644 (file)
index 0000000..4efb5f8
Binary files /dev/null and b/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm differ
diff --git a/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm-manifest.json b/LayoutTests/imported/w3c/web-platform-tests/media-source/webm/test-v-1s-blue.webm-manifest.json
new file mode 100644 (file)
index 0000000..472e4b2
--- /dev/null
@@ -0,0 +1,14 @@
+{
+    "url": "webm/test-v-1s-blue.webm",
+    "init": {
+        "offset": 0,
+        "size": 580
+    },
+    "media": [
+        {
+            "offset": 580,
+            "size": 766,
+            "timestamp": 0.0
+        }
+    ]
+}
\ No newline at end of file
index d7576f9..341b2aa 100644 (file)
@@ -2022,6 +2022,9 @@ webgpu/whlsl/separate-shader-modules/separate-shader-modules-19.html [ Pass Imag
 webgpu/whlsl/separate-shader-modules/separate-shader-modules-24.html [ Pass ImageOnlyFailure ]
 webgpu/whlsl/separate-shader-modules/separate-shader-modules-26.html [ Pass ImageOnlyFailure ]
 
+# The test runner is failing in an unrelated, unexplained, invisible to the human eye, 2 pixels difference.
+webkit.org/b/201401 imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible.html [ Skip ]
+
 # rdar://55484256 (REGRESSION: (Catalina) fast/images/async-image-multiple-clients-repaint.html is a flakey failure)
 [ Catalina+ ] fast/images/async-image-multiple-clients-repaint.html [ Pass Failure ]
 
diff --git a/LayoutTests/platform/mac/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html b/LayoutTests/platform/mac/imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible-expected.html
new file mode 100644 (file)
index 0000000..50d8e36
--- /dev/null
@@ -0,0 +1,36 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <title>HTML MSE test: video is shown</title>
+    <link rel="author" title="Alicia Boya García" href="mailto:aboya@igalia.com">
+    <style>
+        html, body {
+            margin: 0;
+            padding: 0;
+            overflow: hidden;
+            background: white;
+        }
+        body {
+            position: absolute;
+            top: 0;
+            bottom: 0;
+            left: 0;
+            right: 0;
+        }
+        #video {
+            width: 320px;
+            height: 240px;
+            background: #1a23ff;
+        }
+        #explanation {
+            position: absolute;
+            top: 260px;
+        }
+    </style>
+</head>
+<body>
+<div id="video"></div>
+<div id="explanation">A video containing a blue static image should be visible above.</div>
+</body>
+</html>
\ No newline at end of file
index 7477731..c6b769a 100644 (file)
@@ -1,3 +1,27 @@
+2019-10-09  Alicia Boya García  <aboya@igalia.com>
+
+        [MSE][GStreamer] Fix video sometimes not appearing
+        https://bugs.webkit.org/show_bug.cgi?id=201401
+
+        Reviewed by Xabier Rodriguez-Calvar.
+
+        The code in MediaPlayerPrivateGStreamer::changePipelineState() was
+        supposed to call `ensureGLVideoSinkContext()` on upwards transitions
+        to PAUSED but the code did not take into account non-step-by-step
+        state transitions, which happens frequently with playbin3 in the MSE
+        case.
+
+        Before the patch, when playbin3 transitioned from READY to PLAYING
+        without stopping for preroll this call would not be made and the
+        texture IDs received at the sink would not correspond to the
+        compositor GL context, leading to artifacts (often the player controls
+        or a blank screen).
+
+        Test: imported/w3c/web-platform-tests/media-source/mediasource-video-is-visible.html
+
+        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
+        (WebCore::MediaPlayerPrivateGStreamer::changePipelineState):
+
 2019-10-09  Eric Carlson  <eric.carlson@apple.com>
 
         [ Mac WK2 ] Layout Test fast/mediastream/MediaStreamTrack-getSettings.html is a flaky failure
index 79eae70..1a47c15 100644 (file)
@@ -421,7 +421,7 @@ bool MediaPlayerPrivateGStreamer::changePipelineState(GstState newState)
         gst_element_state_get_name(currentState), gst_element_state_get_name(pending));
 
 #if USE(GSTREAMER_GL)
-    if (currentState == GST_STATE_READY && newState == GST_STATE_PAUSED)
+    if (currentState <= GST_STATE_READY && newState >= GST_STATE_PAUSED)
         ensureGLVideoSinkContext();
 #endif