Canvas is tainted when painting a video with MediaStreamTrack
authorcommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 5 Apr 2017 04:48:03 +0000 (04:48 +0000)
committercommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 5 Apr 2017 04:48:03 +0000 (04:48 +0000)
https://bugs.webkit.org/show_bug.cgi?id=170486

Patch by Youenn Fablet <youenn@apple.com> on 2017-04-04
Reviewed by Eric Carlson.

Source/WebCore:

Test: http/tests/media/media-stream/getusermedia-with-canvas.html

Adding the notion of isolated source so that we can later on implement WebRTC isolated tracks.
For now, canvas will not be tainted if painted from a MediaStreamTrack.

* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::didPassCORSAccessCheck):
* platform/mediastream/MediaStreamTrackPrivate.h:
(WebCore::MediaStreamTrackPrivate::isIsolated):
* platform/mediastream/RealtimeMediaSource.h:

LayoutTests:

* http/tests/media/media-stream/getusermedia-with-canvas-expected.txt: Added.
* http/tests/media/media-stream/getusermedia-with-canvas.html: Added.
* http/tests/media/media-stream/resources/getUserMedia-helper.js: Added.
(reject):
(getUserMedia):
(defaultRejectOrCatch):
(setupVideoElementWithStream):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@214928 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/ChangeLog
LayoutTests/http/tests/media/media-stream/getusermedia-with-canvas-expected.txt [new file with mode: 0644]
LayoutTests/http/tests/media/media-stream/getusermedia-with-canvas.html [new file with mode: 0644]
LayoutTests/http/tests/media/media-stream/resources/getUserMedia-helper.js [new file with mode: 0644]
Source/WebCore/ChangeLog
Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h
Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm
Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h
Source/WebCore/platform/mediastream/RealtimeMediaSource.h

index f0fbf7a..02fcb15 100644 (file)
@@ -1,5 +1,20 @@
 2017-04-04  Youenn Fablet  <youenn@apple.com>
 
+        Canvas is tainted when painting a video with MediaStreamTrack
+        https://bugs.webkit.org/show_bug.cgi?id=170486
+
+        Reviewed by Eric Carlson.
+
+        * http/tests/media/media-stream/getusermedia-with-canvas-expected.txt: Added.
+        * http/tests/media/media-stream/getusermedia-with-canvas.html: Added.
+        * http/tests/media/media-stream/resources/getUserMedia-helper.js: Added.
+        (reject):
+        (getUserMedia):
+        (defaultRejectOrCatch):
+        (setupVideoElementWithStream):
+
+2017-04-04  Youenn Fablet  <youenn@apple.com>
+
         [Mac] Add back web audio support for getUserMedia MediaStreamTrack
         https://bugs.webkit.org/show_bug.cgi?id=170482
 
diff --git a/LayoutTests/http/tests/media/media-stream/getusermedia-with-canvas-expected.txt b/LayoutTests/http/tests/media/media-stream/getusermedia-with-canvas-expected.txt
new file mode 100644 (file)
index 0000000..012ecbb
--- /dev/null
@@ -0,0 +1,33 @@
+Tests that re-enabling a video MediaStreamTrack when all tracks were previously disabled causes captured media to display.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS mediaDevices.getUserMedia generated a stream successfully.
+video.srcObject = mediaStream
+
+ === beginning round of pixel tests ===
+PASS pixel was black.
+
+ === all video tracks disabled ===
+PASS pixel was black.
+
+ === video track reenabled, should NOT render current frame ===
+PASS pixel was black.
+
+ ===== play video =====
+video.play()
+
+ === beginning round of pixel tests ===
+PASS pixel was white.
+
+ === all video tracks disabled ===
+PASS pixel was black.
+
+ === video track reenabled, should render current frame ===
+PASS pixel was white.
+
+PASS successfullyParsed is true
+
+TEST COMPLETE
diff --git a/LayoutTests/http/tests/media/media-stream/getusermedia-with-canvas.html b/LayoutTests/http/tests/media/media-stream/getusermedia-with-canvas.html
new file mode 100644 (file)
index 0000000..69ae60d
--- /dev/null
@@ -0,0 +1,137 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <script src="/resources/js-test-pre.js"></script>
+    <script src="./resources/getUserMedia-helper.js"></script>
+</head>
+<body onload="start()">
+<p id="description"></p>
+<div id="console"></div>
+<video controls width="680" height="360"></video>
+<canvas width="680" height="360"></canvas>
+<script>
+    let canvas;
+    let context;
+    let mediaStream;
+    let video;
+    let havePlayed = false;
+
+    let buffer;
+
+    function isPixelBlack(pixel)
+    {
+        return pixel[0] === 0 && pixel[1] === 0 && pixel[2] === 0 && pixel[3] === 255;
+    }
+
+    function isPixelTransparent(pixel)
+    {
+        return pixel[0] === 0 && pixel[1] === 0 && pixel[2] === 0 && pixel[3] === 0;
+    }
+
+    function isPixelWhite(pixel)
+    {
+        return pixel[0] === 255 && pixel[1] === 255 && pixel[2] === 255 && pixel[3] === 255;
+    }
+
+    function canvasShouldBeBlack()
+    {
+        return !(mediaStream.getVideoTracks()[0].enabled && havePlayed);
+    }
+
+    function attempt(numberOfTries, call, callback)
+    {
+        if (numberOfTries <= 0) {
+            testFailed('Pixel check did not succeed after multiple tries.');
+            return;
+        }
+
+        let attemptSucceeded = call();
+        if (attemptSucceeded) {
+            testPassed(canvasShouldBeBlack() ? 'pixel was black.' : 'pixel was white.');
+            callback();
+
+            return;
+        }
+
+        setTimeout(() => { attempt(--numberOfTries, call, callback); }, 50);
+    }
+
+    function repeatWithVideoPlayingAndFinishTest()
+    {
+        if (video.paused) {
+            debug('<br> ===== play video =====');
+            evalAndLog('video.play()');
+            havePlayed = true;
+            beginTestRound();
+        } else {
+            debug('');
+            video.pause();
+            finishJSTest();
+        }
+    }
+
+    function reenableTrack()
+    {
+        mediaStream.getVideoTracks()[0].enabled = true;
+        debug(`<br> === video track reenabled, should${havePlayed ? "" : " NOT"} render current frame ===`);
+
+        // The video is not guaranteed to render non-black frames before the canvas is drawn to and the pixels are checked.
+        // A timeout is used to ensure that the pixel check is done after the video renders non-black frames.
+        attempt(10, checkPixels, repeatWithVideoPlayingAndFinishTest);
+    }
+
+    function checkPixels()
+    {
+        context.clearRect(0, 0, canvas.width, canvas.height);
+        buffer = context.getImageData(30, 242, 1, 1).data;
+        if (!isPixelTransparent(buffer))
+            testFailed('pixel was not transparent after clearing canvas.');
+
+        context.drawImage(video, 0, 0, canvas.width, canvas.height);
+        buffer = context.getImageData(30, 242, 1, 1).data;
+
+        if (!canvasShouldBeBlack())
+            return isPixelWhite(buffer);
+        else
+            return isPixelBlack(buffer);
+    }
+
+    function disableAllTracks()
+    {
+        mediaStream.getVideoTracks()[0].enabled = false;
+        debug('<br> === all video tracks disabled ===');
+
+        // The video is not guaranteed to render black frames before the canvas is drawn to and the pixels are checked.
+        // A timeout is used to ensure that the pixel check is done after the video renders black frames.
+        attempt(10, checkPixels, reenableTrack);
+    }
+
+    function beginTestRound()
+    {
+        debug('<br> === beginning round of pixel tests ===');
+        attempt(10, checkPixels, disableAllTracks);
+    }
+
+    function canplay()
+    {
+        canvas = document.querySelector('canvas');
+        context = canvas.getContext('2d');
+
+        beginTestRound();
+    }
+
+    function start()
+    {
+        description("Tests that re-enabling a video MediaStreamTrack when all tracks were previously disabled causes captured media to display.");
+
+        video = document.querySelector('video');
+        video.addEventListener('canplay', canplay);
+
+        getUserMedia("allow", {video:true}, setupVideoElementWithStream);
+    }
+
+    window.jsTestIsAsync = true;
+</script>
+<script src="/resources/js-test-post.js"></script>
+</body>
+</html>
diff --git a/LayoutTests/http/tests/media/media-stream/resources/getUserMedia-helper.js b/LayoutTests/http/tests/media/media-stream/resources/getUserMedia-helper.js
new file mode 100644 (file)
index 0000000..ce28cb6
--- /dev/null
@@ -0,0 +1,33 @@
+function getUserMedia(permission, constraints, successCallback, errorCallback) {
+    if (window.testRunner)
+        testRunner.setUserMediaPermission(permission == "allow");
+    else {
+        debug("This test can not be run without the testRunner");
+        finishJSTest();
+        return;
+    }
+
+    navigator.mediaDevices
+                .getUserMedia(constraints)
+                .then(successCallback, reject)
+                .catch(defaultRejectOrCatch);
+
+    function reject(e) {
+        if (errorCallback)
+            errorCallback(e);
+        else
+            defaultRejectOrCatch(e);
+    }
+}
+
+function defaultRejectOrCatch(e) {
+    testFailed('getUserMedia failed:' + e);
+    finishJSTest();
+}
+
+function setupVideoElementWithStream(stream)
+{
+    mediaStream = stream;
+    testPassed('mediaDevices.getUserMedia generated a stream successfully.');
+    evalAndLog('video.srcObject = mediaStream');
+}
index 204ae91..a2e0074 100644 (file)
@@ -1,3 +1,22 @@
+2017-04-04  Youenn Fablet  <youenn@apple.com>
+
+        Canvas is tainted when painting a video with MediaStreamTrack
+        https://bugs.webkit.org/show_bug.cgi?id=170486
+
+        Reviewed by Eric Carlson.
+
+        Test: http/tests/media/media-stream/getusermedia-with-canvas.html
+
+        Adding the notion of isolated source so that we can later on implement WebRTC isolated tracks.
+        For now, canvas will not be tainted if painted from a MediaStreamTrack.
+
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
+        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
+        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::didPassCORSAccessCheck):
+        * platform/mediastream/MediaStreamTrackPrivate.h:
+        (WebCore::MediaStreamTrackPrivate::isIsolated):
+        * platform/mediastream/RealtimeMediaSource.h:
+
 2017-04-04  Commit Queue  <commit-queue@webkit.org>
 
         Unreviewed, rolling out r214894, r214895, r214907, r214912,
index c973f10..429f1e4 100644 (file)
@@ -91,6 +91,8 @@ private:
 
     // FIXME(146853): Implement necessary conformations to standard in HTMLMediaElement for MediaStream
 
+    bool didPassCORSAccessCheck() const final;
+
     void load(const String&) override;
 #if ENABLE(MEDIA_SOURCE)
     void load(const String&, MediaSourcePrivateClient*) override;
index 39c4e7a..dec09a2 100644 (file)
@@ -553,6 +553,14 @@ void MediaPlayerPrivateMediaStreamAVFObjC::load(MediaStreamPrivate& stream)
     });
 }
 
+bool MediaPlayerPrivateMediaStreamAVFObjC::didPassCORSAccessCheck() const
+{
+    // We are only doing a check on the active video track since the sole consumer of this API is canvas.
+    // FIXME: We should change the name of didPassCORSAccessCheck if it is expected to stay like this.
+    const auto* track = m_mediaStreamPrivate->activeVideoTrack();
+    return !track || !track->isIsolated();
+}
+
 void MediaPlayerPrivateMediaStreamAVFObjC::cancelLoad()
 {
     LOG(Media, "MediaPlayerPrivateMediaStreamAVFObjC::cancelLoad(%p)", this);
index 2eec54e..6b87f88 100644 (file)
@@ -66,6 +66,8 @@ public:
     void stopProducingData() { m_source->stopProducingData(); }
     bool isProducingData() { return m_source->isProducingData(); }
 
+    bool isIsolated() const { return m_source->isIsolated(); }
+
     bool muted() const;
     void setMuted(bool muted) { m_source->setMuted(muted); }
 
index 1cddf0c..777824e 100644 (file)
@@ -120,6 +120,8 @@ public:
 
     virtual void settingsDidChange();
 
+    virtual bool isIsolated() const { return false; }
+    
     void videoSampleAvailable(MediaSample&);
     void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t);