[GStreamer] Stopping playback of html5 media when receiving a higher priority audio...
authorcommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 14 Mar 2013 14:39:17 +0000 (14:39 +0000)
committercommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 14 Mar 2013 14:39:17 +0000 (14:39 +0000)
https://bugs.webkit.org/show_bug.cgi?id=91611

Source/WebCore:

React to REQUEST_STATE GStreamer message to stop the pipeline when
a higher priority stream is played. When this happens, states are
updated accordingly.

A method was added in the MediaPlayer class and internals to allow
the the test runner to simulate an audio interruption.

Patch by Xabier Rodriguez Calvar <calvaris@igalia.com> on 2013-03-14
Reviewed by Philippe Normand.

Test: media/media-higher-prio-audio-stream.html

* platform/graphics/MediaPlayer.h:
* platform/graphics/MediaPlayer.cpp:
(WebCore):
(MediaPlayer):
(WebCore::MediaPlayer::simulateAudioInterruption): New method
delegating an audio interruption to the private backend to
simulate the use-case where an external application needs
exclusive access to the audio device.
* platform/graphics/MediaPlayerPrivate.h:
(MediaPlayerPrivateInterface):
(WebCore::MediaPlayerPrivateInterface::simulateAudioInterruption):
Added default empty method in the common private header.
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
(WebCore):
(WebCore::MediaPlayerPrivateGStreamer::createAudioSink):
(WebCore::setAudioStreamPropertiesCallback): Hooked to child-added
signal on the audio sink, delegates on setAudioStreamProperties.
(WebCore::MediaPlayerPrivateGStreamer::setAudioStreamProperties):
Sets the audio stream properties.
(WebCore::MediaPlayerPrivateGStreamer::MediaPlayerPrivateGStreamer):
Initializes the new attribute.
(WebCore::MediaPlayerPrivateGStreamer::~MediaPlayerPrivateGStreamer):
Disconnects autoaudiosink signal.
(WebCore::MediaPlayerPrivateGStreamer::changePipelineState):
Changed logging.
(WebCore::MediaPlayerPrivateGStreamer::handleMessage): Reacting to
the REQUEST_STATE message.
(WebCore::MediaPlayerPrivateGStreamer::simulateAudioInterruption):
Added. Injects the REQUEST_STATE message to the pipeline.
(WebCore::MediaPlayerPrivateGStreamer::updateStates): Updating the
playback state if REQUEST_STATE.
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
(MediaPlayerPrivateGStreamer): Added new method and attribute.
* testing/Internals.h:
* testing/Internals.idl:
* testing/Internals.cpp:
(WebCore):
(WebCore::Internals::simulateAudioInterruption): Added to call the
method to stop the element because of a higher prio stream at the
tests.

LayoutTests:

Created test, expected result and updated other ports
expectations.

Patch by Xabier Rodriguez Calvar <calvaris@igalia.com> on 2013-03-14
Reviewed by Philippe Normand.

* media/media-higher-prio-audio-stream-expected.txt: Added.
* media/media-higher-prio-audio-stream.html: Added.
* platform/chromium/TestExpectations: Skipped the new test.
* platform/mac/TestExpectations: Skipped the new test.
* platform/qt/TestExpectations: Skipped the new test for Mac and
Win.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@145811 268f45cc-cd09-0410-ab3c-d52691b4dbfc

15 files changed:
LayoutTests/ChangeLog
LayoutTests/media/media-higher-prio-audio-stream-expected.txt [new file with mode: 0644]
LayoutTests/media/media-higher-prio-audio-stream.html [new file with mode: 0644]
LayoutTests/platform/chromium/TestExpectations
LayoutTests/platform/mac/TestExpectations
LayoutTests/platform/qt/TestExpectations
Source/WebCore/ChangeLog
Source/WebCore/platform/graphics/MediaPlayer.cpp
Source/WebCore/platform/graphics/MediaPlayer.h
Source/WebCore/platform/graphics/MediaPlayerPrivate.h
Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp
Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h
Source/WebCore/testing/Internals.cpp
Source/WebCore/testing/Internals.h
Source/WebCore/testing/Internals.idl

index 9086ce9..3e80458 100644 (file)
@@ -1,3 +1,20 @@
+2013-03-14  Xabier Rodriguez Calvar  <calvaris@igalia.com>
+
+        [GStreamer] Stopping playback of html5 media when receiving a higher priority audio event needs implementation
+        https://bugs.webkit.org/show_bug.cgi?id=91611
+
+        Created test, expected result and updated other ports
+        expectations.
+
+        Reviewed by Philippe Normand.
+
+        * media/media-higher-prio-audio-stream-expected.txt: Added.
+        * media/media-higher-prio-audio-stream.html: Added.
+        * platform/chromium/TestExpectations: Skipped the new test.
+        * platform/mac/TestExpectations: Skipped the new test.
+        * platform/qt/TestExpectations: Skipped the new test for Mac and
+        Win.
+
 2013-03-14  Zoltan Arvai  <zarvai@inf.u-szeged.hu>
 
         [Qt] Unreviewed gardening.
diff --git a/LayoutTests/media/media-higher-prio-audio-stream-expected.txt b/LayoutTests/media/media-higher-prio-audio-stream-expected.txt
new file mode 100644 (file)
index 0000000..baeb952
--- /dev/null
@@ -0,0 +1,12 @@
+Test paused by:
+
+Getting a higher priority audio stream.
+This test does not work in the launchers, only with the test runners.
+EVENT(playing)
+EXPECTED (mediaElement.paused == 'false') OK
+EVENT(pause)
+EXPECTED (mediaElement.paused == 'true') OK
+EXPECTED (mediaElement.ended == 'false') OK
+
+END OF TEST
+
diff --git a/LayoutTests/media/media-higher-prio-audio-stream.html b/LayoutTests/media/media-higher-prio-audio-stream.html
new file mode 100644 (file)
index 0000000..857b173
--- /dev/null
@@ -0,0 +1,46 @@
+<!DOCTYPE html>
+<html>
+    <head>
+        <script src=media-file.js></script>
+        <script src=video-test.js></script>
+
+        <script>
+            var mediaElement;
+
+            function start()
+            {
+                mediaElement = new Audio();
+
+                waitForEvent('error');
+                waitForEvent('playing', playing);
+
+                mediaElement.src = findMediaFile('audio', 'content/test');
+                mediaElement.play();
+            }
+
+            function playing()
+            {
+                testExpected("mediaElement.paused", false);
+                waitForEvent('pause', paused);
+                internals.simulateAudioInterruption(mediaElement);
+            }
+
+            function paused()
+            {
+                testExpected("mediaElement.paused", true);
+                testExpected("mediaElement.ended", false);
+                consoleWrite("");
+                endTest();
+            }
+        </script>
+    </head>
+    <body onload="start()">
+        <p><b>Test paused by:</b>
+        <ol>
+            <li>Getting a higher priority audio stream.</li>
+            <li>This test does not work in the launchers, only with
+            the test runners.</li>
+        </ol>
+        </p>
+    </body>
+</html>
index 0d5f908..310b708 100644 (file)
@@ -4453,3 +4453,6 @@ webkit.org/b/112193 fast/dom/Window/window-postmessage-clone.html [ Failure Pass
 webkit.org/b/112219 fast/layers/no-clipping-overflow-hidden-added-after-transform.html [ ImageOnlyFailure Pass ]
 webkit.org/b/112219 fast/layers/no-clipping-overflow-hidden-hardware-acceleration.html [ ImageOnlyFailure Pass ]
 webkit.org/b/112219 fast/layers/no-clipping-overflow-hidden-added-after-transition.html [ ImageOnlyFailure Pass ]
+
+# Feature not implemented
+webkit.org/b/91611 media/media-higher-prio-audio-stream.html [ Skip ]
index 3c47306..e34d7b6 100644 (file)
@@ -1480,3 +1480,6 @@ webkit.org/b/112176 fast/css/sticky/inline-sticky-abspos-child.html [ ImageOnlyF
 # Once the bug 112176 is fixed, potentially restore the following test expectations.
 # webkit.org/b/105998 [ Lion Release ] fast/css/sticky/inline-sticky-abspos-child.html [ ImageOnlyFailure ]
 # webkit.org/b/105998 [ MountainLion Debug ] fast/css/sticky/inline-sticky-abspos-child.html [ ImageOnlyFailure Pass ]
+
+# Feature not implemented
+webkit.org/b/91611 media/media-higher-prio-audio-stream.html [ Skip ]
index e1a80f7..d3c10db 100644 (file)
@@ -2692,3 +2692,6 @@ webkit.org/b/109179 compositing/transitions/transform-on-large-layer.html [ Imag
 
 # [Qt] Two tests hit assertion fail after r140999.
 webkit.org/b/108257 compositing/overflow/composited-scrolling-creates-a-stacking-container.html [ Skip ]
+
+# Feature not implemented
+webkit.org/b/91611 [ Mac Win ] media/media-higher-prio-audio-stream.html [ Skip ]
index 8d758f6..092345e 100644 (file)
@@ -1,3 +1,60 @@
+2013-03-14  Xabier Rodriguez Calvar  <calvaris@igalia.com>
+
+        [GStreamer] Stopping playback of html5 media when receiving a higher priority audio event needs implementation
+        https://bugs.webkit.org/show_bug.cgi?id=91611
+
+        React to REQUEST_STATE GStreamer message to stop the pipeline when
+        a higher priority stream is played. When this happens, states are
+        updated accordingly.
+
+        A method was added in the MediaPlayer class and internals to allow
+        the the test runner to simulate an audio interruption.
+
+        Reviewed by Philippe Normand.
+
+        Test: media/media-higher-prio-audio-stream.html
+
+        * platform/graphics/MediaPlayer.h:
+        * platform/graphics/MediaPlayer.cpp:
+        (WebCore):
+        (MediaPlayer):
+        (WebCore::MediaPlayer::simulateAudioInterruption): New method
+        delegating an audio interruption to the private backend to
+        simulate the use-case where an external application needs
+        exclusive access to the audio device.
+        * platform/graphics/MediaPlayerPrivate.h:
+        (MediaPlayerPrivateInterface):
+        (WebCore::MediaPlayerPrivateInterface::simulateAudioInterruption):
+        Added default empty method in the common private header.
+        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
+        (WebCore):
+        (WebCore::MediaPlayerPrivateGStreamer::createAudioSink):
+        (WebCore::setAudioStreamPropertiesCallback): Hooked to child-added
+        signal on the audio sink, delegates on setAudioStreamProperties.
+        (WebCore::MediaPlayerPrivateGStreamer::setAudioStreamProperties):
+        Sets the audio stream properties.
+        (WebCore::MediaPlayerPrivateGStreamer::MediaPlayerPrivateGStreamer):
+        Initializes the new attribute.
+        (WebCore::MediaPlayerPrivateGStreamer::~MediaPlayerPrivateGStreamer):
+        Disconnects autoaudiosink signal.
+        (WebCore::MediaPlayerPrivateGStreamer::changePipelineState):
+        Changed logging.
+        (WebCore::MediaPlayerPrivateGStreamer::handleMessage): Reacting to
+        the REQUEST_STATE message.
+        (WebCore::MediaPlayerPrivateGStreamer::simulateAudioInterruption):
+        Added. Injects the REQUEST_STATE message to the pipeline.
+        (WebCore::MediaPlayerPrivateGStreamer::updateStates): Updating the
+        playback state if REQUEST_STATE.
+        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
+        (MediaPlayerPrivateGStreamer): Added new method and attribute.
+        * testing/Internals.h:
+        * testing/Internals.idl:
+        * testing/Internals.cpp:
+        (WebCore):
+        (WebCore::Internals::simulateAudioInterruption): Added to call the
+        method to stop the element because of a higher prio stream at the
+        tests.
+
 2013-03-14  Allan Sandfeld Jensen  <allan.jensen@digia.com>
 
         [Qt] Add support for tiled shadow blur
index 7c94b2b..89c5ba5 100644 (file)
@@ -1144,6 +1144,16 @@ void MediaPlayer::resetMediaEngines()
     installedMediaEngines(ResetEngines);
 }
 
+#if USE(GSTREAMER)
+void MediaPlayer::simulateAudioInterruption()
+{
+    if (!m_private)
+        return;
+
+    m_private->simulateAudioInterruption();
+}
+#endif
+
 }
 
 #endif
index 994c8cd..86ea402 100644 (file)
@@ -462,6 +462,10 @@ public:
     PassRefPtr<PlatformTextTrackMenuInterface> textTrackMenu();
 #endif
 
+#if USE(GSTREAMER)
+    virtual void simulateAudioInterruption();
+#endif
+
 private:
     MediaPlayer(MediaPlayerClient*);
     void loadWithNextMediaEngine(MediaPlayerFactory*);
index 857b495..b79709d 100644 (file)
@@ -188,6 +188,9 @@ public:
     virtual PassRefPtr<PlatformTextTrackMenuInterface> textTrackMenu() { return 0; }
 #endif
 
+#if USE(GSTREAMER)
+    virtual void simulateAudioInterruption() { }
+#endif
 };
 
 }
index 2adfe74..5bd54fc 100644 (file)
@@ -113,6 +113,16 @@ static gboolean mediaPlayerPrivateAudioChangeTimeoutCallback(MediaPlayerPrivateG
     return FALSE;
 }
 
+#ifdef GST_API_VERSION_1
+static void setAudioStreamPropertiesCallback(GstChildProxy*, GObject* object, gchar*,
+    MediaPlayerPrivateGStreamer* player)
+#else
+static void setAudioStreamPropertiesCallback(GstChildProxy*, GObject* object, MediaPlayerPrivateGStreamer* player)
+#endif
+{
+    player->setAudioStreamProperties(object);
+}
+
 static gboolean mediaPlayerPrivateVideoChangeTimeoutCallback(MediaPlayerPrivateGStreamer* player)
 {
     // This is the callback of the timeout source created in ::videoChanged.
@@ -120,6 +130,19 @@ static gboolean mediaPlayerPrivateVideoChangeTimeoutCallback(MediaPlayerPrivateG
     return FALSE;
 }
 
+void MediaPlayerPrivateGStreamer::setAudioStreamProperties(GObject* object)
+{
+    if (g_strcmp0(G_OBJECT_TYPE_NAME(object), "GstPulseSink"))
+        return;
+
+    const char* role = m_player->mediaPlayerClient() && m_player->mediaPlayerClient()->mediaPlayerIsVideo()
+        ? "video" : "music";
+    GstStructure* structure = gst_structure_new("stream-properties", "media.role", G_TYPE_STRING, role, NULL);
+    g_object_set(object, "stream-properties", structure, NULL);
+    gst_structure_free(structure);
+    LOG_MEDIA_MESSAGE("Set media.role as %s at %s", role, gst_element_get_name(GST_ELEMENT(object)));
+}
+
 PassOwnPtr<MediaPlayerPrivateInterface> MediaPlayerPrivateGStreamer::create(MediaPlayer* player)
 {
     return adoptPtr(new MediaPlayerPrivateGStreamer(player));
@@ -187,6 +210,7 @@ MediaPlayerPrivateGStreamer::MediaPlayerPrivateGStreamer(MediaPlayer* player)
     , m_totalBytes(-1)
     , m_originalPreloadWasAutoAndWasOverridden(false)
     , m_preservesPitch(false)
+    , m_requestedState(GST_STATE_VOID_PENDING)
 {
 }
 
@@ -200,6 +224,10 @@ MediaPlayerPrivateGStreamer::~MediaPlayerPrivateGStreamer()
         m_mediaLocations = 0;
     }
 
+    if (m_autoAudioSink)
+        g_signal_handlers_disconnect_by_func(G_OBJECT(m_autoAudioSink.get()),
+            reinterpret_cast<gpointer>(setAudioStreamPropertiesCallback), this);
+
     if (m_playBin) {
         GRefPtr<GstBus> bus = webkitGstPipelineGetBus(GST_PIPELINE(m_playBin.get()));
         ASSERT(bus);
@@ -321,9 +349,14 @@ bool MediaPlayerPrivateGStreamer::changePipelineState(GstState newState)
     GstState pending;
 
     gst_element_get_state(m_playBin.get(), &currentState, &pending, 0);
-    LOG_MEDIA_MESSAGE("Current state: %s, pending: %s", gst_element_state_get_name(currentState), gst_element_state_get_name(pending));
-    if (currentState == newState || pending == newState)
+    if (currentState == newState || pending == newState) {
+        LOG_MEDIA_MESSAGE("Rejected state change to %s from %s with %s pending", gst_element_state_get_name(newState),
+            gst_element_state_get_name(currentState), gst_element_state_get_name(pending));
         return true;
+    }
+
+    LOG_MEDIA_MESSAGE("Changing state change to %s from %s with %s pending", gst_element_state_get_name(newState),
+        gst_element_state_get_name(currentState), gst_element_state_get_name(pending));
 
     GstStateChangeReturn setStateResult = gst_element_set_state(m_playBin.get(), newState);
     GstState pausedOrPlaying = newState == GST_STATE_PLAYING ? GST_STATE_PAUSED : GST_STATE_PLAYING;
@@ -627,6 +660,7 @@ gboolean MediaPlayerPrivateGStreamer::handleMessage(GstMessage* message)
     bool issueError = true;
     bool attemptNextLocation = false;
     const GstStructure* structure = gst_message_get_structure(message);
+    GstState requestedState, currentState;
 
     if (structure) {
         const gchar* messageTypeName = gst_structure_get_name(structure);
@@ -692,12 +726,12 @@ gboolean MediaPlayerPrivateGStreamer::handleMessage(GstMessage* message)
             updateStates();
 
             // Construct a filename for the graphviz dot file output.
-            GstState oldState, newState;
-            gst_message_parse_state_changed(message, &oldState, &newState, 0);
+            GstState newState;
+            gst_message_parse_state_changed(message, &currentState, &newState, 0);
 
             CString dotFileName = String::format("webkit-video.%s_%s",
-                                                 gst_element_state_get_name(oldState),
-                                                 gst_element_state_get_name(newState)).utf8();
+                gst_element_state_get_name(currentState),
+                gst_element_state_get_name(newState)).utf8();
 
             GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS(GST_BIN(m_playBin.get()), GST_DEBUG_GRAPH_SHOW_ALL, dotFileName.data());
         }
@@ -713,6 +747,16 @@ gboolean MediaPlayerPrivateGStreamer::handleMessage(GstMessage* message)
         LOG_MEDIA_MESSAGE("Duration changed");
         durationChanged();
         break;
+    case GST_MESSAGE_REQUEST_STATE:
+        gst_message_parse_request_state(message, &requestedState);
+        gst_element_get_state(m_playBin.get(), &currentState, NULL, 250);
+        if (requestedState < currentState) {
+            LOG_MEDIA_MESSAGE("Element %s requested state change to %s", gst_element_get_name(GST_MESSAGE_SRC(message)),
+                gst_element_state_get_name(requestedState));
+            m_requestedState = requestedState;
+            changePipelineState(requestedState);
+        }
+        break;
     default:
         LOG_MEDIA_MESSAGE("Unhandled GStreamer message type: %s",
                     GST_MESSAGE_TYPE_NAME(message));
@@ -962,6 +1006,7 @@ void MediaPlayerPrivateGStreamer::updateStates()
         &state, &pending, 250 * GST_NSECOND);
 
     bool shouldUpdateAfterSeek = false;
+    bool shouldUpdatePlaybackState = false;
     switch (ret) {
     case GST_STATE_CHANGE_SUCCESS:
         LOG_MEDIA_MESSAGE("State: %s, pending: %s",
@@ -1048,6 +1093,11 @@ void MediaPlayerPrivateGStreamer::updateStates()
             m_seeking = false;
         }
 
+        if (m_requestedState == GST_STATE_PAUSED && state == GST_STATE_PAUSED) {
+            shouldUpdatePlaybackState = true;
+            LOG_MEDIA_MESSAGE("Requested state change to %s was completed", gst_element_state_get_name(state));
+        }
+
         break;
     case GST_STATE_CHANGE_ASYNC:
         LOG_MEDIA_MESSAGE("Async: State: %s, pending: %s",
@@ -1114,12 +1164,17 @@ void MediaPlayerPrivateGStreamer::updateStates()
         break;
     }
 
+    m_requestedState = GST_STATE_VOID_PENDING;
+
     if (seeking())
         m_readyState = MediaPlayer::HaveNothing;
 
     if (shouldUpdateAfterSeek)
         timeChanged();
 
+    if (shouldUpdatePlaybackState)
+        m_player->playbackStateChanged();
+
     if (m_networkState != oldNetworkState) {
         LOG_MEDIA_MESSAGE("Network State Changed from %u to %u",
             oldNetworkState, m_networkState);
@@ -1494,6 +1549,10 @@ void MediaPlayerPrivateGStreamer::createAudioSink()
     GstElement* resample = gst_element_factory_make("audioresample", 0);
     GstElement* sink = gst_element_factory_make("autoaudiosink", 0);
 
+    m_autoAudioSink = sink;
+
+    g_signal_connect(sink, "child-added", G_CALLBACK(setAudioStreamPropertiesCallback), this);
+
     GstElement* audioSink = gst_bin_new("audio-sink");
     gst_bin_add_many(GST_BIN(audioSink), scale, convert, resample, sink, NULL);
 
@@ -1539,6 +1598,12 @@ void MediaPlayerPrivateGStreamer::createGSTPlayBin()
     createAudioSink();
 }
 
+void MediaPlayerPrivateGStreamer::simulateAudioInterruption()
+{
+    GstMessage* message = gst_message_new_request_state(GST_OBJECT(m_playBin.get()), GST_STATE_PAUSED);
+    gst_element_post_message(m_playBin.get(), message);
+}
+
 }
 
 #endif // USE(GSTREAMER)
index 8a1fd48..2bbd8d3 100644 (file)
@@ -88,6 +88,10 @@ public:
     void sourceChanged();
     GstElement* audioSink() const;
 
+    void setAudioStreamProperties(GObject*);
+
+    void simulateAudioInterruption();
+
 private:
     MediaPlayerPrivateGStreamer(MediaPlayer*);
 
@@ -153,6 +157,8 @@ private:
     KURL m_url;
     bool m_originalPreloadWasAutoAndWasOverridden;
     bool m_preservesPitch;
+    GstState m_requestedState;
+    GRefPtr<GstElement> m_autoAudioSink;
 };
 }
 
index c037533..0c24fb2 100644 (file)
@@ -51,6 +51,7 @@
 #include "FrameView.h"
 #include "HTMLContentElement.h"
 #include "HTMLInputElement.h"
+#include "HTMLMediaElement.h"
 #include "HTMLNames.h"
 #include "HTMLTextAreaElement.h"
 #include "HistoryItem.h"
@@ -2071,4 +2072,14 @@ String Internals::getImageSourceURL(Element* element, ExceptionCode& ec)
     return element->imageSourceURL();
 }
 
+void Internals::simulateAudioInterruption(Node* node)
+{
+#if USE(GSTREAMER)
+    HTMLMediaElement* element = toMediaElement(node);
+    element->player()->simulateAudioInterruption();
+#else
+    UNUSED_PARAM(node);
+#endif
+}
+
 }
index 2883fae..3fdb3bc 100644 (file)
@@ -300,6 +300,8 @@ public:
 
     String getImageSourceURL(Element*, ExceptionCode&);
                     
+    void simulateAudioInterruption(Node*);
+
 private:
     explicit Internals(Document*);
     Document* contextDocument() const;
index 6fc26a6..3620d0e 100644 (file)
 
     void forceReload(in boolean endToEnd);
 
+    void simulateAudioInterruption(in Node node);
+
     [Conditional=ENCRYPTED_MEDIA_V2] void initializeMockCDM();
 
     [Conditional=SPEECH_SYNTHESIS] void enableMockSpeechSynthesizer();