[EME][GStreamer] Move the decryptor from AppendPipeline to PlaybackPipeline.
authorcommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 27 Apr 2018 07:33:53 +0000 (07:33 +0000)
committercommit-queue@webkit.org <commit-queue@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 27 Apr 2018 07:33:53 +0000 (07:33 +0000)
https://bugs.webkit.org/show_bug.cgi?id=181855

Patch by Yacine Bandou <yacine.bandou_ext@softathome.com> on 2018-04-27
Reviewed by Xabier Rodriguez-Calvar.

The goal of this move is to handle the limitation of SVP (Secure Video Path) memory size.

When the decryptor is in the AppendPipeline and we use SVP, we buffer in MediaSource queue
the decrypted GstBuffers that are in SVP memory.
This behavior cause an out-of-memory error, because we are limited in SVP memory size.

By moving the decryptor in PlaybackPipeline, we avoid to buffer the decrypted GstBuffers
which use the SVP memory and we buffer the encrypted GstBuffers that are in system memory.

This new architecture also allows to start the buffering before obtaining the DRM license
and it makes easier to manage dynamic change of the license or Key.

The decryptor is auto plugged by GStreamer playbin in PlaybackPipeline.

SVP: Secure Video Path also named trusted or protected video path, it is a memory which is
protected by a hardware access control engine, it is not accessible to other unauthorised
software or hardware components.

Tests:
    media/encrypted-media/clearKey/clearKey-cenc-audio-playback-mse.html
    media/encrypted-media/clearKey/clearKey-cenc-video-playback-mse.html

* platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp:
(webkitMediaCommonEncryptionDecryptSinkEventHandler):
* platform/graphics/gstreamer/mse/AppendPipeline.cpp:
(WebCore::dumpAppendState):
(WebCore::AppendPipeline::AppendPipeline):
(WebCore::AppendPipeline::handleNeedContextSyncMessage):
(WebCore::AppendPipeline::handleAppsrcNeedDataReceived):
(WebCore::AppendPipeline::setAppendState):
(WebCore::AppendPipeline::parseDemuxerSrcPadCaps):
(WebCore::AppendPipeline::appsinkNewSample):
(WebCore::AppendPipeline::connectDemuxerSrcPadToAppsinkFromAnyThread):
(WebCore::AppendPipeline::disconnectDemuxerSrcPadFromAppsinkFromAnyThread):
(WebCore::appendPipelineElementMessageCallback): Deleted.
(WebCore::AppendPipeline::handleElementMessage): Deleted.
(WebCore::AppendPipeline::dispatchPendingDecryptionStructure): Deleted.
(WebCore::AppendPipeline::dispatchDecryptionStructure): Deleted.
* platform/graphics/gstreamer/mse/AppendPipeline.h:
* platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp:
(WebCore::MediaPlayerPrivateGStreamerMSE::attemptToDecryptWithInstance):
* platform/graphics/gstreamer/mse/PlaybackPipeline.cpp:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@231089 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/WebCore/ChangeLog
Source/WebCore/platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp
Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp
Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.h
Source/WebCore/platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp
Source/WebCore/platform/graphics/gstreamer/mse/PlaybackPipeline.cpp

index 7bde60d..624d5ac 100644 (file)
@@ -1,5 +1,55 @@
 2018-04-27  Yacine Bandou  <yacine.bandou_ext@softathome.com>
 
+        [EME][GStreamer] Move the decryptor from AppendPipeline to PlaybackPipeline.
+        https://bugs.webkit.org/show_bug.cgi?id=181855
+
+        Reviewed by Xabier Rodriguez-Calvar.
+
+        The goal of this move is to handle the limitation of SVP (Secure Video Path) memory size.
+
+        When the decryptor is in the AppendPipeline and we use SVP, we buffer in MediaSource queue
+        the decrypted GstBuffers that are in SVP memory.
+        This behavior cause an out-of-memory error, because we are limited in SVP memory size.
+
+        By moving the decryptor in PlaybackPipeline, we avoid to buffer the decrypted GstBuffers
+        which use the SVP memory and we buffer the encrypted GstBuffers that are in system memory.
+
+        This new architecture also allows to start the buffering before obtaining the DRM license
+        and it makes easier to manage dynamic change of the license or Key.
+
+        The decryptor is auto plugged by GStreamer playbin in PlaybackPipeline.
+
+        SVP: Secure Video Path also named trusted or protected video path, it is a memory which is
+        protected by a hardware access control engine, it is not accessible to other unauthorised
+        software or hardware components.
+
+        Tests:
+            media/encrypted-media/clearKey/clearKey-cenc-audio-playback-mse.html
+            media/encrypted-media/clearKey/clearKey-cenc-video-playback-mse.html
+
+        * platform/graphics/gstreamer/eme/WebKitCommonEncryptionDecryptorGStreamer.cpp:
+        (webkitMediaCommonEncryptionDecryptSinkEventHandler):
+        * platform/graphics/gstreamer/mse/AppendPipeline.cpp:
+        (WebCore::dumpAppendState):
+        (WebCore::AppendPipeline::AppendPipeline):
+        (WebCore::AppendPipeline::handleNeedContextSyncMessage):
+        (WebCore::AppendPipeline::handleAppsrcNeedDataReceived):
+        (WebCore::AppendPipeline::setAppendState):
+        (WebCore::AppendPipeline::parseDemuxerSrcPadCaps):
+        (WebCore::AppendPipeline::appsinkNewSample):
+        (WebCore::AppendPipeline::connectDemuxerSrcPadToAppsinkFromAnyThread):
+        (WebCore::AppendPipeline::disconnectDemuxerSrcPadFromAppsinkFromAnyThread):
+        (WebCore::appendPipelineElementMessageCallback): Deleted.
+        (WebCore::AppendPipeline::handleElementMessage): Deleted.
+        (WebCore::AppendPipeline::dispatchPendingDecryptionStructure): Deleted.
+        (WebCore::AppendPipeline::dispatchDecryptionStructure): Deleted.
+        * platform/graphics/gstreamer/mse/AppendPipeline.h:
+        * platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp:
+        (WebCore::MediaPlayerPrivateGStreamerMSE::attemptToDecryptWithInstance):
+        * platform/graphics/gstreamer/mse/PlaybackPipeline.cpp:
+
+2018-04-27  Yacine Bandou  <yacine.bandou_ext@softathome.com>
+
         [EME][GStreamer] Add a new message "decrypt-key-needed" send from the decryptor to the application.
         https://bugs.webkit.org/show_bug.cgi?id=181858
 
index 33a330a..8d1b84e 100644 (file)
@@ -299,23 +299,6 @@ static gboolean webkitMediaCommonEncryptionDecryptSinkEventHandler(GstBaseTransf
     gboolean result = FALSE;
 
     switch (GST_EVENT_TYPE(event)) {
-    case GST_EVENT_PROTECTION: {
-        const char* systemId = nullptr;
-
-        gst_event_parse_protection(event, &systemId, nullptr, nullptr);
-        GST_TRACE_OBJECT(self, "received protection event for %s", systemId);
-
-        if (!g_strcmp0(systemId, klass->protectionSystemId)) {
-            GST_DEBUG_OBJECT(self, "sending protection event to the pipeline");
-            gst_element_post_message(GST_ELEMENT(self),
-                gst_message_new_element(GST_OBJECT(self),
-                    gst_structure_new("drm-key-needed", "event", GST_TYPE_EVENT, event, nullptr)));
-        }
-
-        gst_event_unref(event);
-        result = TRUE;
-        break;
-    }
     case GST_EVENT_CUSTOM_DOWNSTREAM_OOB: {
         if (klass->handleKeyResponse(self, event)) {
             GST_DEBUG_OBJECT(self, "key received");
index c2508c7..691f0c8 100644 (file)
@@ -55,8 +55,6 @@ static const char* dumpAppendState(AppendPipeline::AppendState appendState)
         return "NotStarted";
     case AppendPipeline::AppendState::Ongoing:
         return "Ongoing";
-    case AppendPipeline::AppendState::KeyNegotiation:
-        return "KeyNegotiation";
     case AppendPipeline::AppendState::DataStarve:
         return "DataStarve";
     case AppendPipeline::AppendState::Sampling:
@@ -93,13 +91,6 @@ static void appendPipelineApplicationMessageCallback(GstBus*, GstMessage* messag
     appendPipeline->handleApplicationMessage(message);
 }
 
-#if ENABLE(ENCRYPTED_MEDIA)
-static void appendPipelineElementMessageCallback(GstBus*, GstMessage* message, AppendPipeline* appendPipeline)
-{
-    appendPipeline->handleElementMessage(message);
-}
-#endif
-
 static void appendPipelineStateChangeMessageCallback(GstBus*, GstMessage* message, AppendPipeline* appendPipeline)
 {
     appendPipeline->handleStateChangeMessage(message);
@@ -131,9 +122,6 @@ AppendPipeline::AppendPipeline(Ref<MediaSourceClientGStreamerMSE> mediaSourceCli
 
     g_signal_connect(m_bus.get(), "sync-message::need-context", G_CALLBACK(appendPipelineNeedContextMessageCallback), this);
     g_signal_connect(m_bus.get(), "message::application", G_CALLBACK(appendPipelineApplicationMessageCallback), this);
-#if ENABLE(ENCRYPTED_MEDIA)
-    g_signal_connect(m_bus.get(), "message::element", G_CALLBACK(appendPipelineElementMessageCallback), this);
-#endif
     g_signal_connect(m_bus.get(), "message::state-changed", G_CALLBACK(appendPipelineStateChangeMessageCallback), this);
 
     // We assign the created instances here instead of adoptRef() because gst_bin_add_many()
@@ -275,9 +263,6 @@ void AppendPipeline::handleNeedContextSyncMessage(GstMessage* message)
     const gchar* contextType = nullptr;
     gst_message_parse_context_type(message, &contextType);
     GST_TRACE("context type: %s", contextType);
-    if (!g_strcmp0(contextType, "drm-preferred-decryption-system-id")
-        && m_appendState != AppendPipeline::AppendState::KeyNegotiation)
-        setAppendState(AppendPipeline::AppendState::KeyNegotiation);
 
     // MediaPlayerPrivateGStreamerBase will take care of setting up encryption.
     if (m_playerPrivate)
@@ -329,25 +314,6 @@ void AppendPipeline::handleApplicationMessage(GstMessage* message)
     ASSERT_NOT_REACHED();
 }
 
-#if ENABLE(ENCRYPTED_MEDIA)
-void AppendPipeline::handleElementMessage(GstMessage* message)
-{
-    ASSERT(WTF::isMainThread());
-
-    const GstStructure* structure = gst_message_get_structure(message);
-    GST_TRACE("%s message from %s", gst_structure_get_name(structure), GST_MESSAGE_SRC_NAME(message));
-    if (m_playerPrivate && gst_structure_has_name(structure, "drm-key-needed")) {
-        if (m_appendState != AppendPipeline::AppendState::KeyNegotiation)
-            setAppendState(AppendPipeline::AppendState::KeyNegotiation);
-
-        GST_DEBUG("sending drm-key-needed message from %s to the player", GST_MESSAGE_SRC_NAME(message));
-        GRefPtr<GstEvent> event;
-        gst_structure_get(structure, "event", GST_TYPE_EVENT, &event.outPtr(), nullptr);
-        m_playerPrivate->handleProtectionEvent(event.get());
-    }
-}
-#endif
-
 void AppendPipeline::handleStateChangeMessage(GstMessage* message)
 {
     ASSERT(WTF::isMainThread());
@@ -373,7 +339,7 @@ void AppendPipeline::handleAppsrcNeedDataReceived()
         return;
     }
 
-    ASSERT(m_appendState == AppendState::KeyNegotiation || m_appendState == AppendState::Ongoing || m_appendState == AppendState::Sampling);
+    ASSERT(m_appendState == AppendState::Ongoing || m_appendState == AppendState::Sampling);
     ASSERT(!m_appsrcNeedDataReceived);
 
     GST_TRACE("received need-data from appsrc");
@@ -431,8 +397,7 @@ void AppendPipeline::setAppendState(AppendState newAppendState)
     // NotStarted-->Ongoing-->DataStarve-->NotStarted
     //           |         |            `->Aborting-->NotStarted
     //           |         `->Sampling-ยทยทยท->Sampling-->LastSample-->NotStarted
-    //           |         |                                     `->Aborting-->NotStarted
-    //           |         `->KeyNegotiation-->Ongoing-->[...]
+    //           |                                               `->Aborting-->NotStarted
     //           `->Aborting-->NotStarted
     AppendState oldAppendState = m_appendState;
     AppendState nextAppendState = AppendState::Invalid;
@@ -468,19 +433,8 @@ void AppendPipeline::setAppendState(AppendState newAppendState)
             break;
         }
         break;
-    case AppendState::KeyNegotiation:
-        switch (newAppendState) {
-        case AppendState::Ongoing:
-        case AppendState::Invalid:
-            ok = true;
-            break;
-        default:
-            break;
-        }
-        break;
     case AppendState::Ongoing:
         switch (newAppendState) {
-        case AppendState::KeyNegotiation:
         case AppendState::Sampling:
         case AppendState::Invalid:
             ok = true;
@@ -582,18 +536,7 @@ void AppendPipeline::parseDemuxerSrcPadCaps(GstCaps* demuxerSrcPadCaps)
 
     m_demuxerSrcPadCaps = adoptGRef(demuxerSrcPadCaps);
     m_streamType = WebCore::MediaSourceStreamTypeGStreamer::Unknown;
-#if ENABLE(ENCRYPTED_MEDIA)
-    if (areEncryptedCaps(m_demuxerSrcPadCaps.get())) {
-        // Any previous decryptor should have been removed from the pipeline by disconnectFromAppSinkFromStreamingThread()
-        ASSERT(!m_decryptor);
-        GstStructure* structure = gst_caps_get_structure(m_demuxerSrcPadCaps.get(), 0);
-        m_decryptor = GStreamerEMEUtilities::createDecryptor(gst_structure_get_string(structure, "protection-system"));
-        if (!m_decryptor) {
-            GST_ERROR("decryptor not found for caps: %" GST_PTR_FORMAT, m_demuxerSrcPadCaps.get());
-            return;
-        }
-    }
-#endif
+
     const char* originalMediaType = capsMediaType(m_demuxerSrcPadCaps.get());
     if (!MediaPlayerPrivateGStreamerMSE::supportsCodec(originalMediaType)) {
             m_presentationSize = WebCore::FloatSize();
@@ -673,10 +616,6 @@ void AppendPipeline::appsinkNewSample(GstSample* sample)
     {
         LockHolder locker(m_newSampleLock);
 
-        // If we were in KeyNegotiation but samples are coming, assume we're already OnGoing
-        if (m_appendState == AppendState::KeyNegotiation)
-            setAppendState(AppendState::Ongoing);
-
         // Ignore samples if we're not expecting them. Refuse processing if we're in Invalid state.
         if (m_appendState != AppendState::Ongoing && m_appendState != AppendState::Sampling) {
             GST_WARNING("Unexpected sample, appendState=%s", dumpAppendState(m_appendState));
@@ -1009,28 +948,10 @@ void AppendPipeline::connectDemuxerSrcPadToAppsinkFromAnyThread(GstPad* demuxerS
             currentSrcPad = parserSrcPad;
         }
 
-#if ENABLE(ENCRYPTED_MEDIA)
-        if (m_decryptor) {
-            gst_object_ref(m_decryptor.get());
-            gst_bin_add(GST_BIN(m_pipeline.get()), m_decryptor.get());
-            gst_element_sync_state_with_parent(m_decryptor.get());
-
-            GRefPtr<GstPad> decryptorSinkPad = adoptGRef(gst_element_get_static_pad(m_decryptor.get(), "sink"));
-            GRefPtr<GstPad> decryptorSrcPad = adoptGRef(gst_element_get_static_pad(m_decryptor.get(), "src"));
-
-            gst_pad_link(currentSrcPad.get(), decryptorSinkPad.get());
-            currentSrcPad = decryptorSrcPad;
-        }
-#endif
-
         gst_pad_link(currentSrcPad.get(), appsinkSinkPad.get());
 
         gst_element_sync_state_with_parent(m_appsink.get());
 
-#if ENABLE(ENCRYPTED_MEDIA)
-        if (m_pendingDecryptionStructure)
-            dispatchPendingDecryptionStructure();
-#endif
         gst_element_set_state(m_pipeline.get(), GST_STATE_PAUSED);
         gst_element_sync_state_with_parent(m_appsink.get());
 
@@ -1109,14 +1030,6 @@ void AppendPipeline::disconnectDemuxerSrcPadFromAppsinkFromAnyThread(GstPad*)
 
     GST_DEBUG("Disconnecting appsink");
 
-#if ENABLE(ENCRYPTED_MEDIA)
-    if (m_decryptor) {
-        gst_element_set_state(m_decryptor.get(), GST_STATE_NULL);
-        gst_bin_remove(GST_BIN(m_pipeline.get()), m_decryptor.get());
-        m_decryptor = nullptr;
-    }
-#endif
-
     if (m_parser) {
         gst_element_set_state(m_parser.get(), GST_STATE_NULL);
         gst_bin_remove(GST_BIN(m_pipeline.get()), m_parser.get());
@@ -1126,35 +1039,6 @@ void AppendPipeline::disconnectDemuxerSrcPadFromAppsinkFromAnyThread(GstPad*)
     GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS(GST_BIN(m_pipeline.get()), GST_DEBUG_GRAPH_SHOW_ALL, "pad-removed-after");
 }
 
-#if ENABLE(ENCRYPTED_MEDIA)
-void AppendPipeline::dispatchPendingDecryptionStructure()
-{
-    ASSERT(m_decryptor);
-    ASSERT(m_pendingDecryptionStructure);
-    ASSERT(m_appendState == AppendState::KeyNegotiation);
-    GST_TRACE("dispatching key to append pipeline %p", this);
-
-    // Release the m_pendingDecryptionStructure object since
-    // gst_event_new_custom() takes over ownership of it.
-    gst_element_send_event(m_pipeline.get(), gst_event_new_custom(GST_EVENT_CUSTOM_DOWNSTREAM_OOB, m_pendingDecryptionStructure.release()));
-
-    setAppendState(AppendState::Ongoing);
-}
-
-void AppendPipeline::dispatchDecryptionStructure(GUniquePtr<GstStructure>&& structure)
-{
-    if (m_appendState == AppendState::KeyNegotiation) {
-        GST_TRACE("append pipeline %p in key negotiation", this);
-        m_pendingDecryptionStructure = WTFMove(structure);
-        if (m_decryptor)
-            dispatchPendingDecryptionStructure();
-        else
-            GST_TRACE("no decryptor yet, waiting for it");
-    } else
-        GST_TRACE("append pipeline %p not in key negotiation", this);
-}
-#endif
-
 static void appendPipelineAppsinkCapsChanged(GObject* appsinkPad, GParamSpec*, AppendPipeline* appendPipeline)
 {
     GstStructure* structure = gst_structure_new_empty("appsink-caps-changed");
index 8330466..995c129 100644 (file)
@@ -42,7 +42,7 @@ struct PadProbeInformation {
 
 class AppendPipeline : public ThreadSafeRefCounted<AppendPipeline> {
 public:
-    enum class AppendState { Invalid, NotStarted, Ongoing, KeyNegotiation, DataStarve, Sampling, LastSample, Aborting };
+    enum class AppendState { Invalid, NotStarted, Ongoing, DataStarve, Sampling, LastSample, Aborting };
 
     AppendPipeline(Ref<MediaSourceClientGStreamerMSE>, Ref<SourceBufferPrivateGStreamer>, MediaPlayerPrivateGStreamerMSE&);
     virtual ~AppendPipeline();
@@ -50,9 +50,6 @@ public:
     void handleNeedContextSyncMessage(GstMessage*);
     void handleApplicationMessage(GstMessage*);
     void handleStateChangeMessage(GstMessage*);
-#if ENABLE(ENCRYPTED_MEDIA)
-    void handleElementMessage(GstMessage*);
-#endif
 
     gint id();
     AppendState appendState() { return m_appendState; }
@@ -60,9 +57,6 @@ public:
 
     GstFlowReturn handleNewAppsinkSample(GstElement*);
     GstFlowReturn pushNewBuffer(GstBuffer*);
-#if ENABLE(ENCRYPTED_MEDIA)
-    void dispatchDecryptionStructure(GUniquePtr<GstStructure>&&);
-#endif
 
     // Takes ownership of caps.
     void parseDemuxerSrcPadCaps(GstCaps*);
@@ -98,9 +92,6 @@ private:
     void handleAppsrcNeedDataReceived();
     void removeAppsrcDataLeavingProbe();
     void setAppsrcDataLeavingProbe();
-#if ENABLE(ENCRYPTED_MEDIA)
-    void dispatchPendingDecryptionStructure();
-#endif
 
     Ref<MediaSourceClientGStreamerMSE> m_mediaSourceClient;
     Ref<SourceBufferPrivateGStreamer> m_sourceBufferPrivate;
@@ -118,9 +109,6 @@ private:
     GRefPtr<GstElement> m_appsrc;
     GRefPtr<GstElement> m_demux;
     GRefPtr<GstElement> m_parser; // Optional.
-#if ENABLE(ENCRYPTED_MEDIA)
-    GRefPtr<GstElement> m_decryptor;
-#endif
     // The demuxer has one src stream only, so only one appsink is needed and linked to it.
     GRefPtr<GstElement> m_appsink;
 
@@ -155,9 +143,6 @@ private:
     RefPtr<WebCore::TrackPrivateBase> m_track;
 
     GRefPtr<GstBuffer> m_pendingBuffer;
-#if ENABLE(ENCRYPTED_MEDIA)
-    GUniquePtr<GstStructure> m_pendingDecryptionStructure;
-#endif
 };
 
 } // namespace WebCore.
index a30811f..0436325 100644 (file)
@@ -955,8 +955,7 @@ void MediaPlayerPrivateGStreamerMSE::attemptToDecryptWithInstance(CDMInstance& i
         gst_structure_set_value(structure.get(), "key-ids", &keyIDList);
         gst_structure_set_value(structure.get(), "key-values", &keyValueList);
 
-        for (auto it : m_appendPipelinesMap)
-            it.value->dispatchDecryptionStructure(GUniquePtr<GstStructure>(gst_structure_copy(structure.get())));
+        gst_element_send_event(m_playbackPipeline->pipeline(), gst_event_new_custom(GST_EVENT_CUSTOM_DOWNSTREAM_OOB, gst_structure_copy(structure.get())));
     }
 }
 #endif
index 06d5862..2b69971 100644 (file)
@@ -180,7 +180,9 @@ void PlaybackPipeline::attachTrack(RefPtr<SourceBufferPrivateGStreamer> sourceBu
 
     GUniquePtr<gchar> parserBinName(g_strdup_printf("streamparser%u", padId));
 
-    if (!g_strcmp0(mediaType, "video/x-h264")) {
+    if (areEncryptedCaps(caps)) {
+        GST_DEBUG("It's encrypted content, parsers are not needed before decrypting the content");
+    } else if (!g_strcmp0(mediaType, "video/x-h264")) {
         GRefPtr<GstCaps> filterCaps = adoptGRef(gst_caps_new_simple("video/x-h264", "alignment", G_TYPE_STRING, "au", nullptr));
         GstElement* capsfilter = gst_element_factory_make("capsfilter", nullptr);
         g_object_set(capsfilter, "caps", filterCaps.get(), nullptr);