Refactor the heuristic for showing media controls to take all media sessions into...
authorwenson_hsieh@apple.com <wenson_hsieh@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sun, 4 Sep 2016 00:24:14 +0000 (00:24 +0000)
committerwenson_hsieh@apple.com <wenson_hsieh@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Sun, 4 Sep 2016 00:24:14 +0000 (00:24 +0000)
commit6e16231746372612b68d4e7f2e60d70f816f9d0d
tree5ecfc14b42e55f5ef940002ce74b05e973b581fa
parent13038ab3068983add2c0078c5af5e0ba3f38f2b5
Refactor the heuristic for showing media controls to take all media sessions into account
https://bugs.webkit.org/show_bug.cgi?id=161503
<rdar://problem/28033783>

Reviewed by Darin Adler.

Source/WebCore:

Currently, when selecting a media session to show playback controls for, we grab the first media session that
passes our heuristic. Using this method, we are unable to take additional factors into account, such as whether
another media session's element is scrolled in view, or if another media session has been interacted with more
recently. To address this, we make the following changes:

    1.  Consider the list of all MediaElementSessions.

    2.  Select only the MediaElementSessions capable of showing media controls and sort the list by a special
        heuristic that takes visibility and time of last user interaction into account. The first element on
        this list is the strongest candidate for main content.

    3.  If this strongest candidate is visible in the viewport, or it is playing with audio, we return this
        as the chosen candidate. Otherwise, we return this session only if no other non-candidate video could be
        confused as the main content (i.e. the non-candidate video is not only visible in the viewport, but also
        large enough to be considered main content).

Using this new method of determining the video to show controls for, we retain previous behavior for pages with
a single video. On pages with multiple videos, the above logic ensures that if the current controlled video is
paused, scrolled out of view, and then a new video is scrolled into view, we will either hide media controls to
avoid confusion if that video could be confused for main content (using the mechanism in step 3), or we
hook up the media controls to the new video if it satisfies main content (using the mechanism in step 2).

This patch also adds 6 new TestWebKitAPI unit tests.

* html/HTMLMediaElement.cpp:
(WebCore::mediaElementSessionInfoForSession):
(WebCore::preferMediaControlsForCandidateSessionOverOtherCandidateSession):
(WebCore::mediaSessionMayBeConfusedWithMainContent):
(WebCore::bestMediaSessionForShowingPlaybackControlsManager):
(WebCore::HTMLMediaElement::didAttachRenderers):
(WebCore::HTMLMediaElement::layoutSizeChanged):
(WebCore::HTMLMediaElement::isVisibleInViewportChanged):
(WebCore::HTMLMediaElement::resetPlaybackSessionState):
(WebCore::HTMLMediaElement::isVisibleInViewport):
(WebCore::HTMLMediaElement::updatePlaybackControlsManager):
* html/HTMLMediaElement.h:
* html/MediaElementSession.cpp:
(WebCore::MediaElementSession::removeBehaviorRestriction):
(WebCore::MediaElementSession::canShowControlsManager):
(WebCore::MediaElementSession::isLargeEnoughForMainContent):
(WebCore::MediaElementSession::mostRecentUserInteractionTime):
(WebCore::MediaElementSession::wantsToObserveViewportVisibilityForMediaControls):
(WebCore::MediaElementSession::wantsToObserveViewportVisibilityForAutoplay):
(WebCore::MediaElementSession::resetPlaybackSessionState):
(WebCore::MediaElementSession::canControlControlsManager): Deleted.
* html/MediaElementSession.h:
* platform/audio/PlatformMediaSession.h:
(WebCore::PlatformMediaSession::resetPlaybackSessionState):
(WebCore::PlatformMediaSession::canControlControlsManager): Deleted.
* platform/audio/PlatformMediaSessionManager.cpp:
(WebCore::PlatformMediaSessionManager::currentSessionsMatching):
(WebCore::PlatformMediaSessionManager::currentSessionMatching): Deleted.
* platform/audio/PlatformMediaSessionManager.h:
* platform/cocoa/WebPlaybackSessionModelMediaElement.mm:
(WebPlaybackSessionModelMediaElement::setMediaElement):

Source/WebKit2:

Adds an SPI testing hook for sending the element ID of the currently controlled video element from the web
process to the UI process. See VideoControlsManager.mm in Tools/TestWebKitAPI/ for usage.

* UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _requestControlledElementID]):
(-[WKWebView _handleControlledElementIDResponse:]):
(-[WKWebView _hasActiveVideoForControlsManager]): Deleted.
* UIProcess/API/Cocoa/WKWebViewPrivate.h:
* UIProcess/Cocoa/WebPlaybackSessionManagerProxy.h:
* UIProcess/Cocoa/WebPlaybackSessionManagerProxy.messages.in:
* UIProcess/Cocoa/WebPlaybackSessionManagerProxy.mm:
(WebKit::WebPlaybackSessionManagerProxy::handleControlledElementIDResponse):
(WebKit::WebPlaybackSessionManagerProxy::requestControlledElementID):
* UIProcess/PageClient.h:
* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::requestControlledElementID):
(WebKit::WebPageProxy::handleControlledElementIDResponse):
* UIProcess/WebPageProxy.h:
* UIProcess/mac/PageClientImpl.h:
* UIProcess/mac/PageClientImpl.mm:
(WebKit::PageClientImpl::handleControlledElementIDResponse):
* WebProcess/cocoa/WebPlaybackSessionManager.h:
* WebProcess/cocoa/WebPlaybackSessionManager.messages.in:
* WebProcess/cocoa/WebPlaybackSessionManager.mm:
(WebKit::WebPlaybackSessionManager::handleControlledElementIDRequest):

Tools:

Adds new unit tests verifying the behavior of media playback controls when scrolling another video into view.
Please see the WebCore ChangeLog for more details about this change. Also refactors existing
VideoControlsManager tests by folding duplicated setup and testing logic into helper methods to make the unit
tests more readable.

* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WebKit2Cocoa/VideoControlsManager.mm:
(-[MessageHandler initWithMessage:handler:]):
(-[MessageHandler userContentController:didReceiveScriptMessage:]):
(-[VideoControlsManagerTestWebView performAfterLoading:]):
(-[VideoControlsManagerTestWebView loadTestPageNamed:]):
(-[VideoControlsManagerTestWebView loadTestPageNamed:andExpectControlsManager:afterReceivingMessage:]):
(-[VideoControlsManagerTestWebView performAfterReceivingMessage:action:]):
(-[VideoControlsManagerTestWebView controlledElementID]):
(-[VideoControlsManagerTestWebView _handleControlledElementIDResponse:]):
(TestWebKitAPI::setUpWebViewForTestingVideoControlsManager):
(TestWebKitAPI::TEST):
(-[MediaPlaybackMessageHandler initWithWKWebView:finalMessageString:]): Deleted.
(-[MediaPlaybackMessageHandler userContentController:didReceiveScriptMessage:]): Deleted.
(-[OnLoadMessageHandler initWithWKWebView:handler:]): Deleted.
(-[OnLoadMessageHandler userContentController:didReceiveScriptMessage:]): Deleted.
(-[WKWebView performAfterLoading:]): Deleted.
* TestWebKitAPI/Tests/WebKit2Cocoa/large-video-playing-scroll-away.html: Added.
* TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-autoplaying-click-to-pause.html: Added.
* TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-autoplaying-scroll-to-video.html: Added.
* TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-paused-video-hides-controls.html: Added.
* TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-playing-muted-video-hides-controls.html: Added.
* TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-playing-video-keeps-controls.html: Added.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205412 268f45cc-cd09-0410-ab3c-d52691b4dbfc
32 files changed:
Source/WebCore/ChangeLog
Source/WebCore/html/HTMLMediaElement.cpp
Source/WebCore/html/HTMLMediaElement.h
Source/WebCore/html/MediaElementSession.cpp
Source/WebCore/html/MediaElementSession.h
Source/WebCore/platform/audio/PlatformMediaSession.h
Source/WebCore/platform/audio/PlatformMediaSessionManager.cpp
Source/WebCore/platform/audio/PlatformMediaSessionManager.h
Source/WebCore/platform/cocoa/WebPlaybackSessionModelMediaElement.mm
Source/WebKit2/ChangeLog
Source/WebKit2/UIProcess/API/Cocoa/WKWebView.mm
Source/WebKit2/UIProcess/API/Cocoa/WKWebViewPrivate.h
Source/WebKit2/UIProcess/Cocoa/WebPlaybackSessionManagerProxy.h
Source/WebKit2/UIProcess/Cocoa/WebPlaybackSessionManagerProxy.messages.in
Source/WebKit2/UIProcess/Cocoa/WebPlaybackSessionManagerProxy.mm
Source/WebKit2/UIProcess/PageClient.h
Source/WebKit2/UIProcess/WebPageProxy.cpp
Source/WebKit2/UIProcess/WebPageProxy.h
Source/WebKit2/UIProcess/mac/PageClientImpl.h
Source/WebKit2/UIProcess/mac/PageClientImpl.mm
Source/WebKit2/WebProcess/cocoa/WebPlaybackSessionManager.h
Source/WebKit2/WebProcess/cocoa/WebPlaybackSessionManager.messages.in
Source/WebKit2/WebProcess/cocoa/WebPlaybackSessionManager.mm
Tools/ChangeLog
Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/VideoControlsManager.mm
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/large-video-playing-scroll-away.html [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-autoplaying-click-to-pause.html [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-autoplaying-scroll-to-video.html [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-paused-video-hides-controls.html [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-playing-muted-video-hides-controls.html [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/WebKit2Cocoa/large-videos-playing-video-keeps-controls.html [new file with mode: 0644]