[iOS WK2] Add plumbing for WKContentView to ask the web process for additional drag...
authorwenson_hsieh@apple.com <wenson_hsieh@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 6 Sep 2017 02:41:52 +0000 (02:41 +0000)
committerwenson_hsieh@apple.com <wenson_hsieh@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 6 Sep 2017 02:41:52 +0000 (02:41 +0000)
https://bugs.webkit.org/show_bug.cgi?id=176348
Work towards <rdar://problem/31144674>

Reviewed by Tim Horton.

Source/WebKit:

Adds boilerplate plumbing to request additional items to an existing session. This implements some UI-side logic
in WKContentView to send an additional drag items request and handle a response from the web process.
To start, WebPageProxy::requestAdditionalItemsForDragSession is invoked by -_dragInteraction:
itemsForAddingToSession:withTouchAtPoint:completion: in WKContentView, and the response from the web process is
handled in -[WKContentView _didHandleAdditionalDragItemsRequest:].

* UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _simulateItemsForAddingToSession:atLocation:completion:]):
* UIProcess/API/Cocoa/WKWebViewPrivate.h:

Add SPI to simulate a request for additional drag items. See Tools/ChangeLog for more detail.

* UIProcess/PageClient.h:
* UIProcess/WebPageProxy.h:
* UIProcess/WebPageProxy.messages.in:

Add an IPC message handler for the additional drag items response from the web process.

* UIProcess/ios/DragDropInteractionState.h:
(WebKit::DragDropInteractionState::BlockPtr<void):
* UIProcess/ios/DragDropInteractionState.mm:
(WebKit::DragDropInteractionState::shouldRequestAdditionalItemForDragSession const):
(WebKit::DragDropInteractionState::dragSessionWillRequestAdditionalItem):

Invoke the new additional items completion handler when tearing down to avoid getting UIKit into a bad state.

(WebKit::DragDropInteractionState::dragAndDropSessionsDidEnd):
* UIProcess/ios/PageClientImplIOS.h:
* UIProcess/ios/PageClientImplIOS.mm:
(WebKit::PageClientImpl::didHandleAdditionalDragItemsRequest):
* UIProcess/ios/WKContentViewInteraction.h:
* UIProcess/ios/WKContentViewInteraction.mm:
(-[WKContentView _didHandleAdditionalDragItemsRequest:]):
(-[WKContentView _itemsForBeginningOrAddingToSessionWithRegistrationList:stagedDragSource:]):

Factors common logic to convert a staged WebItemProviderRegistrationInfoList and DragSourceState into a list of
drag items, invoking the private UI delegate in the process. This is called both when starting a drag session,
and adding items to an existing drag session.

(-[WKContentView _dragInteraction:itemsForAddingToSession:withTouchAtPoint:completion:]):

Implements a (proposed) additional drag item delegate that serves as an asynchronous alternative to the API
variant, -_dragInteraction:itemsForAddingToSession:withTouchAtPoint:. See <rdar://problem/33146803> for more
information.

(-[WKContentView dragInteraction:itemsForBeginningSession:]):

Refactored to use -_itemsForBeginningOrAddingToSessionWithRegistrationList:stagedDragSource:.

(-[WKContentView _simulateItemsForAddingToSession:atLocation:completion:]):
* UIProcess/ios/WebPageProxyIOS.mm:
(WebKit::WebPageProxy::didHandleAdditionalDragItemsRequest):
(WebKit::WebPageProxy::requestAdditionalItemsForDragSession):
* WebProcess/WebPage/WebPage.h:
* WebProcess/WebPage/WebPage.messages.in:

Add an IPC message handler for the additional drag items request from the UI process.

* WebProcess/WebPage/ios/WebPageIOS.mm:
(WebKit::WebPage::requestAdditionalItemsForDragSession):

Tools:

Introduces new test infrastructure to simulate tapping to add additional items to the current drag session. We
add -[DataInteractionSimulator runFrom:to:additionalItemRequestLocations:], for which the last argument is a
dictionary mapping progress (a double between 0 and 1) to NSValues representing the location from which the drag
and drop simulation will request an additional drag item. During the simulated drag and drop, when the progress
value exceeds a value in the map of remaining additional item request locations, we halt drag simulation
progress for that runloop and instead request additional items from the location specified.

The only (useful) passing test we can create using the new machinery is one that verifies that preventDefault()
on dragstart works as expected, preventing additional items from being added. While this trivially passes now,
since the web-process-side of the additional items flow is not yet implemented, it should continue to pass after
the web process portion is implemented.

* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WebKitCocoa/selected-text-image-link-and-editable.html: Added.

Add a new test page that contains some non-editable text, an image, a link, and a rich text editable area.

* TestWebKitAPI/Tests/ios/DataInteractionTests.mm:
(TestWebKitAPI::TEST):
* TestWebKitAPI/ios/DataInteractionSimulator.h:
* TestWebKitAPI/ios/DataInteractionSimulator.mm:
(-[MockDragDropSession addItems:]):

Convenience method to add additional mock drag items to a drag or drop session.

(-[DataInteractionSimulator _resetSimulatedState]):
(-[DataInteractionSimulator runFrom:to:]):

Converted into a convenience wrapper around the latter version, passing in nil for the additional item request
locations dictionary.

(-[DataInteractionSimulator runFrom:to:additionalItemRequestLocations:]):
(-[DataInteractionSimulator _enqueuePendingAdditionalItemRequestLocations]):
(-[DataInteractionSimulator _sendQueuedAdditionalItemRequest]):

New helper methods to detect when and where additional items should be "added", and subsequently simulate adding
drag items at these given locations.

(-[DataInteractionSimulator _advanceProgress]):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@221660 268f45cc-cd09-0410-ab3c-d52691b4dbfc

22 files changed:
Source/WebKit/ChangeLog
Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm
Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h
Source/WebKit/UIProcess/PageClient.h
Source/WebKit/UIProcess/WebPageProxy.h
Source/WebKit/UIProcess/WebPageProxy.messages.in
Source/WebKit/UIProcess/ios/DragDropInteractionState.h
Source/WebKit/UIProcess/ios/DragDropInteractionState.mm
Source/WebKit/UIProcess/ios/PageClientImplIOS.h
Source/WebKit/UIProcess/ios/PageClientImplIOS.mm
Source/WebKit/UIProcess/ios/WKContentViewInteraction.h
Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm
Source/WebKit/UIProcess/ios/WebPageProxyIOS.mm
Source/WebKit/WebProcess/WebPage/WebPage.h
Source/WebKit/WebProcess/WebPage/WebPage.messages.in
Source/WebKit/WebProcess/WebPage/ios/WebPageIOS.mm
Tools/ChangeLog
Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj
Tools/TestWebKitAPI/Tests/WebKitCocoa/selected-text-image-link-and-editable.html [new file with mode: 0644]
Tools/TestWebKitAPI/Tests/ios/DataInteractionTests.mm
Tools/TestWebKitAPI/ios/DataInteractionSimulator.h
Tools/TestWebKitAPI/ios/DataInteractionSimulator.mm

index 32bba78..a29e359 100644 (file)
@@ -1,3 +1,72 @@
+2017-09-05  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        [iOS WK2] Add plumbing for WKContentView to ask the web process for additional drag items
+        https://bugs.webkit.org/show_bug.cgi?id=176348
+        Work towards <rdar://problem/31144674>
+
+        Reviewed by Tim Horton.
+
+        Adds boilerplate plumbing to request additional items to an existing session. This implements some UI-side logic
+        in WKContentView to send an additional drag items request and handle a response from the web process. 
+        To start, WebPageProxy::requestAdditionalItemsForDragSession is invoked by -_dragInteraction:
+        itemsForAddingToSession:withTouchAtPoint:completion: in WKContentView, and the response from the web process is
+        handled in -[WKContentView _didHandleAdditionalDragItemsRequest:].
+
+        * UIProcess/API/Cocoa/WKWebView.mm:
+        (-[WKWebView _simulateItemsForAddingToSession:atLocation:completion:]):
+        * UIProcess/API/Cocoa/WKWebViewPrivate.h:
+
+        Add SPI to simulate a request for additional drag items. See Tools/ChangeLog for more detail.
+
+        * UIProcess/PageClient.h:
+        * UIProcess/WebPageProxy.h:
+        * UIProcess/WebPageProxy.messages.in:
+
+        Add an IPC message handler for the additional drag items response from the web process.
+
+        * UIProcess/ios/DragDropInteractionState.h:
+        (WebKit::DragDropInteractionState::BlockPtr<void):
+        * UIProcess/ios/DragDropInteractionState.mm:
+        (WebKit::DragDropInteractionState::shouldRequestAdditionalItemForDragSession const):
+        (WebKit::DragDropInteractionState::dragSessionWillRequestAdditionalItem):
+
+        Invoke the new additional items completion handler when tearing down to avoid getting UIKit into a bad state.
+
+        (WebKit::DragDropInteractionState::dragAndDropSessionsDidEnd):
+        * UIProcess/ios/PageClientImplIOS.h:
+        * UIProcess/ios/PageClientImplIOS.mm:
+        (WebKit::PageClientImpl::didHandleAdditionalDragItemsRequest):
+        * UIProcess/ios/WKContentViewInteraction.h:
+        * UIProcess/ios/WKContentViewInteraction.mm:
+        (-[WKContentView _didHandleAdditionalDragItemsRequest:]):
+        (-[WKContentView _itemsForBeginningOrAddingToSessionWithRegistrationList:stagedDragSource:]):
+
+        Factors common logic to convert a staged WebItemProviderRegistrationInfoList and DragSourceState into a list of
+        drag items, invoking the private UI delegate in the process. This is called both when starting a drag session,
+        and adding items to an existing drag session.
+
+        (-[WKContentView _dragInteraction:itemsForAddingToSession:withTouchAtPoint:completion:]):
+
+        Implements a (proposed) additional drag item delegate that serves as an asynchronous alternative to the API
+        variant, -_dragInteraction:itemsForAddingToSession:withTouchAtPoint:. See <rdar://problem/33146803> for more
+        information.
+
+        (-[WKContentView dragInteraction:itemsForBeginningSession:]):
+
+        Refactored to use -_itemsForBeginningOrAddingToSessionWithRegistrationList:stagedDragSource:.
+
+        (-[WKContentView _simulateItemsForAddingToSession:atLocation:completion:]):
+        * UIProcess/ios/WebPageProxyIOS.mm:
+        (WebKit::WebPageProxy::didHandleAdditionalDragItemsRequest):
+        (WebKit::WebPageProxy::requestAdditionalItemsForDragSession):
+        * WebProcess/WebPage/WebPage.h:
+        * WebProcess/WebPage/WebPage.messages.in:
+
+        Add an IPC message handler for the additional drag items request from the UI process.
+
+        * WebProcess/WebPage/ios/WebPageIOS.mm:
+        (WebKit::WebPage::requestAdditionalItemsForDragSession):
+
 2017-09-05  Alex Christensen  <achristensen@webkit.org>
 
         Add WKUIDelegatePrivate equivalent of WKPageUIClient's mouseDidMoveOverElement
index 3e01dad..534a8fe 100644 (file)
@@ -5831,6 +5831,13 @@ static WebCore::UserInterfaceLayoutDirection toUserInterfaceLayoutDirection(UISe
 #endif
 }
 
+- (void)_simulateItemsForAddingToSession:(id)session atLocation:(CGPoint)location completion:(void(^)(NSArray *))completion
+{
+#if ENABLE(DATA_INTERACTION)
+    [_contentView _simulateItemsForAddingToSession:session atLocation:location completion:completion];
+#endif
+}
+
 - (void)_simulatePrepareForDataInteractionSession:(id)session completion:(dispatch_block_t)completion
 {
 #if ENABLE(DATA_INTERACTION)
index c9583a7..1a7fd3a 100644 (file)
@@ -373,6 +373,7 @@ typedef NS_ENUM(NSInteger, _WKImmediateActionType) {
 - (void)_simulateDataInteractionSessionDidEnd:(id)session WK_API_AVAILABLE(ios(WK_IOS_TBA));
 - (void)_simulateWillBeginDataInteractionWithSession:(id)session WK_API_AVAILABLE(ios(WK_IOS_TBA));
 - (NSArray *)_simulatedItemsForSession:(id)session WK_API_AVAILABLE(ios(WK_IOS_TBA));
+- (void)_simulateItemsForAddingToSession:(id)session atLocation:(CGPoint)location completion:(void(^)(NSArray *))completion WK_API_AVAILABLE(ios(WK_IOS_TBA));
 - (void)_simulatePrepareForDataInteractionSession:(id)session completion:(dispatch_block_t)completion WK_API_AVAILABLE(ios(WK_IOS_TBA));
 - (void)_simulateLongPressActionAtLocation:(CGPoint)location;
 
index 1667066..53fea0a 100644 (file)
@@ -381,6 +381,7 @@ public:
 #if ENABLE(DATA_INTERACTION)
     virtual void didPerformDataInteractionControllerOperation(bool handled) = 0;
     virtual void didHandleStartDataInteractionRequest(bool started) = 0;
+    virtual void didHandleAdditionalDragItemsRequest(bool added) = 0;
     virtual void startDrag(const WebCore::DragItem&, const ShareableBitmap::Handle& image) = 0;
     virtual void didConcludeEditDataInteraction(std::optional<WebCore::TextIndicatorData>) = 0;
     virtual void didChangeDataInteractionCaretRect(const WebCore::IntRect& previousCaretRect, const WebCore::IntRect& caretRect) = 0;
index aee30b3..dd887ec 100644 (file)
@@ -561,7 +561,9 @@ public:
 #if ENABLE(DATA_INTERACTION)
     void didPerformDataInteractionControllerOperation(bool handled);
     void didHandleStartDataInteractionRequest(bool started);
+    void didHandleAdditionalDragItemsRequest(bool added);
     void requestStartDataInteraction(const WebCore::IntPoint& clientPosition, const WebCore::IntPoint& globalPosition);
+    void requestAdditionalItemsForDragSession(const WebCore::IntPoint& clientPosition, const WebCore::IntPoint& globalPosition);
     void didConcludeEditDataInteraction(std::optional<WebCore::TextIndicatorData>);
 #endif
 #endif
index 125741d..c7dac79 100644 (file)
@@ -317,6 +317,7 @@ messages -> WebPageProxy {
 #if ENABLE(DATA_INTERACTION)
     DidPerformDataInteractionControllerOperation(bool handled)
     DidHandleStartDataInteractionRequest(bool started)
+    DidHandleAdditionalDragItemsRequest(bool added)
     DidConcludeEditDataInteraction(std::optional<WebCore::TextIndicatorData> textIndicator)
 #endif
 
index a3d8aeb..3b1c329 100644 (file)
@@ -70,6 +70,8 @@ public:
     void clearStagedDragSource(DidBecomeActive = DidBecomeActive::No);
     UITargetedDragPreview *previewForDragItem(UIDragItem *, UIView *contentView, UIView *previewContainer) const;
     void dragSessionWillDelaySetDownAnimation(dispatch_block_t completion);
+    bool shouldRequestAdditionalItemForDragSession(id <UIDragSession>) const;
+    void dragSessionWillRequestAdditionalItem(void (^completion)(NSArray <UIDragItem *> *));
 
     // These helper methods are unique to UIDropInteraction.
     void dropSessionDidEnterOrUpdate(id <UIDropSession>, const WebCore::DragData&);
@@ -86,6 +88,7 @@ public:
     id<UIDropSession> dropSession() const { return m_dropSession.get(); }
     BlockPtr<void()> takeDragStartCompletionBlock() { return WTFMove(m_dragStartCompletionBlock); }
     BlockPtr<void()> takeDragCancelSetDownBlock() { return WTFMove(m_dragCancelSetDownBlock); }
+    BlockPtr<void(NSArray<UIDragItem *> *)> takeAddDragItemCompletionBlock() { return WTFMove(m_addDragItemCompletionBlock); }
 
 private:
     void updatePreviewsForActiveDragSources();
@@ -99,6 +102,7 @@ private:
     RetainPtr<id <UIDropSession>> m_dropSession;
     BlockPtr<void()> m_dragStartCompletionBlock;
     BlockPtr<void()> m_dragCancelSetDownBlock;
+    BlockPtr<void(NSArray<UIDragItem *> *)> m_addDragItemCompletionBlock;
 
     std::optional<DragSourceState> m_stagedDragSource;
     Vector<DragSourceState> m_activeDragSources;
index 73ff07b..e52dba9 100644 (file)
@@ -167,6 +167,17 @@ void DragDropInteractionState::dragSessionWillDelaySetDownAnimation(dispatch_blo
     m_dragCancelSetDownBlock = completion;
 }
 
+bool DragDropInteractionState::shouldRequestAdditionalItemForDragSession(id <UIDragSession> session) const
+{
+    return m_dragSession == session && !m_addDragItemCompletionBlock && !m_dragStartCompletionBlock;
+}
+
+void DragDropInteractionState::dragSessionWillRequestAdditionalItem(void (^completion)(NSArray <UIDragItem *> *))
+{
+    clearStagedDragSource();
+    m_addDragItemCompletionBlock = completion;
+}
+
 void DragDropInteractionState::dropSessionDidEnterOrUpdate(id <UIDropSession> session, const DragData& dragData)
 {
     m_dropSession = session;
@@ -210,6 +221,9 @@ void DragDropInteractionState::dragAndDropSessionsDidEnd()
     if (auto completionBlock = takeDragCancelSetDownBlock())
         completionBlock();
 
+    if (auto completionBlock = takeAddDragItemCompletionBlock())
+        completionBlock(@[ ]);
+
     if (auto completionBlock = takeDragStartCompletionBlock())
         completionBlock();
 }
index 3c3086b..7f9be7b 100644 (file)
@@ -205,6 +205,7 @@ private:
 #if ENABLE(DATA_INTERACTION)
     void didPerformDataInteractionControllerOperation(bool handled) override;
     void didHandleStartDataInteractionRequest(bool started) override;
+    void didHandleAdditionalDragItemsRequest(bool added) override;
     void startDrag(const WebCore::DragItem&, const ShareableBitmap::Handle& image) override;
     void didConcludeEditDataInteraction(std::optional<WebCore::TextIndicatorData>) override;
     void didChangeDataInteractionCaretRect(const WebCore::IntRect& previousCaretRect, const WebCore::IntRect& caretRect) override;
index 0f2b774..bd0fee4 100644 (file)
@@ -786,6 +786,11 @@ void PageClientImpl::didHandleStartDataInteractionRequest(bool started)
     [m_contentView _didHandleStartDataInteractionRequest:started];
 }
 
+void PageClientImpl::didHandleAdditionalDragItemsRequest(bool added)
+{
+    [m_contentView _didHandleAdditionalDragItemsRequest:added];
+}
+
 void PageClientImpl::startDrag(const DragItem& item, const ShareableBitmap::Handle& image)
 {
     [m_contentView _startDrag:ShareableBitmap::create(image)->makeCGImageCopy() item:item];
index 9f703d8..7df9bd2 100644 (file)
@@ -321,6 +321,7 @@ FOR_EACH_WKCONTENTVIEW_ACTION(DECLARE_WKCONTENTVIEW_ACTION_FOR_WEB_VIEW)
 - (void)_didChangeDragInteractionPolicy;
 - (void)_didPerformDataInteractionControllerOperation:(BOOL)handled;
 - (void)_didHandleStartDataInteractionRequest:(BOOL)started;
+- (void)_didHandleAdditionalDragItemsRequest:(BOOL)added;
 - (void)_startDrag:(RetainPtr<CGImageRef>)image item:(const WebCore::DragItem&)item;
 - (void)_didConcludeEditDataInteraction:(std::optional<WebCore::TextIndicatorData>)data;
 - (void)_didChangeDataInteractionCaretRect:(CGRect)previousRect currentRect:(CGRect)rect;
@@ -333,6 +334,7 @@ FOR_EACH_WKCONTENTVIEW_ACTION(DECLARE_WKCONTENTVIEW_ACTION_FOR_WEB_VIEW)
 - (void)_simulateWillBeginDataInteractionWithSession:(id)session;
 - (NSArray *)_simulatedItemsForSession:(id)session;
 - (void)_simulatePrepareForDataInteractionSession:(id)session completion:(dispatch_block_t)completion;
+- (void)_simulateItemsForAddingToSession:(id)session atLocation:(CGPoint)location completion:(void(^)(NSArray *))completion;
 #endif
 
 - (void)_simulateLongPressActionAtLocation:(CGPoint)location;
index a968961..1063ee2 100644 (file)
@@ -4259,6 +4259,31 @@ static BOOL shouldEnableDragInteractionForPolicy(_WKDragInteractionPolicy policy
     _dragDropInteractionState.stageDragItem(item, dragImage.get());
 }
 
+- (void)_didHandleAdditionalDragItemsRequest:(BOOL)added
+{
+    auto completion = _dragDropInteractionState.takeAddDragItemCompletionBlock();
+    if (!completion)
+        return;
+
+    WebItemProviderRegistrationInfoList *registrationList = [[WebItemProviderPasteboard sharedInstance] takeRegistrationList];
+    if (!added || !registrationList || !_dragDropInteractionState.hasStagedDragSource()) {
+        _dragDropInteractionState.clearStagedDragSource();
+        completion(@[ ]);
+        return;
+    }
+
+    auto stagedDragSource = _dragDropInteractionState.stagedDragSource();
+    NSArray *dragItemsToAdd = [self _itemsForBeginningOrAddingToSessionWithRegistrationList:registrationList stagedDragSource:stagedDragSource];
+
+    RELEASE_LOG(DragAndDrop, "Drag session: %p adding %tu items", _dragDropInteractionState.dragSession(), dragItemsToAdd.count);
+    _dragDropInteractionState.clearStagedDragSource(dragItemsToAdd.count ? DragDropInteractionState::DidBecomeActive::Yes : DragDropInteractionState::DidBecomeActive::No);
+
+    completion(dragItemsToAdd);
+
+    if (dragItemsToAdd.count)
+        _page->didStartDrag();
+}
+
 - (void)_didHandleStartDataInteractionRequest:(BOOL)started
 {
     BlockPtr<void()> savedCompletionBlock = _dragDropInteractionState.takeDragStartCompletionBlock();
@@ -4454,6 +4479,37 @@ static NSArray<UIItemProvider *> *extractItemProvidersFromDropSession(id <UIDrop
     _shouldRestoreCalloutBarAfterDrop = NO;
 }
 
+- (NSArray<UIDragItem *> *)_itemsForBeginningOrAddingToSessionWithRegistrationList:(WebItemProviderRegistrationInfoList *)registrationList stagedDragSource:(const DragSourceState&)stagedDragSource
+{
+    UIItemProvider *defaultItemProvider = registrationList.itemProvider;
+    if (!defaultItemProvider)
+        return @[ ];
+
+    NSArray *adjustedItemProviders;
+    id <WKUIDelegatePrivate> uiDelegate = self.webViewUIDelegate;
+    if ([uiDelegate respondsToSelector:@selector(_webView:adjustedDataInteractionItemProvidersForItemProvider:representingObjects:additionalData:)]) {
+        auto representingObjects = adoptNS([[NSMutableArray alloc] init]);
+        auto additionalData = adoptNS([[NSMutableDictionary alloc] init]);
+        [registrationList enumerateItems:[representingObjects, additionalData] (WebItemProviderRegistrationInfo *item, NSUInteger) {
+            if (item.representingObject)
+                [representingObjects addObject:item.representingObject];
+            if (item.typeIdentifier && item.data)
+                [additionalData setObject:item.data forKey:item.typeIdentifier];
+        }];
+        adjustedItemProviders = [uiDelegate _webView:_webView adjustedDataInteractionItemProvidersForItemProvider:defaultItemProvider representingObjects:representingObjects.get() additionalData:additionalData.get()];
+    } else
+        adjustedItemProviders = @[ defaultItemProvider ];
+
+    NSMutableArray *dragItems = [NSMutableArray arrayWithCapacity:adjustedItemProviders.count];
+    for (UIItemProvider *itemProvider in adjustedItemProviders) {
+        auto item = adoptNS([[UIDragItem alloc] initWithItemProvider:itemProvider]);
+        [item _setPrivateLocalContext:@(stagedDragSource.itemIdentifier)];
+        [dragItems addObject:item.autorelease()];
+    }
+
+    return dragItems;
+}
+
 #pragma mark - UIDragInteractionDelegate
 
 - (BOOL)_dragInteraction:(UIDragInteraction *)interaction shouldDelayCompetingGestureRecognizer:(UIGestureRecognizer *)competingGestureRecognizer
@@ -4477,6 +4533,17 @@ static NSArray<UIItemProvider *> *extractItemProvidersFromDropSession(id <UIDrop
     return dataOwner;
 }
 
+- (void)_dragInteraction:(UIDragInteraction *)interaction itemsForAddingToSession:(id <UIDragSession>)session withTouchAtPoint:(CGPoint)point completion:(void(^)(NSArray<UIDragItem *> *))completion
+{
+    if (!_dragDropInteractionState.shouldRequestAdditionalItemForDragSession(session)) {
+        completion(@[ ]);
+        return;
+    }
+
+    _dragDropInteractionState.dragSessionWillRequestAdditionalItem(completion);
+    _page->requestAdditionalItemsForDragSession(roundedIntPoint(point), roundedIntPoint(point));
+}
+
 - (void)_dragInteraction:(UIDragInteraction *)interaction prepareForSession:(id <UIDragSession>)session completion:(dispatch_block_t)completion
 {
     [self _cancelLongPressGestureRecognizer];
@@ -4515,37 +4582,7 @@ static NSArray<UIItemProvider *> *extractItemProvidersFromDropSession(id <UIDrop
 
     auto stagedDragSource = _dragDropInteractionState.stagedDragSource();
     WebItemProviderRegistrationInfoList *registrationList = [[WebItemProviderPasteboard sharedInstance] takeRegistrationList];
-    UIItemProvider *defaultItemProvider = registrationList.itemProvider;
-    if (!defaultItemProvider) {
-        RELEASE_LOG(DragAndDrop, "Drag session failed: %p (no item providers generated before adjustment)", session);
-        _page->dragCancelled();
-        _dragDropInteractionState.clearStagedDragSource();
-        return @[ ];
-    }
-
-    // Give internal clients such as Mail one final chance to augment the contents of each UIItemProvider before sending the drag items off to UIKit.
-    NSArray *adjustedItemProviders;
-    id <WKUIDelegatePrivate> uiDelegate = self.webViewUIDelegate;
-    if ([uiDelegate respondsToSelector:@selector(_webView:adjustedDataInteractionItemProvidersForItemProvider:representingObjects:additionalData:)]) {
-        auto representingObjects = adoptNS([[NSMutableArray alloc] init]);
-        auto additionalData = adoptNS([[NSMutableDictionary alloc] init]);
-        [registrationList enumerateItems:[representingObjects, additionalData] (WebItemProviderRegistrationInfo *item, NSUInteger) {
-            if (item.representingObject)
-                [representingObjects addObject:item.representingObject];
-            if (item.typeIdentifier && item.data)
-                [additionalData setObject:item.data forKey:item.typeIdentifier];
-        }];
-        adjustedItemProviders = [uiDelegate _webView:_webView adjustedDataInteractionItemProvidersForItemProvider:defaultItemProvider representingObjects:representingObjects.get() additionalData:additionalData.get()];
-    } else
-        adjustedItemProviders = @[ defaultItemProvider ];
-
-    NSMutableArray *dragItems = [NSMutableArray arrayWithCapacity:adjustedItemProviders.count];
-    for (UIItemProvider *itemProvider in adjustedItemProviders) {
-        auto item = adoptNS([[UIDragItem alloc] initWithItemProvider:itemProvider]);
-        [item _setPrivateLocalContext:@(stagedDragSource.itemIdentifier)];
-        [dragItems addObject:item.autorelease()];
-    }
-
+    NSArray *dragItems = [self _itemsForBeginningOrAddingToSessionWithRegistrationList:registrationList stagedDragSource:stagedDragSource];
     if (![dragItems count])
         _page->dragCancelled();
 
@@ -4828,6 +4865,11 @@ static NSArray<UIItemProvider *> *extractItemProvidersFromDropSession(id <UIDrop
     [self _dragInteraction:_dragInteraction.get() prepareForSession:session completion:completion];
 }
 
+- (void)_simulateItemsForAddingToSession:(id)session atLocation:(CGPoint)location completion:(void(^)(NSArray *))completion
+{
+    [self _dragInteraction:_dragInteraction.get() itemsForAddingToSession:session withTouchAtPoint:location completion:completion];
+}
+
 #endif
 
 - (void)_simulateLongPressActionAtLocation:(CGPoint)location
index 211aac7..4728e0d 100644 (file)
@@ -1094,12 +1094,23 @@ void WebPageProxy::didHandleStartDataInteractionRequest(bool started)
     m_pageClient.didHandleStartDataInteractionRequest(started);
 }
 
+void WebPageProxy::didHandleAdditionalDragItemsRequest(bool added)
+{
+    m_pageClient.didHandleAdditionalDragItemsRequest(added);
+}
+
 void WebPageProxy::requestStartDataInteraction(const WebCore::IntPoint& clientPosition, const WebCore::IntPoint& globalPosition)
 {
     if (isValid())
         m_process->send(Messages::WebPage::RequestStartDataInteraction(clientPosition, globalPosition), m_pageID);
 }
 
+void WebPageProxy::requestAdditionalItemsForDragSession(const IntPoint& clientPosition, const IntPoint& globalPosition)
+{
+    if (isValid())
+        m_process->send(Messages::WebPage::RequestAdditionalItemsForDragSession(clientPosition, globalPosition), m_pageID);
+}
+
 void WebPageProxy::didConcludeEditDataInteraction(std::optional<TextIndicatorData> data)
 {
     m_pageClient.didConcludeEditDataInteraction(data);
index a5959a2..e546993 100644 (file)
@@ -1032,6 +1032,7 @@ private:
     RefPtr<WebCore::Range> switchToBlockSelectionAtPoint(const WebCore::IntPoint&, SelectionHandlePosition);
 #if ENABLE(DATA_INTERACTION)
     void requestStartDataInteraction(const WebCore::IntPoint& clientPosition, const WebCore::IntPoint& globalPosition);
+    void requestAdditionalItemsForDragSession(const WebCore::IntPoint& clientPosition, const WebCore::IntPoint& globalPosition);
 #endif
 #endif
 
index 9c71066..0521aa2 100644 (file)
@@ -253,6 +253,7 @@ messages -> WebPage LegacyReceiver {
 
 #if ENABLE(DATA_INTERACTION)
     RequestStartDataInteraction(WebCore::IntPoint clientPosition, WebCore::IntPoint globalPosition)
+    RequestAdditionalItemsForDragSession(WebCore::IntPoint clientPosition, WebCore::IntPoint globalPosition)
 #endif
 
     # Popup menu.
index e0dd7f9..4779161 100644 (file)
@@ -631,6 +631,13 @@ void WebPage::requestStartDataInteraction(const IntPoint& clientPosition, const
     send(Messages::WebPageProxy::DidHandleStartDataInteractionRequest(didStart));
 }
 
+void WebPage::requestAdditionalItemsForDragSession(const IntPoint& clientPosition, const IntPoint& globalPosition)
+{
+    notImplemented();
+
+    send(Messages::WebPageProxy::DidHandleAdditionalDragItemsRequest(false));
+}
+
 void WebPage::didConcludeEditDataInteraction()
 {
     std::optional<TextIndicatorData> textIndicatorData;
index 981e942..6e34073 100644 (file)
@@ -1,3 +1,51 @@
+2017-09-05  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        [iOS WK2] Add plumbing for WKContentView to ask the web process for additional drag items
+        https://bugs.webkit.org/show_bug.cgi?id=176348
+        Work towards <rdar://problem/31144674>
+
+        Reviewed by Tim Horton.
+
+        Introduces new test infrastructure to simulate tapping to add additional items to the current drag session. We
+        add -[DataInteractionSimulator runFrom:to:additionalItemRequestLocations:], for which the last argument is a
+        dictionary mapping progress (a double between 0 and 1) to NSValues representing the location from which the drag
+        and drop simulation will request an additional drag item. During the simulated drag and drop, when the progress
+        value exceeds a value in the map of remaining additional item request locations, we halt drag simulation
+        progress for that runloop and instead request additional items from the location specified.
+
+        The only (useful) passing test we can create using the new machinery is one that verifies that preventDefault()
+        on dragstart works as expected, preventing additional items from being added. While this trivially passes now,
+        since the web-process-side of the additional items flow is not yet implemented, it should continue to pass after
+        the web process portion is implemented.
+
+        * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
+        * TestWebKitAPI/Tests/WebKitCocoa/selected-text-image-link-and-editable.html: Added.
+
+        Add a new test page that contains some non-editable text, an image, a link, and a rich text editable area.
+
+        * TestWebKitAPI/Tests/ios/DataInteractionTests.mm:
+        (TestWebKitAPI::TEST):
+        * TestWebKitAPI/ios/DataInteractionSimulator.h:
+        * TestWebKitAPI/ios/DataInteractionSimulator.mm:
+        (-[MockDragDropSession addItems:]):
+
+        Convenience method to add additional mock drag items to a drag or drop session.
+
+        (-[DataInteractionSimulator _resetSimulatedState]):
+        (-[DataInteractionSimulator runFrom:to:]):
+
+        Converted into a convenience wrapper around the latter version, passing in nil for the additional item request
+        locations dictionary.
+
+        (-[DataInteractionSimulator runFrom:to:additionalItemRequestLocations:]):
+        (-[DataInteractionSimulator _enqueuePendingAdditionalItemRequestLocations]):
+        (-[DataInteractionSimulator _sendQueuedAdditionalItemRequest]):
+
+        New helper methods to detect when and where additional items should be "added", and subsequently simulate adding
+        drag items at these given locations.
+
+        (-[DataInteractionSimulator _advanceProgress]):
+
 2017-09-05  Alex Christensen  <achristensen@webkit.org>
 
         Add WKUIDelegatePrivate equivalent of WKPageUIClient's mouseDidMoveOverElement
index 260caa2..6fca682 100644 (file)
                F4D4F3B61E4E2BCB00BB2767 /* DataInteractionSimulator.mm in Sources */ = {isa = PBXBuildFile; fileRef = F4D4F3B41E4E2BCB00BB2767 /* DataInteractionSimulator.mm */; };
                F4D4F3B91E4E36E400BB2767 /* DataInteractionTests.mm in Sources */ = {isa = PBXBuildFile; fileRef = F4D4F3B71E4E36E400BB2767 /* DataInteractionTests.mm */; };
                F4D5E4E81F0C5D38008C1A49 /* dragstart-clear-selection.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F4D5E4E71F0C5D27008C1A49 /* dragstart-clear-selection.html */; };
+               F4D65DA81F5E4704009D8C27 /* selected-text-image-link-and-editable.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F4D65DA71F5E46C0009D8C27 /* selected-text-image-link-and-editable.html */; };
                F4DEF6ED1E9B4DB60048EF61 /* image-in-link-and-input.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F4DEF6EC1E9B4D950048EF61 /* image-in-link-and-input.html */; };
                F4F137921D9B683E002BEC57 /* large-video-test-now-playing.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F4F137911D9B6832002BEC57 /* large-video-test-now-playing.html */; };
                F4F405BC1D4C0D1C007A9707 /* full-size-autoplaying-video-with-audio.html in Copy Resources */ = {isa = PBXBuildFile; fileRef = F4F405BA1D4C0CF8007A9707 /* full-size-autoplaying-video-with-audio.html */; };
                                F6FDDDD614241C6F004F1729 /* push-state.html in Copy Resources */,
                                A12DDC001E8373E700CF6CAE /* rendered-image-excluding-overflow.html in Copy Resources */,
                                F46849C01EEF5EF300B937FE /* rich-and-plain-text.html in Copy Resources */,
+                               F4D65DA81F5E4704009D8C27 /* selected-text-image-link-and-editable.html in Copy Resources */,
                                7A66BDB81EAF18D500CCC924 /* set-long-title.html in Copy Resources */,
                                52B8CF9815868D9100281053 /* SetDocumentURI.html in Copy Resources */,
                                CEBABD491B71687C0051210A /* should-open-external-schemes.html in Copy Resources */,
                F4D4F3B51E4E2BCB00BB2767 /* DataInteractionSimulator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DataInteractionSimulator.h; sourceTree = "<group>"; };
                F4D4F3B71E4E36E400BB2767 /* DataInteractionTests.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = DataInteractionTests.mm; sourceTree = "<group>"; };
                F4D5E4E71F0C5D27008C1A49 /* dragstart-clear-selection.html */ = {isa = PBXFileReference; lastKnownFileType = text.html; path = "dragstart-clear-selection.html"; sourceTree = "<group>"; };
+               F4D65DA71F5E46C0009D8C27 /* selected-text-image-link-and-editable.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "selected-text-image-link-and-editable.html"; sourceTree = "<group>"; };
                F4DEF6EC1E9B4D950048EF61 /* image-in-link-and-input.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "image-in-link-and-input.html"; sourceTree = "<group>"; };
                F4F137911D9B6832002BEC57 /* large-video-test-now-playing.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "large-video-test-now-playing.html"; sourceTree = "<group>"; };
                F4F405BA1D4C0CF8007A9707 /* full-size-autoplaying-video-with-audio.html */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.html; path = "full-size-autoplaying-video-with-audio.html"; sourceTree = "<group>"; };
                                F41AB99A1EF4692C0083FA08 /* prevent-start.html */,
                                A12DDBFF1E8373C100CF6CAE /* rendered-image-excluding-overflow.html */,
                                F46849BF1EEF5EDC00B937FE /* rich-and-plain-text.html */,
+                               F4D65DA71F5E46C0009D8C27 /* selected-text-image-link-and-editable.html */,
                                C9B4AD291ECA6EA500F5FEA0 /* silence-long.m4a */,
                                F4F405BB1D4C0CF8007A9707 /* skinny-autoplaying-video-with-audio.html */,
                                515BE16E1D4288FF00DD7C68 /* StoreBlobToBeDeleted.html */,
diff --git a/Tools/TestWebKitAPI/Tests/WebKitCocoa/selected-text-image-link-and-editable.html b/Tools/TestWebKitAPI/Tests/WebKitCocoa/selected-text-image-link-and-editable.html
new file mode 100644 (file)
index 0000000..c029f6f
--- /dev/null
@@ -0,0 +1,34 @@
+<!DOCTYPE html>
+<html>
+<head>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+html, body {
+    width: 100%;
+    height: 100%;
+    margin: 0;
+}
+
+#text, #image, #link {
+    height: 100px;
+    font-size: 80px;
+    white-space: nowrap;
+}
+
+#editor {
+    border: black 1px solid;
+    height: calc(100% - 300px);
+}
+</style>
+</head>
+
+<body>
+<div id="text">ABCD</div>
+<div><img id="image" src="apple.gif"></img></div>
+<div><a id="link" href="https://www.apple.com/">A link</a></div>
+<div contenteditable id="editor"></div>
+<script>
+getSelection().setBaseAndExtent(text, 0, text, 1);
+</script>
+</body>
+</html>
index 3f73bf0..55b0bbd 100644 (file)
@@ -1234,6 +1234,21 @@ TEST(DataInteractionTests, DoNotCrashWhenSelectionMovesOffscreenAfterDragStart)
     EXPECT_WK_STREQ("FAR OFFSCREEN", [webView stringByEvaluatingJavaScript:@"getSelection().getRangeAt(0).toString()"]);
 }
 
+TEST(DataInteractionTests, AdditionalItemsCanBePreventedOnDragStart)
+{
+    auto webView = adoptNS([[TestWKWebView alloc] initWithFrame:CGRectMake(0, 0, 320, 500)]);
+    [webView synchronouslyLoadTestPageNamed:@"selected-text-image-link-and-editable"];
+    [webView stringByEvaluatingJavaScript:@"link.addEventListener('dragstart', e => e.preventDefault())"];
+    [webView stringByEvaluatingJavaScript:@"image.addEventListener('dragstart', e => e.preventDefault())"];
+
+    auto simulator = adoptNS([[DataInteractionSimulator alloc] initWithWebView:webView.get()]);
+    [simulator runFrom:CGPointMake(50, 50) to:CGPointMake(50, 400) additionalItemRequestLocations:@{
+        @0.33: [NSValue valueWithCGPoint:CGPointMake(50, 150)],
+        @0.66: [NSValue valueWithCGPoint:CGPointMake(50, 250)]
+    }];
+    EXPECT_WK_STREQ("ABCD", [webView stringByEvaluatingJavaScript:@"editor.textContent"]);
+}
+
 } // namespace TestWebKitAPI
 
 #endif // ENABLE(DATA_INTERACTION)
index 0a0d50d..a1332de 100644 (file)
@@ -86,6 +86,8 @@ extern NSString * const DataInteractionPerformOperationEventName;
 extern NSString * const DataInteractionLeaveEventName;
 extern NSString * const DataInteractionStartEventName;
 
+typedef NSDictionary<NSNumber *, NSValue *> *ProgressToCGPointValueMap;
+
 typedef NS_ENUM(NSInteger, DataInteractionPhase) {
     DataInteractionCancelled = 0,
     DataInteractionBeginning = 1,
@@ -107,6 +109,9 @@ typedef NS_ENUM(NSInteger, DataInteractionPhase) {
     CGPoint _endLocation;
     CGRect _lastKnownDragCaretRect;
 
+    RetainPtr<NSMutableDictionary<NSNumber *, NSValue *>>_remainingAdditionalItemRequestLocationsByProgress;
+    RetainPtr<NSMutableArray<NSValue *>>_queuedAdditionalItemRequestLocations;
+
     bool _isDoneWaitingForInputSession;
     BOOL _shouldPerformOperation;
     double _currentProgress;
@@ -115,7 +120,9 @@ typedef NS_ENUM(NSInteger, DataInteractionPhase) {
 }
 
 - (instancetype)initWithWebView:(TestWKWebView *)webView;
+// The start location, end location, and locations of additional item requests are all in window coordinates.
 - (void)runFrom:(CGPoint)startLocation to:(CGPoint)endLocation;
+- (void)runFrom:(CGPoint)startLocation to:(CGPoint)endLocation additionalItemRequestLocations:(ProgressToCGPointValueMap)additionalItemRequestLocations;
 - (void)waitForInputSession;
 
 @property (nonatomic) BOOL allowsFocusToStartInputSession;
index cc4af79..a4eefd1 100644 (file)
@@ -133,6 +133,17 @@ using namespace TestWebKitAPI;
     _mockItems = items;
 }
 
+- (void)addItems:(NSArray<UIDragItem *> *)items
+{
+    if (![items count])
+        return;
+
+    if (![_mockItems count])
+        _mockItems = items;
+    else
+        _mockItems = [_mockItems arrayByAddingObjectsFromArray:items];
+}
+
 - (CGPoint)locationInView:(UIView *)view
 {
     return [_window convertPoint:_mockLocationInWindow toView:view];
@@ -307,6 +318,8 @@ static NSArray *dataInteractionEventNames()
     _dataOperationSession = nil;
     _shouldPerformOperation = NO;
     _lastKnownDragCaretRect = CGRectZero;
+    _remainingAdditionalItemRequestLocationsByProgress = nil;
+    _queuedAdditionalItemRequestLocations = adoptNS([[NSMutableArray alloc] init]);
 }
 
 - (NSArray *)observedEventNames
@@ -326,6 +339,11 @@ static NSArray *dataInteractionEventNames()
 
 - (void)runFrom:(CGPoint)startLocation to:(CGPoint)endLocation
 {
+    [self runFrom:startLocation to:endLocation additionalItemRequestLocations:nil];
+}
+
+- (void)runFrom:(CGPoint)startLocation to:(CGPoint)endLocation additionalItemRequestLocations:(ProgressToCGPointValueMap)additionalItemRequestLocations
+{
     NSNotificationCenter *defaultCenter = [NSNotificationCenter defaultCenter];
     [defaultCenter addObserver:self selector:@selector(simulateAllTouchesCanceled:) name:TestWebKitAPISimulateCancelAllTouchesNotificationName object:nil];
 
@@ -334,6 +352,9 @@ static NSArray *dataInteractionEventNames()
 
     [self _resetSimulatedState];
 
+    if (additionalItemRequestLocations)
+        _remainingAdditionalItemRequestLocationsByProgress = adoptNS([additionalItemRequestLocations mutableCopy]);
+
     RetainPtr<DataInteractionSimulator> strongSelf = self;
     for (NSString *eventName in dataInteractionEventNames()) {
         DataInteractionSimulator *weakSelf = strongSelf.get();
@@ -391,8 +412,45 @@ static NSArray *dataInteractionEventNames()
         [_webView _simulateDataInteractionSessionDidEnd:_dataInteractionSession.get()];
 }
 
+- (void)_enqueuePendingAdditionalItemRequestLocations
+{
+    NSMutableArray *progressValuesToRemove = [NSMutableArray array];
+    for (NSNumber *progressValue in _remainingAdditionalItemRequestLocationsByProgress.get()) {
+        double progress = progressValue.doubleValue;
+        if (progress > _currentProgress)
+            continue;
+        [progressValuesToRemove addObject:progressValue];
+        [_queuedAdditionalItemRequestLocations addObject:[_remainingAdditionalItemRequestLocationsByProgress objectForKey:progressValue]];
+    }
+
+    for (NSNumber *progressToRemove in progressValuesToRemove)
+        [_remainingAdditionalItemRequestLocationsByProgress removeObjectForKey:progressToRemove];
+}
+
+- (BOOL)_sendQueuedAdditionalItemRequest
+{
+    if (![_queuedAdditionalItemRequestLocations count])
+        return NO;
+
+    RetainPtr<NSValue> requestLocationValue = [_queuedAdditionalItemRequestLocations objectAtIndex:0];
+    [_queuedAdditionalItemRequestLocations removeObjectAtIndex:0];
+
+    auto requestLocation = [[_webView window] convertPoint:[requestLocationValue CGPointValue] toView:_webView.get()];
+    [_webView _simulateItemsForAddingToSession:_dataInteractionSession.get() atLocation:requestLocation completion:[dragSession = _dataInteractionSession, dropSession = _dataOperationSession] (NSArray *items) {
+        [dragSession addItems:items];
+        [dropSession addItems:items];
+    }];
+    return YES;
+}
+
 - (void)_advanceProgress
 {
+    [self _enqueuePendingAdditionalItemRequestLocations];
+    if ([self _sendQueuedAdditionalItemRequest]) {
+        [self _scheduleAdvanceProgress];
+        return;
+    }
+
     _lastKnownDragCaretRect = [_webView _dragCaretRect];
     _currentProgress += progressIncrementStep;
     CGPoint locationInWindow = self._currentLocation;