Web Automation: elements larger than the viewport have incorrect in-view center point
authordrousso@apple.com <drousso@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 15 May 2019 08:37:02 +0000 (08:37 +0000)
committerdrousso@apple.com <drousso@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Wed, 15 May 2019 08:37:02 +0000 (08:37 +0000)
commitaf779fbd37b2d0145aba071edb9f1158032ff04b
tree42bbfaeeb6f1723c1b944d78c914ff88e4dd1562
parent4459934a3d5364f9908131bfa7c77e4e68d5e725
Web Automation: elements larger than the viewport have incorrect in-view center point
https://bugs.webkit.org/show_bug.cgi?id=195696
<rdar://problem/48737122>

Reviewed by Simon Fraser.

Original patch by Brian Burg <bburg@apple.com>.

Source/WebCore:

Some conversion methods do not exist for `FloatRect`/`FloatPoint`. Fill them in as needed,
and export some symbols used by WebDriver code to compute an element's in-view center point
in various coordinate systems.

* dom/TreeScope.h:
* dom/TreeScope.cpp:
(WebCore::TreeScope::elementsFromPoint): Added.
* page/FrameView.h:
* page/FrameView.cpp:
(WebCore::FrameView::absoluteToLayoutViewportPoint const): Added.
(WebCore::FrameView::layoutViewportToAbsoluteRect const): Added.
(WebCore::FrameView::absoluteToLayoutViewportRect const): Added.
* platform/ScrollView.h:
* platform/ScrollView.cpp:
(WebCore::ScrollView::viewToContents const): Added.
(WebCore::ScrollView::contentsToView const): Added.
(WebCore::ScrollView::contentsToRootView const): Added.
* platform/Widget.h:
* platform/Widget.cpp:
(WebCore::Widget::convertToRootView const): Added.
(WebCore::Widget::convertFromRootView const): Added.
(WebCore::Widget::convertToContainingView const): Added.
(WebCore::Widget::convertFromContainingView const): Added.

Source/WebKit:

This seems to be an omission in the specification. While it does mention that the in-view
center point (IVCP) must be within the viewport, the algorithm never intersects the element
bounding box with the viewport rect.

* WebProcess/Automation/WebAutomationSessionProxy.cpp:
(WebKit::WebAutomationSessionProxy::computeElementLayout):
This code is incorrect. For `CoordinateSystem::LayoutViewport`, coordinates should be in
root view coordinates so that it can be later converted to screen and synthesized as a HID
event in screen coordinates. Intersect the element rect and the viewport rect before finding
the center point of the part of the element that's visible in the viewport.

(WebKit::convertRectFromFrameClientToRootView): Added.
(WebKit::convertPointFromFrameClientToRootView): Added.
Added helpers to properly account for scroll contents position on iOS.

* UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::viewportInViewCenterPointOfElement):
Now that we determine whether the element is inside the viewport much earlier, if the
element has no `inViewCenterPoint`, we can return a `TargetOutOfBounds` instead of a more
"generic" `ElementNotInteractable`.

(WebKit::WebAutomationSession::simulateMouseInteraction):
Rename `locationInView` -> `locationInViewport`.

(WebKit::WebAutomationSession::simulateTouchInteraction):
This code is incorrect. The `unobscuredContentRect` is in screen coordinates, but
we are trying to see if (x, y) is outside the size of the viewport assumed to be at (0, 0).
Grab the visual viewport rect and see if the location exceeds the viewport size.

* UIProcess/Automation/ios/WebAutomationSessionIOS.mm:
(WebKit::operator<<):
Add logging helper for `TouchInteraction` enum.

(WebKit::WebAutomationSession::platformSimulateTouchInteraction):
Move local variable.

* UIProcess/Automation/SimulatedInputDispatcher.cpp:
(WebKit::SimulatedInputDispatcher::transitionInputSourceToState):
Fix a typo in logging.

* UIProcess/Automation/Automation.json:
Simplify enum name.

* Platform/Logging.h:
Add logging channel to dump fully resolved interaction details.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@245320 268f45cc-cd09-0410-ab3c-d52691b4dbfc
16 files changed:
Source/WebCore/ChangeLog
Source/WebCore/dom/TreeScope.cpp
Source/WebCore/dom/TreeScope.h
Source/WebCore/page/FrameView.cpp
Source/WebCore/page/FrameView.h
Source/WebCore/platform/ScrollView.cpp
Source/WebCore/platform/ScrollView.h
Source/WebCore/platform/Widget.cpp
Source/WebCore/platform/Widget.h
Source/WebKit/ChangeLog
Source/WebKit/Platform/Logging.h
Source/WebKit/UIProcess/Automation/Automation.json
Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp
Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp
Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm
Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp