WebDriver: implement advance user interactions
authorcarlosgc@webkit.org <carlosgc@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 10 May 2018 06:52:31 +0000 (06:52 +0000)
committercarlosgc@webkit.org <carlosgc@webkit.org@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 10 May 2018 06:52:31 +0000 (06:52 +0000)
https://bugs.webkit.org/show_bug.cgi?id=174616

Reviewed by Brian Burg.

Source/WebDriver:

Add initial implementation of action commands.

* Actions.h: Added.
(WebDriver::Action::Action):
* CommandResult.cpp:
(WebDriver::CommandResult::CommandResult): Handle MoveTargetOutOfBounds error.
(WebDriver::CommandResult::httpStatusCode const): Ditto.
(WebDriver::CommandResult::errorString const): Ditto.
* CommandResult.h:
* Session.cpp:
(WebDriver::Session::webElementIdentifier): Helper to return the web element id.
(WebDriver::Session::createElement): Use webElementIdentifier().
(WebDriver::Session::extractElementID): Ditto.
(WebDriver::Session::virtualKeyForKeySequence): Add more kay codes includes in the spec.
(WebDriver::mouseButtonForAutomation): Helper to get the mouse button string to pass to automation.
(WebDriver::Session::performMouseInteraction): Use mouseButtonForAutomation().
(WebDriver::Session::getOrCreateInputSource): Ensure an input source for given id and add it to the active input
sources.
(WebDriver::Session::inputSourceState): Return the current input source state for the given id.
(WebDriver::Session::computeInViewCenterPointOfElements): Get the in view center point for the list of elements given.
(WebDriver::automationSourceType): Helper to get the input source type to pass to automation.
(WebDriver::Session::performActions): Process the list of action by tick and generate a list of states to pass
to automation.
(WebDriver::Session::releaseActions): Reset input sources and state table and send a message to automation.
* Session.h:
* WebDriverService.cpp:
(WebDriver::processPauseAction):
(WebDriver::processNullAction):
(WebDriver::processKeyAction):
(WebDriver::actionMouseButton):
(WebDriver::processPointerAction):
(WebDriver::processPointerParameters):
(WebDriver::processInputActionSequence):
(WebDriver::WebDriverService::performActions):
(WebDriver::WebDriverService::releaseActions):
* WebDriverService.h:

Source/WebKit:

Handle origin in case of mouse move transitions.

* UIProcess/Automation/Automation.json: Add MouseMoveOrigin enum and pass it as parameter of InputSourceState
together with optional node handle. Also pass the frame handle to performInteractionSequence command to find the
node in the current browsing context.
* UIProcess/Automation/SimulatedInputDispatcher.cpp:
(WebKit::SimulatedInputKeyFrame::keyFrameToResetInputSources): Ensure we reset the location.
(WebKit::SimulatedInputDispatcher::resolveLocation): Helper to resolve destination location based on current
location and mouse move origin.
(WebKit::SimulatedInputDispatcher::transitionInputSourceToState): Use resolveLocation() in mouse transitions.
(WebKit::SimulatedInputDispatcher::run): Receive and save the frame ID.
(WebKit::SimulatedInputDispatcher::finishDispatching): Reset the frame ID.
* UIProcess/Automation/SimulatedInputDispatcher.h:
* UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::computeElementLayout): Use even numbers for the callback ID to not conflict with
viewportInViewCenterPointOfElement() callbacks.
(WebKit::WebAutomationSession::didComputeElementLayout): Handle computeElementLayout() or
viewportInViewCenterPointOfElement() requests by calling the right callback depending on whether the ID is odd
or even number.
(WebKit::WebAutomationSession::viewportInViewCenterPointOfElement): Send ComputeElementLayout message to the
WebProcess using odd numbers for the callback ID to not conflict with computeElementLayout() callbacks.
(WebKit::WebAutomationSession::performInteractionSequence): Handle the mouse origin and element handle.
(WebKit::WebAutomationSession::cancelInteractionSequence): Pass the frame ID to the input dispatcher.
* UIProcess/Automation/WebAutomationSession.h:
* UIProcess/Automation/WebAutomationSessionMacros.h:

WebDriverTests:

Update test expectations.

* TestExpectations.json:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@231632 268f45cc-cd09-0410-ab3c-d52691b4dbfc

17 files changed:
Source/WebDriver/Actions.h [new file with mode: 0644]
Source/WebDriver/ChangeLog
Source/WebDriver/CommandResult.cpp
Source/WebDriver/CommandResult.h
Source/WebDriver/Session.cpp
Source/WebDriver/Session.h
Source/WebDriver/WebDriverService.cpp
Source/WebDriver/WebDriverService.h
Source/WebKit/ChangeLog
Source/WebKit/UIProcess/Automation/Automation.json
Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp
Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.h
Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp
Source/WebKit/UIProcess/Automation/WebAutomationSession.h
Source/WebKit/UIProcess/Automation/WebAutomationSessionMacros.h
WebDriverTests/ChangeLog
WebDriverTests/TestExpectations.json

diff --git a/Source/WebDriver/Actions.h b/Source/WebDriver/Actions.h
new file mode 100644 (file)
index 0000000..fdc9ac6
--- /dev/null
@@ -0,0 +1,78 @@
+/*
+ * Copyright (C) 2018 Igalia S.L.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include <wtf/text/WTFString.h>
+
+namespace WebDriver {
+
+enum class MouseButton { None, Left, Middle, Right };
+enum class PointerType { Mouse, Pen, Touch };
+
+struct InputSource {
+    enum class Type { None, Key, Pointer };
+
+    Type type;
+    std::optional<PointerType> pointerType;
+};
+
+struct PointerParameters {
+    PointerType pointerType { PointerType::Mouse };
+};
+
+struct PointerOrigin {
+    enum class Type { Viewport, Pointer, Element };
+
+    Type type;
+    std::optional<String> elementID;
+};
+
+struct Action {
+    enum class Type { None, Key, Pointer };
+    enum class Subtype { Pause, PointerUp, PointerDown, PointerMove, PointerCancel, KeyUp, KeyDown };
+
+    Action(const String& id, Type type, Subtype subtype)
+        : id(id)
+        , type(type)
+        , subtype(subtype)
+    {
+    }
+
+    String id;
+    Type type;
+    Subtype subtype;
+    std::optional<unsigned> duration;
+
+    std::optional<PointerType> pointerType;
+    std::optional<MouseButton> button;
+    std::optional<PointerOrigin> origin;
+    std::optional<int64_t> x;
+    std::optional<int64_t> y;
+
+    std::optional<String> key;
+};
+
+} // WebDriver
index 3833c86..9e5e0a0 100644 (file)
@@ -1,3 +1,47 @@
+2018-05-09  Carlos Garcia Campos  <cgarcia@igalia.com>
+
+        WebDriver: implement advance user interactions
+        https://bugs.webkit.org/show_bug.cgi?id=174616
+
+        Reviewed by Brian Burg.
+
+        Add initial implementation of action commands.
+
+        * Actions.h: Added.
+        (WebDriver::Action::Action):
+        * CommandResult.cpp:
+        (WebDriver::CommandResult::CommandResult): Handle MoveTargetOutOfBounds error.
+        (WebDriver::CommandResult::httpStatusCode const): Ditto.
+        (WebDriver::CommandResult::errorString const): Ditto.
+        * CommandResult.h:
+        * Session.cpp:
+        (WebDriver::Session::webElementIdentifier): Helper to return the web element id.
+        (WebDriver::Session::createElement): Use webElementIdentifier().
+        (WebDriver::Session::extractElementID): Ditto.
+        (WebDriver::Session::virtualKeyForKeySequence): Add more kay codes includes in the spec.
+        (WebDriver::mouseButtonForAutomation): Helper to get the mouse button string to pass to automation.
+        (WebDriver::Session::performMouseInteraction): Use mouseButtonForAutomation().
+        (WebDriver::Session::getOrCreateInputSource): Ensure an input source for given id and add it to the active input
+        sources.
+        (WebDriver::Session::inputSourceState): Return the current input source state for the given id.
+        (WebDriver::Session::computeInViewCenterPointOfElements): Get the in view center point for the list of elements given.
+        (WebDriver::automationSourceType): Helper to get the input source type to pass to automation.
+        (WebDriver::Session::performActions): Process the list of action by tick and generate a list of states to pass
+        to automation.
+        (WebDriver::Session::releaseActions): Reset input sources and state table and send a message to automation.
+        * Session.h:
+        * WebDriverService.cpp:
+        (WebDriver::processPauseAction):
+        (WebDriver::processNullAction):
+        (WebDriver::processKeyAction):
+        (WebDriver::actionMouseButton):
+        (WebDriver::processPointerAction):
+        (WebDriver::processPointerParameters):
+        (WebDriver::processInputActionSequence):
+        (WebDriver::WebDriverService::performActions):
+        (WebDriver::WebDriverService::releaseActions):
+        * WebDriverService.h:
+
 2018-03-05  Carlos Garcia Campos  <cgarcia@igalia.com>
 
         WebDriver: Also ignore NoSuchwindow errors when waiting for navigation to complete
index fa8fa88..cb9add5 100644 (file)
@@ -108,6 +108,8 @@ CommandResult::CommandResult(RefPtr<JSON::Value>&& result, std::optional<ErrorCo
             m_errorCode = ErrorCode::UnableToCaptureScreen;
         else if (errorName == "UnexpectedAlertOpen")
             m_errorCode = ErrorCode::UnexpectedAlertOpen;
+        else if (errorName == "TargetOutOfBounds")
+            m_errorCode = ErrorCode::MoveTargetOutOfBounds;
 
         break;
     }
@@ -148,6 +150,7 @@ unsigned CommandResult::httpStatusCode() const
     case ErrorCode::Timeout:
         return 408;
     case ErrorCode::JavascriptError:
+    case ErrorCode::MoveTargetOutOfBounds:
     case ErrorCode::SessionNotCreated:
     case ErrorCode::UnableToCaptureScreen:
     case ErrorCode::UnexpectedAlertOpen:
@@ -201,6 +204,8 @@ String CommandResult::errorString() const
         return ASCIILiteral("timeout");
     case ErrorCode::UnableToCaptureScreen:
         return ASCIILiteral("unable to capture screen");
+    case ErrorCode::MoveTargetOutOfBounds:
+        return ASCIILiteral("move target out of bounds");
     case ErrorCode::UnexpectedAlertOpen:
         return ASCIILiteral("unexpected alert open");
     case ErrorCode::UnknownCommand:
index e5ca8c1..29dda75 100644 (file)
@@ -44,6 +44,7 @@ public:
         InvalidSelector,
         InvalidSessionID,
         JavascriptError,
+        MoveTargetOutOfBounds,
         NoSuchAlert,
         NoSuchCookie,
         NoSuchElement,
index 29e52c2..3d95ab4 100644 (file)
 #include "SessionHost.h"
 #include "WebDriverAtoms.h"
 #include <wtf/CryptographicallyRandomNumber.h>
+#include <wtf/HashSet.h>
 #include <wtf/HexNumber.h>
+#include <wtf/NeverDestroyed.h>
 
 namespace WebDriver {
 
-// The web element identifier is a constant defined by the spec in Section 11 Elements.
-// https://www.w3.org/TR/webdriver/#elements
-static const String webElementIdentifier = ASCIILiteral("element-6066-11e4-a52e-4f735466cecf");
-
 // https://w3c.github.io/webdriver/webdriver-spec.html#dfn-session-script-timeout
 static const Seconds defaultScriptTimeout = 30_s;
 // https://w3c.github.io/webdriver/webdriver-spec.html#dfn-session-page-load-timeout
@@ -45,6 +43,14 @@ static const Seconds defaultPageLoadTimeout = 300_s;
 // https://w3c.github.io/webdriver/webdriver-spec.html#dfn-session-implicit-wait-timeout
 static const Seconds defaultImplicitWaitTimeout = 0_s;
 
+const String& Session::webElementIdentifier()
+{
+    // The web element identifier is a constant defined by the spec in Section 11 Elements.
+    // https://www.w3.org/TR/webdriver/#elements
+    static NeverDestroyed<String> webElementID { ASCIILiteral("element-6066-11e4-a52e-4f735466cecf") };
+    return webElementID;
+}
+
 Session::Session(std::unique_ptr<SessionHost>&& host)
     : m_host(WTFMove(host))
     , m_scriptTimeout(defaultScriptTimeout)
@@ -779,7 +785,7 @@ RefPtr<JSON::Object> Session::createElement(RefPtr<JSON::Value>&& value)
         return nullptr;
 
     RefPtr<JSON::Object> elementObject = JSON::Object::create();
-    elementObject->setString(webElementIdentifier, elementID);
+    elementObject->setString(webElementIdentifier(), elementID);
     return elementObject;
 }
 
@@ -803,7 +809,7 @@ String Session::extractElementID(JSON::Value& value)
         return emptyString();
 
     String elementID;
-    if (!valueObject->getString(webElementIdentifier, elementID))
+    if (!valueObject->getString(webElementIdentifier(), elementID))
         return emptyString();
 
     return elementID;
@@ -1504,12 +1510,15 @@ String Session::virtualKeyForKeySequence(const String& keySequence, KeyModifier&
     case 0xE007U:
         return ASCIILiteral("Enter");
     case 0xE008U:
+    case 0xE050U:
         modifier = KeyModifier::Shift;
         return ASCIILiteral("Shift");
     case 0xE009U:
+    case 0xE051U:
         modifier = KeyModifier::Control;
         return ASCIILiteral("Control");
     case 0xE00AU:
+    case 0xE052U:
         modifier = KeyModifier::Alternate;
         return ASCIILiteral("Alternate");
     case 0xE00BU:
@@ -1519,24 +1528,34 @@ String Session::virtualKeyForKeySequence(const String& keySequence, KeyModifier&
     case 0xE00DU:
         return ASCIILiteral("Space");
     case 0xE00EU:
+    case 0xE054U:
         return ASCIILiteral("PageUp");
     case 0xE00FU:
+    case 0xE055U:
         return ASCIILiteral("PageDown");
     case 0xE010U:
+    case 0xE056U:
         return ASCIILiteral("End");
     case 0xE011U:
+    case 0xE057U:
         return ASCIILiteral("Home");
     case 0xE012U:
+    case 0xE058U:
         return ASCIILiteral("LeftArrow");
     case 0xE013U:
+    case 0xE059U:
         return ASCIILiteral("UpArrow");
     case 0xE014U:
+    case 0xE05AU:
         return ASCIILiteral("RightArrow");
     case 0xE015U:
+    case 0xE05BU:
         return ASCIILiteral("DownArrow");
     case 0xE016U:
+    case 0xE05CU:
         return ASCIILiteral("Insert");
     case 0xE017U:
+    case 0xE05DU:
         return ASCIILiteral("Delete");
     case 0xE018U:
         return ASCIILiteral("Semicolon");
@@ -1599,6 +1618,7 @@ String Session::virtualKeyForKeySequence(const String& keySequence, KeyModifier&
     case 0xE03CU:
         return ASCIILiteral("Function12");
     case 0xE03DU:
+    case 0xE053U:
         modifier = KeyModifier::Meta;
         return ASCIILiteral("Meta");
     default:
@@ -1769,6 +1789,22 @@ void Session::executeScript(const String& script, RefPtr<JSON::Array>&& argument
     });
 }
 
+static String mouseButtonForAutomation(MouseButton button)
+{
+    switch (button) {
+    case MouseButton::None:
+        return ASCIILiteral("None");
+    case MouseButton::Left:
+        return ASCIILiteral("Left");
+    case MouseButton::Middle:
+        return ASCIILiteral("Middle");
+    case MouseButton::Right:
+        return ASCIILiteral("Right");
+    }
+
+    RELEASE_ASSERT_NOT_REACHED();
+}
+
 void Session::performMouseInteraction(int x, int y, MouseButton button, MouseInteraction interaction, Function<void (CommandResult&&)>&& completionHandler)
 {
     RefPtr<JSON::Object> parameters = JSON::Object::create();
@@ -1777,20 +1813,7 @@ void Session::performMouseInteraction(int x, int y, MouseButton button, MouseInt
     position->setInteger(ASCIILiteral("x"), x);
     position->setInteger(ASCIILiteral("y"), y);
     parameters->setObject(ASCIILiteral("position"), WTFMove(position));
-    switch (button) {
-    case MouseButton::None:
-        parameters->setString(ASCIILiteral("button"), ASCIILiteral("None"));
-        break;
-    case MouseButton::Left:
-        parameters->setString(ASCIILiteral("button"), ASCIILiteral("Left"));
-        break;
-    case MouseButton::Middle:
-        parameters->setString(ASCIILiteral("button"), ASCIILiteral("Middle"));
-        break;
-    case MouseButton::Right:
-        parameters->setString(ASCIILiteral("button"), ASCIILiteral("Right"));
-        break;
-    }
+    parameters->setString(ASCIILiteral("button"), mouseButtonForAutomation(button));
     switch (interaction) {
     case MouseInteraction::Move:
         parameters->setString(ASCIILiteral("interaction"), ASCIILiteral("Move"));
@@ -2059,6 +2082,195 @@ void Session::deleteAllCookies(Function<void (CommandResult&&)>&& completionHand
     });
 }
 
+InputSource& Session::getOrCreateInputSource(const String& id, InputSource::Type type, std::optional<PointerType> pointerType)
+{
+    auto addResult = m_activeInputSources.add(id, InputSource());
+    if (addResult.isNewEntry)
+        addResult.iterator->value = { type, pointerType };
+    return addResult.iterator->value;
+}
+
+Session::InputSourceState& Session::inputSourceState(const String& id)
+{
+    return m_inputStateTable.ensure(id, [] { return InputSourceState(); }).iterator->value;
+}
+
+static const char* automationSourceType(InputSource::Type type)
+{
+    switch (type) {
+    case InputSource::Type::None:
+        return "Null";
+    case InputSource::Type::Pointer:
+        return "Mouse";
+    case InputSource::Type::Key:
+        return "Keyboard";
+    }
+    RELEASE_ASSERT_NOT_REACHED();
+}
+
+static const char* automationOriginType(PointerOrigin::Type type)
+{
+    switch (type) {
+    case PointerOrigin::Type::Viewport:
+        return "Viewport";
+    case PointerOrigin::Type::Pointer:
+        return "Pointer";
+    case PointerOrigin::Type::Element:
+        return "Element";
+    }
+    RELEASE_ASSERT_NOT_REACHED();
+}
+
+void Session::performActions(Vector<Vector<Action>>&& actionsByTick, Function<void (CommandResult&&)>&& completionHandler)
+{
+    if (!m_toplevelBrowsingContext) {
+        completionHandler(CommandResult::fail(CommandResult::ErrorCode::NoSuchWindow));
+        return;
+    }
+
+    handleUserPrompts([this, actionsByTick = WTFMove(actionsByTick), completionHandler = WTFMove(completionHandler)](CommandResult&& result) mutable {
+        if (result.isError()) {
+            completionHandler(WTFMove(result));
+            return;
+        }
+
+        // First check if we have actions and whether we need to resolve any pointer move element origin.
+        unsigned actionsCount = 0;
+        for (const auto& tick : actionsByTick)
+            actionsCount += tick.size();
+        if (!actionsCount) {
+            completionHandler(CommandResult::success());
+            return;
+        }
+
+        RefPtr<JSON::Object> parameters = JSON::Object::create();
+        parameters->setString(ASCIILiteral("handle"), m_toplevelBrowsingContext.value());
+        if (m_currentBrowsingContext)
+            parameters->setString(ASCIILiteral("frameHandle"), m_currentBrowsingContext.value());
+        RefPtr<JSON::Array> inputSources = JSON::Array::create();
+        for (const auto& inputSource : m_activeInputSources) {
+            RefPtr<JSON::Object> inputSourceObject = JSON::Object::create();
+            inputSourceObject->setString(ASCIILiteral("sourceId"), inputSource.key);
+            inputSourceObject->setString(ASCIILiteral("sourceType"), automationSourceType(inputSource.value.type));
+            inputSources->pushObject(WTFMove(inputSourceObject));
+        }
+        parameters->setArray(ASCIILiteral("inputSources"), WTFMove(inputSources));
+        RefPtr<JSON::Array> steps = JSON::Array::create();
+        for (const auto& tick : actionsByTick) {
+            RefPtr<JSON::Array> states = JSON::Array::create();
+            for (const auto& action : tick) {
+                RefPtr<JSON::Object> state = JSON::Object::create();
+                auto& currentState = inputSourceState(action.id);
+                state->setString(ASCIILiteral("sourceId"), action.id);
+                switch (action.type) {
+                case Action::Type::None:
+                    state->setDouble(ASCIILiteral("duration"), action.duration.value());
+                    break;
+                case Action::Type::Pointer: {
+                    switch (action.subtype) {
+                    case Action::Subtype::PointerUp:
+                        currentState.pressedButton = std::nullopt;
+                        break;
+                    case Action::Subtype::PointerDown:
+                        currentState.pressedButton = action.button.value();
+                        break;
+                    case Action::Subtype::PointerMove: {
+                        state->setString(ASCIILiteral("origin"), automationOriginType(action.origin->type));
+                        RefPtr<JSON::Object> location = JSON::Object::create();
+                        location->setInteger(ASCIILiteral("x"), action.x.value());
+                        location->setInteger(ASCIILiteral("y"), action.y.value());
+                        state->setObject(ASCIILiteral("location"), WTFMove(location));
+                        if (action.origin->type == PointerOrigin::Type::Element)
+                            state->setString(ASCIILiteral("nodeHandle"), action.origin->elementID.value());
+                        FALLTHROUGH;
+                    }
+                    case Action::Subtype::Pause:
+                        if (action.duration)
+                            state->setDouble(ASCIILiteral("duration"), action.duration.value());
+                        break;
+                    case Action::Subtype::PointerCancel:
+                        currentState.pressedButton = std::nullopt;
+                        break;
+                    case Action::Subtype::KeyUp:
+                    case Action::Subtype::KeyDown:
+                        ASSERT_NOT_REACHED();
+                    }
+                    if (currentState.pressedButton)
+                        state->setString(ASCIILiteral("pressedButton"), mouseButtonForAutomation(currentState.pressedButton.value()));
+                    break;
+                }
+                case Action::Type::Key:
+                    switch (action.subtype) {
+                    case Action::Subtype::KeyUp:
+                        if (currentState.pressedVirtualKey)
+                            currentState.pressedVirtualKey = std::nullopt;
+                        else
+                            currentState.pressedKey = std::nullopt;
+                        break;
+                    case Action::Subtype::KeyDown: {
+                        KeyModifier modifier;
+                        auto virtualKey = virtualKeyForKeySequence(action.key.value(), modifier);
+                        if (!virtualKey.isNull())
+                            currentState.pressedVirtualKey = virtualKey;
+                        else
+                            currentState.pressedKey = action.key.value();
+                        break;
+                    }
+                    case Action::Subtype::Pause:
+                        if (action.duration)
+                            state->setDouble(ASCIILiteral("duration"), action.duration.value());
+                        break;
+                    case Action::Subtype::PointerUp:
+                    case Action::Subtype::PointerDown:
+                    case Action::Subtype::PointerMove:
+                    case Action::Subtype::PointerCancel:
+                        ASSERT_NOT_REACHED();
+                    }
+                    if (currentState.pressedKey)
+                        state->setString(ASCIILiteral("pressedCharKey"), currentState.pressedKey.value());
+                    if (currentState.pressedVirtualKey)
+                        state->setString(ASCIILiteral("pressedVirtualKey"), currentState.pressedVirtualKey.value());
+                    break;
+                }
+                states->pushObject(WTFMove(state));
+            }
+            RefPtr<JSON::Object> stepStates = JSON::Object::create();
+            stepStates->setArray(ASCIILiteral("states"), WTFMove(states));
+            steps->pushObject(WTFMove(stepStates));
+        }
+
+        parameters->setArray(ASCIILiteral("steps"), WTFMove(steps));
+        m_host->sendCommandToBackend(ASCIILiteral("performInteractionSequence"), WTFMove(parameters), [this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler)] (SessionHost::CommandResponse&& response) {
+            if (response.isError) {
+                completionHandler(CommandResult::fail(WTFMove(response.responseObject)));
+                return;
+            }
+            completionHandler(CommandResult::success());
+        });
+    });
+}
+
+void Session::releaseActions(Function<void (CommandResult&&)>&& completionHandler)
+{
+    if (!m_toplevelBrowsingContext) {
+        completionHandler(CommandResult::fail(CommandResult::ErrorCode::NoSuchWindow));
+        return;
+    }
+
+    m_activeInputSources.clear();
+    m_inputStateTable.clear();
+
+    RefPtr<JSON::Object> parameters = JSON::Object::create();
+    parameters->setString(ASCIILiteral("handle"), m_toplevelBrowsingContext.value());
+    m_host->sendCommandToBackend(ASCIILiteral("cancelInteractionSequence"), WTFMove(parameters), [this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler)](SessionHost::CommandResponse&& response) {
+        if (response.isError) {
+            completionHandler(CommandResult::fail(WTFMove(response.responseObject)));
+            return;
+        }
+        completionHandler(CommandResult::success());
+    });
+}
+
 void Session::dismissAlert(Function<void (CommandResult&&)>&& completionHandler)
 {
     if (!m_toplevelBrowsingContext) {
index 42b3bba..ec6430a 100644 (file)
@@ -25,6 +25,7 @@
 
 #pragma once
 
+#include "Actions.h"
 #include "Capabilities.h"
 #include <wtf/Forward.h>
 #include <wtf/Function.h>
@@ -52,6 +53,7 @@ public:
     Seconds scriptTimeout() const  { return m_scriptTimeout; }
     Seconds pageLoadTimeout() const { return m_pageLoadTimeout; }
     Seconds implicitWaitTimeout() const { return m_implicitWaitTimeout; }
+    static const String& webElementIdentifier();
 
     enum class FindElementsMode { Single, Multiple };
     enum class ExecuteScriptMode { Sync, Async };
@@ -66,6 +68,8 @@ public:
         std::optional<uint64_t> expiry;
     };
 
+    InputSource& getOrCreateInputSource(const String& id, InputSource::Type, std::optional<PointerType>);
+
     void waitForNavigationToComplete(Function<void (CommandResult&&)>&&);
     void createTopLevelBrowsingContext(Function<void (CommandResult&&)>&&);
     void close(Function<void (CommandResult&&)>&&);
@@ -106,6 +110,8 @@ public:
     void addCookie(const Cookie&, Function<void (CommandResult&&)>&&);
     void deleteCookie(const String& name, Function<void (CommandResult&&)>&&);
     void deleteAllCookies(Function<void (CommandResult&&)>&&);
+    void performActions(Vector<Vector<Action>>&&, Function<void (CommandResult&&)>&&);
+    void releaseActions(Function<void (CommandResult&&)>&&);
     void dismissAlert(Function<void (CommandResult&&)>&&);
     void acceptAlert(Function<void (CommandResult&&)>&&);
     void getAlertText(Function<void (CommandResult&&)>&&);
@@ -159,7 +165,6 @@ private:
 
     void selectOptionElement(const String& elementID, Function<void (CommandResult&&)>&&);
 
-    enum class MouseButton { None, Left, Middle, Right };
     enum class MouseInteraction { Move, Down, Up, SingleClick, DoubleClick };
     void performMouseInteraction(int x, int y, MouseButton, MouseInteraction, Function<void (CommandResult&&)>&&);
 
@@ -179,12 +184,25 @@ private:
     String virtualKeyForKeySequence(const String& keySequence, KeyModifier&);
     void performKeyboardInteractions(Vector<KeyboardInteraction>&&, Function<void (CommandResult&&)>&&);
 
+    struct InputSourceState {
+        enum class Type { Null, Key, Pointer };
+
+        Type type;
+        String subtype;
+        std::optional<MouseButton> pressedButton;
+        std::optional<String> pressedKey;
+        std::optional<String> pressedVirtualKey;
+    };
+    InputSourceState& inputSourceState(const String& id);
+
     std::unique_ptr<SessionHost> m_host;
     Seconds m_scriptTimeout;
     Seconds m_pageLoadTimeout;
     Seconds m_implicitWaitTimeout;
     std::optional<String> m_toplevelBrowsingContext;
     std::optional<String> m_currentBrowsingContext;
+    HashMap<String, InputSource> m_activeInputSources;
+    HashMap<String, InputSourceState> m_inputStateTable;
 };
 
 } // WebDriver
index 73ebd57..ee3fc6e 100644 (file)
@@ -151,6 +151,9 @@ const WebDriverService::Command WebDriverService::s_commands[] = {
     { HTTPMethod::Delete, "/session/$sessionId/cookie/$name", &WebDriverService::deleteCookie },
     { HTTPMethod::Delete, "/session/$sessionId/cookie", &WebDriverService::deleteAllCookies },
 
+    { HTTPMethod::Post, "/session/$sessionId/actions", &WebDriverService::performActions },
+    { HTTPMethod::Delete, "/session/$sessionId/actions", &WebDriverService::releaseActions },
+
     { HTTPMethod::Post, "/session/$sessionId/alert/dismiss", &WebDriverService::dismissAlert },
     { HTTPMethod::Post, "/session/$sessionId/alert/accept", &WebDriverService::acceptAlert },
     { HTTPMethod::Get, "/session/$sessionId/alert/text", &WebDriverService::getAlertText },
@@ -1587,6 +1590,371 @@ void WebDriverService::deleteAllCookies(RefPtr<JSON::Object>&& parameters, Funct
     });
 }
 
+static bool processPauseAction(JSON::Object& actionItem, Action& action, std::optional<String>& errorMessage)
+{
+    RefPtr<JSON::Value> durationValue;
+    if (!actionItem.getValue(ASCIILiteral("duration"), durationValue)) {
+        errorMessage = String("The parameter 'duration' is missing in pause action");
+        return false;
+    }
+
+    auto duration = unsignedValue(*durationValue);
+    if (!duration) {
+        errorMessage = String("The parameter 'duration' is invalid in pause action");
+        return false;
+    }
+
+    action.duration = duration.value();
+    return true;
+}
+
+static std::optional<Action> processNullAction(const String& id, JSON::Object& actionItem, std::optional<String>& errorMessage)
+{
+    String subtype;
+    actionItem.getString(ASCIILiteral("type"), subtype);
+    if (subtype != "pause") {
+        errorMessage = String("The parameter 'type' in null action is invalid or missing");
+        return std::nullopt;
+    }
+
+    Action action(id, Action::Type::None, Action::Subtype::Pause);
+    if (!processPauseAction(actionItem, action, errorMessage))
+        return std::nullopt;
+
+    return action;
+}
+
+static std::optional<Action> processKeyAction(const String& id, JSON::Object& actionItem, std::optional<String>& errorMessage)
+{
+    Action::Subtype actionSubtype;
+    String subtype;
+    actionItem.getString(ASCIILiteral("type"), subtype);
+    if (subtype == "pause")
+        actionSubtype = Action::Subtype::Pause;
+    else if (subtype == "keyUp")
+        actionSubtype = Action::Subtype::KeyUp;
+    else if (subtype == "keyDown")
+        actionSubtype = Action::Subtype::KeyDown;
+    else {
+        errorMessage = String("The parameter 'type' of key action is invalid");
+        return std::nullopt;
+    }
+
+    Action action(id, Action::Type::Key, actionSubtype);
+
+    switch (actionSubtype) {
+    case Action::Subtype::Pause:
+        if (!processPauseAction(actionItem, action, errorMessage))
+            return std::nullopt;
+        break;
+    case Action::Subtype::KeyUp:
+    case Action::Subtype::KeyDown: {
+        RefPtr<JSON::Value> keyValue;
+        if (!actionItem.getValue(ASCIILiteral("value"), keyValue)) {
+            errorMessage = String("The paramater 'value' is missing for key up/down action");
+            return std::nullopt;
+        }
+        String key;
+        if (!keyValue->asString(key) || key.isEmpty()) {
+            errorMessage = String("The paramater 'value' is invalid for key up/down action");
+            return std::nullopt;
+        }
+        // FIXME: check single unicode code point.
+        action.key = key;
+        break;
+    }
+    case Action::Subtype::PointerUp:
+    case Action::Subtype::PointerDown:
+    case Action::Subtype::PointerMove:
+    case Action::Subtype::PointerCancel:
+        ASSERT_NOT_REACHED();
+    }
+
+    return action;
+}
+
+static MouseButton actionMouseButton(unsigned button)
+{
+    // MouseEvent.button
+    // https://www.w3.org/TR/uievents/#ref-for-dom-mouseevent-button-1
+    switch (button) {
+    case 0:
+        return MouseButton::Left;
+    case 1:
+        return MouseButton::Middle;
+    case 2:
+        return MouseButton::Right;
+    }
+
+    return MouseButton::None;
+}
+
+static std::optional<Action> processPointerAction(const String& id, PointerParameters& parameters, JSON::Object& actionItem, std::optional<String>& errorMessage)
+{
+    Action::Subtype actionSubtype;
+    String subtype;
+    actionItem.getString(ASCIILiteral("type"), subtype);
+    if (subtype == "pause")
+        actionSubtype = Action::Subtype::Pause;
+    else if (subtype == "pointerUp")
+        actionSubtype = Action::Subtype::PointerUp;
+    else if (subtype == "pointerDown")
+        actionSubtype = Action::Subtype::PointerDown;
+    else if (subtype == "pointerMove")
+        actionSubtype = Action::Subtype::PointerMove;
+    else if (subtype == "pointerCancel")
+        actionSubtype = Action::Subtype::PointerCancel;
+    else {
+        errorMessage = String("The parameter 'type' of pointer action is invalid");
+        return std::nullopt;
+    }
+
+    Action action(id, Action::Type::Pointer, actionSubtype);
+    action.pointerType = parameters.pointerType;
+
+    switch (actionSubtype) {
+    case Action::Subtype::Pause:
+        if (!processPauseAction(actionItem, action, errorMessage))
+            return std::nullopt;
+        break;
+    case Action::Subtype::PointerUp:
+    case Action::Subtype::PointerDown: {
+        RefPtr<JSON::Value> buttonValue;
+        if (!actionItem.getValue(ASCIILiteral("button"), buttonValue)) {
+            errorMessage = String("The paramater 'button' is missing for pointer up/down action");
+            return std::nullopt;
+        }
+        auto button = unsignedValue(*buttonValue);
+        if (!button) {
+            errorMessage = String("The paramater 'button' is invalid for pointer up/down action");
+            return std::nullopt;
+        }
+        action.button = actionMouseButton(button.value());
+        break;
+    }
+    case Action::Subtype::PointerMove: {
+        RefPtr<JSON::Value> durationValue;
+        if (actionItem.getValue(ASCIILiteral("duration"), durationValue)) {
+            auto duration = unsignedValue(*durationValue);
+            if (!duration) {
+                errorMessage = String("The parameter 'duration' is invalid in pointer move action");
+                return std::nullopt;
+            }
+            action.duration = duration.value();
+        }
+
+        RefPtr<JSON::Value> originValue;
+        if (actionItem.getValue(ASCIILiteral("origin"), originValue)) {
+            if (originValue->type() == JSON::Value::Type::Object) {
+                RefPtr<JSON::Object> originObject;
+                originValue->asObject(originObject);
+                String elementID;
+                if (!originObject->getString(Session::webElementIdentifier(), elementID)) {
+                    errorMessage = String("The parameter 'origin' is not a valid web element object in pointer move action");
+                    return std::nullopt;
+                }
+                action.origin = PointerOrigin { PointerOrigin::Type::Element, elementID };
+            } else {
+                String origin;
+                originValue->asString(origin);
+                if (origin == "viewport")
+                    action.origin = PointerOrigin { PointerOrigin::Type::Viewport, std::nullopt };
+                else if (origin == "pointer")
+                    action.origin = PointerOrigin { PointerOrigin::Type::Pointer, std::nullopt };
+                else {
+                    errorMessage = String("The parameter 'origin' is invalid in pointer move action");
+                    return std::nullopt;
+                }
+            }
+        } else
+            action.origin = PointerOrigin { PointerOrigin::Type::Viewport, std::nullopt };
+
+        RefPtr<JSON::Value> xValue;
+        if (actionItem.getValue(ASCIILiteral("x"), xValue)) {
+            auto x = valueAsNumberInRange(*xValue, INT_MIN);
+            if (!x) {
+                errorMessage = String("The paramater 'x' is invalid for pointer move action");
+                return std::nullopt;
+            }
+            action.x = x.value();
+        }
+
+        RefPtr<JSON::Value> yValue;
+        if (actionItem.getValue(ASCIILiteral("y"), yValue)) {
+            auto y = valueAsNumberInRange(*yValue, INT_MIN);
+            if (!y) {
+                errorMessage = String("The paramater 'y' is invalid for pointer move action");
+                return std::nullopt;
+            }
+            action.y = y.value();
+        }
+        break;
+    }
+    case Action::Subtype::PointerCancel:
+        break;
+    case Action::Subtype::KeyUp:
+    case Action::Subtype::KeyDown:
+        ASSERT_NOT_REACHED();
+    }
+
+    return action;
+}
+
+static std::optional<PointerParameters> processPointerParameters(JSON::Object& actionSequence, std::optional<String>& errorMessage)
+{
+    PointerParameters parameters;
+    RefPtr<JSON::Value> parametersDataValue;
+    if (!actionSequence.getValue(ASCIILiteral("parameters"), parametersDataValue))
+        return parameters;
+
+    RefPtr<JSON::Object> parametersData;
+    if (!parametersDataValue->asObject(parametersData)) {
+        errorMessage = String("Action sequence pointer parameters is not an object");
+        return std::nullopt;
+    }
+
+    String pointerType;
+    if (!parametersData->getString(ASCIILiteral("pointerType"), pointerType))
+        return parameters;
+
+    if (pointerType == "mouse")
+        parameters.pointerType = PointerType::Mouse;
+    else if (pointerType == "pen")
+        parameters.pointerType = PointerType::Pen;
+    else if (pointerType == "touch")
+        parameters.pointerType = PointerType::Touch;
+    else {
+        errorMessage = String("The parameter 'pointerType' in action sequence pointer parameters is invalid");
+        return std::nullopt;
+    }
+
+    return parameters;
+}
+
+static std::optional<Vector<Action>> processInputActionSequence(Session& session, JSON::Value& actionSequenceValue, std::optional<String>& errorMessage)
+{
+    RefPtr<JSON::Object> actionSequence;
+    if (!actionSequenceValue.asObject(actionSequence)) {
+        errorMessage = String("The action sequence is not an object");
+        return std::nullopt;
+    }
+
+    String type;
+    actionSequence->getString(ASCIILiteral("type"), type);
+    InputSource::Type inputSourceType;
+    if (type == "key")
+        inputSourceType = InputSource::Type::Key;
+    else if (type == "pointer")
+        inputSourceType = InputSource::Type::Pointer;
+    else if (type == "none")
+        inputSourceType = InputSource::Type::None;
+    else {
+        errorMessage = String("The parameter 'type' is invalid or missing in action sequence");
+        return std::nullopt;
+    }
+
+    String id;
+    if (!actionSequence->getString(ASCIILiteral("id"), id)) {
+        errorMessage = String("The parameter 'id' is invalid or missing in action sequence");
+        return std::nullopt;
+    }
+
+    std::optional<PointerParameters> parameters;
+    std::optional<PointerType> pointerType;
+    if (inputSourceType == InputSource::Type::Pointer) {
+        parameters = processPointerParameters(*actionSequence, errorMessage);
+        if (!parameters)
+            return std::nullopt;
+
+        pointerType = parameters->pointerType;
+    }
+
+    auto& inputSource = session.getOrCreateInputSource(id, inputSourceType, pointerType);
+    if (inputSource.type != inputSourceType) {
+        errorMessage = String("Action sequence type doesn't match input source type");
+        return std::nullopt;
+    }
+
+    if (inputSource.type ==  InputSource::Type::Pointer && inputSource.pointerType != pointerType) {
+        errorMessage = String("Action sequence pointer type doesn't match input source pointer type");
+        return std::nullopt;
+    }
+
+    RefPtr<JSON::Array> actionItems;
+    if (!actionSequence->getArray(ASCIILiteral("actions"), actionItems)) {
+        errorMessage = String("The parameter 'actions' is invalid or not present in action sequence");
+        return std::nullopt;
+    }
+
+    Vector<Action> actions;
+    unsigned actionItemsLength = actionItems->length();
+    for (unsigned i = 0; i < actionItemsLength; ++i) {
+        auto actionItemValue = actionItems->get(i);
+        RefPtr<JSON::Object> actionItem;
+        if (!actionItemValue->asObject(actionItem)) {
+            errorMessage = String("An action in action sequence is not an object");
+            return std::nullopt;
+        }
+
+        std::optional<Action> action;
+        if (inputSourceType == InputSource::Type::None)
+            action = processNullAction(id, *actionItem, errorMessage);
+        else if (inputSourceType == InputSource::Type::Key)
+            action = processKeyAction(id, *actionItem, errorMessage);
+        else if (inputSourceType == InputSource::Type::Pointer)
+            action = processPointerAction(id, parameters.value(), *actionItem, errorMessage);
+        if (!action)
+            return std::nullopt;
+
+        actions.append(action.value());
+    }
+
+    return actions;
+}
+
+void WebDriverService::performActions(RefPtr<JSON::Object>&& parameters, Function<void (CommandResult&&)>&& completionHandler)
+{
+    // §17.5 Perform Actions.
+    // https://w3c.github.io/webdriver/webdriver-spec.html#perform-actions
+    if (!findSessionOrCompleteWithError(*parameters, completionHandler))
+        return;
+
+    RefPtr<JSON::Array> actionsArray;
+    if (!parameters->getArray(ASCIILiteral("actions"), actionsArray)) {
+        completionHandler(CommandResult::fail(CommandResult::ErrorCode::InvalidArgument, String("The paramater 'actions' is invalid or not present")));
+        return;
+    }
+
+    std::optional<String> errorMessage;
+    Vector<Vector<Action>> actionsByTick;
+    unsigned actionsArrayLength = actionsArray->length();
+    for (unsigned i = 0; i < actionsArrayLength; ++i) {
+        auto actionSequence = actionsArray->get(i);
+        auto inputSourceActions = processInputActionSequence(*m_session, *actionSequence, errorMessage);
+        if (!inputSourceActions) {
+            completionHandler(CommandResult::fail(CommandResult::ErrorCode::InvalidArgument, errorMessage.value()));
+            return;
+        }
+        for (unsigned i = 0; i < inputSourceActions->size(); ++i) {
+            if (actionsByTick.size() < i + 1)
+                actionsByTick.append({ });
+            actionsByTick[i].append(inputSourceActions.value()[i]);
+        }
+    }
+
+    m_session->performActions(WTFMove(actionsByTick), WTFMove(completionHandler));
+}
+
+void WebDriverService::releaseActions(RefPtr<JSON::Object>&& parameters, Function<void (CommandResult&&)>&& completionHandler)
+{
+    // §17.5 Release Actions.
+    // https://w3c.github.io/webdriver/webdriver-spec.html#release-actions
+    if (!findSessionOrCompleteWithError(*parameters, completionHandler))
+        return;
+
+    m_session->releaseActions(WTFMove(completionHandler));
+}
+
 void WebDriverService::dismissAlert(RefPtr<JSON::Object>&& parameters, Function<void (CommandResult&&)>&& completionHandler)
 {
     // §18.1 Dismiss Alert.
index 40e37ed..7ea45bd 100644 (file)
@@ -103,6 +103,8 @@ private:
     void addCookie(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
     void deleteCookie(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
     void deleteAllCookies(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
+    void performActions(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
+    void releaseActions(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
     void dismissAlert(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
     void acceptAlert(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
     void getAlertText(RefPtr<JSON::Object>&&, Function<void (CommandResult&&)>&&);
index d61d891..90ca7af 100644 (file)
@@ -1,3 +1,36 @@
+2018-05-09  Carlos Garcia Campos  <cgarcia@igalia.com>
+
+        WebDriver: implement advance user interactions
+        https://bugs.webkit.org/show_bug.cgi?id=174616
+
+        Reviewed by Brian Burg.
+
+        Handle origin in case of mouse move transitions.
+
+        * UIProcess/Automation/Automation.json: Add MouseMoveOrigin enum and pass it as parameter of InputSourceState
+        together with optional node handle. Also pass the frame handle to performInteractionSequence command to find the
+        node in the current browsing context.
+        * UIProcess/Automation/SimulatedInputDispatcher.cpp:
+        (WebKit::SimulatedInputKeyFrame::keyFrameToResetInputSources): Ensure we reset the location.
+        (WebKit::SimulatedInputDispatcher::resolveLocation): Helper to resolve destination location based on current
+        location and mouse move origin.
+        (WebKit::SimulatedInputDispatcher::transitionInputSourceToState): Use resolveLocation() in mouse transitions.
+        (WebKit::SimulatedInputDispatcher::run): Receive and save the frame ID.
+        (WebKit::SimulatedInputDispatcher::finishDispatching): Reset the frame ID.
+        * UIProcess/Automation/SimulatedInputDispatcher.h:
+        * UIProcess/Automation/WebAutomationSession.cpp:
+        (WebKit::WebAutomationSession::computeElementLayout): Use even numbers for the callback ID to not conflict with
+        viewportInViewCenterPointOfElement() callbacks.
+        (WebKit::WebAutomationSession::didComputeElementLayout): Handle computeElementLayout() or
+        viewportInViewCenterPointOfElement() requests by calling the right callback depending on whether the ID is odd
+        or even number.
+        (WebKit::WebAutomationSession::viewportInViewCenterPointOfElement): Send ComputeElementLayout message to the
+        WebProcess using odd numbers for the callback ID to not conflict with computeElementLayout() callbacks.
+        (WebKit::WebAutomationSession::performInteractionSequence): Handle the mouse origin and element handle.
+        (WebKit::WebAutomationSession::cancelInteractionSequence): Pass the frame ID to the input dispatcher.
+        * UIProcess/Automation/WebAutomationSession.h:
+        * UIProcess/Automation/WebAutomationSessionMacros.h:
+
 2018-05-09  Tim Horton  <timothy_horton@apple.com>
 
         Remove the unused HAVE_OS_ACTIVITY
index c317e1a..42556b5 100644 (file)
             ]
         },
         {
+            "id": "MouseMoveOrigin",
+            "type": "string",
+            "description": "Enumerates different origin types that can be used in mouse move interactions.",
+            "enum": [
+                "Viewport",
+                "Pointer",
+                "Element"
+            ]
+        },
+        {
             "id": "InputSourceState",
             "type": "object",
             "description": "A new state for a specific input source. All state-related fields are optional and must be applicable to the InputSource referenced by 'sourceId'. If no state-related fields are specified, the state is assumed to remain the same as in the previous step (i.e., 'sustained').",
                 { "name": "pressedCharKey", "type": "string", "optional": true, "description": "For 'keyboard' input sources, specifies a character key that has 'pressed' state. Unmentioned character keys are assumed to have a 'released' state." },
                 { "name": "pressedVirtualKey", "$ref": "VirtualKey", "optional": true, "description": "For 'keyboard' input sources, specifies a virtual key that has a 'pressed' state. Unmentioned virtual keys are assumed to have a 'released' state." },
                 { "name": "pressedButton", "$ref": "MouseButton", "optional": true, "description": "For 'mouse' input sources, specifies which mouse button has a 'pressed' state. Unmentioned mouse buttons are assumed to have a 'released' state." },
+                { "name": "origin", "$ref": "MouseMoveOrigin", "optional": true, "description": "For 'mouse' input sources, specifies the origin type of a mouse move transition. Defaults to 'Viewport' if omitted."},
+                { "name": "nodeHandle", "$ref": "NodeHandle", "optional": true, "description": "The handle of the element to use as origin when origin type is 'Element'."},
                 { "name": "location", "$ref": "Point", "optional": true, "description": "For 'mouse' or 'touch' input sources, specifies a location in view coordinates to which the input source should transition. Transitioning to this state may interpolate intemediate input source states to better simulate real user movements and gestures." },
                 { "name": "duration", "type": "integer", "optional": true, "description": "The minimum number of milliseconds that must elapse while the relevant input source transitions to this state." }
             ]
             "description": "Perform multiple simulated interactions over time using a list of input sources and a list of steps, where each step specifies a state for each input source at the time that step is performed.",
             "parameters": [
                 { "name": "handle", "$ref": "BrowsingContextHandle", "description": "The browsing context to be interacted with." },
+                { "name": "frameHandle", "$ref": "FrameHandle", "optional": true, "description": "The handle for the frame in which to search for the elements in case of an 'Element' type MouseMoveOrigin. The main frame is used if this parameter empty string or excluded." },
                 { "name": "inputSources", "type": "array", "items": { "$ref": "InputSource" }, "description": "All input sources that are used to perform this interaction sequence." },
                 { "name": "steps", "type": "array", "items": { "$ref": "InteractionStep" }, "description": "A list of steps that are executed in order." }
             ],
             "name": "cancelInteractionSequence",
             "description": "Cancel an active interaction sequence that is currently in progress.",
             "parameters": [
-                { "name": "handle", "$ref": "BrowsingContextHandle", "description": "The browsing context to be interacted with." }
+                { "name": "handle", "$ref": "BrowsingContextHandle", "description": "The browsing context to be interacted with." },
+                { "name": "frameHandle", "$ref": "FrameHandle", "optional": true, "description": "The handle for the frame passed to performInteractionSequence. The main frame is used if this parameter empty string or excluded." }
             ],
             "async": true
         },
index 7d582bb..f47174b 100644 (file)
@@ -66,8 +66,12 @@ SimulatedInputKeyFrame SimulatedInputKeyFrame::keyFrameToResetInputSources(HashS
     Vector<SimulatedInputKeyFrame::StateEntry> entries;
     entries.reserveCapacity(inputSources.size());
 
-    for (auto& inputSource : inputSources)
-        entries.uncheckedAppend(std::pair<SimulatedInputSource&, SimulatedInputSourceState> { inputSource.get(), SimulatedInputSourceState::emptyState() });
+    for (auto& inputSource : inputSources) {
+        auto emptyState = SimulatedInputSourceState::emptyState();
+        // Ensure we reset the location.
+        emptyState.location = WebCore::IntPoint();
+        entries.uncheckedAppend(std::pair<SimulatedInputSource&, SimulatedInputSourceState> { inputSource.get(), WTFMove(emptyState) });
+    }
 
     return SimulatedInputKeyFrame(WTFMove(entries));
 }
@@ -174,11 +178,44 @@ void SimulatedInputDispatcher::transitionBetweenKeyFrames(const SimulatedInputKe
     transitionToNextInputSourceState();
 }
 
-void SimulatedInputDispatcher::transitionInputSourceToState(SimulatedInputSource& inputSource, const SimulatedInputSourceState& newState, AutomationCompletionHandler&& completionHandler)
+void SimulatedInputDispatcher::resolveLocation(const WebCore::IntPoint& currentLocation, std::optional<WebCore::IntPoint> location, MouseMoveOrigin origin, std::optional<String> nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&& completionHandler)
+{
+    if (!location) {
+        completionHandler(currentLocation, std::nullopt);
+        return;
+    }
+
+    switch (origin) {
+    case MouseMoveOrigin::Viewport:
+        completionHandler(location.value(), std::nullopt);
+        break;
+    case MouseMoveOrigin::Pointer: {
+        WebCore::IntPoint destination(currentLocation);
+        destination.moveBy(location.value());
+        completionHandler(destination, std::nullopt);
+        break;
+    }
+    case MouseMoveOrigin::Element: {
+        m_client.viewportInViewCenterPointOfElement(m_page, m_frameID.value(), nodeHandle.value(), [destination = location.value(), completionHandler = WTFMove(completionHandler)](std::optional<WebCore::IntPoint> inViewCenterPoint, std::optional<AutomationCommandError> error) mutable {
+            if (error) {
+                completionHandler(std::nullopt, error);
+                return;
+            }
+
+            ASSERT(inViewCenterPoint);
+            destination.moveBy(inViewCenterPoint.value());
+            completionHandler(destination, std::nullopt);
+        });
+        break;
+    }
+    }
+}
+
+void SimulatedInputDispatcher::transitionInputSourceToState(SimulatedInputSource& inputSource, SimulatedInputSourceState& newState, AutomationCompletionHandler&& completionHandler)
 {
     // Make cases and conditionals more readable by aliasing pre/post states as 'a' and 'b'.
-    SimulatedInputSourceState a = inputSource.state;
-    SimulatedInputSourceState b = newState;
+    SimulatedInputSourceState& a = inputSource.state;
+    SimulatedInputSourceState& b = newState;
 
     AutomationCompletionHandler eventDispatchFinished = [&inputSource, &newState, completionHandler = WTFMove(completionHandler)](std::optional<AutomationCommandError> error) {
         if (error) {
@@ -196,16 +233,24 @@ void SimulatedInputDispatcher::transitionInputSourceToState(SimulatedInputSource
         eventDispatchFinished(std::nullopt);
         break;
     case SimulatedInputSource::Type::Mouse: {
-        // The "dispatch a pointer{Down,Up,Move} action" algorithms (§17.4 Dispatching Actions).
-        if (!a.pressedMouseButton && b.pressedMouseButton)
-            m_client.simulateMouseInteraction(m_page, MouseInteraction::Down, b.pressedMouseButton.value(), b.location.value(), WTFMove(eventDispatchFinished));
-        else if (a.pressedMouseButton && !b.pressedMouseButton)
-            m_client.simulateMouseInteraction(m_page, MouseInteraction::Up, a.pressedMouseButton.value(), b.location.value(), WTFMove(eventDispatchFinished));
-        else if (a.location != b.location) {
-            // FIXME: This does not interpolate mousemoves per the "perform a pointer move" algorithm (§17.4 Dispatching Actions).
-            m_client.simulateMouseInteraction(m_page, MouseInteraction::Move, b.pressedMouseButton.value_or(MouseButton::NoButton), b.location.value(), WTFMove(eventDispatchFinished));
-        } else
-            eventDispatchFinished(std::nullopt);
+        resolveLocation(a.location.value_or(WebCore::IntPoint()), b.location, b.origin.value_or(MouseMoveOrigin::Viewport), b.nodeHandle, [this, &a, &b, eventDispatchFinished = WTFMove(eventDispatchFinished)](std::optional<WebCore::IntPoint> location, std::optional<AutomationCommandError> error) mutable {
+            if (error) {
+                eventDispatchFinished(error);
+                return;
+            }
+            RELEASE_ASSERT(location);
+            b.location = location;
+            // The "dispatch a pointer{Down,Up,Move} action" algorithms (§17.4 Dispatching Actions).
+            if (!a.pressedMouseButton && b.pressedMouseButton)
+                m_client.simulateMouseInteraction(m_page, MouseInteraction::Down, b.pressedMouseButton.value(), b.location.value(), WTFMove(eventDispatchFinished));
+            else if (a.pressedMouseButton && !b.pressedMouseButton)
+                m_client.simulateMouseInteraction(m_page, MouseInteraction::Up, a.pressedMouseButton.value(), b.location.value(), WTFMove(eventDispatchFinished));
+            else if (a.location != b.location) {
+                // FIXME: This does not interpolate mousemoves per the "perform a pointer move" algorithm (§17.4 Dispatching Actions).
+                m_client.simulateMouseInteraction(m_page, MouseInteraction::Move, b.pressedMouseButton.value_or(MouseButton::NoButton), b.location.value(), WTFMove(eventDispatchFinished));
+            } else
+                eventDispatchFinished(std::nullopt);
+        });
         break;
     }
     case SimulatedInputSource::Type::Keyboard:
@@ -225,7 +270,7 @@ void SimulatedInputDispatcher::transitionInputSourceToState(SimulatedInputSource
     }
 }
 
-void SimulatedInputDispatcher::run(Vector<SimulatedInputKeyFrame>&& keyFrames, HashSet<Ref<SimulatedInputSource>>& inputSources, AutomationCompletionHandler&& completionHandler)
+void SimulatedInputDispatcher::run(uint64_t frameID, Vector<SimulatedInputKeyFrame>&& keyFrames, HashSet<Ref<SimulatedInputSource>>& inputSources, AutomationCompletionHandler&& completionHandler)
 {
     ASSERT(!isActive());
     if (isActive()) {
@@ -233,6 +278,7 @@ void SimulatedInputDispatcher::run(Vector<SimulatedInputKeyFrame>&& keyFrames, H
         return;
     }
 
+    m_frameID = frameID;
     m_runCompletionHandler = WTFMove(completionHandler);
     for (const Ref<SimulatedInputSource>& inputSource : inputSources)
         m_inputSources.add(inputSource.copyRef());
@@ -261,6 +307,7 @@ void SimulatedInputDispatcher::finishDispatching(std::optional<AutomationCommand
     m_keyFrameTransitionDurationTimer.stop();
 
     auto finish = std::exchange(m_runCompletionHandler, nullptr);
+    m_frameID = std::nullopt;
     m_keyframes.clear();
     m_inputSources.clear();
     m_keyframeIndex = 0;
index a602c2c..d04cdde 100644 (file)
@@ -39,6 +39,7 @@ namespace Inspector { namespace Protocol { namespace Automation {
 enum class ErrorMessage;
 enum class KeyboardInteractionType;
 enum class MouseInteraction;
+enum class MouseMoveOrigin;
 enum class VirtualKey;
 } } }
 
@@ -54,11 +55,14 @@ using VirtualKey = Inspector::Protocol::Automation::VirtualKey;
 using CharKey = char; // For WebDriver, this only needs to support ASCII characters on 102-key keyboard.
 using MouseButton = WebMouseEvent::Button;
 using MouseInteraction = Inspector::Protocol::Automation::MouseInteraction;
+using MouseMoveOrigin = Inspector::Protocol::Automation::MouseMoveOrigin;
 
 struct SimulatedInputSourceState {
     std::optional<CharKey> pressedCharKey;
     std::optional<VirtualKey> pressedVirtualKey;
     std::optional<MouseButton> pressedMouseButton;
+    std::optional<MouseMoveOrigin> origin;
+    std::optional<String> nodeHandle;
     std::optional<WebCore::IntPoint> location;
     std::optional<Seconds> duration;
 
@@ -112,6 +116,7 @@ public:
         virtual ~Client() { }
         virtual void simulateMouseInteraction(WebPageProxy&, MouseInteraction, WebMouseEvent::Button, const WebCore::IntPoint& locationInView, AutomationCompletionHandler&&) = 0;
         virtual void simulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, std::optional<VirtualKey>, std::optional<CharKey>, AutomationCompletionHandler&&) = 0;
+        virtual void viewportInViewCenterPointOfElement(WebPageProxy&, uint64_t frameID, const String& nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&&) = 0;
     };
 
     static Ref<SimulatedInputDispatcher> create(WebPageProxy& page, SimulatedInputDispatcher::Client& client)
@@ -121,7 +126,7 @@ public:
 
     ~SimulatedInputDispatcher();
 
-    void run(Vector<SimulatedInputKeyFrame>&& keyFrames, HashSet<Ref<SimulatedInputSource>>& inputSources, AutomationCompletionHandler&&);
+    void run(uint64_t frameID, Vector<SimulatedInputKeyFrame>&& keyFrames, HashSet<Ref<SimulatedInputSource>>& inputSources, AutomationCompletionHandler&&);
     void cancel();
 
     bool isActive() const;
@@ -133,15 +138,18 @@ private:
     void transitionBetweenKeyFrames(const SimulatedInputKeyFrame&, const SimulatedInputKeyFrame&, AutomationCompletionHandler&&);
 
     void transitionToNextInputSourceState();
-    void transitionInputSourceToState(SimulatedInputSource&, const SimulatedInputSourceState& newState, AutomationCompletionHandler&&);
+    void transitionInputSourceToState(SimulatedInputSource&, SimulatedInputSourceState& newState, AutomationCompletionHandler&&);
     void finishDispatching(std::optional<AutomationCommandError>);
 
     void keyFrameTransitionDurationTimerFired();
     bool isKeyFrameTransitionComplete() const;
 
+    void resolveLocation(const WebCore::IntPoint& currentLocation, std::optional<WebCore::IntPoint> location, MouseMoveOrigin, std::optional<String> nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&&);
+
     WebPageProxy& m_page;
     SimulatedInputDispatcher::Client& m_client;
 
+    std::optional<uint64_t> m_frameID;
     AutomationCompletionHandler m_runCompletionHandler;
     AutomationCompletionHandler m_keyFrameTransitionCompletionHandler;
     RunLoop::Timer<SimulatedInputDispatcher> m_keyFrameTransitionDurationTimer;
index 1e9c39d..83c9fc2 100644 (file)
@@ -1,3 +1,4 @@
+
 /*
  * Copyright (C) 2016, 2017 Apple Inc. All rights reserved.
  *
@@ -977,7 +978,8 @@ void WebAutomationSession::computeElementLayout(const String& browsingContextHan
     if (!coordinateSystem)
         ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "The parameter 'coordinateSystem' is invalid.");
 
-    uint64_t callbackID = m_nextComputeElementLayoutCallbackID++;
+    // Start at 2 and use only even numbers to not conflict with m_nextViewportInViewCenterPointOfElementCallbackID.
+    uint64_t callbackID = m_nextComputeElementLayoutCallbackID += 2;
     m_computeElementLayoutCallbacks.set(callbackID, WTFMove(callback));
 
     bool scrollIntoViewIfNeeded = optionalScrollIntoViewIfNeeded ? *optionalScrollIntoViewIfNeeded : false;
@@ -986,6 +988,17 @@ void WebAutomationSession::computeElementLayout(const String& browsingContextHan
 
 void WebAutomationSession::didComputeElementLayout(uint64_t callbackID, WebCore::IntRect rect, std::optional<WebCore::IntPoint> inViewCenterPoint, bool isObscured, const String& errorType)
 {
+    if (callbackID % 2 == 1) {
+        ASSERT(inViewCenterPoint);
+        if (auto callback = m_viewportInViewCenterPointOfElementCallbacks.take(callbackID)) {
+            std::optional<AutomationCommandError> error;
+            if (!errorType.isEmpty())
+                error = AUTOMATION_COMMAND_ERROR_WITH_MESSAGE(errorType);
+            callback(inViewCenterPoint, error);
+        }
+        return;
+    }
+
     auto callback = m_computeElementLayoutCallbacks.take(callbackID);
     if (!callback)
         return;
@@ -1403,6 +1416,15 @@ SimulatedInputSource* WebAutomationSession::inputSourceForType(SimulatedInputSou
 }
 
 // SimulatedInputDispatcher::Client API
+void WebAutomationSession::viewportInViewCenterPointOfElement(WebPageProxy& page, uint64_t frameID, const String& nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&& completionHandler)
+{
+    // Start at 3 and use only odd numbers to not conflict with m_nextComputeElementLayoutCallbackID.
+    uint64_t callbackID = m_nextViewportInViewCenterPointOfElementCallbackID += 2;
+    m_viewportInViewCenterPointOfElementCallbacks.set(callbackID, WTFMove(completionHandler));
+
+    page.process().send(Messages::WebAutomationSessionProxy::ComputeElementLayout(page.pageID(), frameID, nodeHandle, false, CoordinateSystem::LayoutViewport, callbackID), 0);
+}
+
 void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInteraction interaction, WebMouseEvent::Button mouseButton, const WebCore::IntPoint& locationInViewport, CompletionHandler<void(std::optional<AutomationCommandError>)>&& completionHandler)
 {
     WebCore::IntPoint locationInView = WebCore::IntPoint(locationInViewport.x(), locationInViewport.y() + page.topContentInset());
@@ -1663,7 +1685,7 @@ static SimulatedInputSource::Type simulatedInputSourceTypeFromProtocolSourceType
 }
 #endif // USE(APPKIT) || PLATFORM(GTK)
 
-void WebAutomationSession::performInteractionSequence(const String& handle, const JSON::Array& inputSources, const JSON::Array& steps, Ref<WebAutomationSession::PerformInteractionSequenceCallback>&& callback)
+void WebAutomationSession::performInteractionSequence(const String& handle, const String* optionalFrameHandle, const JSON::Array& inputSources, const JSON::Array& steps, Ref<WebAutomationSession::PerformInteractionSequenceCallback>&& callback)
 {
     // This command implements WebKit support for §17.5 Perform Actions.
 
@@ -1674,6 +1696,10 @@ void WebAutomationSession::performInteractionSequence(const String& handle, cons
     if (!page)
         ASYNC_FAIL_WITH_PREDEFINED_ERROR(WindowNotFound);
 
+    auto frameID = webFrameIDForHandle(optionalFrameHandle ? *optionalFrameHandle : emptyString());
+    if (!frameID)
+        ASYNC_FAIL_WITH_PREDEFINED_ERROR(FrameNotFound);
+
     HashMap<String, Ref<SimulatedInputSource>> sourceIdToInputSourceMap;
     HashMap<SimulatedInputSource::Type, String, WTF::IntHash<SimulatedInputSource::Type>, WTF::StrongEnumHashTraits<SimulatedInputSource::Type>> typeToSourceIdMap;
 
@@ -1758,6 +1784,17 @@ void WebAutomationSession::performInteractionSequence(const String& handle, cons
                 sourceState.pressedMouseButton = protocolMouseButtonToWebMouseEventButton(protocolButton.value_or(Inspector::Protocol::Automation::MouseButton::None));
             }
 
+            String originString;
+            if (stateObject->getString(ASCIILiteral("origin"), originString))
+                sourceState.origin = Inspector::Protocol::AutomationHelpers::parseEnumValueFromString<Inspector::Protocol::Automation::MouseMoveOrigin>(originString);
+
+            if (sourceState.origin && sourceState.origin.value() == Inspector::Protocol::Automation::MouseMoveOrigin::Element) {
+                String nodeHandleString;
+                if (!stateObject->getString(ASCIILiteral("nodeHandle"), nodeHandleString))
+                    ASYNC_FAIL_WITH_PREDEFINED_ERROR_AND_DETAILS(InvalidParameter, "Node handle not provided for 'Element' origin");
+                sourceState.nodeHandle = nodeHandleString;
+            }
+
             RefPtr<JSON::Object> locationObject;
             if (stateObject->getObject(ASCIILiteral("location"), locationObject)) {
                 int x, y;
@@ -1782,7 +1819,7 @@ void WebAutomationSession::performInteractionSequence(const String& handle, cons
     }
 
     // Delegate the rest of §17.4 Dispatching Actions to the dispatcher.
-    inputDispatcher.run(WTFMove(keyFrames), m_inputSources, [protectedThis = makeRef(*this), callback = WTFMove(callback)](std::optional<AutomationCommandError> error) {
+    inputDispatcher.run(frameID.value(), WTFMove(keyFrames), m_inputSources, [protectedThis = makeRef(*this), callback = WTFMove(callback)](std::optional<AutomationCommandError> error) {
         if (error)
             callback->sendFailure(error.value().toProtocolString());
         else
@@ -1791,7 +1828,7 @@ void WebAutomationSession::performInteractionSequence(const String& handle, cons
 #endif // PLATFORM(COCOA) || PLATFORM(GTK)
 }
 
-void WebAutomationSession::cancelInteractionSequence(const String& handle, Ref<CancelInteractionSequenceCallback>&& callback)
+void WebAutomationSession::cancelInteractionSequence(const String& handle, const String* optionalFrameHandle, Ref<CancelInteractionSequenceCallback>&& callback)
 {
     // This command implements WebKit support for §17.6 Release Actions.
 
@@ -1802,11 +1839,15 @@ void WebAutomationSession::cancelInteractionSequence(const String& handle, Ref<C
     if (!page)
         ASYNC_FAIL_WITH_PREDEFINED_ERROR(WindowNotFound);
 
+    auto frameID = webFrameIDForHandle(optionalFrameHandle ? *optionalFrameHandle : emptyString());
+    if (!frameID)
+        ASYNC_FAIL_WITH_PREDEFINED_ERROR(FrameNotFound);
+
     Vector<SimulatedInputKeyFrame> keyFrames({ SimulatedInputKeyFrame::keyFrameToResetInputSources(m_inputSources) });
     SimulatedInputDispatcher& inputDispatcher = inputDispatcherForPage(*page);
     inputDispatcher.cancel();
     
-    inputDispatcher.run(WTFMove(keyFrames), m_inputSources, [protectedThis = makeRef(*this), callback = WTFMove(callback)](std::optional<AutomationCommandError> error) {
+    inputDispatcher.run(frameID.value(), WTFMove(keyFrames), m_inputSources, [protectedThis = makeRef(*this), callback = WTFMove(callback)](std::optional<AutomationCommandError> error) {
         if (error)
             callback->sendFailure(error.value().toProtocolString());
         else
index 4d248cc..bd24ddc 100644 (file)
@@ -138,6 +138,7 @@ public:
     // SimulatedInputDispatcher::Client API
     void simulateMouseInteraction(WebPageProxy&, MouseInteraction, WebMouseEvent::Button, const WebCore::IntPoint& locationInView, AutomationCompletionHandler&&) final;
     void simulateKeyboardInteraction(WebPageProxy&, KeyboardInteraction, std::optional<VirtualKey>, std::optional<CharKey>, AutomationCompletionHandler&&) final;
+    void viewportInViewCenterPointOfElement(WebPageProxy&, uint64_t frameID, const String& nodeHandle, Function<void (std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>&&) final;
 
     // Inspector::AutomationBackendDispatcherHandler API
     // NOTE: the set of declarations included in this interface depend on the "platform" property in Automation.json
@@ -159,8 +160,8 @@ public:
     void evaluateJavaScriptFunction(const String& browsingContextHandle, const String* optionalFrameHandle, const String& function, const JSON::Array& arguments, const bool* optionalExpectsImplicitCallbackArgument, const int* optionalCallbackTimeout, Ref<Inspector::AutomationBackendDispatcherHandler::EvaluateJavaScriptFunctionCallback>&&) override;
     void performMouseInteraction(const String& handle, const JSON::Object& requestedPosition, const String& mouseButton, const String& mouseInteraction, const JSON::Array& keyModifiers, Ref<PerformMouseInteractionCallback>&&) final;
     void performKeyboardInteractions(const String& handle, const JSON::Array& interactions, Ref<PerformKeyboardInteractionsCallback>&&) override;
-    void performInteractionSequence(const String& handle, const JSON::Array& sources, const JSON::Array& steps, Ref<PerformInteractionSequenceCallback>&&) override;
-    void cancelInteractionSequence(const String& handle, Ref<CancelInteractionSequenceCallback>&&) override;
+    void performInteractionSequence(const String& handle, const String* optionalFrameHandle, const JSON::Array& sources, const JSON::Array& steps, Ref<PerformInteractionSequenceCallback>&&) override;
+    void cancelInteractionSequence(const String& handle, const String* optionalFrameHandle, Ref<CancelInteractionSequenceCallback>&&) override;
     void takeScreenshot(const String& handle, const String* optionalFrameHandle, const String* optionalNodeHandle, const bool* optionalScrollIntoViewIfNeeded, const bool* optionalClipToViewport, Ref<TakeScreenshotCallback>&&) override;
     void resolveChildFrameHandle(const String& browsingContextHandle, const String* optionalFrameHandle, const int* optionalOrdinal, const String* optionalName, const String* optionalNodeHandle, Ref<ResolveChildFrameHandleCallback>&&) override;
     void resolveParentFrameHandle(const String& browsingContextHandle, const String& frameHandle, Ref<ResolveParentFrameHandleCallback>&&) override;
@@ -276,9 +277,14 @@ private:
     uint64_t m_nextResolveParentFrameCallbackID { 1 };
     HashMap<uint64_t, RefPtr<Inspector::AutomationBackendDispatcherHandler::ResolveParentFrameHandleCallback>> m_resolveParentFrameHandleCallbacks;
 
-    uint64_t m_nextComputeElementLayoutCallbackID { 1 };
+    // Start at 2 and use only even numbers to not conflict with m_nextViewportInViewCenterPointOfElementCallbackID.
+    uint64_t m_nextComputeElementLayoutCallbackID { 2 };
     HashMap<uint64_t, RefPtr<Inspector::AutomationBackendDispatcherHandler::ComputeElementLayoutCallback>> m_computeElementLayoutCallbacks;
 
+    // Start at 3 and use only odd numbers to not conflict with m_nextComputeElementLayoutCallbackID.
+    uint64_t m_nextViewportInViewCenterPointOfElementCallbackID { 3 };
+    HashMap<uint64_t, Function<void(std::optional<WebCore::IntPoint>, std::optional<AutomationCommandError>)>> m_viewportInViewCenterPointOfElementCallbacks;
+
     uint64_t m_nextScreenshotCallbackID { 1 };
     HashMap<uint64_t, RefPtr<Inspector::AutomationBackendDispatcherHandler::TakeScreenshotCallback>> m_screenshotCallbacks;
 
index 17e9a37..3487fa9 100644 (file)
@@ -39,6 +39,7 @@
 #define STRING_FOR_PREDEFINED_ERROR_MESSAGE_AND_DETAILS(errorMessage, detailsString) makeString(Inspector::Protocol::AutomationHelpers::getEnumConstantValue(VALIDATED_ERROR_MESSAGE(errorMessage)), errorNameAndDetailsSeparator, detailsString)
 
 #define AUTOMATION_COMMAND_ERROR_WITH_NAME(errorName) AutomationCommandError(Inspector::Protocol::Automation::ErrorMessage::errorName)
+#define AUTOMATION_COMMAND_ERROR_WITH_MESSAGE(errorString) AutomationCommandError(VALIDATED_ERROR_MESSAGE(errorString))
 
 // Convenience macros for filling in the error string of synchronous commands in bailout branches.
 #define SYNC_FAIL_WITH_PREDEFINED_ERROR(errorName) \
index 09fec45..72d8897 100644 (file)
@@ -1,3 +1,14 @@
+2018-05-09  Carlos Garcia Campos  <cgarcia@igalia.com>
+
+        WebDriver: implement advance user interactions
+        https://bugs.webkit.org/show_bug.cgi?id=174616
+
+        Reviewed by Brian Burg.
+
+        Update test expectations.
+
+        * TestExpectations.json:
+
 2018-04-25  Carlos Garcia Campos  <cgarcia@igalia.com>
 
         Unreviewed gardening. Update expectations for new tests added in r230953.
index 29080dc..8f881b4 100644 (file)
         }
     },
     "imported/selenium/py/test/selenium/webdriver/common/interactions_tests.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "testClickingOnFormElements": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "testSelectingMultipleItems": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "testSendingKeysToActiveElementWithModifier": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/selenium/py/test/selenium/webdriver/common/position_and_size_tests.py": {
         "subtests": {
         }
     },
     "imported/selenium/py/test/selenium/webdriver/common/w3c_interaction_tests.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "testSendingKeysToActiveElementWithModifier": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/selenium/py/test/selenium/webdriver/common/window_tests.py": {
         "subtests": {
         }
     },
     "imported/w3c/webdriver/tests/actions/key.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "test_single_printable_key_sends_correct_events[\\xe0-]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_printable_key_sends_correct_events[\\u0416-]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_printable_key_sends_correct_events[\\u2603-]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_printable_key_sends_correct_events[\\uf6c2-]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_emoji_records_correct_key[\\U0001f604]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_emoji_records_correct_key[\\U0001f60d]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_modifier_key_sends_correct_events[\\ue053-OSRight-Meta]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_single_modifier_key_sends_correct_events[\\ue009-ControlLeft-Control]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_sequence_of_keydown_printable_keys_sends_events": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/w3c/webdriver/tests/actions/key_shortcuts.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "test_mod_a_and_backspace_deletes_all_text": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_mod_a_mod_c_right_mod_v_pastes_text": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_mod_a_mod_x_deletes_all_text": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/w3c/webdriver/tests/actions/modifier_click.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
-    },
-    "imported/w3c/webdriver/tests/actions/mouse.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
-    },
-    "imported/w3c/webdriver/tests/actions/mouse_dblclick.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
-    },
-    "imported/w3c/webdriver/tests/actions/mouse_pause_dblclick.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "test_many_modifiers_click": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/w3c/webdriver/tests/actions/pointer_origin.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "test_element_larger_than_viewport": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/w3c/webdriver/tests/actions/sequence.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "test_release_char_sequence_sends_keyup_events_in_reverse": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/w3c/webdriver/tests/actions/special_keys.py": {
-        "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/174616"}}
+        "subtests": {
+            "test_webdriver_special_key_sends_keydown[F12-expected10]": {
+                "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F11-expected47]": {
+                "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F5-expected55]": {
+                "expected": {"all": {"status": ["SKIP"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[SHIFT-expected3]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_ARROWRIGHT-expected4]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[PAGE_UP-expected6]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_PAGEUP-expected7]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[META-expected11]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NULL-expected15]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[SUBTRACT-expected16]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[CONTROL-expected17]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_META-expected19]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[SEMICOLON-expected20]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD4-expected22]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_ALT-expected25]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[DECIMAL-expected27]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_DELETE-expected29]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[PAGE_DOWN-expected30]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[PAUSE-expected31]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_ARROWUP-expected34]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[CLEAR-expected36]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_ARROWLEFT-expected37]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[EQUALS-expected38]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_PAGEDOWN-expected39]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[ADD-expected40]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD1-expected41]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_INSERT-expected42]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[ENTER-expected43]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[CANCEL-expected44]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD6-expected45]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_END-expected48]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD7-expected49]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD2-expected50]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F5-expected55]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F6-expected56]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F6-expected56]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F7-expected57]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F7-expected57]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F8-expected58]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F8-expected58]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F9-expected59]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[F9-expected59]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD8-expected60]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD8-expected60]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD5-expected61]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[NUMPAD5-expected61]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_CONTROL-expected62]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_CONTROL-expected62]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_HOME-expected63]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_HOME-expected63]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[ZENKAKUHANKAKU-expected64]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[ZENKAKUHANKAKU-expected64]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_SHIFT-expected65]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_SHIFT-expected65]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[SEPARATOR-expected66]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[SEPARATOR-expected66]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[ALT-expected67]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[ALT-expected67]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_ARROWDOWN-expected68]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[R_ARROWDOWN-expected68]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[DELETE-expected69]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_webdriver_special_key_sends_keydown[DELETE-expected69]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_multiple_codepoint_keys_behave_correctly[f]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_multiple_codepoint_keys_behave_correctly[f]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_multiple_codepoint_keys_behave_correctly[\u0ba8\u0bbf]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_multiple_codepoint_keys_behave_correctly[\u0ba8\u0bbf]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_multiple_codepoint_keys_behave_correctly[\u1100\u1161\u11a8]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_multiple_codepoint_keys_behave_correctly[\u1100\u1161\u11a8]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[fa]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[fa]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[\u0ba8\u0bbfb]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[\u0ba8\u0bbfb]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[\u0ba8\u0bbf\u0ba8]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[\u0ba8\u0bbf\u0ba8]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[\u1100\u1161\u11a8c]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            },
+            "test_invalid_multiple_codepoint_keys_fail[\u1100\u1161\u11a8c]": {
+                "expected": {"all": {"status": ["FAIL"], "bug": "webkit.org/b/184967"}}
+            }
+        }
     },
     "imported/w3c/webdriver/tests/contexts/maximize_window.py": {
         "subtests": {