Teach TextIndicator to estimate the background color of the given Range
authorwenson_hsieh@apple.com <wenson_hsieh@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 21 Mar 2017 21:25:47 +0000 (21:25 +0000)
committerwenson_hsieh@apple.com <wenson_hsieh@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 21 Mar 2017 21:25:47 +0000 (21:25 +0000)
https://bugs.webkit.org/show_bug.cgi?id=169869
<rdar://problem/31127272>

Reviewed by Anders Carlsson and Simon Fraser.

Source/WebCore:

Introduces a simple heuristic to compute a background color that is appropriate to use as a border surrounding
the snapshot of the given Range. This work is only carried out if TextIndicatorOptionComputeEstimatedBackgroundColor
is specified. The details of how this background color is estimated (as well as when to fall back) can be improved
in several ways (one idea is to sample colors along the edge of the snapshot). For the time being, this patch
naively walks up the render tree in search of enclosing parent renderers that have background colors. If any
renderers have a style that is deemed too complex to capture in a single background color, then fall back to the
base document background color; otherwise, the estimated background color is the result of blending these
background colors together.

* page/TextIndicator.cpp:
(WebCore::styleContainsComplexBackground):

Bail out of the background color codepath if the renderer has backdrop filters, has a background image, or uses
a non-normal blend mode.

(WebCore::fallbackBackgroundColorForTextSelection):
(WebCore::estimatedBackgroundColorForRange):
(WebCore::initializeIndicator):
* page/TextIndicator.h:

Source/WebKit/mac:

Plumb the estimated background color for WebKit1 clients through a new property in WebUITextIndicatorData.

* WebView/WebView.mm:
(-[WebUITextIndicatorData initWithImage:textIndicatorData:scale:]):
(-[WebUITextIndicatorData dealloc]):
* WebView/WebViewPrivate.h:

Source/WebKit2:

Send the estimated background color over XPC, and adopt TextIndicatorOptionComputeEstimatedBackgroundColor when
snapshotting after performing an edit data interaction.

* Shared/WebCoreArgumentCoders.cpp:
(IPC::ArgumentCoder<TextIndicatorData>::encode):
(IPC::ArgumentCoder<TextIndicatorData>::decode):
* WebProcess/WebPage/ios/WebPageIOS.mm:
(WebKit::WebPage::didConcludeEditDataInteraction):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@214231 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/WebCore/ChangeLog
Source/WebCore/page/TextIndicator.cpp
Source/WebCore/page/TextIndicator.h
Source/WebKit/mac/ChangeLog
Source/WebKit/mac/WebView/WebView.mm
Source/WebKit/mac/WebView/WebViewPrivate.h
Source/WebKit2/ChangeLog
Source/WebKit2/Shared/WebCoreArgumentCoders.cpp
Source/WebKit2/WebProcess/WebPage/ios/WebPageIOS.mm

index c7457c7..27ee2a2 100644 (file)
@@ -1,3 +1,31 @@
+2017-03-19  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        Teach TextIndicator to estimate the background color of the given Range
+        https://bugs.webkit.org/show_bug.cgi?id=169869
+        <rdar://problem/31127272>
+
+        Reviewed by Anders Carlsson and Simon Fraser.
+
+        Introduces a simple heuristic to compute a background color that is appropriate to use as a border surrounding
+        the snapshot of the given Range. This work is only carried out if TextIndicatorOptionComputeEstimatedBackgroundColor
+        is specified. The details of how this background color is estimated (as well as when to fall back) can be improved
+        in several ways (one idea is to sample colors along the edge of the snapshot). For the time being, this patch
+        naively walks up the render tree in search of enclosing parent renderers that have background colors. If any
+        renderers have a style that is deemed too complex to capture in a single background color, then fall back to the
+        base document background color; otherwise, the estimated background color is the result of blending these
+        background colors together.
+
+        * page/TextIndicator.cpp:
+        (WebCore::styleContainsComplexBackground):
+
+        Bail out of the background color codepath if the renderer has backdrop filters, has a background image, or uses
+        a non-normal blend mode.
+
+        (WebCore::fallbackBackgroundColorForTextSelection):
+        (WebCore::estimatedBackgroundColorForRange):
+        (WebCore::initializeIndicator):
+        * page/TextIndicator.h:
+
 2017-03-20  Jer Noble  <jer.noble@apple.com>
 
         Crash in WebCore: WebCore::CARingBuffer::getCurrentFrameBounds + 28
index d57b68d..6c8fb18 100644 (file)
@@ -39,6 +39,7 @@
 #include "IntRect.h"
 #include "NodeTraversal.h"
 #include "Range.h"
+#include "RenderElement.h"
 #include "RenderObject.h"
 
 #if PLATFORM(IOS)
@@ -200,8 +201,61 @@ static void getSelectionRectsForRange(Vector<FloatRect>& resultingRects, const R
 
 #endif
 
+static bool styleContainsComplexBackground(const RenderStyle& style)
+{
+    if (style.hasBlendMode())
+        return true;
+
+    if (style.hasBackgroundImage())
+        return true;
+
+    if (style.hasBackdropFilter())
+        return true;
+
+    return false;
+}
+
+static Color estimatedBackgroundColorForRange(const Range& range, const Frame& frame)
+{
+    auto estimatedBackgroundColor = frame.view() ? frame.view()->documentBackgroundColor() : Color::transparent;
+
+    RenderElement* renderer = nullptr;
+    auto commonAncestor = range.commonAncestorContainer();
+    while (commonAncestor) {
+        if (is<RenderElement>(commonAncestor->renderer())) {
+            renderer = downcast<RenderElement>(commonAncestor->renderer());
+            break;
+        }
+        commonAncestor = commonAncestor->parentOrShadowHostElement();
+    }
+
+    auto boundingRectForRange = enclosingIntRect(range.absoluteBoundingRect());
+    Vector<Color> parentRendererBackgroundColors;
+    for (; !!renderer; renderer = renderer->parent()) {
+        auto absoluteBoundingBox = renderer->absoluteBoundingBoxRect();
+        auto& style = renderer->style();
+        if (!absoluteBoundingBox.contains(boundingRectForRange) || !style.hasBackground())
+            continue;
+
+        if (styleContainsComplexBackground(style))
+            return estimatedBackgroundColor;
+
+        auto visitedDependentBackgroundColor = style.visitedDependentColor(CSSPropertyBackgroundColor);
+        if (visitedDependentBackgroundColor != Color::transparent)
+            parentRendererBackgroundColors.append(visitedDependentBackgroundColor);
+    }
+    parentRendererBackgroundColors.reverse();
+    for (auto backgroundColor : parentRendererBackgroundColors)
+        estimatedBackgroundColor = estimatedBackgroundColor.blend(backgroundColor);
+
+    return estimatedBackgroundColor;
+}
+
 static bool initializeIndicator(TextIndicatorData& data, Frame& frame, const Range& range, FloatSize margin, bool indicatesCurrentSelection)
 {
+    if (data.options & TextIndicatorOptionComputeEstimatedBackgroundColor)
+        data.estimatedBackgroundColor = estimatedBackgroundColorForRange(range, frame);
+
     Vector<FloatRect> textRects;
 
     // FIXME (138888): Ideally we wouldn't remove the margin in this case, but we need to
index 25ba5c3..a4fa168 100644 (file)
@@ -88,6 +88,10 @@ enum TextIndicatorOption : uint16_t {
     // selection rects that would enclose the given Range instead.
     // Currently, this is only supported on iOS.
     TextIndicatorOptionUseSelectionRectForSizing = 1 << 9,
+
+    // Compute a background color to use when rendering a platter around the content image, falling back to a default if the
+    // content's background is too complex to be captured by a single color.
+    TextIndicatorOptionComputeEstimatedBackgroundColor = 1 << 10,
 };
 typedef uint16_t TextIndicatorOptions;
 
@@ -100,6 +104,7 @@ struct TextIndicatorData {
     RefPtr<Image> contentImageWithHighlight;
     RefPtr<Image> contentImageWithoutSelection;
     RefPtr<Image> contentImage;
+    Color estimatedBackgroundColor;
     TextIndicatorPresentationTransition presentationTransition;
     TextIndicatorOptions options;
 };
index 869af68..be20cfd 100644 (file)
@@ -1,3 +1,18 @@
+2017-03-19  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        Teach TextIndicator to estimate the background color of the given Range
+        https://bugs.webkit.org/show_bug.cgi?id=169869
+        <rdar://problem/31127272>
+
+        Reviewed by Anders Carlsson and Simon Fraser.
+
+        Plumb the estimated background color for WebKit1 clients through a new property in WebUITextIndicatorData.
+
+        * WebView/WebView.mm:
+        (-[WebUITextIndicatorData initWithImage:textIndicatorData:scale:]):
+        (-[WebUITextIndicatorData dealloc]):
+        * WebView/WebViewPrivate.h:
+
 2017-03-20  Dan Bernstein  <mitz@apple.com>
 
         [Xcode] Incremental builds of WebKitLegacy are slow because of the Postprocess Headers phase
index dab80e7..eabe53b 100644 (file)
@@ -638,6 +638,7 @@ private:
 @synthesize contentImageWithoutSelection=_contentImageWithoutSelection;
 @synthesize contentImageWithoutSelectionRectInRootViewCoordinates=_contentImageWithoutSelectionRectInRootViewCoordinates;
 @synthesize contentImage=_contentImage;
+@synthesize estimatedBackgroundColor=_estimatedBackgroundColor;
 
 @end
 
@@ -670,6 +671,9 @@ private:
         }
     }
 
+    if (indicatorData.options & TextIndicatorOptionComputeEstimatedBackgroundColor)
+        _estimatedBackgroundColor = [[[getUIColorClass() alloc] initWithCGColor:cachedCGColor(indicatorData.estimatedBackgroundColor)] retain];
+
     return self;
 }
 
@@ -690,6 +694,7 @@ private:
     [_contentImageWithHighlight release];
     [_contentImageWithoutSelection release];
     [_contentImage release];
+    [_estimatedBackgroundColor release];
     
     [super dealloc];
 }
index 4012ef1..53c8649 100644 (file)
@@ -57,6 +57,7 @@
 
 #define WEBKIT_DI_BLOCK 1
 
+@class UIColor;
 @class UIImage;
 @class NSError;
 @class WebFrame;
@@ -193,6 +194,7 @@ typedef enum {
 @property (nonatomic, retain) UIImage *contentImage;
 @property (nonatomic, retain) UIImage *contentImageWithoutSelection;
 @property (nonatomic, assign) CGRect contentImageWithoutSelectionRectInRootViewCoordinates;
+@property (nonatomic, retain) UIColor *estimatedBackgroundColor;
 @end
 
 #if !TARGET_OS_IPHONE
index 2bda3cf..509b5fa 100644 (file)
@@ -1,3 +1,20 @@
+2017-03-19  Wenson Hsieh  <wenson_hsieh@apple.com>
+
+        Teach TextIndicator to estimate the background color of the given Range
+        https://bugs.webkit.org/show_bug.cgi?id=169869
+        <rdar://problem/31127272>
+
+        Reviewed by Anders Carlsson and Simon Fraser.
+
+        Send the estimated background color over XPC, and adopt TextIndicatorOptionComputeEstimatedBackgroundColor when
+        snapshotting after performing an edit data interaction.
+
+        * Shared/WebCoreArgumentCoders.cpp:
+        (IPC::ArgumentCoder<TextIndicatorData>::encode):
+        (IPC::ArgumentCoder<TextIndicatorData>::decode):
+        * WebProcess/WebPage/ios/WebPageIOS.mm:
+        (WebKit::WebPage::didConcludeEditDataInteraction):
+
 2017-03-21  Chris Dumez  <cdumez@apple.com>
 
         [WK2] Call processDidCrash() right away when terminating a process that exceeded background CPU limit
index 8848c0f..b493c5f 100644 (file)
@@ -2048,6 +2048,7 @@ void ArgumentCoder<TextIndicatorData>::encode(Encoder& encoder, const TextIndica
     encoder << textIndicatorData.textRectsInBoundingRectCoordinates;
     encoder << textIndicatorData.contentImageWithoutSelectionRectInRootViewCoordinates;
     encoder << textIndicatorData.contentImageScaleFactor;
+    encoder << textIndicatorData.estimatedBackgroundColor;
     encoder.encodeEnum(textIndicatorData.presentationTransition);
     encoder << static_cast<uint64_t>(textIndicatorData.options);
 
@@ -2073,6 +2074,9 @@ bool ArgumentCoder<TextIndicatorData>::decode(Decoder& decoder, TextIndicatorDat
     if (!decoder.decode(textIndicatorData.contentImageScaleFactor))
         return false;
 
+    if (!decoder.decode(textIndicatorData.estimatedBackgroundColor))
+        return false;
+
     if (!decoder.decodeEnum(textIndicatorData.presentationTransition))
         return false;
 
index e08612d..d6c32e0 100644 (file)
@@ -631,10 +631,10 @@ void WebPage::didConcludeEditDataInteraction()
 {
     std::optional<TextIndicatorData> textIndicatorData;
 
-    static auto defaultEditDragTextIndicatorOptions = TextIndicatorOptionIncludeSnapshotOfAllVisibleContentWithoutSelection | TextIndicatorOptionDoNotClipToVisibleRect | TextIndicatorOptionPaintAllContent | TextIndicatorOptionIncludeMarginIfRangeMatchesSelection | TextIndicatorOptionPaintBackgrounds | TextIndicatorOptionUseSelectionRectForSizing | TextIndicatorOptionIncludeSnapshotWithSelectionHighlight;
+    static auto defaultEditDataInteractionTextIndicatorOptions = TextIndicatorOptionIncludeSnapshotOfAllVisibleContentWithoutSelection | TextIndicatorOptionDoNotClipToVisibleRect | TextIndicatorOptionPaintAllContent | TextIndicatorOptionIncludeMarginIfRangeMatchesSelection | TextIndicatorOptionPaintBackgrounds | TextIndicatorOptionComputeEstimatedBackgroundColor| TextIndicatorOptionUseSelectionRectForSizing | TextIndicatorOptionIncludeSnapshotWithSelectionHighlight;
     auto& frame = m_page->focusController().focusedOrMainFrame();
     if (auto range = frame.selection().selection().toNormalizedRange()) {
-        if (auto textIndicator = TextIndicator::createWithRange(*range, defaultEditDragTextIndicatorOptions, TextIndicatorPresentationTransition::None))
+        if (auto textIndicator = TextIndicator::createWithRange(*range, defaultEditDataInteractionTextIndicatorOptions, TextIndicatorPresentationTransition::None))
             textIndicatorData = textIndicator->data();
     }