[WebGL] Simulated vertexAttrib0 can sometimes cause OUT_OF_MEMORY errors
authordino@apple.com <dino@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 12 Jan 2018 23:01:32 +0000 (23:01 +0000)
committerdino@apple.com <dino@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Fri, 12 Jan 2018 23:01:32 +0000 (23:01 +0000)
commit71d3fa4b2d7969dbb03e6e350bb8ef90748b78d8
tree43fd863ec22cd4e08e6e0458bbbc2ba232e64dd0
parentcef5c46e72242a3916ee3444d554165a3c7dd7e0
[WebGL] Simulated vertexAttrib0 can sometimes cause OUT_OF_MEMORY errors
https://bugs.webkit.org/show_bug.cgi?id=181558
<rdar://problem/36189833>

Reviewed by Eric Carlson.

Source/WebCore:

Very large element indices in the ELEMENT_ARRAY_BUFFER meant that
our simulated vertexAttrib0 buffer might be too large. We need
to check for out-of-memory, but we can also detect some of the issues
earlier in our validation code. Additionally, make sure that we don't
accidentally cast an unsigned to a signed.

Test: fast/canvas/webgl/simulated-vertexAttrib0-invalid-indicies.html

* html/canvas/WebGL2RenderingContext.cpp:
(WebCore::WebGL2RenderingContext::validateIndexArrayConservative): Update validation
code to look for overflow, rather than relying on looking for sign changes.
* html/canvas/WebGLRenderingContext.cpp:
(WebCore::WebGLRenderingContext::validateIndexArrayConservative): Ditto.
* html/canvas/WebGLRenderingContextBase.cpp:
(WebCore::WebGLRenderingContextBase::validateIndexArrayPrecise):
(WebCore::WebGLRenderingContextBase::drawArrays): Check that we were able to simulate.
(WebCore::WebGLRenderingContextBase::drawElements):
(WebCore::WebGLRenderingContextBase::validateSimulatedVertexAttrib0): Update validation code, and
use GC3Duint, since that's what the indicies are.
(WebCore::WebGLRenderingContextBase::simulateVertexAttrib0): Ditto.
(WebCore::WebGLRenderingContextBase::drawArraysInstanced): Check that we were able to simulate.
(WebCore::WebGLRenderingContextBase::drawElementsInstanced):
* html/canvas/WebGLRenderingContextBase.h:

LayoutTests:

* fast/canvas/webgl/simulated-vertexAttrib0-invalid-indicies-expected.txt: Added.
* fast/canvas/webgl/simulated-vertexAttrib0-invalid-indicies.html: Added.
* platform/mac/TestExpectations: Test crashes on Sierra and earlier.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@226916 268f45cc-cd09-0410-ab3c-d52691b4dbfc
LayoutTests/ChangeLog
LayoutTests/fast/canvas/webgl/simulated-vertexAttrib0-invalid-indicies-expected.txt [new file with mode: 0644]
LayoutTests/fast/canvas/webgl/simulated-vertexAttrib0-invalid-indicies.html [new file with mode: 0755]
LayoutTests/platform/mac/TestExpectations
Source/WebCore/ChangeLog
Source/WebCore/html/canvas/WebGL2RenderingContext.cpp
Source/WebCore/html/canvas/WebGLRenderingContext.cpp
Source/WebCore/html/canvas/WebGLRenderingContextBase.cpp
Source/WebCore/html/canvas/WebGLRenderingContextBase.h