Optimize SharedArrayBuffer in the DFG+FTL
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 20 Apr 2017 17:55:44 +0000 (17:55 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 20 Apr 2017 17:55:44 +0000 (17:55 +0000)
https://bugs.webkit.org/show_bug.cgi?id=164108

Reviewed by Saam Barati.

JSTests:

Added a fairly comprehensive test of the intrinsics. This creates a function for each possible
combination of type and operation, and then first uses it nicely and then tries a bunch of
erroneous conditions like OOB.

* stress/SharedArrayBuffer-opt.js: Added.
(string_appeared_here.switch):
(string_appeared_here.str):
(runAtomic):
(shouldFail):
(Symbol):
(string_appeared_here.a.of.arrays.m.of.atomics):
* stress/SharedArrayBuffer.js:

Source/JavaScriptCore:

This adds atomics intrinsics to the DFG and wires them through to the DFG and FTL backends. This
was super easy in the FTL since B3 already has comprehensive atomic intrinsics, which are more
powerful than what we need right now. In the DFG backend, I went with an easy-to-write
implementation that just reduces everything to a weak CAS loop. It's very inefficient with
registers (it needs ~8) but it's the DFG backend, so it's not obvious how much we care.

To make the rare cases easy to handle, I refactored AtomicsObject.cpp so that the operations for
the slow paths can share code with the native functions.

This also fixes register handling in the X86 implementations of CAS, in the case that
expectedAndResult is not %rax. This also fixes the ARM64 implementation of branchWeakCAS.

I adapted the CascadeLock from WTF/benchmarks/ToyLocks.h as a microbenchmark of lock performance.
This benchmark performs 2.5x faster, in both the contended and uncontended case, thanks to this
change. It's still about 3x slower than native. I investigated this only a bit. I suspect that
the story will be different in asm.js code, which will get constant-folding of the typed array
backing store by virtue of how it uses lexically scoped variables as pointers to the heap arrays.
It's worth noting that the native lock I was comparing against, the very nicely-tuned
CascadeLock, is at the very high end of lock throughput under virtually all conditions
(uncontended, microcontended, held for a long time). I also compared to WTF::Lock and others, and
the only ones that performed better in this microbenchmark were spinlocks. I don't recommend
using those. So, when I say this is 3x slower than native, I really mean that it's 3x slower than
the fastest native lock that I have in my arsenal.

Also worth noting is that I experimented with exposing Atomics.yield(), which uses sched_yield,
as a way of testing if adding a yield loop to the JS cascadeLock would help. It does not help. I
did not investigate why.

* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::append):
* assembler/CPU.h:
(JSC::is64Bit):
(JSC::is32Bit):
* b3/B3Common.h:
(JSC::B3::is64Bit): Deleted.
(JSC::B3::is32Bit): Deleted.
* b3/B3LowerToAir.cpp:
(JSC::B3::Air::LowerToAir::appendTrapping):
(JSC::B3::Air::LowerToAir::appendCAS):
(JSC::B3::Air::LowerToAir::appendGeneralAtomic):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGByteCodeParser.cpp:
(JSC::DFG::ByteCodeParser::handleIntrinsicCall):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGNode.h:
(JSC::DFG::Node::hasHeapPrediction):
(JSC::DFG::Node::hasArrayMode):
* dfg/DFGNodeType.h:
(JSC::DFG::isAtomicsIntrinsic):
(JSC::DFG::numExtraAtomicsArgs):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSSALoweringPhase.cpp:
(JSC::DFG::SSALoweringPhase::handleNode):
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::loadFromIntTypedArray):
(JSC::DFG::SpeculativeJIT::setIntTypedArrayLoadResult):
(JSC::DFG::SpeculativeJIT::compileGetByValOnIntTypedArray):
(JSC::DFG::SpeculativeJIT::getIntTypedArrayStoreOperand):
(JSC::DFG::SpeculativeJIT::compilePutByValForIntTypedArray):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::callOperation):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* ftl/FTLAbstractHeapRepository.cpp:
(JSC::FTL::AbstractHeapRepository::decorateFencedAccess):
(JSC::FTL::AbstractHeapRepository::computeRangesAndDecorateInstructions):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileAtomicsReadModifyWrite):
(JSC::FTL::DFG::LowerDFGToB3::compileAtomicsIsLockFree):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compilePutByVal):
(JSC::FTL::DFG::LowerDFGToB3::pointerIntoTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::loadFromIntTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::storeType):
(JSC::FTL::DFG::LowerDFGToB3::setIntTypedArrayLoadResult):
(JSC::FTL::DFG::LowerDFGToB3::getIntTypedArrayStoreOperand):
(JSC::FTL::DFG::LowerDFGToB3::vmCall):
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::store):
(JSC::FTL::Output::store32As8):
(JSC::FTL::Output::store32As16):
(JSC::FTL::Output::atomicXchgAdd):
(JSC::FTL::Output::atomicXchgAnd):
(JSC::FTL::Output::atomicXchgOr):
(JSC::FTL::Output::atomicXchgSub):
(JSC::FTL::Output::atomicXchgXor):
(JSC::FTL::Output::atomicXchg):
(JSC::FTL::Output::atomicStrongCAS):
* ftl/FTLOutput.h:
(JSC::FTL::Output::store32):
(JSC::FTL::Output::store64):
(JSC::FTL::Output::storePtr):
(JSC::FTL::Output::storeFloat):
(JSC::FTL::Output::storeDouble):
* jit/JITOperations.h:
* runtime/AtomicsObject.cpp:
(JSC::atomicsFuncAdd):
(JSC::atomicsFuncAnd):
(JSC::atomicsFuncCompareExchange):
(JSC::atomicsFuncExchange):
(JSC::atomicsFuncIsLockFree):
(JSC::atomicsFuncLoad):
(JSC::atomicsFuncOr):
(JSC::atomicsFuncStore):
(JSC::atomicsFuncSub):
(JSC::atomicsFuncWait):
(JSC::atomicsFuncWake):
(JSC::atomicsFuncXor):
(JSC::operationAtomicsAdd):
(JSC::operationAtomicsAnd):
(JSC::operationAtomicsCompareExchange):
(JSC::operationAtomicsExchange):
(JSC::operationAtomicsIsLockFree):
(JSC::operationAtomicsLoad):
(JSC::operationAtomicsOr):
(JSC::operationAtomicsStore):
(JSC::operationAtomicsSub):
(JSC::operationAtomicsXor):
* runtime/AtomicsObject.h:

Source/WTF:

Made small changes as part of benchmarking the JS versions of these locks.

* benchmarks/LockSpeedTest.cpp:
* benchmarks/ToyLocks.h:
* wtf/Range.h:
(WTF::Range::dump):

LayoutTests:

Add a test of futex performance.

* workers/sab/cascade_lock-worker.js: Added.
(onmessage):
* workers/sab/cascade_lock.html: Added.
* workers/sab/worker-resources.js:
(cascadeLockSlow):
(cascadeLock):
(cascadeUnlock):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@215565 268f45cc-cd09-0410-ab3c-d52691b4dbfc

46 files changed:
JSTests/ChangeLog
JSTests/stress/SharedArrayBuffer-opt.js [new file with mode: 0644]
JSTests/stress/SharedArrayBuffer.js
JSTests/stress/atomics-known-int-use.js [new file with mode: 0644]
JSTests/stress/isLockFree.js [new file with mode: 0644]
LayoutTests/ChangeLog
LayoutTests/workers/sab/cascade_lock-expected.txt [new file with mode: 0644]
LayoutTests/workers/sab/cascade_lock-worker.js [new file with mode: 0644]
LayoutTests/workers/sab/cascade_lock.html [new file with mode: 0644]
LayoutTests/workers/sab/worker-resources.js
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/assembler/AbstractMacroAssembler.h
Source/JavaScriptCore/assembler/CPU.h
Source/JavaScriptCore/assembler/MacroAssemblerARM64.h
Source/JavaScriptCore/assembler/MacroAssemblerX86Common.h
Source/JavaScriptCore/assembler/MacroAssemblerX86_64.h
Source/JavaScriptCore/b3/B3Common.h
Source/JavaScriptCore/b3/B3LowerToAir.cpp
Source/JavaScriptCore/b3/air/AirAllocateRegistersByGraphColoring.h
Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h
Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp
Source/JavaScriptCore/dfg/DFGClobberize.h
Source/JavaScriptCore/dfg/DFGDoesGC.cpp
Source/JavaScriptCore/dfg/DFGFixupPhase.cpp
Source/JavaScriptCore/dfg/DFGNode.h
Source/JavaScriptCore/dfg/DFGNodeType.h
Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp
Source/JavaScriptCore/dfg/DFGSSALoweringPhase.cpp
Source/JavaScriptCore/dfg/DFGSafeToExecute.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp
Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp
Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h
Source/JavaScriptCore/ftl/FTLCapabilities.cpp
Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
Source/JavaScriptCore/ftl/FTLOutput.cpp
Source/JavaScriptCore/ftl/FTLOutput.h
Source/JavaScriptCore/jit/JITOperations.h
Source/JavaScriptCore/runtime/AtomicsObject.cpp
Source/JavaScriptCore/runtime/AtomicsObject.h
Source/WTF/ChangeLog
Source/WTF/benchmarks/LockSpeedTest.cpp
Source/WTF/benchmarks/ToyLocks.h
Source/WTF/wtf/Range.h

index 8955b10..c1baffb 100644 (file)
@@ -1,3 +1,23 @@
+2017-04-15  Filip Pizlo  <fpizlo@apple.com>
+
+        Optimize SharedArrayBuffer in the DFG+FTL
+        https://bugs.webkit.org/show_bug.cgi?id=164108
+
+        Reviewed by Saam Barati.
+        
+        Added a fairly comprehensive test of the intrinsics. This creates a function for each possible
+        combination of type and operation, and then first uses it nicely and then tries a bunch of
+        erroneous conditions like OOB.
+
+        * stress/SharedArrayBuffer-opt.js: Added.
+        (string_appeared_here.switch):
+        (string_appeared_here.str):
+        (runAtomic):
+        (shouldFail):
+        (Symbol):
+        (string_appeared_here.a.of.arrays.m.of.atomics):
+        * stress/SharedArrayBuffer.js:
+
 2017-04-19  Mark Lam  <mark.lam@apple.com>
 
         B3StackmapSpecial should handle when stackmap values are not recoverable from a Def'ed arg.
diff --git a/JSTests/stress/SharedArrayBuffer-opt.js b/JSTests/stress/SharedArrayBuffer-opt.js
new file mode 100644 (file)
index 0000000..e241a48
--- /dev/null
@@ -0,0 +1,133 @@
+var dv = new DataView(new SharedArrayBuffer(128));
+var i8a = new Int8Array(new SharedArrayBuffer(128));
+var i16a = new Int16Array(new SharedArrayBuffer(128));
+var i32a = new Int32Array(new SharedArrayBuffer(128));
+var u8a = new Uint8Array(new SharedArrayBuffer(128));
+var u8ca = new Uint8ClampedArray(new SharedArrayBuffer(128));
+var u16a = new Uint16Array(new SharedArrayBuffer(128));
+var u32a = new Uint32Array(new SharedArrayBuffer(128));
+var f32a = new Float32Array(new SharedArrayBuffer(128));
+var f64a = new Float64Array(new SharedArrayBuffer(128));
+
+var arrays = [i8a, i16a, i32a, u8a, u16a, u32a];
+
+var atomics = new Map();
+var genericAtomics = new Map();
+for (var a of arrays) {
+    var map = new Map();
+    atomics.set(a, map);
+}
+var count = 0;
+for (var op of ["add", "and", "compareExchange", "exchange", "load", "or", "store", "sub", "xor"]) {
+    var numExtraArgs;
+    switch (op) {
+    case "compareExchange":
+        numExtraArgs = 2;
+        break;
+    case "load":
+        numExtraArgs = 0;
+        break;
+    default:
+        numExtraArgs = 1;
+        break;
+    }
+    
+    function str() {
+        var str = "(function (array" + count + ", index";
+        for (var i = 0; i < numExtraArgs; ++i)
+            str += ", a" + i;
+        str += ") { return Atomics." + op + "(array" + count + ", index";
+        for (var i = 0; i < numExtraArgs; ++i)
+            str += ", a" + i;
+        str += "); })";
+        count++;
+        return str;
+    }
+    
+    var f = eval(str());
+    noInline(f);
+    // Warm it up on crazy.
+    for (var i = 0; i < 10000; ++i)
+        f(arrays[i % arrays.length], 0, 0, 0);
+    genericAtomics.set(op, f);
+    
+    for (var a of arrays) {
+        var map = atomics.get(a);
+        
+        var f = eval(str());
+        noInline(f);
+        
+        // Warm it up on something easy.
+        for (var i = 0; i < 10000; ++i)
+            f(a, 0, 0, 0);
+        
+        map.set(op, f);
+    }
+}
+
+function runAtomic(array, index, init, name, args, expectedResult, expectedOutcome)
+{
+    for (var f of [{name: "specialized", func: atomics.get(array).get(name)},
+                   {name: "generic", func: genericAtomics.get(name)}]) {
+        array[index] = init;
+        var result = f.func(array, index, ...args);
+        if (result != expectedResult)
+            throw new Error("Expected Atomics." + name + "(array, " + index + ", " + args.join(", ") + ") to return " + expectedResult + " but returned " + result + " for " + Object.prototype.toString.apply(array) + " and " + f.name);
+        if (array[index] !== expectedOutcome)
+            throw new Error("Expected Atomics." + name + "(array, " + index + ", " + args.join(", ") + ") to result in array[" + index + "] = " + expectedOutcome + " but got " + array[index] + " for " + Object.prototype.toString.apply(array) + " and " + f.name);
+    }
+}
+
+for (var a of arrays) {
+    runAtomic(a, 0, 13, "add", [42], 13, 55);
+    runAtomic(a, 0, 13, "and", [42], 13, 8);
+    runAtomic(a, 0, 13, "compareExchange", [25, 42], 13, 13);
+    runAtomic(a, 0, 13, "compareExchange", [13, 42], 13, 42);
+    runAtomic(a, 0, 13, "exchange", [42], 13, 42);
+    runAtomic(a, 0, 13, "load", [], 13, 13);
+    runAtomic(a, 0, 13, "or", [42], 13, 47);
+    runAtomic(a, 0, 13, "store", [42], 42, 42);
+    runAtomic(a, 0, 42, "sub", [13], 42, 29);
+    runAtomic(a, 0, 13, "xor", [42], 13, 39);
+}
+
+function shouldFail(f, name)
+{
+    try {
+        f();
+    } catch (e) {
+        if (e.name == name.name)
+            return;
+        throw new Error(f + " threw the wrong error: " + e);
+    }
+    throw new Error(f + " succeeded!");
+}
+
+for (var bad of [void 0, null, false, true, 1, 0.5, Symbol(), {}, "hello", dv, u8ca, f32a, f64a]) {
+    shouldFail(() => genericAtomics.get("add")(bad, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("and")(bad, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("compareExchange")(bad, 0, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("exchange")(bad, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("load")(bad, 0), TypeError);
+    shouldFail(() => genericAtomics.get("or")(bad, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("store")(bad, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("sub")(bad, 0, 0), TypeError);
+    shouldFail(() => genericAtomics.get("xor")(bad, 0, 0), TypeError);
+}
+
+for (var idx of [-1, -1000000000000, 10000, 10000000000000, "hello"]) {
+    for (var a of arrays) {
+        for (var m of [atomics.get(a), genericAtomics]) {
+            shouldFail(() => m.get("add")(a, idx, 0), RangeError);
+            shouldFail(() => m.get("and")(a, idx, 0), RangeError);
+            shouldFail(() => m.get("compareExchange")(a, idx, 0, 0), RangeError);
+            shouldFail(() => m.get("exchange")(a, idx, 0), RangeError);
+            shouldFail(() => m.get("load")(a, idx), RangeError);
+            shouldFail(() => m.get("or")(a, idx, 0), RangeError);
+            shouldFail(() => m.get("store")(a, idx, 0), RangeError);
+            shouldFail(() => m.get("sub")(a, idx, 0), RangeError);
+            shouldFail(() => m.get("xor")(a, idx, 0), RangeError);
+        }
+    }
+}
+
index 27ba2fb..213d5c0 100644 (file)
@@ -85,7 +85,7 @@ for (bad of [void 0, null, false, true, 1, 0.5, Symbol(), {}, "hello", dv, i8a,
 for (idx of [-1, -1000000000000, 10000, 10000000000000, "hello"]) {
     for (a of [i8a, i16a, i32a, u8a, u16a, u32a]) {
         shouldFail(() => Atomics.add(a, idx, 0), RangeError);
-        shouldFail(() => Atomics.add(a, idx, 0), RangeError);
+        shouldFail(() => Atomics.and(a, idx, 0), RangeError);
         shouldFail(() => Atomics.compareExchange(a, idx, 0, 0), RangeError);
         shouldFail(() => Atomics.exchange(a, idx, 0), RangeError);
         shouldFail(() => Atomics.load(a, idx), RangeError);
diff --git a/JSTests/stress/atomics-known-int-use.js b/JSTests/stress/atomics-known-int-use.js
new file mode 100644 (file)
index 0000000..4e0a96c
--- /dev/null
@@ -0,0 +1,31 @@
+// Break type inference.
+var o = {f: 42.5};
+
+function foo(a, i) {
+    return Atomics.exchange(a, i.f, 42);
+}
+
+noInline(foo);
+
+var array = new Int32Array(new SharedArrayBuffer(4));
+
+for (var i = 0; i < 10000; ++i) {
+    array[0] = 13;
+    var result = foo(array, {f: 0});
+    if (result != 13)
+        throw "Error in loop: bad result: " + result;
+    if (array[0] != 42)
+        throw "Error in loop: bad value in array: " + array[0];
+}
+
+var success = false;
+try {
+    array[0] = 14;
+    var result = foo(array, {f: 42.5});
+    success = true;
+} catch (e) {
+    if (e.name != "RangeError")
+        throw "Error: bad error type: " + e;
+}
+if (success)
+    throw "Error: expected to fail, but didn't."
diff --git a/JSTests/stress/isLockFree.js b/JSTests/stress/isLockFree.js
new file mode 100644 (file)
index 0000000..1d0af22
--- /dev/null
@@ -0,0 +1,73 @@
+function foo(bytes) {
+    return Atomics.isLockFree(bytes);
+}
+noInline(foo);
+
+function foo0(bytes) {
+    return Atomics.isLockFree(0);
+}
+noInline(foo0);
+
+function foo1(bytes) {
+    return Atomics.isLockFree(1);
+}
+noInline(foo1);
+
+function foo2(bytes) {
+    return Atomics.isLockFree(2);
+}
+noInline(foo2);
+
+function foo3(bytes) {
+    return Atomics.isLockFree(3);
+}
+noInline(foo3);
+
+function foo4(bytes) {
+    return Atomics.isLockFree(4);
+}
+noInline(foo4);
+
+function foo5(bytes) {
+    return Atomics.isLockFree(5);
+}
+noInline(foo5);
+
+for (var i = 0; i < 10000; ++i) {
+    var result = foo(0);
+    if (result !== false)
+        throw new Error("Bad result: " + result);
+    var result = foo(1);
+    if (result !== true)
+        throw new Error("Bad result: " + result);
+    var result = foo(2);
+    if (result !== true)
+        throw new Error("Bad result: " + result);
+    var result = foo(3);
+    if (result !== false)
+        throw new Error("Bad result: " + result);
+    var result = foo(4);
+    if (result !== true)
+        throw new Error("Bad result: " + result);
+    var result = foo(5);
+    if (result !== false)
+        throw new Error("Bad result: " + result);
+    var result = foo0();
+    if (result !== false)
+        throw new Error("Bad result: " + result);
+    var result = foo1();
+    if (result !== true)
+        throw new Error("Bad result: " + result);
+    var result = foo2();
+    if (result !== true)
+        throw new Error("Bad result: " + result);
+    var result = foo3();
+    if (result !== false)
+        throw new Error("Bad result: " + result);
+    var result = foo4();
+    if (result !== true)
+        throw new Error("Bad result: " + result);
+    var result = foo5();
+    if (result !== false)
+        throw new Error("Bad result: " + result);
+}
index cd992ca..0a64871 100644 (file)
@@ -1,3 +1,20 @@
+2017-04-18  Filip Pizlo  <fpizlo@apple.com>
+
+        Optimize SharedArrayBuffer in the DFG+FTL
+        https://bugs.webkit.org/show_bug.cgi?id=164108
+
+        Reviewed by Saam Barati.
+        
+        Add a test of futex performance.
+
+        * workers/sab/cascade_lock-worker.js: Added.
+        (onmessage):
+        * workers/sab/cascade_lock.html: Added.
+        * workers/sab/worker-resources.js:
+        (cascadeLockSlow):
+        (cascadeLock):
+        (cascadeUnlock):
+
 2017-04-20  Jon Lee  <jonlee@apple.com>
 
         Update pip placard to "picture in picture"
diff --git a/LayoutTests/workers/sab/cascade_lock-expected.txt b/LayoutTests/workers/sab/cascade_lock-expected.txt
new file mode 100644 (file)
index 0000000..84d0902
--- /dev/null
@@ -0,0 +1,7 @@
+Starting worker!
+Starting worker!
+Starting worker!
+Starting worker!
+All workers done!
+Test passed!
+
diff --git a/LayoutTests/workers/sab/cascade_lock-worker.js b/LayoutTests/workers/sab/cascade_lock-worker.js
new file mode 100644 (file)
index 0000000..358ae0d
--- /dev/null
@@ -0,0 +1,34 @@
+importScripts("worker-resources.js");
+
+onmessage = function (event) {
+    var memory = event.data;
+    var workPerCriticalSection = memory[1];
+    var workBetweenCriticalSections = memory[2];
+    
+    var doubleMemory = new Float64Array(memory.buffer);
+    
+    var lockIdx = 0;
+    var countIdx = 2;
+    var count = 10000000;
+    
+    postMessage("Started!");
+    postMessage("Memory: " + memory);
+    
+    var value = 1;
+    var localWord = 0;
+    for (var i = 0; i < count; ++i) {
+        cascadeLock(memory, lockIdx);
+        for (var j = workPerCriticalSection; j--;) {
+            doubleMemory[countIdx] += value;
+        }
+        cascadeUnlock(memory, lockIdx);
+        for (var j = workBetweenCriticalSections; j--;) {
+            localWord += value;
+            value = localWord;
+        }
+    }
+    
+    postMessage("All done!");
+    postMessage("Memory: " + memory);
+    postMessage("done");
+}
diff --git a/LayoutTests/workers/sab/cascade_lock.html b/LayoutTests/workers/sab/cascade_lock.html
new file mode 100644 (file)
index 0000000..db93e5e
--- /dev/null
@@ -0,0 +1,101 @@
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<html>
+<head>
+</head>
+<body>
+<script>
+function getOrCreate(id, tagName)
+{
+    var element = document.getElementById(id);
+    if (element)
+        return element;
+    
+    element = document.createElement(tagName);
+    element.id = id;
+    var parent = document.body || document.documentElement;
+    var refNode = parent.firstChild;
+    
+    parent.insertBefore(element, refNode);
+    return element;
+}
+
+function debug(msg)
+{
+    var span = document.createElement("span");
+    getOrCreate("console", "div").appendChild(span); // insert it first so XHTML knows the namespace
+    span.innerHTML = msg + '<br />';
+}
+
+if (window.testRunner) {
+    testRunner.dumpAsText();
+    testRunner.waitUntilDone();
+}
+
+var verbose = false;
+var measureTime = false;
+
+var sab = new SharedArrayBuffer(32);
+
+var workPerCriticalSection = 1;
+var workBetweenCriticalSections = 0;
+
+var memory = new Int32Array(sab);
+var doubleMemory = new Float64Array(sab);
+memory[1] = workPerCriticalSection;
+memory[2] = workBetweenCriticalSections;
+
+var numWorkers = 0;
+function startWorker(file)
+{
+    if (verbose)
+        debug("Starting worker: " + file);
+    numWorkers++;
+    var worker = new Worker(file);
+    worker.onmessage = function(event) {
+        if (event.data == "done") {
+            if (verbose)
+                debug("Finished worker: " + file);
+            if (--numWorkers)
+                return;
+            debug("All workers done!");
+            done();
+            return;
+        }
+        if (event.data.indexOf("Error") == 0) {
+            debug("Test failed: "+ event.data);
+            if (window.testRunner)
+                testRunner.notifyDone();
+        }
+        
+        if (verbose)
+            debug("Event from " + file + ": " + event.data);
+    };
+    worker.postMessage(memory);
+}
+
+var count = 10000000;
+var numThreads = 4;
+
+function done()
+{
+    if (memory[0] != 0)
+        throw "Error: bad value at memory[0]: " + memory[0];
+    if (doubleMemory[2] != count * numThreads)
+        throw "Error: bad value at doubleMemory[1]: " + doubleMemory[1];
+    if (measureTime)
+        debug("Speed: " + count / (Date.now() - before) + " KHz.");
+    debug("Test passed!");
+
+    if (window.testRunner)
+        testRunner.notifyDone();
+}
+
+var before = Date.now();
+for (var i = 0; i < numThreads; ++i) {
+    debug("Starting worker!");
+    startWorker("cascade_lock-worker.js");
+}
+
+</script>
+</body>
+</html>
index d537817..e560602 100644 (file)
@@ -51,3 +51,36 @@ function checkBufferSharing(shouldShareBuffer)
     }
 }
 
+var cascadeLockUnlocked = 0;
+var cascadeLockLocked = 1;
+var cascadeLockLockedAndParked = 2;
+
+function cascadeLockSlow(memory, index)
+{
+    var desiredState = cascadeLockLocked;
+    for (;;) {
+        if (Atomics.compareExchange(memory, index, cascadeLockUnlocked, desiredState) == cascadeLockUnlocked)
+            return;
+        
+        desiredState = cascadeLockLockedAndParked;
+        Atomics.compareExchange(memory, index, cascadeLockLocked, cascadeLockLockedAndParked);
+        Atomics.wait(memory, index, cascadeLockLockedAndParked);
+    }
+}
+
+function cascadeLock(memory, index)
+{
+    if (Atomics.compareExchange(memory, index, cascadeLockUnlocked, cascadeLockLocked) == cascadeLockUnlocked)
+        return;
+    
+    cascadeLockSlow(memory, index);
+}
+
+function cascadeUnlock(memory, index)
+{
+    if (Atomics.exchange(memory, index, cascadeLockUnlocked) == cascadeLockLocked)
+        return;
+
+    Atomics.wake(memory, index, 1);
+}
+
index 6854ad2..51e6c43 100644 (file)
@@ -1,3 +1,144 @@
+2017-04-15  Filip Pizlo  <fpizlo@apple.com>
+
+        Optimize SharedArrayBuffer in the DFG+FTL
+        https://bugs.webkit.org/show_bug.cgi?id=164108
+
+        Reviewed by Saam Barati.
+        
+        This adds atomics intrinsics to the DFG and wires them through to the DFG and FTL backends. This
+        was super easy in the FTL since B3 already has comprehensive atomic intrinsics, which are more
+        powerful than what we need right now. In the DFG backend, I went with an easy-to-write
+        implementation that just reduces everything to a weak CAS loop. It's very inefficient with
+        registers (it needs ~8) but it's the DFG backend, so it's not obvious how much we care.
+        
+        To make the rare cases easy to handle, I refactored AtomicsObject.cpp so that the operations for
+        the slow paths can share code with the native functions.
+        
+        This also fixes register handling in the X86 implementations of CAS, in the case that
+        expectedAndResult is not %rax. This also fixes the ARM64 implementation of branchWeakCAS.
+        
+        I adapted the CascadeLock from WTF/benchmarks/ToyLocks.h as a microbenchmark of lock performance.
+        This benchmark performs 2.5x faster, in both the contended and uncontended case, thanks to this
+        change. It's still about 3x slower than native. I investigated this only a bit. I suspect that
+        the story will be different in asm.js code, which will get constant-folding of the typed array
+        backing store by virtue of how it uses lexically scoped variables as pointers to the heap arrays.
+        It's worth noting that the native lock I was comparing against, the very nicely-tuned
+        CascadeLock, is at the very high end of lock throughput under virtually all conditions
+        (uncontended, microcontended, held for a long time). I also compared to WTF::Lock and others, and
+        the only ones that performed better in this microbenchmark were spinlocks. I don't recommend
+        using those. So, when I say this is 3x slower than native, I really mean that it's 3x slower than
+        the fastest native lock that I have in my arsenal.
+        
+        Also worth noting is that I experimented with exposing Atomics.yield(), which uses sched_yield,
+        as a way of testing if adding a yield loop to the JS cascadeLock would help. It does not help. I
+        did not investigate why.
+
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::AbstractMacroAssembler::JumpList::append):
+        * assembler/CPU.h:
+        (JSC::is64Bit):
+        (JSC::is32Bit):
+        * b3/B3Common.h:
+        (JSC::B3::is64Bit): Deleted.
+        (JSC::B3::is32Bit): Deleted.
+        * b3/B3LowerToAir.cpp:
+        (JSC::B3::Air::LowerToAir::appendTrapping):
+        (JSC::B3::Air::LowerToAir::appendCAS):
+        (JSC::B3::Air::LowerToAir::appendGeneralAtomic):
+        * dfg/DFGAbstractInterpreterInlines.h:
+        (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
+        * dfg/DFGByteCodeParser.cpp:
+        (JSC::DFG::ByteCodeParser::handleIntrinsicCall):
+        * dfg/DFGClobberize.h:
+        (JSC::DFG::clobberize):
+        * dfg/DFGDoesGC.cpp:
+        (JSC::DFG::doesGC):
+        * dfg/DFGFixupPhase.cpp:
+        (JSC::DFG::FixupPhase::fixupNode):
+        * dfg/DFGNode.h:
+        (JSC::DFG::Node::hasHeapPrediction):
+        (JSC::DFG::Node::hasArrayMode):
+        * dfg/DFGNodeType.h:
+        (JSC::DFG::isAtomicsIntrinsic):
+        (JSC::DFG::numExtraAtomicsArgs):
+        * dfg/DFGPredictionPropagationPhase.cpp:
+        * dfg/DFGSSALoweringPhase.cpp:
+        (JSC::DFG::SSALoweringPhase::handleNode):
+        * dfg/DFGSafeToExecute.h:
+        (JSC::DFG::safeToExecute):
+        * dfg/DFGSpeculativeJIT.cpp:
+        (JSC::DFG::SpeculativeJIT::loadFromIntTypedArray):
+        (JSC::DFG::SpeculativeJIT::setIntTypedArrayLoadResult):
+        (JSC::DFG::SpeculativeJIT::compileGetByValOnIntTypedArray):
+        (JSC::DFG::SpeculativeJIT::getIntTypedArrayStoreOperand):
+        (JSC::DFG::SpeculativeJIT::compilePutByValForIntTypedArray):
+        * dfg/DFGSpeculativeJIT.h:
+        (JSC::DFG::SpeculativeJIT::callOperation):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        * ftl/FTLAbstractHeapRepository.cpp:
+        (JSC::FTL::AbstractHeapRepository::decorateFencedAccess):
+        (JSC::FTL::AbstractHeapRepository::computeRangesAndDecorateInstructions):
+        * ftl/FTLAbstractHeapRepository.h:
+        * ftl/FTLCapabilities.cpp:
+        (JSC::FTL::canCompile):
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileNode):
+        (JSC::FTL::DFG::LowerDFGToB3::compileAtomicsReadModifyWrite):
+        (JSC::FTL::DFG::LowerDFGToB3::compileAtomicsIsLockFree):
+        (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
+        (JSC::FTL::DFG::LowerDFGToB3::compilePutByVal):
+        (JSC::FTL::DFG::LowerDFGToB3::pointerIntoTypedArray):
+        (JSC::FTL::DFG::LowerDFGToB3::loadFromIntTypedArray):
+        (JSC::FTL::DFG::LowerDFGToB3::storeType):
+        (JSC::FTL::DFG::LowerDFGToB3::setIntTypedArrayLoadResult):
+        (JSC::FTL::DFG::LowerDFGToB3::getIntTypedArrayStoreOperand):
+        (JSC::FTL::DFG::LowerDFGToB3::vmCall):
+        * ftl/FTLOutput.cpp:
+        (JSC::FTL::Output::store):
+        (JSC::FTL::Output::store32As8):
+        (JSC::FTL::Output::store32As16):
+        (JSC::FTL::Output::atomicXchgAdd):
+        (JSC::FTL::Output::atomicXchgAnd):
+        (JSC::FTL::Output::atomicXchgOr):
+        (JSC::FTL::Output::atomicXchgSub):
+        (JSC::FTL::Output::atomicXchgXor):
+        (JSC::FTL::Output::atomicXchg):
+        (JSC::FTL::Output::atomicStrongCAS):
+        * ftl/FTLOutput.h:
+        (JSC::FTL::Output::store32):
+        (JSC::FTL::Output::store64):
+        (JSC::FTL::Output::storePtr):
+        (JSC::FTL::Output::storeFloat):
+        (JSC::FTL::Output::storeDouble):
+        * jit/JITOperations.h:
+        * runtime/AtomicsObject.cpp:
+        (JSC::atomicsFuncAdd):
+        (JSC::atomicsFuncAnd):
+        (JSC::atomicsFuncCompareExchange):
+        (JSC::atomicsFuncExchange):
+        (JSC::atomicsFuncIsLockFree):
+        (JSC::atomicsFuncLoad):
+        (JSC::atomicsFuncOr):
+        (JSC::atomicsFuncStore):
+        (JSC::atomicsFuncSub):
+        (JSC::atomicsFuncWait):
+        (JSC::atomicsFuncWake):
+        (JSC::atomicsFuncXor):
+        (JSC::operationAtomicsAdd):
+        (JSC::operationAtomicsAnd):
+        (JSC::operationAtomicsCompareExchange):
+        (JSC::operationAtomicsExchange):
+        (JSC::operationAtomicsIsLockFree):
+        (JSC::operationAtomicsLoad):
+        (JSC::operationAtomicsOr):
+        (JSC::operationAtomicsStore):
+        (JSC::operationAtomicsSub):
+        (JSC::operationAtomicsXor):
+        * runtime/AtomicsObject.h:
+
 2017-04-19  Youenn Fablet  <youenn@apple.com>
 
         [Mac] Allow customizing H264 encoder
index baba57b..a38e65a 100644 (file)
@@ -117,6 +117,15 @@ public:
     
     struct BaseIndex;
     
+    static RegisterID withSwappedRegister(RegisterID original, RegisterID left, RegisterID right)
+    {
+        if (original == left)
+            return right;
+        if (original == right)
+            return left;
+        return original;
+    }
+    
     // Address:
     //
     // Describes a simple base-offset address.
@@ -132,6 +141,11 @@ public:
             return Address(base, offset + additionalOffset);
         }
         
+        Address withSwappedRegister(RegisterID left, RegisterID right)
+        {
+            return Address(AbstractMacroAssembler::withSwappedRegister(base, left, right), offset);
+        }
+        
         BaseIndex indexedBy(RegisterID index, Scale) const;
         
         RegisterID base;
@@ -201,6 +215,11 @@ public:
         {
             return BaseIndex(base, index, scale, offset + additionalOffset);
         }
+
+        BaseIndex withSwappedRegister(RegisterID left, RegisterID right)
+        {
+            return BaseIndex(AbstractMacroAssembler::withSwappedRegister(base, left, right), AbstractMacroAssembler::withSwappedRegister(index, left, right), scale, offset);
+        }
     };
 
     // AbsoluteAddress:
@@ -702,7 +721,8 @@ public:
         
         void append(Jump jump)
         {
-            m_jumps.append(jump);
+            if (jump.isSet())
+                m_jumps.append(jump);
         }
         
         void append(const JumpList& other)
index d5cc921..b88cb41 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2008, 2012-2015 Apple Inc. All rights reserved.
+ * Copyright (C) 2008-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -65,6 +65,20 @@ inline bool isX86_64()
 #endif
 }
 
+inline bool is64Bit()
+{
+#if USE(JSVALUE64)
+    return true;
+#else
+    return false;
+#endif
+}
+
+inline bool is32Bit()
+{
+    return !is64Bit();
+}
+
 inline bool isMIPS()
 {
 #if CPU(MIPS)
index 823510e..abcbd1e 100644 (file)
@@ -3508,82 +3508,98 @@ public:
         m_assembler.stlxr<64>(result, src, extractSimpleAddress(address));
     }
     
-    void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<8>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<16>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<32>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<64>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<8>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<16>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<32>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<64>(cond, expectedAndResult, newValue, address, result);
     }
     
-    JumpList branchAtomicWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<8>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<16>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<32>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<64>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<8>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<16>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<32>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<64>(cond, expectedAndClobbered, newValue, address);
     }
@@ -4172,7 +4188,7 @@ private:
     template<int datasize>
     void storeCondRel(RegisterID src, RegisterID dest, RegisterID result)
     {
-        m_assembler.stlxr<datasize>(src, dest, result);
+        m_assembler.stlxr<datasize>(result, src, dest);
     }
     
     template<int datasize>
@@ -4213,8 +4229,8 @@ private:
         done.link(this);
     }
     
-    template<int datasize>
-    void atomicRelaxedStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<int datasize, typename AddressType>
+    void atomicRelaxedStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         signExtend<datasize>(expectedAndResult, expectedAndResult);
         
@@ -4237,8 +4253,8 @@ private:
         done.link(this);
     }
     
-    template<int datasize>
-    JumpList branchAtomicWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<int datasize, typename AddressType>
+    JumpList branchAtomicWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         signExtend<datasize>(expectedAndClobbered, expectedAndClobbered);
         
@@ -4265,8 +4281,8 @@ private:
         RELEASE_ASSERT_NOT_REACHED();
     }
     
-    template<int datasize>
-    JumpList branchAtomicRelaxedWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<int datasize, typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         signExtend<datasize>(expectedAndClobbered, expectedAndClobbered);
         
@@ -4303,6 +4319,17 @@ private:
         return memoryTempRegister;
     }
 
+    // This uses both the memory and data temp, but only returns the memorty temp. So you can use the
+    // data temp after this finishes.
+    RegisterID extractSimpleAddress(BaseIndex address)
+    {
+        RegisterID result = getCachedMemoryTempRegisterIDAndInvalidate();
+        lshift64(address.index, TrustedImm32(address.scale), result);
+        add64(address.base, result);
+        add64(TrustedImm32(address.offset), result);
+        return result;
+    }
+
     Jump jumpAfterFloatingPointCompare(DoubleCondition cond)
     {
         if (cond == DoubleNotEqual) {
index b86c7da..df007d4 100644 (file)
@@ -3023,92 +3023,92 @@ public:
     
     void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS8(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS8(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS16(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS16(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS32(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS32(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     // If you use weak CAS, you cannot rely on expectedAndClobbered to have any particular value after
@@ -4072,9 +4072,10 @@ protected:
         srcIsNonZero.link(this);
     }
     
-    template<typename Func>
-    void atomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID result, const Func& func)
+    template<typename AddressType, typename Func>
+    void atomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID result, AddressType& address, const Func& func)
     {
+        address = address.withSwappedRegister(X86Registers::eax, expectedAndResult);
         swap(expectedAndResult, X86Registers::eax);
         m_assembler.lock();
         func();
@@ -4082,18 +4083,20 @@ protected:
         set32(x86Condition(cond), result);
     }
 
-    template<typename Func>
-    void atomicStrongCAS(RegisterID expectedAndResult, const Func& func)
+    template<typename AddressType, typename Func>
+    void atomicStrongCAS(RegisterID expectedAndResult, AddressType& address, const Func& func)
     {
+        address = address.withSwappedRegister(X86Registers::eax, expectedAndResult);
         swap(expectedAndResult, X86Registers::eax);
         m_assembler.lock();
         func();
         swap(expectedAndResult, X86Registers::eax);
     }
 
-    template<typename Func>
-    Jump branchAtomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, const Func& func)
+    template<typename AddressType, typename Func>
+    Jump branchAtomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, AddressType& address, const Func& func)
     {
+        address = address.withSwappedRegister(X86Registers::eax, expectedAndResult);
         swap(expectedAndResult, X86Registers::eax);
         m_assembler.lock();
         func();
index 2343e95..6e7ad5e 100644 (file)
@@ -1435,32 +1435,32 @@ public:
     
     void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS64(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS64(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address, RegisterID result)
index 9d1812a..e3b0f20 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
+ * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
 
 #if ENABLE(B3_JIT)
 
+#include "CPU.h"
 #include "JSExportMacros.h"
 #include "Options.h"
 #include <wtf/Optional.h>
 
 namespace JSC { namespace B3 {
 
-inline bool is64Bit() { return sizeof(void*) == 8; }
-inline bool is32Bit() { return !is64Bit(); }
-
 enum B3ComplitationMode {
     B3Mode,
     AirMode
index 4a7e0af..8c6f736 100644 (file)
@@ -1133,6 +1133,12 @@ private:
         m_insts.last().append(Inst(opcode, m_value, std::forward<Arguments>(arguments)...));
     }
     
+    template<typename... Arguments>
+    void appendTrapping(Air::Opcode opcode, Arguments&&... arguments)
+    {
+        m_insts.last().append(trappingInst(m_value, opcode, m_value, std::forward<Arguments>(arguments)...));
+    }
+    
     void append(Inst&& inst)
     {
         m_insts.last().append(WTFMove(inst));
@@ -2174,13 +2180,13 @@ private:
         if (isX86()) {
             append(relaxedMoveForType(atomic->accessType()), immOrTmp(atomic->child(0)), m_eax);
             if (returnsOldValue) {
-                append(OPCODE_FOR_WIDTH(AtomicStrongCAS, width), m_eax, newValueTmp, address);
+                appendTrapping(OPCODE_FOR_WIDTH(AtomicStrongCAS, width), m_eax, newValueTmp, address);
                 append(relaxedMoveForType(atomic->accessType()), m_eax, valueResultTmp);
             } else if (isBranch) {
-                append(OPCODE_FOR_WIDTH(BranchAtomicStrongCAS, width), Arg::statusCond(MacroAssembler::Success), m_eax, newValueTmp, address);
+                appendTrapping(OPCODE_FOR_WIDTH(BranchAtomicStrongCAS, width), Arg::statusCond(MacroAssembler::Success), m_eax, newValueTmp, address);
                 m_blockToBlock[m_block]->setSuccessors(success, failure);
             } else
-                append(OPCODE_FOR_WIDTH(AtomicStrongCAS, width), Arg::statusCond(invert ? MacroAssembler::Failure : MacroAssembler::Success), m_eax, tmp(atomic->child(1)), address, boolResultTmp);
+                appendTrapping(OPCODE_FOR_WIDTH(AtomicStrongCAS, width), Arg::statusCond(invert ? MacroAssembler::Failure : MacroAssembler::Success), m_eax, tmp(atomic->child(1)), address, boolResultTmp);
             return;
         }
         
@@ -2230,11 +2236,11 @@ private:
         append(Air::Jump);
         beginBlock->setSuccessors(reloopBlock);
         
-        reloopBlock->append(loadLinkOpcode(width, atomic->hasFence()), m_value, address, valueResultTmp);
+        reloopBlock->append(trappingInst(m_value, loadLinkOpcode(width, atomic->hasFence()), m_value, address, valueResultTmp));
         reloopBlock->append(OPCODE_FOR_CANONICAL_WIDTH(Branch, width), m_value, Arg::relCond(MacroAssembler::NotEqual), valueResultTmp, expectedValueTmp);
         reloopBlock->setSuccessors(comparisonFail, storeBlock);
         
-        storeBlock->append(storeCondOpcode(width, atomic->hasFence()), m_value, newValueTmp, address, successBoolResultTmp);
+        storeBlock->append(trappingInst(m_value, storeCondOpcode(width, atomic->hasFence()), m_value, newValueTmp, address, successBoolResultTmp));
         if (isBranch) {
             storeBlock->append(BranchTest32, m_value, Arg::resCond(MacroAssembler::Zero), boolResultTmp, boolResultTmp);
             storeBlock->setSuccessors(success, weakFail);
@@ -2264,7 +2270,7 @@ private:
         
         if (isStrong && hasFence) {
             Tmp tmp = m_code.newTmp(GP);
-            strongFailBlock->append(storeCondOpcode(width, atomic->hasFence()), m_value, valueResultTmp, address, tmp);
+            strongFailBlock->append(trappingInst(m_value, storeCondOpcode(width, atomic->hasFence()), m_value, valueResultTmp, address, tmp));
             strongFailBlock->append(BranchTest32, m_value, Arg::resCond(MacroAssembler::Zero), tmp, tmp);
             strongFailBlock->setSuccessors(failure, reloopBlock);
         }
@@ -2338,7 +2344,7 @@ private:
             RELEASE_ASSERT(isARM64());
             prepareOpcode = loadLinkOpcode(atomic->accessWidth(), atomic->hasFence());
         }
-        reloopBlock->append(prepareOpcode, m_value, address, oldValue);
+        reloopBlock->append(trappingInst(m_value, prepareOpcode, m_value, address, oldValue));
         
         if (opcode != Air::Nop) {
             // FIXME: If we ever have to write this again, we need to find a way to share the code with
@@ -2364,11 +2370,11 @@ private:
         if (isX86()) {
             Air::Opcode casOpcode = OPCODE_FOR_WIDTH(BranchAtomicStrongCAS, atomic->accessWidth());
             reloopBlock->append(relaxedMoveForType(atomic->type()), m_value, oldValue, m_eax);
-            reloopBlock->append(casOpcode, m_value, Arg::statusCond(MacroAssembler::Success), m_eax, newValue, address);
+            reloopBlock->append(trappingInst(m_value, casOpcode, m_value, Arg::statusCond(MacroAssembler::Success), m_eax, newValue, address));
         } else {
             RELEASE_ASSERT(isARM64());
             Tmp boolResult = m_code.newTmp(GP);
-            reloopBlock->append(storeCondOpcode(atomic->accessWidth(), atomic->hasFence()), m_value, newValue, address, boolResult);
+            reloopBlock->append(trappingInst(m_value, storeCondOpcode(atomic->accessWidth(), atomic->hasFence()), m_value, newValue, address, boolResult));
             reloopBlock->append(BranchTest32, m_value, Arg::resCond(MacroAssembler::Zero), boolResult, boolResult);
         }
         reloopBlock->setSuccessors(doneBlock, reloopBlock);
index 5cb2c47..75d4e40 100644 (file)
@@ -36,6 +36,12 @@ class Code;
 
 inline bool useIRC()
 {
+    // FIXME: Currently, the Briggs allocator has bugs that our regression tests detect. Hence, it is
+    // unconditionally disabled.
+    // https://bugs.webkit.org/show_bug.cgi?id=170948
+    if (true)
+        return true;
+    
     return Options::airForceIRCAllocator()
         || (!isARM64() && !Options::airForceBriggsAllocator());
 }
index cf710b6..8d71a83 100644 (file)
@@ -546,6 +546,13 @@ bool AbstractInterpreter<AbstractStateType>::executeEffects(unsigned clobberLimi
         }
         break;
     }
+        
+    case AtomicsIsLockFree: {
+        if (node->child1().useKind() != Int32Use)
+            clobberWorld(node->origin.semantic, clobberLimit);
+        forNode(node).setType(SpecBoolInt32);
+        break;
+    }
 
     case ArithClz32: {
         JSValue operand = forNode(node->child1()).value();
@@ -1506,7 +1513,18 @@ bool AbstractInterpreter<AbstractStateType>::executeEffects(unsigned clobberLimi
         forNode(node).set(m_graph, m_vm.stringStructure.get());
         break;
             
-    case GetByVal: {
+    case GetByVal:
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor: {
+        if (node->op() != GetByVal)
+            clobberWorld(node->origin.semantic, clobberLimit);
         switch (node->arrayMode().type()) {
         case Array::SelectUsingPredictions:
         case Array::Unprofiled:
index e72fa64..224ab14 100644 (file)
@@ -184,7 +184,7 @@ private:
     // Helper for min and max.
     template<typename ChecksFunctor>
     bool handleMinMax(int resultOperand, NodeType op, int registerOffset, int argumentCountIncludingThis, const ChecksFunctor& insertChecks);
-
+    
     void refineStatically(CallLinkStatus&, Node* callTarget);
     // Handle calls. This resolves issues surrounding inlining and intrinsics.
     enum Terminality { Terminal, NonTerminal };
@@ -2362,6 +2362,94 @@ bool ByteCodeParser::handleIntrinsicCall(Node* callee, int resultOperand, Intrin
             return false;
         }
     }
+        
+    case AtomicsAddIntrinsic:
+    case AtomicsAndIntrinsic:
+    case AtomicsCompareExchangeIntrinsic:
+    case AtomicsExchangeIntrinsic:
+    case AtomicsIsLockFreeIntrinsic:
+    case AtomicsLoadIntrinsic:
+    case AtomicsOrIntrinsic:
+    case AtomicsStoreIntrinsic:
+    case AtomicsSubIntrinsic:
+    case AtomicsXorIntrinsic: {
+        if (!is64Bit())
+            return false;
+        
+        NodeType op = LastNodeType;
+        unsigned numArgs = 0; // Number of actual args; we add one for the backing store pointer.
+        switch (intrinsic) {
+        case AtomicsAddIntrinsic:
+            op = AtomicsAdd;
+            numArgs = 3;
+            break;
+        case AtomicsAndIntrinsic:
+            op = AtomicsAnd;
+            numArgs = 3;
+            break;
+        case AtomicsCompareExchangeIntrinsic:
+            op = AtomicsCompareExchange;
+            numArgs = 4;
+            break;
+        case AtomicsExchangeIntrinsic:
+            op = AtomicsExchange;
+            numArgs = 3;
+            break;
+        case AtomicsIsLockFreeIntrinsic:
+            // This gets no backing store, but we need no special logic for this since this also does
+            // not need varargs.
+            op = AtomicsIsLockFree;
+            numArgs = 1;
+            break;
+        case AtomicsLoadIntrinsic:
+            op = AtomicsLoad;
+            numArgs = 2;
+            break;
+        case AtomicsOrIntrinsic:
+            op = AtomicsOr;
+            numArgs = 3;
+            break;
+        case AtomicsStoreIntrinsic:
+            op = AtomicsStore;
+            numArgs = 3;
+            break;
+        case AtomicsSubIntrinsic:
+            op = AtomicsSub;
+            numArgs = 3;
+            break;
+        case AtomicsXorIntrinsic:
+            op = AtomicsXor;
+            numArgs = 3;
+            break;
+        default:
+            RELEASE_ASSERT_NOT_REACHED();
+            break;
+        }
+        
+        if (static_cast<unsigned>(argumentCountIncludingThis) < 1 + numArgs)
+            return false;
+        
+        insertChecks();
+        
+        Vector<Node*, 3> args;
+        for (unsigned i = 0; i < numArgs; ++i)
+            args.append(get(virtualRegisterForArgument(1 + i, registerOffset)));
+        
+        Node* result;
+        if (numArgs + 1 <= 3) {
+            while (args.size() < 3)
+                args.append(nullptr);
+            result = addToGraph(op, OpInfo(ArrayMode(Array::SelectUsingPredictions).asWord()), OpInfo(prediction), args[0], args[1], args[2]);
+        } else {
+            for (Node* node : args)
+                addVarArgChild(node);
+            addVarArgChild(nullptr);
+            result = addToGraph(Node::VarArg, op, OpInfo(ArrayMode(Array::SelectUsingPredictions).asWord()), OpInfo(prediction));
+        }
+        
+        set(VirtualRegister(resultOperand), result);
+        return true;
+    }
 
     case ParseIntIntrinsic: {
         if (argumentCountIncludingThis < 2)
index d2ceefe..e7576cd 100644 (file)
@@ -181,6 +181,15 @@ void clobberize(Graph& graph, Node* node, const ReadFunctor& read, const WriteFu
         def(PureValue(node));
         return;
 
+    case AtomicsIsLockFree:
+        if (node->child1().useKind() == Int32Use)
+            def(PureValue(node));
+        else {
+            read(World);
+            write(Heap);
+        }
+        return;
+        
     case ArithCos:
     case ArithFRound:
     case ArithLog:
@@ -570,6 +579,28 @@ void clobberize(Graph& graph, Node* node, const ReadFunctor& read, const WriteFu
         write(Heap);
         return;
 
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor: {
+        unsigned numExtraArgs = numExtraAtomicsArgs(node->op());
+        Edge storageEdge = graph.child(node, 2 + numExtraArgs);
+        if (!storageEdge) {
+            read(World);
+            write(Heap);
+            return;
+        }
+        read(TypedArrayProperties);
+        read(MiscFields);
+        write(TypedArrayProperties);
+        return;
+    }
+
     case CallEval:
         ASSERT(!node->origin.semantic.inlineCallFrame);
         read(AbstractHeap(Stack, graph.m_codeBlock->scopeRegister()));
index 87eb27c..92faabb 100644 (file)
@@ -270,6 +270,16 @@ bool doesGC(Graph& graph, Node* node)
     case ResolveScopeForHoistingFuncDeclInEval:
     case ResolveScope:
     case NukeStructureAndSetButterfly:
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor:
+    case AtomicsIsLockFree:
         return false;
 
     case CreateActivation:
index e45d9c4..9008385 100644 (file)
@@ -909,6 +909,73 @@ private:
             break;
         }
             
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor: {
+            Edge& base = m_graph.child(node, 0);
+            Edge& index = m_graph.child(node, 1);
+            
+            bool badNews = false;
+            for (unsigned i = numExtraAtomicsArgs(node->op()); i--;) {
+                Edge& child = m_graph.child(node, 2 + i);
+                // NOTE: DFG is not smart enough to handle double->int conversions in atomics. So, we
+                // just call the function when that happens. But the FTL is totally cool with those
+                // conversions.
+                if (!child->shouldSpeculateInt32()
+                    && !child->shouldSpeculateAnyInt()
+                    && !(child->shouldSpeculateNumberOrBoolean() && isFTL(m_graph.m_plan.mode)))
+                    badNews = true;
+            }
+            
+            if (badNews) {
+                node->setArrayMode(ArrayMode(Array::Generic));
+                break;
+            }
+            
+            node->setArrayMode(
+                node->arrayMode().refine(
+                    m_graph, node, base->prediction(), index->prediction()));
+            
+            if (node->arrayMode().type() == Array::Generic)
+                break;
+            
+            for (unsigned i = numExtraAtomicsArgs(node->op()); i--;) {
+                Edge& child = m_graph.child(node, 2 + i);
+                if (child->shouldSpeculateInt32())
+                    fixIntOrBooleanEdge(child);
+                else if (child->shouldSpeculateAnyInt())
+                    fixEdge<Int52RepUse>(child);
+                else {
+                    RELEASE_ASSERT(child->shouldSpeculateNumberOrBoolean() && isFTL(m_graph.m_plan.mode));
+                    fixDoubleOrBooleanEdge(child);
+                }
+            }
+            
+            blessArrayOperation(base, index, m_graph.child(node, 2 + numExtraAtomicsArgs(node->op())));
+            fixEdge<CellUse>(base);
+            fixEdge<Int32Use>(index);
+            
+            if (node->arrayMode().type() == Array::Uint32Array) {
+                // NOTE: This means basically always doing Int52.
+                if (node->shouldSpeculateAnyInt() && enableInt52())
+                    node->setResult(NodeResultInt52);
+                else
+                    node->setResult(NodeResultDouble);
+            }
+            break;
+        }
+            
+        case AtomicsIsLockFree:
+            if (node->child1()->shouldSpeculateInt32())
+                fixIntOrBooleanEdge(node->child1());
+            break;
+            
         case ArrayPush: {
             // May need to refine the array mode in case the value prediction contravenes
             // the array prediction. For example, we may have evidence showing that the
index 7804502..7e5ebe2 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2011-2016 Apple Inc. All rights reserved.
+ * Copyright (C) 2011-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -1489,6 +1489,15 @@ public:
         case CallDOMGetter:
         case CallDOM:
         case ParseInt:
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor:
             return true;
         default:
             return false;
@@ -1806,6 +1815,15 @@ public:
         case ArrayPush:
         case ArrayPop:
         case HasIndexedProperty:
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor:
             return true;
         default:
             return false;
index 57348b9..053243b 100644 (file)
@@ -247,6 +247,18 @@ namespace JSC { namespace DFG {
     macro(CheckTypeInfoFlags, NodeMustGenerate) /* Takes an OpInfo with the flags you want to test are set */\
     macro(ParseInt, NodeMustGenerate | NodeResultJS) \
     \
+    /* Atomics object functions. */\
+    macro(AtomicsAdd, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsAnd, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsCompareExchange, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsExchange, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsIsLockFree, NodeResultBoolean) \
+    macro(AtomicsLoad, NodeResultJS | NodeMustGenerate) \
+    macro(AtomicsOr, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsStore, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsSub, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(AtomicsXor, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    \
     /* Optimizations for array mutation. */\
     macro(ArrayPush, NodeResultJS | NodeMustGenerate) \
     macro(ArrayPop, NodeResultJS | NodeMustGenerate) \
@@ -443,6 +455,48 @@ inline NodeFlags defaultFlags(NodeType op)
     }
 }
 
+inline bool isAtomicsIntrinsic(NodeType op)
+{
+    switch (op) {
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor:
+    case AtomicsIsLockFree:
+        return true;
+    default:
+        return false;
+    }
+}
+
+static const unsigned maxNumExtraAtomicsArgs = 2;
+
+inline unsigned numExtraAtomicsArgs(NodeType op)
+{
+    switch (op) {
+    case AtomicsLoad:
+        return 0;
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsExchange:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor:
+        return 1;
+    case AtomicsCompareExchange:
+        return 2;
+    default:
+        RELEASE_ASSERT_NOT_REACHED();
+        return 0;
+    }
+}
+
 } } // namespace JSC::DFG
 
 #endif // ENABLE(DFG_JIT)
index 68ed03e..b623ac5 100644 (file)
@@ -334,14 +334,25 @@ private:
             break;
         }
 
-        case GetByVal: {
-            if (!node->child1()->prediction())
+        case GetByVal:
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor: {
+            Edge child1 = m_graph.child(node, 0);
+            if (!child1->prediction())
                 break;
             
+            Edge child2 = m_graph.child(node, 1);
             ArrayMode arrayMode = node->arrayMode().refine(
                 m_graph, node,
-                node->child1()->prediction(),
-                node->child2()->prediction(),
+                child1->prediction(),
+                child2->prediction(),
                 SpecNone);
             
             switch (arrayMode.type()) {
@@ -364,7 +375,7 @@ private:
                 changed |= mergePrediction(SpecFullDouble);
                 break;
             case Array::Uint32Array:
-                if (isInt32SpeculationForArithmetic(node->getHeapPrediction()))
+                if (isInt32SpeculationForArithmetic(node->getHeapPrediction()) && node->op() == GetByVal)
                     changed |= mergePrediction(SpecInt32Only);
                 else if (enableInt52())
                     changed |= mergePrediction(SpecAnyInt);
@@ -384,7 +395,7 @@ private:
             }
             break;
         }
-
+            
         case ToThis: {
             // ToThis in methods for primitive types should speculate primitive types in strict mode.
             ECMAMode ecmaMode = m_graph.executableFor(node->origin.semantic)->isStrictMode() ? StrictMode : NotStrictMode;
@@ -994,10 +1005,24 @@ private:
         case ArithAbs:
         case GetByVal:
         case ToThis:
-        case ToPrimitive: {
+        case ToPrimitive: 
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor: {
             m_dependentNodes.append(m_currentNode);
             break;
         }
+            
+        case AtomicsIsLockFree: {
+            setPrediction(SpecBoolean);
+            break;
+        }
 
         case PutByValAlias:
         case DoubleAsInt32:
index 3214cff..7f35455 100644 (file)
@@ -69,8 +69,17 @@ private:
     {
         switch (m_node->op()) {
         case GetByVal:
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor:
         case HasIndexedProperty:
-            lowerBoundsCheck(m_node->child1(), m_node->child2(), m_node->child3());
+            lowerBoundsCheck(m_graph.child(m_node, 0), m_graph.child(m_node, 1), m_graph.child(m_node, 2));
             break;
             
         case PutByVal:
index f6c89bd..1ea9c0b 100644 (file)
@@ -383,6 +383,16 @@ bool safeToExecute(AbstractStateType& state, Graph& graph, Node* node)
     case GetMapBucket:
     case LoadFromJSMapBucket:
     case IsNonEmptyMapBucket:
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor:
+    case AtomicsIsLockFree:
         return true;
 
     case ArraySlice: {
index bc33280..ab39a22 100644 (file)
@@ -2739,24 +2739,8 @@ JITCompiler::Jump SpeculativeJIT::jumpForTypedArrayIsNeuteredIfOutOfBounds(Node*
     return done;
 }
 
-void SpeculativeJIT::compileGetByValOnIntTypedArray(Node* node, TypedArrayType type)
+void SpeculativeJIT::loadFromIntTypedArray(GPRReg storageReg, GPRReg propertyReg, GPRReg resultReg, TypedArrayType type)
 {
-    ASSERT(isInt(type));
-    
-    SpeculateCellOperand base(this, node->child1());
-    SpeculateStrictInt32Operand property(this, node->child2());
-    StorageOperand storage(this, node->child3());
-
-    GPRReg baseReg = base.gpr();
-    GPRReg propertyReg = property.gpr();
-    GPRReg storageReg = storage.gpr();
-
-    GPRTemporary result(this);
-    GPRReg resultReg = result.gpr();
-
-    ASSERT(node->arrayMode().alreadyChecked(m_jit.graph(), node, m_state.forNode(node->child1())));
-
-    emitTypedArrayBoundsCheck(node, baseReg, propertyReg);
     switch (elementSize(type)) {
     case 1:
         if (isSigned(type))
@@ -2776,13 +2760,17 @@ void SpeculativeJIT::compileGetByValOnIntTypedArray(Node* node, TypedArrayType t
     default:
         CRASH();
     }
+}
+
+void SpeculativeJIT::setIntTypedArrayLoadResult(Node* node, GPRReg resultReg, TypedArrayType type, bool canSpeculate)
+{
     if (elementSize(type) < 4 || isSigned(type)) {
         int32Result(resultReg, node);
         return;
     }
     
     ASSERT(elementSize(type) == 4 && !isSigned(type));
-    if (node->shouldSpeculateInt32()) {
+    if (node->shouldSpeculateInt32() && canSpeculate) {
         speculationCheck(Overflow, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::LessThan, resultReg, TrustedImm32(0)));
         int32Result(resultReg, node);
         return;
@@ -2804,29 +2792,38 @@ void SpeculativeJIT::compileGetByValOnIntTypedArray(Node* node, TypedArrayType t
     doubleResult(fresult.fpr(), node);
 }
 
-void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg property, Node* node, TypedArrayType type)
+void SpeculativeJIT::compileGetByValOnIntTypedArray(Node* node, TypedArrayType type)
 {
     ASSERT(isInt(type));
     
-    StorageOperand storage(this, m_jit.graph().varArgChild(node, 3));
+    SpeculateCellOperand base(this, node->child1());
+    SpeculateStrictInt32Operand property(this, node->child2());
+    StorageOperand storage(this, node->child3());
+
+    GPRReg baseReg = base.gpr();
+    GPRReg propertyReg = property.gpr();
     GPRReg storageReg = storage.gpr();
-    
-    Edge valueUse = m_jit.graph().varArgChild(node, 2);
-    
-    GPRTemporary value;
-#if USE(JSVALUE32_64)
-    GPRTemporary propertyTag;
-    GPRTemporary valueTag;
-#endif
 
-    GPRReg valueGPR = InvalidGPRReg;
+    GPRTemporary result(this);
+    GPRReg resultReg = result.gpr();
+
+    ASSERT(node->arrayMode().alreadyChecked(m_jit.graph(), node, m_state.forNode(node->child1())));
+
+    emitTypedArrayBoundsCheck(node, baseReg, propertyReg);
+    loadFromIntTypedArray(storageReg, propertyReg, resultReg, type);
+    bool canSpeculate = true;
+    setIntTypedArrayLoadResult(node, resultReg, type, canSpeculate);
+}
+
+bool SpeculativeJIT::getIntTypedArrayStoreOperand(
+    GPRTemporary& value,
+    GPRReg property,
 #if USE(JSVALUE32_64)
-    GPRReg propertyTagGPR = InvalidGPRReg;
-    GPRReg valueTagGPR = InvalidGPRReg;
+    GPRTemporary& propertyTag,
+    GPRTemporary& valueTag,
 #endif
-
-    JITCompiler::JumpList slowPathCases;
-    
+    Edge valueUse, JITCompiler::JumpList& slowPathCases, bool isClamped)
+{
     bool isAppropriateConstant = false;
     if (valueUse->isConstant()) {
         JSValue jsValue = valueUse->asJSValue();
@@ -2834,24 +2831,20 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
         SpeculatedType actualType = speculationFromValue(jsValue);
         isAppropriateConstant = (expectedType | actualType) == expectedType;
     }
-
+    
     if (isAppropriateConstant) {
         JSValue jsValue = valueUse->asJSValue();
         if (!jsValue.isNumber()) {
             terminateSpeculativeExecution(Uncountable, JSValueRegs(), 0);
-            noResult(node);
-            return;
+            return false;
         }
         double d = jsValue.asNumber();
-        if (isClamped(type)) {
-            ASSERT(elementSize(type) == 1);
+        if (isClamped)
             d = clampDoubleToByte(d);
-        }
         GPRTemporary scratch(this);
         GPRReg scratchReg = scratch.gpr();
         m_jit.move(Imm32(toInt32(d)), scratchReg);
         value.adopt(scratch);
-        valueGPR = scratchReg;
     } else {
         switch (valueUse.useKind()) {
         case Int32Use: {
@@ -2859,12 +2852,9 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
             GPRTemporary scratch(this);
             GPRReg scratchReg = scratch.gpr();
             m_jit.move(valueOp.gpr(), scratchReg);
-            if (isClamped(type)) {
-                ASSERT(elementSize(type) == 1);
+            if (isClamped)
                 compileClampIntegerToByte(m_jit, scratchReg);
-            }
             value.adopt(scratch);
-            valueGPR = scratchReg;
             break;
         }
             
@@ -2874,8 +2864,7 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
             GPRTemporary scratch(this);
             GPRReg scratchReg = scratch.gpr();
             m_jit.move(valueOp.gpr(), scratchReg);
-            if (isClamped(type)) {
-                ASSERT(elementSize(type) == 1);
+            if (isClamped) {
                 MacroAssembler::Jump inBounds = m_jit.branch64(
                     MacroAssembler::BelowOrEqual, scratchReg, JITCompiler::TrustedImm64(0xff));
                 MacroAssembler::Jump tooBig = m_jit.branch64(
@@ -2888,14 +2877,13 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
                 inBounds.link(&m_jit);
             }
             value.adopt(scratch);
-            valueGPR = scratchReg;
             break;
         }
 #endif // USE(JSVALUE64)
             
         case DoubleRepUse: {
-            if (isClamped(type)) {
-                ASSERT(elementSize(type) == 1);
+            RELEASE_ASSERT(!isAtomicsIntrinsic(m_currentNode->op()));
+            if (isClamped) {
                 SpeculateDoubleOperand valueOp(this, valueUse);
                 GPRTemporary result(this);
                 FPRTemporary floatScratch(this);
@@ -2903,16 +2891,15 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
                 GPRReg gpr = result.gpr();
                 compileClampDoubleToByte(m_jit, gpr, fpr, floatScratch.fpr());
                 value.adopt(result);
-                valueGPR = gpr;
             } else {
 #if USE(JSVALUE32_64)
                 GPRTemporary realPropertyTag(this);
                 propertyTag.adopt(realPropertyTag);
-                propertyTagGPR = propertyTag.gpr();
+                GPRReg propertyTagGPR = propertyTag.gpr();
 
                 GPRTemporary realValueTag(this);
                 valueTag.adopt(realValueTag);
-                valueTagGPR = valueTag.gpr();
+                GPRReg valueTagGPR = valueTag.gpr();
 #endif
                 SpeculateDoubleOperand valueOp(this, valueUse);
                 GPRTemporary result(this);
@@ -2930,6 +2917,7 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
                 m_jit.or64(GPRInfo::tagTypeNumberRegister, property);
                 boxDouble(fpr, gpr);
 #else
+                UNUSED_PARAM(property);
                 m_jit.move(TrustedImm32(JSValue::Int32Tag), propertyTagGPR);
                 boxDouble(fpr, valueTagGPR, gpr);
 #endif
@@ -2937,7 +2925,6 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
 
                 fixed.link(&m_jit);
                 value.adopt(result);
-                valueGPR = gpr;
             }
             break;
         }
@@ -2947,7 +2934,43 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
             break;
         }
     }
+    return true;
+}
+
+void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg property, Node* node, TypedArrayType type)
+{
+    ASSERT(isInt(type));
+    
+    StorageOperand storage(this, m_jit.graph().varArgChild(node, 3));
+    GPRReg storageReg = storage.gpr();
     
+    Edge valueUse = m_jit.graph().varArgChild(node, 2);
+    
+    GPRTemporary value;
+#if USE(JSVALUE32_64)
+    GPRTemporary propertyTag;
+    GPRTemporary valueTag;
+#endif
+
+    JITCompiler::JumpList slowPathCases;
+    
+    bool result = getIntTypedArrayStoreOperand(
+        value, property,
+#if USE(JSVALUE32_64)
+        propertyTag, valueTag,
+#endif
+        valueUse, slowPathCases, isClamped(type));
+    if (!result) {
+        noResult(node);
+        return;
+    }
+
+    GPRReg valueGPR = value.gpr();
+#if USE(JSVALUE32_64)
+    GPRReg propertyTagGPR = propertyTag.gpr();
+    GPRReg valueTagGPR = valueTag.gpr();
+#endif
+
     ASSERT_UNUSED(valueGPR, valueGPR != property);
     ASSERT(valueGPR != base);
     ASSERT(valueGPR != storageReg);
@@ -2998,6 +3021,7 @@ void SpeculativeJIT::compilePutByValForIntTypedArray(GPRReg base, GPRReg propert
         }
 #endif
     }
+    
     noResult(node);
 }
 
index 75070a6..87b0173 100644 (file)
@@ -1824,6 +1824,11 @@ public:
         m_jit.setupArgumentsWithExecState(arg1, arg2, arg3);
         return appendCallSetResult(operation, result);
     }
+    JITCompiler::Call callOperation(J_JITOperation_EJJJJ operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg arg3, GPRReg arg4)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, arg3, arg4);
+        return appendCallSetResult(operation, result);
+    }
     JITCompiler::Call callOperation(J_JITOperation_ECC operation, GPRReg result, GPRReg arg1, GPRReg arg2)
     {
         m_jit.setupArgumentsWithExecState(arg1, arg2);
@@ -2837,6 +2842,17 @@ public:
     void compilePutByValForIntTypedArray(GPRReg base, GPRReg property, Node*, TypedArrayType);
     void compileGetByValOnFloatTypedArray(Node*, TypedArrayType);
     void compilePutByValForFloatTypedArray(GPRReg base, GPRReg property, Node*, TypedArrayType);
+    // If this returns false it means that we terminated speculative execution.
+    bool getIntTypedArrayStoreOperand(
+        GPRTemporary& value,
+        GPRReg property,
+#if USE(JSVALUE32_64)
+        GPRTemporary& propertyTag,
+        GPRTemporary& valueTag,
+#endif
+        Edge valueUse, JITCompiler::JumpList& slowPathCases, bool isClamped = false);
+    void loadFromIntTypedArray(GPRReg storageReg, GPRReg propertyReg, GPRReg resultReg, TypedArrayType);
+    void setIntTypedArrayLoadResult(Node*, GPRReg resultReg, TypedArrayType, bool canSpeculate = false);
     template <typename ClassType> void compileNewFunctionCommon(GPRReg, RegisteredStructure, GPRReg, GPRReg, GPRReg, MacroAssembler::JumpList&, size_t, FunctionExecutable*, ptrdiff_t, ptrdiff_t, ptrdiff_t);
     void compileNewFunction(Node*);
     void compileSetFunctionName(Node*);
index 07169f7..c327d66 100644 (file)
@@ -5668,6 +5668,16 @@ void SpeculativeJIT::compile(Node* node)
     case PhantomCreateRest:
     case PhantomSpread:
     case PhantomNewArrayWithSpread:
+    case AtomicsIsLockFree:
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor:
         DFG_CRASH(m_jit.graph(), node, "unexpected node in DFG backend");
         break;
     }
index a9c2e4b..ff2acc4 100644 (file)
@@ -29,6 +29,7 @@
 #if ENABLE(DFG_JIT)
 
 #include "ArrayPrototype.h"
+#include "AtomicsObject.h"
 #include "CallFrameShuffler.h"
 #include "DFGAbstractInterpreterInlines.h"
 #include "DFGCallArrayAllocatorSlowPathGenerator.h"
@@ -3176,6 +3177,248 @@ void SpeculativeJIT::compile(Node* node)
 
         break;
     }
+        
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor: {
+        unsigned numExtraArgs = numExtraAtomicsArgs(node->op());
+        Edge baseEdge = m_jit.graph().child(node, 0);
+        Edge indexEdge = m_jit.graph().child(node, 1);
+        Edge argEdges[maxNumExtraAtomicsArgs];
+        for (unsigned i = numExtraArgs; i--;)
+            argEdges[i] = m_jit.graph().child(node, 2 + i);
+        Edge storageEdge = m_jit.graph().child(node, 2 + numExtraArgs);
+
+        GPRReg baseGPR;
+        GPRReg indexGPR;
+        GPRReg argGPRs[2];
+        GPRReg resultGPR;
+
+        auto callSlowPath = [&] () {
+            switch (node->op()) {
+            case AtomicsAdd:
+                callOperation(operationAtomicsAdd, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            case AtomicsAnd:
+                callOperation(operationAtomicsAnd, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            case AtomicsCompareExchange:
+                callOperation(operationAtomicsCompareExchange, resultGPR, baseGPR, indexGPR, argGPRs[0], argGPRs[1]);
+                break;
+            case AtomicsExchange:
+                callOperation(operationAtomicsExchange, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            case AtomicsLoad:
+                callOperation(operationAtomicsLoad, resultGPR, baseGPR, indexGPR);
+                break;
+            case AtomicsOr:
+                callOperation(operationAtomicsOr, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            case AtomicsStore:
+                callOperation(operationAtomicsStore, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            case AtomicsSub:
+                callOperation(operationAtomicsSub, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            case AtomicsXor:
+                callOperation(operationAtomicsXor, resultGPR, baseGPR, indexGPR, argGPRs[0]);
+                break;
+            default:
+                RELEASE_ASSERT_NOT_REACHED();
+                break;
+            }
+        };
+        
+        if (!storageEdge) {
+            // We are in generic mode!
+            JSValueOperand base(this, baseEdge);
+            JSValueOperand index(this, indexEdge);
+            std::optional<JSValueOperand> args[2];
+            baseGPR = base.gpr();
+            indexGPR = index.gpr();
+            for (unsigned i = numExtraArgs; i--;) {
+                args[i].emplace(this, argEdges[i]);
+                argGPRs[i] = args[i]->gpr();
+            }
+            
+            GPRFlushedCallResult result(this);
+            resultGPR = result.gpr();
+            
+            flushRegisters();
+            callSlowPath();
+            m_jit.exceptionCheck();
+            
+            jsValueResult(resultGPR, node);
+            break;
+        }
+        
+        TypedArrayType type = node->arrayMode().typedArrayType();
+        
+        SpeculateCellOperand base(this, baseEdge);
+        SpeculateStrictInt32Operand index(this, indexEdge);
+
+        baseGPR = base.gpr();
+        indexGPR = index.gpr();
+        
+        emitTypedArrayBoundsCheck(node, baseGPR, indexGPR);
+        
+        GPRTemporary args[2];
+        
+        JITCompiler::JumpList slowPathCases;
+        
+        bool ok = true;
+        for (unsigned i = numExtraArgs; i--;) {
+            if (!getIntTypedArrayStoreOperand(args[i], indexGPR, argEdges[i], slowPathCases)) {
+                noResult(node);
+                ok = false;
+            }
+            argGPRs[i] = args[i].gpr();
+        }
+        if (!ok)
+            break;
+        
+        StorageOperand storage(this, storageEdge);
+        GPRTemporary oldValue(this);
+        GPRTemporary result(this);
+        GPRTemporary newValue(this);
+        GPRReg storageGPR = storage.gpr();
+        GPRReg oldValueGPR = oldValue.gpr();
+        resultGPR = result.gpr();
+        GPRReg newValueGPR = newValue.gpr();
+        
+        // FIXME: It shouldn't be necessary to nop-pad between register allocation and a jump label.
+        // https://bugs.webkit.org/show_bug.cgi?id=170974
+        m_jit.nop();
+        
+        JITCompiler::Label loop = m_jit.label();
+        
+        loadFromIntTypedArray(storageGPR, indexGPR, oldValueGPR, type);
+        m_jit.move(oldValueGPR, newValueGPR);
+        m_jit.move(oldValueGPR, resultGPR);
+        
+        switch (node->op()) {
+        case AtomicsAdd:
+            m_jit.add32(argGPRs[0], newValueGPR);
+            break;
+        case AtomicsAnd:
+            m_jit.and32(argGPRs[0], newValueGPR);
+            break;
+        case AtomicsCompareExchange: {
+            switch (elementSize(type)) {
+            case 1:
+                if (isSigned(type))
+                    m_jit.signExtend8To32(argGPRs[0], argGPRs[0]);
+                else
+                    m_jit.and32(TrustedImm32(0xff), argGPRs[0]);
+                break;
+            case 2:
+                if (isSigned(type))
+                    m_jit.signExtend16To32(argGPRs[0], argGPRs[0]);
+                else
+                    m_jit.and32(TrustedImm32(0xffff), argGPRs[0]);
+                break;
+            case 4:
+                break;
+            default:
+                RELEASE_ASSERT_NOT_REACHED();
+                break;
+            }
+            JITCompiler::Jump fail = m_jit.branch32(JITCompiler::NotEqual, oldValueGPR, argGPRs[0]);
+            m_jit.move(argGPRs[1], newValueGPR);
+            fail.link(&m_jit);
+            break;
+        }
+        case AtomicsExchange:
+            m_jit.move(argGPRs[0], newValueGPR);
+            break;
+        case AtomicsLoad:
+            break;
+        case AtomicsOr:
+            m_jit.or32(argGPRs[0], newValueGPR);
+            break;
+        case AtomicsStore:
+            m_jit.move(argGPRs[0], newValueGPR);
+            m_jit.move(argGPRs[0], resultGPR);
+            break;
+        case AtomicsSub:
+            m_jit.sub32(argGPRs[0], newValueGPR);
+            break;
+        case AtomicsXor:
+            m_jit.xor32(argGPRs[0], newValueGPR);
+            break;
+        default:
+            RELEASE_ASSERT_NOT_REACHED();
+            break;
+        }
+        
+        JITCompiler::JumpList success;
+        switch (elementSize(type)) {
+        case 1:
+            success = m_jit.branchAtomicWeakCAS8(JITCompiler::Success, oldValueGPR, newValueGPR, JITCompiler::BaseIndex(storageGPR, indexGPR, MacroAssembler::TimesOne));
+            break;
+        case 2:
+            success = m_jit.branchAtomicWeakCAS16(JITCompiler::Success, oldValueGPR, newValueGPR, JITCompiler::BaseIndex(storageGPR, indexGPR, MacroAssembler::TimesTwo));
+            break;
+        case 4:
+            success = m_jit.branchAtomicWeakCAS32(JITCompiler::Success, oldValueGPR, newValueGPR, JITCompiler::BaseIndex(storageGPR, indexGPR, MacroAssembler::TimesFour));
+            break;
+        default:
+            RELEASE_ASSERT_NOT_REACHED();
+            break;
+        }
+        m_jit.jump().linkTo(loop, &m_jit);
+        
+        if (!slowPathCases.empty()) {
+            slowPathCases.link(&m_jit);
+            silentSpillAllRegisters(resultGPR);
+            // Since we spilled, we can do things to registers.
+            m_jit.boxCell(baseGPR, JSValueRegs(baseGPR));
+            m_jit.boxInt32(indexGPR, JSValueRegs(indexGPR));
+            for (unsigned i = numExtraArgs; i--;)
+                m_jit.boxInt32(argGPRs[i], JSValueRegs(argGPRs[i]));
+            callSlowPath();
+            silentFillAllRegisters(resultGPR);
+            m_jit.exceptionCheck();
+        }
+        
+        success.link(&m_jit);
+        setIntTypedArrayLoadResult(node, resultGPR, type);
+        break;
+    }
+        
+    case AtomicsIsLockFree: {
+        if (node->child1().useKind() != Int32Use) {
+            JSValueOperand operand(this, node->child1());
+            GPRReg operandGPR = operand.gpr();
+            GPRFlushedCallResult result(this);
+            GPRReg resultGPR = result.gpr();
+            flushRegisters();
+            callOperation(operationAtomicsIsLockFree, resultGPR, operandGPR);
+            m_jit.exceptionCheck();
+            jsValueResult(resultGPR, node);
+            break;
+        }
+
+        SpeculateInt32Operand operand(this, node->child1());
+        GPRTemporary result(this);
+        GPRReg operandGPR = operand.gpr();
+        GPRReg resultGPR = result.gpr();
+        m_jit.move(TrustedImm32(ValueTrue), resultGPR);
+        JITCompiler::JumpList done;
+        done.append(m_jit.branch32(JITCompiler::Equal, operandGPR, TrustedImm32(4)));
+        done.append(m_jit.branch32(JITCompiler::Equal, operandGPR, TrustedImm32(1)));
+        done.append(m_jit.branch32(JITCompiler::Equal, operandGPR, TrustedImm32(2)));
+        m_jit.move(TrustedImm32(ValueFalse), resultGPR);
+        done.link(&m_jit);
+        jsValueResult(resultGPR, node);
+        break;
+    }
 
     case RegExpExec: {
         bool sample = false;
index 0b48c4b..5903a8e 100644 (file)
@@ -127,6 +127,11 @@ void AbstractHeapRepository::decorateFenceWrite(const AbstractHeap* heap, Value*
     m_heapForFenceWrite.append(HeapForValue(heap, value));
 }
 
+void AbstractHeapRepository::decorateFencedAccess(const AbstractHeap* heap, Value* value)
+{
+    m_heapForFencedAccess.append(HeapForValue(heap, value));
+}
+
 void AbstractHeapRepository::computeRangesAndDecorateInstructions()
 {
     root.compute();
@@ -156,6 +161,8 @@ void AbstractHeapRepository::computeRangesAndDecorateInstructions()
         entry.value->as<FenceValue>()->read = rangeFor(entry.heap);
     for (HeapForValue entry : m_heapForFenceWrite)
         entry.value->as<FenceValue>()->write = rangeFor(entry.heap);
+    for (HeapForValue entry : m_heapForFencedAccess)
+        entry.value->as<MemoryValue>()->setFenceRange(rangeFor(entry.heap));
 }
 
 } } // namespace JSC::FTL
index 4268179..2011af8 100644 (file)
@@ -224,6 +224,7 @@ public:
     void decoratePatchpointWrite(const AbstractHeap*, B3::Value*);
     void decorateFenceRead(const AbstractHeap*, B3::Value*);
     void decorateFenceWrite(const AbstractHeap*, B3::Value*);
+    void decorateFencedAccess(const AbstractHeap*, B3::Value*);
 
     void computeRangesAndDecorateInstructions();
 
@@ -251,6 +252,7 @@ private:
     Vector<HeapForValue> m_heapForPatchpointWrite;
     Vector<HeapForValue> m_heapForFenceRead;
     Vector<HeapForValue> m_heapForFenceWrite;
+    Vector<HeapForValue> m_heapForFencedAccess;
 };
 
 } } // namespace JSC::FTL
index a69ece8..7a2f8ac 100644 (file)
@@ -286,6 +286,16 @@ inline CapabilityLevel canCompile(Node* node)
     case CallDOMGetter:
     case ArraySlice:
     case ParseInt:
+    case AtomicsAdd:
+    case AtomicsAnd:
+    case AtomicsCompareExchange:
+    case AtomicsExchange:
+    case AtomicsLoad:
+    case AtomicsOr:
+    case AtomicsStore:
+    case AtomicsSub:
+    case AtomicsXor:
+    case AtomicsIsLockFree:
         // These are OK.
         break;
 
index 1600b8f..5e978e4 100644 (file)
@@ -30,6 +30,7 @@
 
 #include "AirGenerationContext.h"
 #include "AllowMacroScratchRegisterUsage.h"
+#include "AtomicsObject.h"
 #include "B3CheckValue.h"
 #include "B3FenceValue.h"
 #include "B3PatchpointValue.h"
@@ -699,6 +700,20 @@ private:
         case PutByValWithThis:
             compilePutByValWithThis();
             break;
+        case AtomicsAdd:
+        case AtomicsAnd:
+        case AtomicsCompareExchange:
+        case AtomicsExchange:
+        case AtomicsLoad:
+        case AtomicsOr:
+        case AtomicsStore:
+        case AtomicsSub:
+        case AtomicsXor:
+            compileAtomicsReadModifyWrite();
+            break;
+        case AtomicsIsLockFree:
+            compileAtomicsIsLockFree();
+            break;
         case DefineDataProperty:
             compileDefineDataProperty();
             break;
@@ -2944,6 +2959,167 @@ private:
         vmCall(Void, m_out.operation(m_graph.isStrictModeFor(m_node->origin.semantic) ? operationPutByValWithThisStrict : operationPutByValWithThis),
             m_callFrame, base, thisValue, property, value);
     }
+    
+    void compileAtomicsReadModifyWrite()
+    {
+        TypedArrayType type = m_node->arrayMode().typedArrayType();
+        unsigned numExtraArgs = numExtraAtomicsArgs(m_node->op());
+        Edge baseEdge = m_graph.child(m_node, 0);
+        Edge indexEdge = m_graph.child(m_node, 1);
+        Edge argEdges[maxNumExtraAtomicsArgs];
+        for (unsigned i = numExtraArgs; i--;)
+            argEdges[i] = m_graph.child(m_node, 2 + i);
+        Edge storageEdge = m_graph.child(m_node, 2 + numExtraArgs);
+        
+        auto operation = [&] () -> LValue {
+            switch (m_node->op()) {
+            case AtomicsAdd:
+                return m_out.operation(operationAtomicsAdd);
+            case AtomicsAnd:
+                return m_out.operation(operationAtomicsAnd);
+            case AtomicsCompareExchange:
+                return m_out.operation(operationAtomicsCompareExchange);
+            case AtomicsExchange:
+                return m_out.operation(operationAtomicsExchange);
+            case AtomicsLoad:
+                return m_out.operation(operationAtomicsLoad);
+            case AtomicsOr:
+                return m_out.operation(operationAtomicsOr);
+            case AtomicsStore:
+                return m_out.operation(operationAtomicsStore);
+            case AtomicsSub:
+                return m_out.operation(operationAtomicsSub);
+            case AtomicsXor:
+                return m_out.operation(operationAtomicsXor);
+            default:
+                RELEASE_ASSERT_NOT_REACHED();
+                break;
+            }
+        };
+        
+        if (!storageEdge) {
+            Vector<LValue> args;
+            args.append(m_callFrame);
+            args.append(lowJSValue(baseEdge));
+            args.append(lowJSValue(indexEdge));
+            for (unsigned i = 0; i < numExtraArgs; ++i)
+                args.append(lowJSValue(argEdges[i]));
+            LValue result = vmCall(Int64, operation(), args);
+            setJSValue(result);
+            return;
+        }
+        
+        LValue index = lowInt32(indexEdge);
+        LValue args[2];
+        for (unsigned i = numExtraArgs; i--;)
+            args[i] = getIntTypedArrayStoreOperand(argEdges[i]);
+        LValue storage = lowStorage(storageEdge);
+        
+        TypedPointer pointer = pointerIntoTypedArray(storage, index, type);
+        Width width = widthForBytes(elementSize(type));
+        
+        LValue atomicValue;
+        LValue result;
+        
+        auto sanitizeResult = [&] (LValue value) -> LValue {
+            if (isSigned(type)) {
+                switch (elementSize(type)) {
+                case 1:
+                    value = m_out.bitAnd(value, m_out.constInt32(0xff));
+                    break;
+                case 2:
+                    value = m_out.bitAnd(value, m_out.constInt32(0xffff));
+                    break;
+                case 4:
+                    break;
+                default:
+                    RELEASE_ASSERT_NOT_REACHED();
+                    break;
+                }
+            }
+            return value;
+        };
+        
+        switch (m_node->op()) {
+        case AtomicsAdd:
+            atomicValue = m_out.atomicXchgAdd(args[0], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsAnd:
+            atomicValue = m_out.atomicXchgAnd(args[0], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsCompareExchange:
+            atomicValue = m_out.atomicStrongCAS(args[0], args[1], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsExchange:
+            atomicValue = m_out.atomicXchg(args[0], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsLoad:
+            atomicValue = loadFromIntTypedArray(pointer, type);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsOr:
+            atomicValue = m_out.atomicXchgOr(args[0], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsStore:
+            atomicValue = m_out.store(args[0], pointer, storeType(type));
+            result = args[0];
+            break;
+        case AtomicsSub:
+            atomicValue = m_out.atomicXchgSub(args[0], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        case AtomicsXor:
+            atomicValue = m_out.atomicXchgXor(args[0], pointer, width);
+            result = sanitizeResult(atomicValue);
+            break;
+        default:
+            RELEASE_ASSERT_NOT_REACHED();
+            break;
+        }
+        // Signify that the state against which the atomic operations are serialized is confined to just
+        // the typed array storage, since that's as precise of an abstraction as we can have of shared
+        // array buffer storage.
+        m_heaps.decorateFencedAccess(&m_heaps.typedArrayProperties, atomicValue);
+        
+        setIntTypedArrayLoadResult(result, type);
+    }
+    
+    void compileAtomicsIsLockFree()
+    {
+        if (m_node->child1().useKind() != Int32Use) {
+            setJSValue(vmCall(Int64, m_out.operation(operationAtomicsIsLockFree), m_callFrame, lowJSValue(m_node->child1())));
+            return;
+        }
+        
+        LValue bytes = lowInt32(m_node->child1());
+        
+        LBasicBlock trueCase = m_out.newBlock();
+        LBasicBlock falseCase = m_out.newBlock();
+        LBasicBlock continuation = m_out.newBlock();
+        
+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(trueCase);
+        
+        Vector<SwitchCase> cases;
+        cases.append(SwitchCase(m_out.constInt32(1), trueCase, Weight()));
+        cases.append(SwitchCase(m_out.constInt32(2), trueCase, Weight()));
+        cases.append(SwitchCase(m_out.constInt32(4), trueCase, Weight()));
+        m_out.switchInstruction(bytes, cases, falseCase, Weight());
+        
+        m_out.appendTo(trueCase, falseCase);
+        ValueFromBlock trueValue = m_out.anchor(m_out.booleanTrue);
+        m_out.jump(continuation);
+        m_out.appendTo(falseCase, continuation);
+        ValueFromBlock falseValue = m_out.anchor(m_out.booleanFalse);
+        m_out.jump(continuation);
+        
+        m_out.appendTo(continuation, lastNext);
+        setBoolean(m_out.phi(Int32, trueValue, falseValue));
+    }
 
     void compileDefineDataProperty()
     {
@@ -3426,48 +3602,12 @@ private:
             TypedArrayType type = m_node->arrayMode().typedArrayType();
             
             if (isTypedView(type)) {
-                TypedPointer pointer = TypedPointer(
-                    m_heaps.typedArrayProperties,
-                    m_out.add(
-                        storage,
-                        m_out.shl(
-                            m_out.zeroExtPtr(index),
-                            m_out.constIntPtr(logElementSize(type)))));
+                TypedPointer pointer = pointerIntoTypedArray(storage, index, type);
                 
                 if (isInt(type)) {
-                    LValue result;
-                    switch (elementSize(type)) {
-                    case 1:
-                        result = isSigned(type) ? m_out.load8SignExt32(pointer) :  m_out.load8ZeroExt32(pointer);
-                        break;
-                    case 2:
-                        result = isSigned(type) ? m_out.load16SignExt32(pointer) :  m_out.load16ZeroExt32(pointer);
-                        break;
-                    case 4:
-                        result = m_out.load32(pointer);
-                        break;
-                    default:
-                        DFG_CRASH(m_graph, m_node, "Bad element size");
-                    }
-                    
-                    if (elementSize(type) < 4 || isSigned(type)) {
-                        setInt32(result);
-                        return;
-                    }
-
-                    if (m_node->shouldSpeculateInt32()) {
-                        speculate(
-                            Overflow, noValue(), 0, m_out.lessThan(result, m_out.int32Zero));
-                        setInt32(result);
-                        return;
-                    }
-                    
-                    if (m_node->shouldSpeculateAnyInt()) {
-                        setStrictInt52(m_out.zeroExt(result, Int64));
-                        return;
-                    }
-                    
-                    setDouble(m_out.unsignedToDouble(result));
+                    LValue result = loadFromIntTypedArray(pointer, type);
+                    bool canSpeculate = true;
+                    setIntTypedArrayLoadResult(result, type, canSpeculate);
                     return;
                 }
             
@@ -3670,106 +3810,20 @@ private:
                             m_out.zeroExt(index, pointerType()),
                             m_out.constIntPtr(logElementSize(type)))));
                 
-                Output::StoreType storeType;
                 LValue valueToStore;
                 
                 if (isInt(type)) {
-                    LValue intValue;
-                    switch (child3.useKind()) {
-                    case Int52RepUse:
-                    case Int32Use: {
-                        if (child3.useKind() == Int32Use)
-                            intValue = lowInt32(child3);
-                        else
-                            intValue = m_out.castToInt32(lowStrictInt52(child3));
-
-                        if (isClamped(type)) {
-                            ASSERT(elementSize(type) == 1);
-                            
-                            LBasicBlock atLeastZero = m_out.newBlock();
-                            LBasicBlock continuation = m_out.newBlock();
-                            
-                            Vector<ValueFromBlock, 2> intValues;
-                            intValues.append(m_out.anchor(m_out.int32Zero));
-                            m_out.branch(
-                                m_out.lessThan(intValue, m_out.int32Zero),
-                                unsure(continuation), unsure(atLeastZero));
-                            
-                            LBasicBlock lastNext = m_out.appendTo(atLeastZero, continuation);
-                            
-                            intValues.append(m_out.anchor(m_out.select(
-                                m_out.greaterThan(intValue, m_out.constInt32(255)),
-                                m_out.constInt32(255),
-                                intValue)));
-                            m_out.jump(continuation);
-                            
-                            m_out.appendTo(continuation, lastNext);
-                            intValue = m_out.phi(Int32, intValues);
-                        }
-                        break;
-                    }
-                        
-                    case DoubleRepUse: {
-                        LValue doubleValue = lowDouble(child3);
-                        
-                        if (isClamped(type)) {
-                            ASSERT(elementSize(type) == 1);
-                            
-                            LBasicBlock atLeastZero = m_out.newBlock();
-                            LBasicBlock withinRange = m_out.newBlock();
-                            LBasicBlock continuation = m_out.newBlock();
-                            
-                            Vector<ValueFromBlock, 3> intValues;
-                            intValues.append(m_out.anchor(m_out.int32Zero));
-                            m_out.branch(
-                                m_out.doubleLessThanOrUnordered(doubleValue, m_out.doubleZero),
-                                unsure(continuation), unsure(atLeastZero));
-                            
-                            LBasicBlock lastNext = m_out.appendTo(atLeastZero, withinRange);
-                            intValues.append(m_out.anchor(m_out.constInt32(255)));
-                            m_out.branch(
-                                m_out.doubleGreaterThan(doubleValue, m_out.constDouble(255)),
-                                unsure(continuation), unsure(withinRange));
-                            
-                            m_out.appendTo(withinRange, continuation);
-                            intValues.append(m_out.anchor(m_out.doubleToInt(doubleValue)));
-                            m_out.jump(continuation);
-                            
-                            m_out.appendTo(continuation, lastNext);
-                            intValue = m_out.phi(Int32, intValues);
-                        } else
-                            intValue = doubleToInt32(doubleValue);
-                        break;
-                    }
-                        
-                    default:
-                        DFG_CRASH(m_graph, m_node, "Bad use kind");
-                    }
+                    LValue intValue = getIntTypedArrayStoreOperand(child3, isClamped(type));
 
                     valueToStore = intValue;
-                    switch (elementSize(type)) {
-                    case 1:
-                        storeType = Output::Store32As8;
-                        break;
-                    case 2:
-                        storeType = Output::Store32As16;
-                        break;
-                    case 4:
-                        storeType = Output::Store32;
-                        break;
-                    default:
-                        DFG_CRASH(m_graph, m_node, "Bad element size");
-                    }
                 } else /* !isInt(type) */ {
                     LValue value = lowDouble(child3);
                     switch (type) {
                     case TypeFloat32:
                         valueToStore = m_out.doubleToFloat(value);
-                        storeType = Output::StoreFloat;
                         break;
                     case TypeFloat64:
                         valueToStore = value;
-                        storeType = Output::StoreDouble;
                         break;
                     default:
                         DFG_CRASH(m_graph, m_node, "Bad typed array type");
@@ -3777,7 +3831,7 @@ private:
                 }
 
                 if (m_node->arrayMode().isInBounds() || m_node->op() == PutByValAlias)
-                    m_out.store(valueToStore, pointer, storeType);
+                    m_out.store(valueToStore, pointer, storeType(type));
                 else {
                     LBasicBlock isInBounds = m_out.newBlock();
                     LBasicBlock isOutOfBounds = m_out.newBlock();
@@ -3788,7 +3842,7 @@ private:
                         unsure(isOutOfBounds), unsure(isInBounds));
                     
                     LBasicBlock lastNext = m_out.appendTo(isInBounds, isOutOfBounds);
-                    m_out.store(valueToStore, pointer, storeType);
+                    m_out.store(valueToStore, pointer, storeType(type));
                     m_out.jump(continuation);
 
                     m_out.appendTo(isOutOfBounds, continuation);
@@ -11752,6 +11806,151 @@ private:
         functor(TypeofType::Undefined);
     }
     
+    TypedPointer pointerIntoTypedArray(LValue storage, LValue index, TypedArrayType type)
+    {
+        return TypedPointer(
+            m_heaps.typedArrayProperties,
+            m_out.add(
+                storage,
+                m_out.shl(
+                    m_out.zeroExtPtr(index),
+                    m_out.constIntPtr(logElementSize(type)))));
+    }
+    
+    LValue loadFromIntTypedArray(TypedPointer pointer, TypedArrayType type)
+    {
+        switch (elementSize(type)) {
+        case 1:
+            return isSigned(type) ? m_out.load8SignExt32(pointer) : m_out.load8ZeroExt32(pointer);
+        case 2:
+            return isSigned(type) ? m_out.load16SignExt32(pointer) : m_out.load16ZeroExt32(pointer);
+        case 4:
+            return m_out.load32(pointer);
+        default:
+            DFG_CRASH(m_graph, m_node, "Bad element size");
+        }
+    }
+    
+    Output::StoreType storeType(TypedArrayType type)
+    {
+        if (isInt(type)) {
+            switch (elementSize(type)) {
+            case 1:
+                return Output::Store32As8;
+            case 2:
+                return Output::Store32As16;
+            case 4:
+                return Output::Store32;
+            default:
+                DFG_CRASH(m_graph, m_node, "Bad element size");
+                return Output::Store32;
+            }
+        }
+        switch (type) {
+        case TypeFloat32:
+            return Output::StoreFloat;
+        case TypeFloat64:
+            return Output::StoreDouble;
+        default:
+            DFG_CRASH(m_graph, m_node, "Bad typed array type");
+        }
+    }
+    
+    void setIntTypedArrayLoadResult(LValue result, TypedArrayType type, bool canSpeculate = false)
+    {
+        if (elementSize(type) < 4 || isSigned(type)) {
+            setInt32(result);
+            return;
+        }
+        
+        if (m_node->shouldSpeculateInt32() && canSpeculate) {
+            speculate(
+                Overflow, noValue(), 0, m_out.lessThan(result, m_out.int32Zero));
+            setInt32(result);
+            return;
+        }
+        
+        if (m_node->shouldSpeculateAnyInt()) {
+            setStrictInt52(m_out.zeroExt(result, Int64));
+            return;
+        }
+        
+        setDouble(m_out.unsignedToDouble(result));
+    }
+    
+    LValue getIntTypedArrayStoreOperand(Edge edge, bool isClamped = false)
+    {
+        LValue intValue;
+        switch (edge.useKind()) {
+        case Int52RepUse:
+        case Int32Use: {
+            if (edge.useKind() == Int32Use)
+                intValue = lowInt32(edge);
+            else
+                intValue = m_out.castToInt32(lowStrictInt52(edge));
+
+            if (isClamped) {
+                LBasicBlock atLeastZero = m_out.newBlock();
+                LBasicBlock continuation = m_out.newBlock();
+                            
+                Vector<ValueFromBlock, 2> intValues;
+                intValues.append(m_out.anchor(m_out.int32Zero));
+                m_out.branch(
+                    m_out.lessThan(intValue, m_out.int32Zero),
+                    unsure(continuation), unsure(atLeastZero));
+                            
+                LBasicBlock lastNext = m_out.appendTo(atLeastZero, continuation);
+                            
+                intValues.append(m_out.anchor(m_out.select(
+                    m_out.greaterThan(intValue, m_out.constInt32(255)),
+                    m_out.constInt32(255),
+                    intValue)));
+                m_out.jump(continuation);
+                            
+                m_out.appendTo(continuation, lastNext);
+                intValue = m_out.phi(Int32, intValues);
+            }
+            break;
+        }
+                        
+        case DoubleRepUse: {
+            LValue doubleValue = lowDouble(edge);
+                        
+            if (isClamped) {
+                LBasicBlock atLeastZero = m_out.newBlock();
+                LBasicBlock withinRange = m_out.newBlock();
+                LBasicBlock continuation = m_out.newBlock();
+                            
+                Vector<ValueFromBlock, 3> intValues;
+                intValues.append(m_out.anchor(m_out.int32Zero));
+                m_out.branch(
+                    m_out.doubleLessThanOrUnordered(doubleValue, m_out.doubleZero),
+                    unsure(continuation), unsure(atLeastZero));
+                            
+                LBasicBlock lastNext = m_out.appendTo(atLeastZero, withinRange);
+                intValues.append(m_out.anchor(m_out.constInt32(255)));
+                m_out.branch(
+                    m_out.doubleGreaterThan(doubleValue, m_out.constDouble(255)),
+                    unsure(continuation), unsure(withinRange));
+                            
+                m_out.appendTo(withinRange, continuation);
+                intValues.append(m_out.anchor(m_out.doubleToInt(doubleValue)));
+                m_out.jump(continuation);
+                            
+                m_out.appendTo(continuation, lastNext);
+                intValue = m_out.phi(Int32, intValues);
+            } else
+                intValue = doubleToInt32(doubleValue);
+            break;
+        }
+                        
+        default:
+            DFG_CRASH(m_graph, m_node, "Bad use kind");
+        }
+        
+        return intValue;
+    }
+    
     LValue doubleToInt32(LValue doubleValue, double low, double high, bool isSigned = true)
     {
         LBasicBlock greatEnough = m_out.newBlock();
@@ -13398,10 +13597,10 @@ private:
     }
 
     template<typename... Args>
-    LValue vmCall(LType type, LValue function, Args... args)
+    LValue vmCall(LType type, LValue function, Args&&... args)
     {
         callPreflight();
-        LValue result = m_out.call(type, function, args...);
+        LValue result = m_out.call(type, function, std::forward<Args>(args)...);
         callCheck();
         return result;
     }
index bd65e64..dff75a4 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2013-2016 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -29,6 +29,7 @@
 #if ENABLE(FTL_JIT)
 
 #include "B3ArgumentRegValue.h"
+#include "B3AtomicValue.h"
 #include "B3BasicBlockInlines.h"
 #include "B3CCallValue.h"
 #include "B3Const32Value.h"
@@ -440,10 +441,11 @@ LValue Output::load16ZeroExt32(TypedPointer pointer)
     return load;
 }
 
-void Output::store(LValue value, TypedPointer pointer)
+LValue Output::store(LValue value, TypedPointer pointer)
 {
     LValue store = m_block->appendNew<MemoryValue>(m_proc, Store, origin(), value, pointer.value());
     m_heaps->decorateMemory(pointer.heap(), store);
+    return store;
 }
 
 FenceValue* Output::fence(const AbstractHeap* read, const AbstractHeap* write)
@@ -454,16 +456,18 @@ FenceValue* Output::fence(const AbstractHeap* read, const AbstractHeap* write)
     return result;
 }
 
-void Output::store32As8(LValue value, TypedPointer pointer)
+LValue Output::store32As8(LValue value, TypedPointer pointer)
 {
     LValue store = m_block->appendNew<MemoryValue>(m_proc, Store8, origin(), value, pointer.value());
     m_heaps->decorateMemory(pointer.heap(), store);
+    return store;
 }
 
-void Output::store32As16(LValue value, TypedPointer pointer)
+LValue Output::store32As16(LValue value, TypedPointer pointer)
 {
     LValue store = m_block->appendNew<MemoryValue>(m_proc, Store16, origin(), value, pointer.value());
     m_heaps->decorateMemory(pointer.heap(), store);
+    return store;
 }
 
 LValue Output::baseIndex(LValue base, LValue index, Scale scale, ptrdiff_t offset)
@@ -663,6 +667,55 @@ LValue Output::select(LValue value, LValue taken, LValue notTaken)
     return m_block->appendNew<B3::Value>(m_proc, B3::Select, origin(), value, taken, notTaken);
 }
 
+LValue Output::atomicXchgAdd(LValue operand, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicXchgAdd, origin(), width, operand, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
+LValue Output::atomicXchgAnd(LValue operand, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicXchgAnd, origin(), width, operand, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
+LValue Output::atomicXchgOr(LValue operand, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicXchgOr, origin(), width, operand, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
+LValue Output::atomicXchgSub(LValue operand, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicXchgSub, origin(), width, operand, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
+LValue Output::atomicXchgXor(LValue operand, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicXchgXor, origin(), width, operand, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
+LValue Output::atomicXchg(LValue operand, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicXchg, origin(), width, operand, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
+LValue Output::atomicStrongCAS(LValue expected, LValue newValue, TypedPointer pointer, Width width)
+{
+    LValue result = m_block->appendNew<AtomicValue>(m_proc, AtomicStrongCAS, origin(), width, expected, newValue, pointer.value(), 0, HeapRange(), HeapRange());
+    m_heaps->decorateMemory(pointer.heap(), result);
+    return result;
+}
+
 void Output::jump(LBasicBlock destination)
 {
     m_block->appendNewControlValue(m_proc, B3::Jump, origin(), B3::FrequentedBlock(destination));
@@ -776,32 +829,26 @@ LValue Output::load(TypedPointer pointer, LoadType type)
     return nullptr;
 }
 
-void Output::store(LValue value, TypedPointer pointer, StoreType type)
+LValue Output::store(LValue value, TypedPointer pointer, StoreType type)
 {
     switch (type) {
     case Store32As8:
-        store32As8(value, pointer);
-        return;
+        return store32As8(value, pointer);
     case Store32As16:
-        store32As16(value, pointer);
-        return;
+        return store32As16(value, pointer);
     case Store32:
-        store32(value, pointer);
-        return;
+        return store32(value, pointer);
     case Store64:
-        store64(value, pointer);
-        return;
+        return store64(value, pointer);
     case StorePtr:
-        storePtr(value, pointer);
-        return;
+        return storePtr(value, pointer);
     case StoreFloat:
-        storeFloat(value, pointer);
-        return;
+        return storeFloat(value, pointer);
     case StoreDouble:
-        storeDouble(value, pointer);
-        return;
+        return storeDouble(value, pointer);
     }
     RELEASE_ASSERT_NOT_REACHED();
+    return nullptr;
 }
 
 TypedPointer Output::absolute(const void* address)
index 91e548c..ccd7cbd 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2013-2016 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -35,6 +35,7 @@
 #include "B3FrequentedBlock.h"
 #include "B3Procedure.h"
 #include "B3SwitchValue.h"
+#include "B3Width.h"
 #include "FTLAbbreviatedTypes.h"
 #include "FTLAbstractHeapRepository.h"
 #include "FTLCommonValues.h"
@@ -209,7 +210,7 @@ public:
     LValue fround(LValue);
 
     LValue load(TypedPointer, LType);
-    void store(LValue, TypedPointer);
+    LValue store(LValue, TypedPointer);
     B3::FenceValue* fence(const AbstractHeap* read, const AbstractHeap* write);
 
     LValue load8SignExt32(TypedPointer);
@@ -221,32 +222,32 @@ public:
     LValue loadPtr(TypedPointer pointer) { return load(pointer, B3::pointerType()); }
     LValue loadFloat(TypedPointer pointer) { return load(pointer, B3::Float); }
     LValue loadDouble(TypedPointer pointer) { return load(pointer, B3::Double); }
-    void store32As8(LValue, TypedPointer);
-    void store32As16(LValue, TypedPointer);
-    void store32(LValue value, TypedPointer pointer)
+    LValue store32As8(LValue, TypedPointer);
+    LValue store32As16(LValue, TypedPointer);
+    LValue store32(LValue value, TypedPointer pointer)
     {
         ASSERT(value->type() == B3::Int32);
-        store(value, pointer);
+        return store(value, pointer);
     }
-    void store64(LValue value, TypedPointer pointer)
+    LValue store64(LValue value, TypedPointer pointer)
     {
         ASSERT(value->type() == B3::Int64);
-        store(value, pointer);
+        return store(value, pointer);
     }
-    void storePtr(LValue value, TypedPointer pointer)
+    LValue storePtr(LValue value, TypedPointer pointer)
     {
         ASSERT(value->type() == B3::pointerType());
-        store(value, pointer);
+        return store(value, pointer);
     }
-    void storeFloat(LValue value, TypedPointer pointer)
+    LValue storeFloat(LValue value, TypedPointer pointer)
     {
         ASSERT(value->type() == B3::Float);
-        store(value, pointer);
+        return store(value, pointer);
     }
-    void storeDouble(LValue value, TypedPointer pointer)
+    LValue storeDouble(LValue value, TypedPointer pointer)
     {
         ASSERT(value->type() == B3::Double);
-        store(value, pointer);
+        return store(value, pointer);
     }
 
     enum LoadType {
@@ -273,7 +274,7 @@ public:
         StoreDouble
     };
 
-    void store(LValue, TypedPointer, StoreType);
+    LValue store(LValue, TypedPointer, StoreType);
 
     LValue addPtr(LValue value, ptrdiff_t immediate = 0)
     {
@@ -369,6 +370,16 @@ public:
     LValue testNonZeroPtr(LValue value, LValue mask) { return notNull(bitAnd(value, mask)); }
 
     LValue select(LValue value, LValue taken, LValue notTaken);
+    
+    // These are relaxed atomics by default. Use AbstractHeapRepository::decorateFencedAccess() with a
+    // non-null heap to make them seq_cst fenced.
+    LValue atomicXchgAdd(LValue operand, TypedPointer pointer, B3::Width);
+    LValue atomicXchgAnd(LValue operand, TypedPointer pointer, B3::Width);
+    LValue atomicXchgOr(LValue operand, TypedPointer pointer, B3::Width);
+    LValue atomicXchgSub(LValue operand, TypedPointer pointer, B3::Width);
+    LValue atomicXchgXor(LValue operand, TypedPointer pointer, B3::Width);
+    LValue atomicXchg(LValue operand, TypedPointer pointer, B3::Width);
+    LValue atomicStrongCAS(LValue expected, LValue newValue, TypedPointer pointer, B3::Width);
 
     template<typename VectorType>
     LValue call(LType type, LValue function, const VectorType& vector)
index 0f99e03..1b538ae 100644 (file)
@@ -147,10 +147,10 @@ typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJArp)(ExecState*, Encoded
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJI)(ExecState*, EncodedJSValue, UniquedStringImpl*);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJ)(ExecState*, EncodedJSValue, EncodedJSValue);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJJ)(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue);
+typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJJJ)(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, EncodedJSValue);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJAp)(ExecState*, EncodedJSValue, EncodedJSValue, ArrayProfile*);
-typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJBy)(ExecState*, EncodedJSValue, EncodedJSValue, ByValInfo*);
-typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJJ)(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJArp)(ExecState*, EncodedJSValue, EncodedJSValue, ArithProfile*);
+typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJBy)(ExecState*, EncodedJSValue, EncodedJSValue, ByValInfo*);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJJMic)(ExecState*, EncodedJSValue, EncodedJSValue, void*);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJMic)(ExecState*, EncodedJSValue, void*);
 typedef EncodedJSValue (JIT_OPERATION *J_JITOperation_EJscI)(ExecState*, JSScope*, UniquedStringImpl*);
index f9e723c..b3566d1 100644 (file)
@@ -26,6 +26,7 @@
 #include "config.h"
 #include "AtomicsObject.h"
 
+#include "FrameTracers.h"
 #include "JSCInlines.h"
 #include "JSTypedArrays.h"
 #include "ObjectPrototype.h"
@@ -87,14 +88,14 @@ void AtomicsObject::finishCreation(VM& vm, JSGlobalObject* globalObject)
 
 namespace {
 
-template<unsigned numExtraArgs, typename Adaptor, typename Func>
-EncodedJSValue atomicOperationWithArgsCase(ExecState* exec, ThrowScope& scope, JSArrayBufferView* typedArrayView, unsigned accessIndex, const Func& func)
+template<typename Adaptor, typename Func>
+EncodedJSValue atomicOperationWithArgsCase(ExecState* exec, const JSValue* args, ThrowScope& scope, JSArrayBufferView* typedArrayView, unsigned accessIndex, const Func& func)
 {
     JSGenericTypedArrayView<Adaptor>* typedArray = jsCast<JSGenericTypedArrayView<Adaptor>*>(typedArrayView);
     
-    double extraArgs[numExtraArgs + 1]; // Add 1 to avoid 0 size array error in VS.
-    for (unsigned i = 0; i < numExtraArgs; ++i) {
-        double value = exec->argument(2 + i).toInteger(exec);
+    double extraArgs[Func::numExtraArgs + 1]; // Add 1 to avoid 0 size array error in VS.
+    for (unsigned i = 0; i < Func::numExtraArgs; ++i) {
+        double value = args[2 + i].toInteger(exec);
         RETURN_IF_EXCEPTION(scope, JSValue::encode(jsUndefined()));
         extraArgs[i] = value;
     }
@@ -102,10 +103,9 @@ EncodedJSValue atomicOperationWithArgsCase(ExecState* exec, ThrowScope& scope, J
     return JSValue::encode(func(typedArray->typedVector() + accessIndex, extraArgs));
 }
 
-unsigned validatedAccessIndex(VM& vm, ExecState* exec, JSArrayBufferView* typedArrayView)
+unsigned validatedAccessIndex(VM& vm, ExecState* exec, JSValue accessIndexValue, JSArrayBufferView* typedArrayView)
 {
     auto scope = DECLARE_THROW_SCOPE(vm);
-    JSValue accessIndexValue = exec->argument(1);
     if (UNLIKELY(!accessIndexValue.isInt32())) {
         double accessIndexDouble = accessIndexValue.toNumber(exec);
         RETURN_IF_EXCEPTION(scope, 0);
@@ -130,13 +130,12 @@ unsigned validatedAccessIndex(VM& vm, ExecState* exec, JSArrayBufferView* typedA
     return accessIndex;
 }
 
-template<unsigned numExtraArgs, typename Func>
-EncodedJSValue atomicOperationWithArgs(ExecState* exec, const Func& func)
+template<typename Func>
+EncodedJSValue atomicOperationWithArgs(VM& vm, ExecState* exec, const JSValue* args, const Func& func)
 {
-    VM& vm = exec->vm();
     auto scope = DECLARE_THROW_SCOPE(vm);
 
-    JSValue typedArrayValue = exec->argument(0);
+    JSValue typedArrayValue = args[0];
     if (!typedArrayValue.isCell()) {
         throwTypeError(exec, scope, ASCIILiteral("Typed array argument must be a cell."));
         return JSValue::encode(jsUndefined());
@@ -164,72 +163,138 @@ EncodedJSValue atomicOperationWithArgs(ExecState* exec, const Func& func)
         return JSValue::encode(jsUndefined());
     }
     
-    unsigned accessIndex = validatedAccessIndex(vm, exec, typedArrayView);
+    unsigned accessIndex = validatedAccessIndex(vm, exec, args[1], typedArrayView);
     RETURN_IF_EXCEPTION(scope, JSValue::encode(jsUndefined()));
     
     switch (type) {
     case Int8ArrayType:
-        return atomicOperationWithArgsCase<numExtraArgs, Int8Adaptor>(exec, scope, typedArrayView, accessIndex, func);
+        return atomicOperationWithArgsCase<Int8Adaptor>(exec, args, scope, typedArrayView, accessIndex, func);
     case Int16ArrayType:
-        return atomicOperationWithArgsCase<numExtraArgs, Int16Adaptor>(exec, scope, typedArrayView, accessIndex, func);
+        return atomicOperationWithArgsCase<Int16Adaptor>(exec, args, scope, typedArrayView, accessIndex, func);
     case Int32ArrayType:
-        return atomicOperationWithArgsCase<numExtraArgs, Int32Adaptor>(exec, scope, typedArrayView, accessIndex, func);
+        return atomicOperationWithArgsCase<Int32Adaptor>(exec, args, scope, typedArrayView, accessIndex, func);
     case Uint8ArrayType:
-        return atomicOperationWithArgsCase<numExtraArgs, Uint8Adaptor>(exec, scope, typedArrayView, accessIndex, func);
+        return atomicOperationWithArgsCase<Uint8Adaptor>(exec, args, scope, typedArrayView, accessIndex, func);
     case Uint16ArrayType:
-        return atomicOperationWithArgsCase<numExtraArgs, Uint16Adaptor>(exec, scope, typedArrayView, accessIndex, func);
+        return atomicOperationWithArgsCase<Uint16Adaptor>(exec, args, scope, typedArrayView, accessIndex, func);
     case Uint32ArrayType:
-        return atomicOperationWithArgsCase<numExtraArgs, Uint32Adaptor>(exec, scope, typedArrayView, accessIndex, func);
+        return atomicOperationWithArgsCase<Uint32Adaptor>(exec, args, scope, typedArrayView, accessIndex, func);
     default:
         RELEASE_ASSERT_NOT_REACHED();
         return JSValue::encode(jsUndefined());
     }
 }
 
-} // anonymous namespace
-
-EncodedJSValue JSC_HOST_CALL atomicsFuncAdd(ExecState* exec)
+template<typename Func>
+EncodedJSValue atomicOperationWithArgs(ExecState* exec, const Func& func)
 {
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            return jsNumber(WTF::atomicExchangeAdd(ptr, toInt32(args[0])));
-        });
+    JSValue args[2 + Func::numExtraArgs];
+    for (unsigned i = 2 + Func::numExtraArgs; i--;)
+        args[i] = exec->argument(i);
+    return atomicOperationWithArgs(exec->vm(), exec, args, func);
 }
 
-EncodedJSValue JSC_HOST_CALL atomicsFuncAnd(ExecState* exec)
-{
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            return jsNumber(WTF::atomicExchangeAnd(ptr, toInt32(args[0])));
-        });
-}
+struct AddFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        return jsNumber(WTF::atomicExchangeAdd(ptr, toInt32(args[0])));
+    }
+};
 
-EncodedJSValue JSC_HOST_CALL atomicsFuncCompareExchange(ExecState* exec)
-{
-    return atomicOperationWithArgs<2>(
-        exec, [&] (auto* ptr, const double* args) {
-            typedef typename std::remove_pointer<decltype(ptr)>::type T;
-            T expected = static_cast<T>(toInt32(args[0]));
-            T newValue = static_cast<T>(toInt32(args[1]));
-            return jsNumber(WTF::atomicCompareExchangeStrong(ptr, expected, newValue));
-        });
-}
+struct AndFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        return jsNumber(WTF::atomicExchangeAnd(ptr, toInt32(args[0])));
+    }
+};
 
-EncodedJSValue JSC_HOST_CALL atomicsFuncExchange(ExecState* exec)
-{
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            typedef typename std::remove_pointer<decltype(ptr)>::type T;
-            return jsNumber(WTF::atomicExchange(ptr, static_cast<T>(toInt32(args[0]))));
-        });
-}
+struct CompareExchangeFunc {
+    static const unsigned numExtraArgs = 2;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        T expected = static_cast<T>(toInt32(args[0]));
+        T newValue = static_cast<T>(toInt32(args[1]));
+        return jsNumber(WTF::atomicCompareExchangeStrong(ptr, expected, newValue));
+    }
+};
 
-EncodedJSValue JSC_HOST_CALL atomicsFuncIsLockFree(ExecState* exec)
+struct ExchangeFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        return jsNumber(WTF::atomicExchange(ptr, static_cast<T>(toInt32(args[0]))));
+    }
+};
+
+struct LoadFunc {
+    static const unsigned numExtraArgs = 0;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double*) const
+    {
+        return jsNumber(WTF::atomicLoad(ptr));
+    }
+};
+
+struct OrFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        return jsNumber(WTF::atomicExchangeOr(ptr, toInt32(args[0])));
+    }
+};
+
+struct StoreFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        double valueAsInt = args[0];
+        T valueAsT = static_cast<T>(toInt32(valueAsInt));
+        WTF::atomicStore(ptr, valueAsT);
+        return jsNumber(valueAsInt);
+    }
+};
+
+struct SubFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        return jsNumber(WTF::atomicExchangeSub(ptr, toInt32(args[0])));
+    }
+};
+
+struct XorFunc {
+    static const unsigned numExtraArgs = 1;
+    
+    template<typename T>
+    JSValue operator()(T* ptr, const double* args) const
+    {
+        return jsNumber(WTF::atomicExchangeXor(ptr, toInt32(args[0])));
+    }
+};
+
+EncodedJSValue isLockFree(ExecState* exec, JSValue arg)
 {
     VM& vm = exec->vm();
     auto scope = DECLARE_THROW_SCOPE(vm);
 
-    int32_t size = exec->argument(0).toInt32(exec);
+    int32_t size = arg.toInt32(exec);
     RETURN_IF_EXCEPTION(scope, JSValue::encode(jsUndefined()));
     
     bool result;
@@ -246,40 +311,51 @@ EncodedJSValue JSC_HOST_CALL atomicsFuncIsLockFree(ExecState* exec)
     return JSValue::encode(jsBoolean(result));
 }
 
+} // anonymous namespace
+
+EncodedJSValue JSC_HOST_CALL atomicsFuncAdd(ExecState* exec)
+{
+    return atomicOperationWithArgs(exec, AddFunc());
+}
+
+EncodedJSValue JSC_HOST_CALL atomicsFuncAnd(ExecState* exec)
+{
+    return atomicOperationWithArgs(exec, AndFunc());
+}
+
+EncodedJSValue JSC_HOST_CALL atomicsFuncCompareExchange(ExecState* exec)
+{
+    return atomicOperationWithArgs(exec, CompareExchangeFunc());
+}
+
+EncodedJSValue JSC_HOST_CALL atomicsFuncExchange(ExecState* exec)
+{
+    return atomicOperationWithArgs(exec, ExchangeFunc());
+}
+
+EncodedJSValue JSC_HOST_CALL atomicsFuncIsLockFree(ExecState* exec)
+{
+    return isLockFree(exec, exec->argument(0));
+}
+
 EncodedJSValue JSC_HOST_CALL atomicsFuncLoad(ExecState* exec)
 {
-    return atomicOperationWithArgs<0>(
-        exec, [&] (auto* ptr, const double*) {
-            return jsNumber(WTF::atomicLoad(ptr));
-        });
+    return atomicOperationWithArgs(exec, LoadFunc());
 }
 
 EncodedJSValue JSC_HOST_CALL atomicsFuncOr(ExecState* exec)
 {
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            return jsNumber(WTF::atomicExchangeOr(ptr, toInt32(args[0])));
-        });
+    return atomicOperationWithArgs(exec, OrFunc());
 }
 
 EncodedJSValue JSC_HOST_CALL atomicsFuncStore(ExecState* exec)
 {
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            typedef typename std::remove_pointer<decltype(ptr)>::type T;
-            double valueAsInt = args[0];
-            T valueAsT = static_cast<T>(toInt32(valueAsInt));
-            WTF::atomicStore(ptr, valueAsT);
-            return jsNumber(valueAsInt);
-        });
+    return atomicOperationWithArgs(exec, StoreFunc());
 }
 
 EncodedJSValue JSC_HOST_CALL atomicsFuncSub(ExecState* exec)
 {
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            return jsNumber(WTF::atomicExchangeSub(ptr, toInt32(args[0])));
-        });
+    return atomicOperationWithArgs(exec, SubFunc());
 }
 
 EncodedJSValue JSC_HOST_CALL atomicsFuncWait(ExecState* exec)
@@ -298,7 +374,7 @@ EncodedJSValue JSC_HOST_CALL atomicsFuncWait(ExecState* exec)
         return JSValue::encode(jsUndefined());
     }
 
-    unsigned accessIndex = validatedAccessIndex(vm, exec, typedArray);
+    unsigned accessIndex = validatedAccessIndex(vm, exec, exec->argument(1), typedArray);
     RETURN_IF_EXCEPTION(scope, JSValue::encode(jsUndefined()));
     
     int32_t* ptr = typedArray->typedVector() + accessIndex;
@@ -368,7 +444,7 @@ EncodedJSValue JSC_HOST_CALL atomicsFuncWake(ExecState* exec)
         return JSValue::encode(jsUndefined());
     }
 
-    unsigned accessIndex = validatedAccessIndex(vm, exec, typedArray);
+    unsigned accessIndex = validatedAccessIndex(vm, exec, exec->argument(1), typedArray);
     RETURN_IF_EXCEPTION(scope, JSValue::encode(jsUndefined()));
     
     int32_t* ptr = typedArray->typedVector() + accessIndex;
@@ -386,10 +462,86 @@ EncodedJSValue JSC_HOST_CALL atomicsFuncWake(ExecState* exec)
 
 EncodedJSValue JSC_HOST_CALL atomicsFuncXor(ExecState* exec)
 {
-    return atomicOperationWithArgs<1>(
-        exec, [&] (auto* ptr, const double* args) {
-            return jsNumber(WTF::atomicExchangeXor(ptr, toInt32(args[0])));
-        });
+    return atomicOperationWithArgs(exec, XorFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsAdd(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, AddFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsAnd(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, AndFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsCompareExchange(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue expected, EncodedJSValue newValue)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(expected), JSValue::decode(newValue)};
+    return atomicOperationWithArgs(vm, exec, args, CompareExchangeFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsExchange(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, ExchangeFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsIsLockFree(ExecState* exec, EncodedJSValue size)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    return isLockFree(exec, JSValue::decode(size));
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsLoad(ExecState* exec, EncodedJSValue base, EncodedJSValue index)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index)};
+    return atomicOperationWithArgs(vm, exec, args, LoadFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsOr(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, OrFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsStore(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, StoreFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsSub(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, SubFunc());
+}
+
+EncodedJSValue JIT_OPERATION operationAtomicsXor(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    JSValue args[] = {JSValue::decode(base), JSValue::decode(index), JSValue::decode(operand)};
+    return atomicOperationWithArgs(vm, exec, args, XorFunc());
 }
 
 } // namespace JSC
index e8144ce..a05f82a 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2016 Apple Inc. All rights reserved.
+ * Copyright (C) 2016-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -46,5 +46,16 @@ protected:
     void finishCreation(VM&, JSGlobalObject*);
 };
 
+EncodedJSValue JIT_OPERATION operationAtomicsAdd(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+EncodedJSValue JIT_OPERATION operationAtomicsAnd(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+EncodedJSValue JIT_OPERATION operationAtomicsCompareExchange(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue expected, EncodedJSValue newValue);
+EncodedJSValue JIT_OPERATION operationAtomicsExchange(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+EncodedJSValue JIT_OPERATION operationAtomicsIsLockFree(ExecState* exec, EncodedJSValue size);
+EncodedJSValue JIT_OPERATION operationAtomicsLoad(ExecState* exec, EncodedJSValue base, EncodedJSValue index);
+EncodedJSValue JIT_OPERATION operationAtomicsOr(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+EncodedJSValue JIT_OPERATION operationAtomicsStore(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+EncodedJSValue JIT_OPERATION operationAtomicsSub(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+EncodedJSValue JIT_OPERATION operationAtomicsXor(ExecState* exec, EncodedJSValue base, EncodedJSValue index, EncodedJSValue operand);
+
 } // namespace JSC
 
index d6179a1..7da6db7 100644 (file)
@@ -1,3 +1,17 @@
+2017-04-20  Filip Pizlo  <fpizlo@apple.com>
+
+        Optimize SharedArrayBuffer in the DFG+FTL
+        https://bugs.webkit.org/show_bug.cgi?id=164108
+
+        Reviewed by Saam Barati.
+        
+        Made small changes as part of benchmarking the JS versions of these locks.
+
+        * benchmarks/LockSpeedTest.cpp:
+        * benchmarks/ToyLocks.h:
+        * wtf/Range.h:
+        (WTF::Range::dump):
+
 2017-04-19  Youenn Fablet  <youenn@apple.com>
 
         [Win] Activate streams API by default
index 82d39f8..0d6235d 100644 (file)
@@ -53,7 +53,7 @@ double secondsPerTest;
     
 NO_RETURN void usage()
 {
-    printf("Usage: LockSpeedTest yieldspinlock|pausespinlock|wordlock|lock|barginglock|bargingwordlock|thunderlock|thunderwordlock|cascadelock|cascadewordlockhandofflock|mutex|all <num thread groups> <num threads per group> <work per critical section> <work between critical sections> <spin limit> <seconds per test>\n");
+    printf("Usage: LockSpeedTest yieldspinlock|pausespinlock|wordlock|lock|barginglock|bargingwordlock|thunderlock|thunderwordlock|cascadelock|cascadewordlock|handofflock|mutex|all <num thread groups> <num threads per group> <work per critical section> <work between critical sections> <spin limit> <seconds per test>\n");
     exit(1);
 }
 
@@ -77,7 +77,7 @@ struct Benchmark {
     {
         std::unique_ptr<WithPadding<LockType>[]> locks = std::make_unique<WithPadding<LockType>[]>(numThreadGroups);
         std::unique_ptr<WithPadding<double>[]> words = std::make_unique<WithPadding<double>[]>(numThreadGroups);
-        std::unique_ptr<RefPtr<Thread>[]> threads = std::make_unique<Refptr<Thread>[]>(numThreadGroups * numThreadsPerGroup);
+        std::unique_ptr<RefPtr<Thread>[]> threads = std::make_unique<RefPtr<Thread>[]>(numThreadGroups * numThreadsPerGroup);
 
         volatile bool keepGoing = true;
 
@@ -100,13 +100,11 @@ struct Benchmark {
                             locks[threadGroupIndex].value.lock();
                             for (unsigned j = workPerCriticalSection; j--;) {
                                 words[threadGroupIndex].value += value;
-                                words[threadGroupIndex].value *= 1.01;
                                 value = words[threadGroupIndex].value;
                             }
                             locks[threadGroupIndex].value.unlock();
                             for (unsigned j = workBetweenCriticalSections; j--;) {
                                 localWord += value;
-                                localWord *= 1.01;
                                 value = localWord;
                             }
                             myNumIterations++;
index 43ad796..3869b0d 100644 (file)
@@ -432,7 +432,7 @@ private:
             
             if (m_state.compareExchangeWeak(state, state + parkedCountUnit)) {
                 bool result = ParkingLot::compareAndPark(&m_state, state + parkedCountUnit).wasUnparked;
-                m_state.exchangeAndAdd(-parkedCountUnit);
+                m_state.exchangeAdd(-parkedCountUnit);
                 if (result)
                     return;
             }
index 272a50f..deba7e2 100644 (file)
@@ -119,7 +119,7 @@ public:
             out.print("Top");
             return;
         }
-        if (m_begin == m_end + 1) {
+        if (m_begin + 1 == m_end) {
             out.print(m_begin);
             return;
         }