DFG and FTL should be able to use DirectCall ICs when they proved the callee or its...
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 18 Oct 2016 18:30:05 +0000 (18:30 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 18 Oct 2016 18:30:05 +0000 (18:30 +0000)
https://bugs.webkit.org/show_bug.cgi?id=163371

Reviewed by Geoffrey Garen and Saam Barati.

JSTests:

Add microbenchmarks for all of the cases that this patch optimizes.

* microbenchmarks/direct-call-arity-mismatch.js: Added.
(foo):
(bar):
* microbenchmarks/direct-call.js: Added.
(foo):
(bar):
* microbenchmarks/direct-construct-arity-mismatch.js: Added.
(Foo):
(bar):
* microbenchmarks/direct-construct.js: Added.
(Foo):
(bar):
* microbenchmarks/direct-tail-call-arity-mismatch.js: Added.
(foo):
(bar):
* microbenchmarks/direct-tail-call-inlined-caller-arity-mismatch.js: Added.
(foo):
(bar):
(baz):
* microbenchmarks/direct-tail-call-inlined-caller.js: Added.
(foo):
(bar):
(baz):
* microbenchmarks/direct-tail-call.js: Added.
(foo):
(bar):

Source/JavaScriptCore:

This adds a new kind of call inline cache for when the DFG can prove what the callee
executable is. In those cases, we can skip some of the things that the traditional call IC
would do:

- No need to check who the callee is.
- No need to do arity checks.

This case isn't as simple as just emitting a call instruction since the callee may not be
compiled at the time that the caller is compiled. So, we need lazy resolution. Also, the
callee may be jettisoned independently of the caller, so we need to be able to revert the
call to an unlinked state. This means that we need almost all of the things that
CallLinkInfo has. CallLinkInfo already knows about different kinds of calls. This patch
teaches it about new "Direct" call types.

The direct non-tail call IC looks like this:

        set up arguments
    FastPath:
        call _SlowPath
        lea -FrameSize(%rbp), %rsp

    SlowPath:
        pop
        call operationLinkDirectCall
        check exception
        jmp FastPath

The job of operationLinkDirectCall is to link the fast path's call entrypoint of the callee.
This means that in steady state, a call is just that: a call. There are no extra branches or
checks.

The direct tail call IC is a bit more complicated because the act of setting up arguments
destroys our frame, which would prevent us from being able to throw an exception if we
failed to compile the callee. So, direct tail call ICs look like this:

        jmp _SlowPath
    FastPath:
        set up arguments
        jmp 0 // patch to jump to callee

    SlowPath:
        silent spill
        call operationLinkDirectCall
        silent fill
        check exception
        jmp FastPath

The jmp to the slow path is patched to be a fall-through jmp when we link the call.

Direct calls mean less code at call sites, fewer checks on the steady state call fast path,
and no need for arity fixup. This looks like a slight speed-up (~0.8%) on both Octane and
AsmBench.

* assembler/ARM64Assembler.h:
(JSC::ARM64Assembler::relinkJumpToNop):
* assembler/ARMv7Assembler.h:
(JSC::ARMv7Assembler::relinkJumpToNop):
(JSC::ARMv7Assembler::relinkJump): Deleted.
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::repatchJumpToNop):
(JSC::AbstractMacroAssembler::repatchJump): Deleted.
* assembler/X86Assembler.h:
(JSC::X86Assembler::relinkJumpToNop):
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::CallLinkInfo):
(JSC::CallLinkInfo::callReturnLocation):
(JSC::CallLinkInfo::patchableJump):
(JSC::CallLinkInfo::hotPathBegin):
(JSC::CallLinkInfo::slowPathStart):
(JSC::CallLinkInfo::setCallee):
(JSC::CallLinkInfo::clearCallee):
(JSC::CallLinkInfo::callee):
(JSC::CallLinkInfo::setCodeBlock):
(JSC::CallLinkInfo::clearCodeBlock):
(JSC::CallLinkInfo::codeBlock):
(JSC::CallLinkInfo::setLastSeenCallee):
(JSC::CallLinkInfo::clearLastSeenCallee):
(JSC::CallLinkInfo::lastSeenCallee):
(JSC::CallLinkInfo::haveLastSeenCallee):
(JSC::CallLinkInfo::setExecutableDuringCompilation):
(JSC::CallLinkInfo::executable):
(JSC::CallLinkInfo::setMaxNumArguments):
(JSC::CallLinkInfo::visitWeak):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::specializationKindFor):
(JSC::CallLinkInfo::callModeFor):
(JSC::CallLinkInfo::isDirect):
(JSC::CallLinkInfo::nearCallMode):
(JSC::CallLinkInfo::isLinked):
(JSC::CallLinkInfo::setCallLocations):
(JSC::CallLinkInfo::addressOfMaxNumArguments):
(JSC::CallLinkInfo::maxNumArguments):
(JSC::CallLinkInfo::isTailCall): Deleted.
(JSC::CallLinkInfo::setUpCallFromFTL): Deleted.
(JSC::CallLinkInfo::callReturnLocation): Deleted.
(JSC::CallLinkInfo::hotPathBegin): Deleted.
(JSC::CallLinkInfo::callee): Deleted.
(JSC::CallLinkInfo::setLastSeenCallee): Deleted.
(JSC::CallLinkInfo::clearLastSeenCallee): Deleted.
(JSC::CallLinkInfo::lastSeenCallee): Deleted.
(JSC::CallLinkInfo::haveLastSeenCallee): Deleted.
* bytecode/CallLinkStatus.cpp:
(JSC::CallLinkStatus::computeDFGStatuses):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
* bytecode/UnlinkedFunctionExecutable.h:
* bytecode/ValueRecovery.h:
(JSC::ValueRecovery::forEachReg):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGBasicBlock.h:
(JSC::DFG::BasicBlock::findTerminal):
* dfg/DFGByteCodeParser.cpp:
(JSC::DFG::ByteCodeParser::addCallWithoutSettingResult):
(JSC::DFG::ByteCodeParser::handleCall):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGGraph.cpp:
(JSC::DFG::Graph::parameterSlotsForArgCount):
* dfg/DFGGraph.h:
* dfg/DFGInPlaceAbstractState.cpp:
(JSC::DFG::InPlaceAbstractState::mergeToSuccessors):
* dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::link):
* dfg/DFGJITCompiler.h:
(JSC::DFG::JITCompiler::addJSDirectCall):
(JSC::DFG::JITCompiler::addJSDirectTailCall):
(JSC::DFG::JITCompiler::JSCallRecord::JSCallRecord):
(JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
(JSC::DFG::JITCompiler::JSDirectTailCallRecord::JSDirectTailCallRecord):
(JSC::DFG::JITCompiler::currentJSCallIndex): Deleted.
* dfg/DFGNode.cpp:
(JSC::DFG::Node::convertToDirectCall):
* dfg/DFGNode.h:
(JSC::DFG::Node::isTerminal):
(JSC::DFG::Node::hasHeapPrediction):
(JSC::DFG::Node::hasCellOperand):
* dfg/DFGNodeType.h:
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::callOperation):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
* interpreter/Interpreter.cpp:
(JSC::Interpreter::execute):
(JSC::Interpreter::executeCall):
(JSC::Interpreter::executeConstruct):
(JSC::Interpreter::prepareForRepeatCall):
* jit/JIT.cpp:
(JSC::JIT::link):
* jit/JITCall.cpp:
(JSC::JIT::compileSetupVarargsFrame):
* jit/JITCall32_64.cpp:
(JSC::JIT::compileSetupVarargsFrame):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/Repatch.cpp:
(JSC::linkDirectFor):
(JSC::revertCall):
* jit/Repatch.h:
* llint/LLIntSlowPaths.cpp:
(JSC::LLInt::setUpCall):
* runtime/Executable.cpp:
(JSC::ScriptExecutable::prepareForExecutionImpl):
* runtime/Executable.h:
(JSC::ScriptExecutable::prepareForExecution):
* runtime/Options.h:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@207475 268f45cc-cd09-0410-ab3c-d52691b4dbfc

54 files changed:
JSTests/ChangeLog
JSTests/microbenchmarks/direct-call-arity-mismatch.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-call.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-construct-arity-mismatch.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-construct.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-tail-call-arity-mismatch.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-tail-call-inlined-caller-arity-mismatch.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-tail-call-inlined-caller.js [new file with mode: 0644]
JSTests/microbenchmarks/direct-tail-call.js [new file with mode: 0644]
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/assembler/ARM64Assembler.h
Source/JavaScriptCore/assembler/ARMv7Assembler.h
Source/JavaScriptCore/assembler/AbstractMacroAssembler.h
Source/JavaScriptCore/assembler/X86Assembler.h
Source/JavaScriptCore/bytecode/CallLinkInfo.cpp
Source/JavaScriptCore/bytecode/CallLinkInfo.h
Source/JavaScriptCore/bytecode/CallLinkStatus.cpp
Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp
Source/JavaScriptCore/bytecode/UnlinkedFunctionExecutable.h
Source/JavaScriptCore/bytecode/ValueRecovery.h
Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h
Source/JavaScriptCore/dfg/DFGBasicBlock.h
Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp
Source/JavaScriptCore/dfg/DFGClobberize.h
Source/JavaScriptCore/dfg/DFGDoesGC.cpp
Source/JavaScriptCore/dfg/DFGFixupPhase.cpp
Source/JavaScriptCore/dfg/DFGGraph.cpp
Source/JavaScriptCore/dfg/DFGGraph.h
Source/JavaScriptCore/dfg/DFGInPlaceAbstractState.cpp
Source/JavaScriptCore/dfg/DFGJITCompiler.cpp
Source/JavaScriptCore/dfg/DFGJITCompiler.h
Source/JavaScriptCore/dfg/DFGNode.cpp
Source/JavaScriptCore/dfg/DFGNode.h
Source/JavaScriptCore/dfg/DFGNodeType.h
Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp
Source/JavaScriptCore/dfg/DFGSafeToExecute.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp
Source/JavaScriptCore/dfg/DFGStrengthReductionPhase.cpp
Source/JavaScriptCore/ftl/FTLCapabilities.cpp
Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
Source/JavaScriptCore/interpreter/Interpreter.cpp
Source/JavaScriptCore/jit/JIT.cpp
Source/JavaScriptCore/jit/JITCall.cpp
Source/JavaScriptCore/jit/JITCall32_64.cpp
Source/JavaScriptCore/jit/JITOperations.cpp
Source/JavaScriptCore/jit/JITOperations.h
Source/JavaScriptCore/jit/Repatch.cpp
Source/JavaScriptCore/jit/Repatch.h
Source/JavaScriptCore/llint/LLIntSlowPaths.cpp
Source/JavaScriptCore/runtime/Executable.cpp
Source/JavaScriptCore/runtime/Executable.h
Source/JavaScriptCore/runtime/Options.h

index ca831b3..83933e8 100644 (file)
@@ -1,3 +1,39 @@
+2016-10-15  Filip Pizlo  <fpizlo@apple.com>
+
+        DFG and FTL should be able to use DirectCall ICs when they proved the callee or its executable
+        https://bugs.webkit.org/show_bug.cgi?id=163371
+
+        Reviewed by Geoffrey Garen and Saam Barati.
+        
+        Add microbenchmarks for all of the cases that this patch optimizes.
+
+        * microbenchmarks/direct-call-arity-mismatch.js: Added.
+        (foo):
+        (bar):
+        * microbenchmarks/direct-call.js: Added.
+        (foo):
+        (bar):
+        * microbenchmarks/direct-construct-arity-mismatch.js: Added.
+        (Foo):
+        (bar):
+        * microbenchmarks/direct-construct.js: Added.
+        (Foo):
+        (bar):
+        * microbenchmarks/direct-tail-call-arity-mismatch.js: Added.
+        (foo):
+        (bar):
+        * microbenchmarks/direct-tail-call-inlined-caller-arity-mismatch.js: Added.
+        (foo):
+        (bar):
+        (baz):
+        * microbenchmarks/direct-tail-call-inlined-caller.js: Added.
+        (foo):
+        (bar):
+        (baz):
+        * microbenchmarks/direct-tail-call.js: Added.
+        (foo):
+        (bar):
+
 2016-10-18  Caitlin Potter  <caitp@igalia.com>
 
         [JSC] ES6 Method functions should not have prototype
diff --git a/JSTests/microbenchmarks/direct-call-arity-mismatch.js b/JSTests/microbenchmarks/direct-call-arity-mismatch.js
new file mode 100644 (file)
index 0000000..6a24500
--- /dev/null
@@ -0,0 +1,21 @@
+function foo(a, b, c, d, e, f, g)
+{
+    return 42;
+}
+
+function bar()
+{
+    return foo();
+}
+
+noInline(foo);
+noInline(bar);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = bar();
+        if (result != 42)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-call.js b/JSTests/microbenchmarks/direct-call.js
new file mode 100644 (file)
index 0000000..4bf0338
--- /dev/null
@@ -0,0 +1,21 @@
+function foo()
+{
+    return 42;
+}
+
+function bar()
+{
+    return foo();
+}
+
+noInline(foo);
+noInline(bar);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = bar();
+        if (result != 42)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-construct-arity-mismatch.js b/JSTests/microbenchmarks/direct-construct-arity-mismatch.js
new file mode 100644 (file)
index 0000000..d29e60c
--- /dev/null
@@ -0,0 +1,21 @@
+function Foo(a, b, c, d, e, f, g)
+{
+    this.f = 42;
+}
+
+function bar()
+{
+    return new Foo();
+}
+
+noInline(Foo);
+noInline(bar);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = bar();
+        if (result.f != 42)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-construct.js b/JSTests/microbenchmarks/direct-construct.js
new file mode 100644 (file)
index 0000000..e426861
--- /dev/null
@@ -0,0 +1,21 @@
+function Foo()
+{
+    this.f = 42;
+}
+
+function bar()
+{
+    return new Foo();
+}
+
+noInline(Foo);
+noInline(bar);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = bar();
+        if (result.f != 42)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-tail-call-arity-mismatch.js b/JSTests/microbenchmarks/direct-tail-call-arity-mismatch.js
new file mode 100644 (file)
index 0000000..3faedbc
--- /dev/null
@@ -0,0 +1,23 @@
+"use strict";
+
+function foo(a, b, c, d, e, f, g)
+{
+    return 42;
+}
+
+function bar()
+{
+    return foo();
+}
+
+noInline(foo);
+noInline(bar);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = bar();
+        if (result != 42)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-tail-call-inlined-caller-arity-mismatch.js b/JSTests/microbenchmarks/direct-tail-call-inlined-caller-arity-mismatch.js
new file mode 100644 (file)
index 0000000..00bc01d
--- /dev/null
@@ -0,0 +1,28 @@
+"use strict";
+
+function foo(a, b, c, d, e, f, g)
+{
+    return 42;
+}
+
+function bar()
+{
+    return foo();
+}
+
+function baz()
+{
+    return bar() + 1;
+}
+
+noInline(foo);
+noInline(baz);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = baz();
+        if (result != 43)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-tail-call-inlined-caller.js b/JSTests/microbenchmarks/direct-tail-call-inlined-caller.js
new file mode 100644 (file)
index 0000000..0af500b
--- /dev/null
@@ -0,0 +1,28 @@
+"use strict";
+
+function foo()
+{
+    return 42;
+}
+
+function bar()
+{
+    return foo();
+}
+
+function baz()
+{
+    return bar() + 1;
+}
+
+noInline(foo);
+noInline(baz);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = baz();
+        if (result != 43)
+            throw "Error: bad result: " + result;
+    }
+})();
+
diff --git a/JSTests/microbenchmarks/direct-tail-call.js b/JSTests/microbenchmarks/direct-tail-call.js
new file mode 100644 (file)
index 0000000..d3c3086
--- /dev/null
@@ -0,0 +1,23 @@
+"use strict";
+
+function foo()
+{
+    return 42;
+}
+
+function bar()
+{
+    return foo();
+}
+
+noInline(foo);
+noInline(bar);
+
+(function() {
+    for (var i = 0; i < 10000000; ++i) {
+        var result = bar();
+        if (result != 42)
+            throw "Error: bad result: " + result;
+    }
+})();
+
index 6052446..40a8931 100644 (file)
@@ -1,3 +1,195 @@
+2016-10-15  Filip Pizlo  <fpizlo@apple.com>
+
+        DFG and FTL should be able to use DirectCall ICs when they proved the callee or its executable
+        https://bugs.webkit.org/show_bug.cgi?id=163371
+
+        Reviewed by Geoffrey Garen and Saam Barati.
+        
+        This adds a new kind of call inline cache for when the DFG can prove what the callee
+        executable is. In those cases, we can skip some of the things that the traditional call IC
+        would do:
+        
+        - No need to check who the callee is.
+        - No need to do arity checks.
+        
+        This case isn't as simple as just emitting a call instruction since the callee may not be
+        compiled at the time that the caller is compiled. So, we need lazy resolution. Also, the
+        callee may be jettisoned independently of the caller, so we need to be able to revert the
+        call to an unlinked state. This means that we need almost all of the things that
+        CallLinkInfo has. CallLinkInfo already knows about different kinds of calls. This patch
+        teaches it about new "Direct" call types.
+        
+        The direct non-tail call IC looks like this:
+        
+                set up arguments
+            FastPath:
+                call _SlowPath
+                lea -FrameSize(%rbp), %rsp
+            
+            SlowPath:
+                pop
+                call operationLinkDirectCall
+                check exception
+                jmp FastPath
+        
+        The job of operationLinkDirectCall is to link the fast path's call entrypoint of the callee.
+        This means that in steady state, a call is just that: a call. There are no extra branches or
+        checks.
+        
+        The direct tail call IC is a bit more complicated because the act of setting up arguments
+        destroys our frame, which would prevent us from being able to throw an exception if we
+        failed to compile the callee. So, direct tail call ICs look like this:
+        
+                jmp _SlowPath
+            FastPath:
+                set up arguments
+                jmp 0 // patch to jump to callee
+            
+            SlowPath:
+                silent spill
+                call operationLinkDirectCall
+                silent fill
+                check exception
+                jmp FastPath
+        
+        The jmp to the slow path is patched to be a fall-through jmp when we link the call.
+        
+        Direct calls mean less code at call sites, fewer checks on the steady state call fast path,
+        and no need for arity fixup. This looks like a slight speed-up (~0.8%) on both Octane and
+        AsmBench.
+
+        * assembler/ARM64Assembler.h:
+        (JSC::ARM64Assembler::relinkJumpToNop):
+        * assembler/ARMv7Assembler.h:
+        (JSC::ARMv7Assembler::relinkJumpToNop):
+        (JSC::ARMv7Assembler::relinkJump): Deleted.
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::AbstractMacroAssembler::repatchJumpToNop):
+        (JSC::AbstractMacroAssembler::repatchJump): Deleted.
+        * assembler/X86Assembler.h:
+        (JSC::X86Assembler::relinkJumpToNop):
+        * bytecode/CallLinkInfo.cpp:
+        (JSC::CallLinkInfo::CallLinkInfo):
+        (JSC::CallLinkInfo::callReturnLocation):
+        (JSC::CallLinkInfo::patchableJump):
+        (JSC::CallLinkInfo::hotPathBegin):
+        (JSC::CallLinkInfo::slowPathStart):
+        (JSC::CallLinkInfo::setCallee):
+        (JSC::CallLinkInfo::clearCallee):
+        (JSC::CallLinkInfo::callee):
+        (JSC::CallLinkInfo::setCodeBlock):
+        (JSC::CallLinkInfo::clearCodeBlock):
+        (JSC::CallLinkInfo::codeBlock):
+        (JSC::CallLinkInfo::setLastSeenCallee):
+        (JSC::CallLinkInfo::clearLastSeenCallee):
+        (JSC::CallLinkInfo::lastSeenCallee):
+        (JSC::CallLinkInfo::haveLastSeenCallee):
+        (JSC::CallLinkInfo::setExecutableDuringCompilation):
+        (JSC::CallLinkInfo::executable):
+        (JSC::CallLinkInfo::setMaxNumArguments):
+        (JSC::CallLinkInfo::visitWeak):
+        * bytecode/CallLinkInfo.h:
+        (JSC::CallLinkInfo::specializationKindFor):
+        (JSC::CallLinkInfo::callModeFor):
+        (JSC::CallLinkInfo::isDirect):
+        (JSC::CallLinkInfo::nearCallMode):
+        (JSC::CallLinkInfo::isLinked):
+        (JSC::CallLinkInfo::setCallLocations):
+        (JSC::CallLinkInfo::addressOfMaxNumArguments):
+        (JSC::CallLinkInfo::maxNumArguments):
+        (JSC::CallLinkInfo::isTailCall): Deleted.
+        (JSC::CallLinkInfo::setUpCallFromFTL): Deleted.
+        (JSC::CallLinkInfo::callReturnLocation): Deleted.
+        (JSC::CallLinkInfo::hotPathBegin): Deleted.
+        (JSC::CallLinkInfo::callee): Deleted.
+        (JSC::CallLinkInfo::setLastSeenCallee): Deleted.
+        (JSC::CallLinkInfo::clearLastSeenCallee): Deleted.
+        (JSC::CallLinkInfo::lastSeenCallee): Deleted.
+        (JSC::CallLinkInfo::haveLastSeenCallee): Deleted.
+        * bytecode/CallLinkStatus.cpp:
+        (JSC::CallLinkStatus::computeDFGStatuses):
+        * bytecode/PolymorphicAccess.cpp:
+        (JSC::AccessCase::generateImpl):
+        * bytecode/UnlinkedFunctionExecutable.h:
+        * bytecode/ValueRecovery.h:
+        (JSC::ValueRecovery::forEachReg):
+        * dfg/DFGAbstractInterpreterInlines.h:
+        (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
+        * dfg/DFGBasicBlock.h:
+        (JSC::DFG::BasicBlock::findTerminal):
+        * dfg/DFGByteCodeParser.cpp:
+        (JSC::DFG::ByteCodeParser::addCallWithoutSettingResult):
+        (JSC::DFG::ByteCodeParser::handleCall):
+        * dfg/DFGClobberize.h:
+        (JSC::DFG::clobberize):
+        * dfg/DFGDoesGC.cpp:
+        (JSC::DFG::doesGC):
+        * dfg/DFGFixupPhase.cpp:
+        (JSC::DFG::FixupPhase::fixupNode):
+        * dfg/DFGGraph.cpp:
+        (JSC::DFG::Graph::parameterSlotsForArgCount):
+        * dfg/DFGGraph.h:
+        * dfg/DFGInPlaceAbstractState.cpp:
+        (JSC::DFG::InPlaceAbstractState::mergeToSuccessors):
+        * dfg/DFGJITCompiler.cpp:
+        (JSC::DFG::JITCompiler::link):
+        * dfg/DFGJITCompiler.h:
+        (JSC::DFG::JITCompiler::addJSDirectCall):
+        (JSC::DFG::JITCompiler::addJSDirectTailCall):
+        (JSC::DFG::JITCompiler::JSCallRecord::JSCallRecord):
+        (JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
+        (JSC::DFG::JITCompiler::JSDirectTailCallRecord::JSDirectTailCallRecord):
+        (JSC::DFG::JITCompiler::currentJSCallIndex): Deleted.
+        * dfg/DFGNode.cpp:
+        (JSC::DFG::Node::convertToDirectCall):
+        * dfg/DFGNode.h:
+        (JSC::DFG::Node::isTerminal):
+        (JSC::DFG::Node::hasHeapPrediction):
+        (JSC::DFG::Node::hasCellOperand):
+        * dfg/DFGNodeType.h:
+        * dfg/DFGPredictionPropagationPhase.cpp:
+        * dfg/DFGSafeToExecute.h:
+        (JSC::DFG::safeToExecute):
+        * dfg/DFGSpeculativeJIT.h:
+        (JSC::DFG::SpeculativeJIT::callOperation):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGStrengthReductionPhase.cpp:
+        (JSC::DFG::StrengthReductionPhase::handleNode):
+        * ftl/FTLCapabilities.cpp:
+        (JSC::FTL::canCompile):
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileNode):
+        (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
+        (JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
+        (JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
+        (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
+        * interpreter/Interpreter.cpp:
+        (JSC::Interpreter::execute):
+        (JSC::Interpreter::executeCall):
+        (JSC::Interpreter::executeConstruct):
+        (JSC::Interpreter::prepareForRepeatCall):
+        * jit/JIT.cpp:
+        (JSC::JIT::link):
+        * jit/JITCall.cpp:
+        (JSC::JIT::compileSetupVarargsFrame):
+        * jit/JITCall32_64.cpp:
+        (JSC::JIT::compileSetupVarargsFrame):
+        * jit/JITOperations.cpp:
+        * jit/JITOperations.h:
+        * jit/Repatch.cpp:
+        (JSC::linkDirectFor):
+        (JSC::revertCall):
+        * jit/Repatch.h:
+        * llint/LLIntSlowPaths.cpp:
+        (JSC::LLInt::setUpCall):
+        * runtime/Executable.cpp:
+        (JSC::ScriptExecutable::prepareForExecutionImpl):
+        * runtime/Executable.h:
+        (JSC::ScriptExecutable::prepareForExecution):
+        * runtime/Options.h:
+
 2016-10-18  Yusuke Suzuki  <utatane.tea@gmail.com>
 
         [DOMJIT] Not emit exception case if it is not necessary
index 488ae84..1eb2018 100644 (file)
@@ -2679,6 +2679,11 @@ public:
         cacheFlush(from, sizeof(int));
     }
     
+    static void relinkJumpToNop(void* from)
+    {
+        relinkJump(from, static_cast<char*>(from) + 4);
+    }
+    
     static void relinkCall(void* from, void* to)
     {
         relinkJumpOrCall<true>(reinterpret_cast<int*>(from) - 1, reinterpret_cast<const int*>(from) - 1, to);
index 496c201..b6db616 100644 (file)
@@ -2256,6 +2256,11 @@ public:
 
         cacheFlush(reinterpret_cast<uint16_t*>(from) - 5, 5 * sizeof(uint16_t));
     }
+
+    static void relinkJumpToNop(void* from)
+    {
+        relinkJump(from, from);
+    }
     
     static void relinkCall(void* from, void* to)
     {
index 192db17..6f55d7f 100644 (file)
@@ -932,6 +932,11 @@ public:
     {
         AssemblerType::relinkJump(jump.dataLocation(), destination.dataLocation());
     }
+    
+    static void repatchJumpToNop(CodeLocationJump jump)
+    {
+        AssemblerType::relinkJumpToNop(jump.dataLocation());
+    }
 
     static void repatchNearCall(CodeLocationNearCall nearCall, CodeLocationLabel destination)
     {
index 0f6d7b5..73e3e99 100644 (file)
@@ -2778,6 +2778,11 @@ public:
         setRel32(from, to);
     }
     
+    static void relinkJumpToNop(void* from)
+    {
+        setInt32(from, 0);
+    }
+    
     static void relinkCall(void* from, void* to)
     {
         setRel32(from, to);
index b190e23..c6c3738 100644 (file)
@@ -59,7 +59,9 @@ CallLinkInfo::CallLinkInfo()
     , m_hasSeenClosure(false)
     , m_clearedByGC(false)
     , m_allowStubs(true)
+    , m_isLinked(false)
     , m_callType(None)
+    , m_calleeGPR(255)
     , m_maxNumArguments(0)
     , m_slowPathCount(0)
 {
@@ -84,30 +86,123 @@ void CallLinkInfo::clearStub()
 
 void CallLinkInfo::unlink(VM& vm)
 {
-    if (!isLinked()) {
-        // We could be called even if we're not linked anymore because of how polymorphic calls
-        // work. Each callsite within the polymorphic call stub may separately ask us to unlink().
-        RELEASE_ASSERT(!isOnList());
-        return;
-    }
-    
-    unlinkFor(vm, *this);
+    // We could be called even if we're not linked anymore because of how polymorphic calls
+    // work. Each callsite within the polymorphic call stub may separately ask us to unlink().
+    if (isLinked())
+        unlinkFor(vm, *this);
 
-    // It will be on a list if the callee has a code block.
-    if (isOnList())
-        remove();
+    // Either we were unlinked, in which case we should not have been on any list, or we unlinked
+    // ourselves so that we're not on any list anymore.
+    RELEASE_ASSERT(!isOnList());
+}
+
+CodeLocationNearCall CallLinkInfo::callReturnLocation()
+{
+    RELEASE_ASSERT(!isDirect());
+    return CodeLocationNearCall(m_callReturnLocationOrPatchableJump, Regular);
+}
+
+CodeLocationJump CallLinkInfo::patchableJump()
+{
+    RELEASE_ASSERT(callType() == DirectTailCall);
+    return CodeLocationJump(m_callReturnLocationOrPatchableJump);
+}
+
+CodeLocationDataLabelPtr CallLinkInfo::hotPathBegin()
+{
+    RELEASE_ASSERT(!isDirect());
+    return CodeLocationDataLabelPtr(m_hotPathBeginOrSlowPathStart);
+}
+
+CodeLocationLabel CallLinkInfo::slowPathStart()
+{
+    RELEASE_ASSERT(isDirect());
+    return m_hotPathBeginOrSlowPathStart;
 }
 
 void CallLinkInfo::setCallee(VM& vm, JSCell* owner, JSFunction* callee)
 {
-    MacroAssembler::repatchPointer(m_hotPathBegin, callee);
-    m_callee.set(vm, owner, callee);
+    RELEASE_ASSERT(!isDirect());
+    MacroAssembler::repatchPointer(hotPathBegin(), callee);
+    m_calleeOrCodeBlock.set(vm, owner, callee);
+    m_isLinked = true;
 }
 
 void CallLinkInfo::clearCallee()
 {
-    MacroAssembler::repatchPointer(m_hotPathBegin, nullptr);
-    m_callee.clear();
+    RELEASE_ASSERT(!isDirect());
+    MacroAssembler::repatchPointer(hotPathBegin(), nullptr);
+    m_calleeOrCodeBlock.clear();
+    m_isLinked = false;
+}
+
+JSFunction* CallLinkInfo::callee()
+{
+    RELEASE_ASSERT(!isDirect());
+    return jsCast<JSFunction*>(m_calleeOrCodeBlock.get());
+}
+
+void CallLinkInfo::setCodeBlock(VM& vm, JSCell* owner, FunctionCodeBlock* codeBlock)
+{
+    RELEASE_ASSERT(isDirect());
+    m_calleeOrCodeBlock.setMayBeNull(vm, owner, codeBlock);
+    m_isLinked = true;
+}
+
+void CallLinkInfo::clearCodeBlock()
+{
+    RELEASE_ASSERT(isDirect());
+    m_calleeOrCodeBlock.clear();
+    m_isLinked = false;
+}
+
+FunctionCodeBlock* CallLinkInfo::codeBlock()
+{
+    RELEASE_ASSERT(isDirect());
+    return jsCast<FunctionCodeBlock*>(m_calleeOrCodeBlock.get());
+}
+
+void CallLinkInfo::setLastSeenCallee(VM& vm, const JSCell* owner, JSFunction* callee)
+{
+    RELEASE_ASSERT(!isDirect());
+    m_lastSeenCalleeOrExecutable.set(vm, owner, callee);
+}
+
+void CallLinkInfo::clearLastSeenCallee()
+{
+    RELEASE_ASSERT(!isDirect());
+    m_lastSeenCalleeOrExecutable.clear();
+}
+
+JSFunction* CallLinkInfo::lastSeenCallee()
+{
+    RELEASE_ASSERT(!isDirect());
+    return jsCast<JSFunction*>(m_lastSeenCalleeOrExecutable.get());
+}
+
+bool CallLinkInfo::haveLastSeenCallee()
+{
+    RELEASE_ASSERT(!isDirect());
+    return !!m_lastSeenCalleeOrExecutable;
+}
+
+void CallLinkInfo::setExecutableDuringCompilation(ExecutableBase* executable)
+{
+    RELEASE_ASSERT(isDirect());
+    m_lastSeenCalleeOrExecutable.setWithoutWriteBarrier(executable);
+}
+
+ExecutableBase* CallLinkInfo::executable()
+{
+    RELEASE_ASSERT(isDirect());
+    return jsCast<ExecutableBase*>(m_lastSeenCalleeOrExecutable.get());
+}
+
+void CallLinkInfo::setMaxNumArguments(unsigned value)
+{
+    RELEASE_ASSERT(isDirect());
+    RELEASE_ASSERT(value);
+    m_maxNumArguments = value;
 }
 
 void CallLinkInfo::visitWeak(VM& vm)
@@ -131,19 +226,37 @@ void CallLinkInfo::visitWeak(VM& vm)
                 unlink(vm);
                 m_clearedByGC = true;
             }
-        } else if (!Heap::isMarked(m_callee.get())) {
+        } else if (!Heap::isMarked(m_calleeOrCodeBlock.get())) {
+            if (isDirect()) {
+                if (Options::verboseOSR()) {
+                    dataLog(
+                        "Clearing call to ", RawPointer(codeBlock()), " (",
+                        pointerDump(codeBlock()), ").\n");
+                }
+            } else {
+                if (Options::verboseOSR()) {
+                    dataLog(
+                        "Clearing call to ",
+                        RawPointer(callee()), " (",
+                        callee()->executable()->hashFor(specializationKind()),
+                        ").\n");
+                }
+                handleSpecificCallee(callee());
+            }
+            unlink(vm);
+        } else if (isDirect() && !Heap::isMarked(m_lastSeenCalleeOrExecutable.get())) {
             if (Options::verboseOSR()) {
                 dataLog(
-                    "Clearing call to ",
-                    RawPointer(m_callee.get()), " (",
-                    m_callee.get()->executable()->hashFor(specializationKind()),
-                    ").\n");
+                    "Clearing call to ", RawPointer(executable()),
+                    " because the executable is dead.\n");
             }
-            handleSpecificCallee(m_callee.get());
             unlink(vm);
+            // We should only get here once the owning CodeBlock is dying, since the executable must
+            // already be in the owner's weak references.
+            m_lastSeenCalleeOrExecutable.clear();
         }
     }
-    if (haveLastSeenCallee() && !Heap::isMarked(lastSeenCallee())) {
+    if (!isDirect() && haveLastSeenCallee() && !Heap::isMarked(lastSeenCallee())) {
         handleSpecificCallee(lastSeenCallee());
         clearLastSeenCallee();
     }
index 91005b2..0a91020 100644 (file)
@@ -28,7 +28,6 @@
 #include "CallMode.h"
 #include "CodeLocation.h"
 #include "CodeSpecializationKind.h"
-#include "JSFunction.h"
 #include "PolymorphicCallStubRoutine.h"
 #include "WriteBarrier.h"
 #include <wtf/SentinelLinkedList.h>
@@ -37,12 +36,26 @@ namespace JSC {
 
 #if ENABLE(JIT)
 
+class FunctionCodeBlock;
+class JSFunction;
 enum OpcodeID : unsigned;
 struct CallFrameShuffleData;
 
 class CallLinkInfo : public BasicRawSentinelNode<CallLinkInfo> {
 public:
-    enum CallType { None, Call, CallVarargs, Construct, ConstructVarargs, TailCall, TailCallVarargs };
+    enum CallType {
+        None,
+        Call,
+        CallVarargs,
+        Construct,
+        ConstructVarargs,
+        TailCall,
+        TailCallVarargs,
+        DirectCall,
+        DirectConstruct,
+        DirectTailCall
+    };
+    
     static CallType callTypeFor(OpcodeID opcodeID);
 
     static bool isVarargsCallType(CallType callType)
@@ -64,24 +77,27 @@ public:
     
     static CodeSpecializationKind specializationKindFor(CallType callType)
     {
-        return specializationFromIsConstruct(callType == Construct || callType == ConstructVarargs);
+        return specializationFromIsConstruct(callType == Construct || callType == ConstructVarargs || callType == DirectConstruct);
     }
     CodeSpecializationKind specializationKind() const
     {
         return specializationKindFor(static_cast<CallType>(m_callType));
     }
-
+    
     static CallMode callModeFor(CallType callType)
     {
         switch (callType) {
         case Call:
         case CallVarargs:
+        case DirectCall:
             return CallMode::Regular;
         case TailCall:
         case TailCallVarargs:
+        case DirectTailCall:
             return CallMode::Tail;
         case Construct:
         case ConstructVarargs:
+        case DirectConstruct:
             return CallMode::Construct;
         case None:
             RELEASE_ASSERT_NOT_REACHED();
@@ -89,23 +105,56 @@ public:
 
         RELEASE_ASSERT_NOT_REACHED();
     }
+    
+    static bool isDirect(CallType callType)
+    {
+        switch (callType) {
+        case DirectCall:
+        case DirectTailCall:
+        case DirectConstruct:
+            return true;
+        case Call:
+        case CallVarargs:
+        case TailCall:
+        case TailCallVarargs:
+        case Construct:
+        case ConstructVarargs:
+            return false;
+        case None:
+            RELEASE_ASSERT_NOT_REACHED();
+            return false;
+        }
 
+        RELEASE_ASSERT_NOT_REACHED();
+        return false;
+    }
+    
     CallMode callMode() const
     {
         return callModeFor(static_cast<CallType>(m_callType));
     }
 
+    bool isDirect()
+    {
+        return isDirect(static_cast<CallType>(m_callType));
+    }
+
     bool isTailCall() const
     {
         return callMode() == CallMode::Tail;
     }
+    
+    NearCallMode nearCallMode() const
+    {
+        return isTailCall() ? Tail : Regular;
+    }
 
     bool isVarargs() const
     {
         return isVarargsCallType(static_cast<CallType>(m_callType));
     }
 
-    bool isLinked() { return m_stub || m_callee; }
+    bool isLinked() { return m_stub || m_calleeOrCodeBlock; }
     void unlink(VM&);
 
     void setUpCall(CallType callType, CodeOrigin codeOrigin, unsigned calleeGPR)
@@ -115,11 +164,13 @@ public:
         m_calleeGPR = calleeGPR;
     }
 
-    void setCallLocations(CodeLocationNearCall callReturnLocation, CodeLocationDataLabelPtr hotPathBegin,
+    void setCallLocations(
+        CodeLocationLabel callReturnLocationOrPatchableJump,
+        CodeLocationLabel hotPathBeginOrSlowPathStart,
         CodeLocationNearCall hotPathOther)
     {
-        m_callReturnLocation = callReturnLocation;
-        m_hotPathBegin = hotPathBegin;
+        m_callReturnLocationOrPatchableJump = callReturnLocationOrPatchableJump;
+        m_hotPathBeginOrSlowPathStart = hotPathBeginOrSlowPathStart;
         m_hotPathOther = hotPathOther;
     }
 
@@ -130,27 +181,10 @@ public:
         m_allowStubs = false;
     }
 
-    void setUpCallFromFTL(CallType callType, CodeOrigin codeOrigin,
-        CodeLocationNearCall callReturnLocation, CodeLocationDataLabelPtr hotPathBegin,
-        CodeLocationNearCall hotPathOther, unsigned calleeGPR)
-    {
-        m_callType = callType;
-        m_codeOrigin = codeOrigin;
-        m_callReturnLocation = callReturnLocation;
-        m_hotPathBegin = hotPathBegin;
-        m_hotPathOther = hotPathOther;
-        m_calleeGPR = calleeGPR;
-    }
-
-    CodeLocationNearCall callReturnLocation()
-    {
-        return m_callReturnLocation;
-    }
-
-    CodeLocationDataLabelPtr hotPathBegin()
-    {
-        return m_hotPathBegin;
-    }
+    CodeLocationNearCall callReturnLocation();
+    CodeLocationJump patchableJump();
+    CodeLocationDataLabelPtr hotPathBegin();
+    CodeLocationLabel slowPathStart();
 
     CodeLocationNearCall hotPathOther()
     {
@@ -158,34 +192,21 @@ public:
     }
 
     void setCallee(VM&, JSCell*, JSFunction* callee);
-
     void clearCallee();
+    JSFunction* callee();
 
-    JSFunction* callee()
-    {
-        return m_callee.get();
-    }
-
-    void setLastSeenCallee(VM& vm, const JSCell* owner, JSFunction* callee)
-    {
-        m_lastSeenCallee.set(vm, owner, callee);
-    }
-
-    void clearLastSeenCallee()
-    {
-        m_lastSeenCallee.clear();
-    }
-
-    JSFunction* lastSeenCallee()
-    {
-        return m_lastSeenCallee.get();
-    }
-
-    bool haveLastSeenCallee()
-    {
-        return !!m_lastSeenCallee;
-    }
+    void setCodeBlock(VM&, JSCell*, FunctionCodeBlock*);
+    void clearCodeBlock();
+    FunctionCodeBlock* codeBlock();
 
+    void setLastSeenCallee(VM& vm, const JSCell* owner, JSFunction* callee);
+    void clearLastSeenCallee();
+    JSFunction* lastSeenCallee();
+    bool haveLastSeenCallee();
+    
+    void setExecutableDuringCompilation(ExecutableBase*);
+    ExecutableBase* executable();
+    
     void setStub(PassRefPtr<PolymorphicCallStubRoutine> newStub)
     {
         clearStub();
@@ -254,15 +275,17 @@ public:
         return static_cast<CallType>(m_callType);
     }
 
-    uint8_t* addressOfMaxNumArguments()
+    uint32_t* addressOfMaxNumArguments()
     {
         return &m_maxNumArguments;
     }
 
-    uint8_t maxNumArguments()
+    uint32_t maxNumArguments()
     {
         return m_maxNumArguments;
     }
+    
+    void setMaxNumArguments(unsigned);
 
     static ptrdiff_t offsetOfSlowPathCount()
     {
@@ -304,11 +327,11 @@ public:
     }
 
 private:
-    CodeLocationNearCall m_callReturnLocation;
-    CodeLocationDataLabelPtr m_hotPathBegin;
+    CodeLocationLabel m_callReturnLocationOrPatchableJump;
+    CodeLocationLabel m_hotPathBeginOrSlowPathStart;
     CodeLocationNearCall m_hotPathOther;
-    WriteBarrier<JSFunction> m_callee;
-    WriteBarrier<JSFunction> m_lastSeenCallee;
+    WriteBarrier<JSCell> m_calleeOrCodeBlock;
+    WriteBarrier<JSCell> m_lastSeenCalleeOrExecutable;
     RefPtr<PolymorphicCallStubRoutine> m_stub;
     RefPtr<JITStubRoutine> m_slowStub;
     std::unique_ptr<CallFrameShuffleData> m_frameShuffleData;
@@ -316,9 +339,10 @@ private:
     bool m_hasSeenClosure : 1;
     bool m_clearedByGC : 1;
     bool m_allowStubs : 1;
+    bool m_isLinked : 1;
     unsigned m_callType : 4; // CallType
     unsigned m_calleeGPR : 8;
-    uint8_t m_maxNumArguments; // Only used for varargs calls.
+    uint32_t m_maxNumArguments; // For varargs: the profiled maximum number of arguments. For direct: the number of stack slots allocated for arguments.
     uint32_t m_slowPathCount;
     CodeOrigin m_codeOrigin;
 };
index d909508..faaf711 100644 (file)
@@ -258,6 +258,12 @@ void CallLinkStatus::computeDFGStatuses(
     CodeBlock* baselineCodeBlock = dfgCodeBlock->alternative();
     for (auto iter = dfgCodeBlock->callLinkInfosBegin(); !!iter; ++iter) {
         CallLinkInfo& info = **iter;
+        if (info.isDirect()) {
+            // If the DFG was able to get a direct call then probably so will we. However, there is
+            // a remote chance that it's bad news to lose information about what the DFG did. We'd
+            // ideally like to just know that the DFG had emitted a DirectCall.
+            continue;
+        }
         CodeOrigin codeOrigin = info.codeOrigin();
         
         // Check if we had already previously made a terrible mistake in the FTL for this
index e70f80f..cddf1c6 100644 (file)
@@ -1109,8 +1109,8 @@ void AccessCase::generateImpl(AccessGenerationState& state)
             jit.addLinkTask(
                 [=, &vm] (LinkBuffer& linkBuffer) {
                     m_rareData->callLinkInfo->setCallLocations(
-                        linkBuffer.locationOfNearCall(slowPathCall),
-                        linkBuffer.locationOf(addressOfLinkFunctionCheck),
+                        CodeLocationLabel(linkBuffer.locationOfNearCall(slowPathCall)),
+                        CodeLocationLabel(linkBuffer.locationOf(addressOfLinkFunctionCheck)),
                         linkBuffer.locationOfNearCall(fastPathCall));
 
                     linkBuffer.link(
index e734713..bac4054 100644 (file)
@@ -76,7 +76,7 @@ public:
     const Identifier& ecmaName() const { return m_ecmaName; }
     void setEcmaName(const Identifier& name) { m_ecmaName = name; }
     const Identifier& inferredName() const { return m_inferredName; }
-    unsigned parameterCount() const { return m_parameterCount; };
+    unsigned parameterCount() const { return m_parameterCount; }; // Excluding 'this'!
     unsigned functionLength() const { return m_functionLength; }
     SourceParseMode parseMode() const { return static_cast<SourceParseMode>(m_sourceParseMode); };
 
index 8e88a14..c98fd20 100644 (file)
@@ -375,6 +375,33 @@ public:
     JSValue recover(ExecState*) const;
     
 #if ENABLE(JIT)
+    template<typename Func>
+    void forEachReg(const Func& func)
+    {
+        switch (m_technique) {
+        case InGPR:
+        case UnboxedInt32InGPR:
+        case UnboxedBooleanInGPR:
+        case UnboxedCellInGPR:
+        case UnboxedInt52InGPR:
+        case UnboxedStrictInt52InGPR:
+            func(gpr());
+            return;
+        case InFPR:
+        case UnboxedDoubleInFPR:
+            func(fpr());
+            return;
+#if USE(JSVALUE32_64)
+        case InPair:
+            func(jsValueRegs().payloadGPR());
+            func(jsValueRegs().tagGPR());
+            return;
+#endif
+        default:
+            return;
+        }
+    }
+    
     void dumpInContext(PrintStream& out, DumpContext* context) const;
     void dump(PrintStream& out) const;
 #endif
index 11a28aa..a12f09c 100644 (file)
@@ -1779,6 +1779,7 @@ bool AbstractInterpreter<AbstractStateType>::executeEffects(unsigned clobberLimi
         break;
 
     case TailCall:
+    case DirectTailCall:
     case TailCallVarargs:
     case TailCallForwardVarargs:
         clobberWorld(node->origin.semantic, clobberLimit);
@@ -2824,6 +2825,9 @@ bool AbstractInterpreter<AbstractStateType>::executeEffects(unsigned clobberLimi
     case ConstructForwardVarargs:
     case TailCallForwardVarargsInlinedCaller:
     case CallEval:
+    case DirectCall:
+    case DirectConstruct:
+    case DirectTailCallInlinedCaller:
         clobberWorld(node->origin.semantic, clobberLimit);
         forNode(node).makeHeapTop();
         break;
index 8e402e7..f458c0f 100644 (file)
@@ -90,6 +90,7 @@ struct BasicBlock : RefCounted<BasicBlock> {
             case Switch:
             case Return:
             case TailCall:
+            case DirectTailCall:
             case TailCallVarargs:
             case TailCallForwardVarargs:
             case Unreachable:
index e5ef405..e47ea47 100644 (file)
@@ -805,9 +805,7 @@ private:
         OpInfo prediction)
     {
         addVarArgChild(callee);
-        size_t frameSize = CallFrame::headerSizeInRegisters + argCount;
-        size_t alignedFrameSize = WTF::roundUpToMultipleOf(stackAlignmentRegisters(), frameSize);
-        size_t parameterSlots = alignedFrameSize - CallerFrameAndPC::sizeInRegisters;
+        size_t parameterSlots = Graph::parameterSlotsForArgCount(argCount);
 
         if (parameterSlots > m_parameterSlots)
             m_parameterSlots = parameterSlots;
@@ -1294,15 +1292,13 @@ ByteCodeParser::Terminality ByteCodeParser::handleCall(
     
     unsigned nextOffset = m_currentIndex + instructionSize;
     
-    OpInfo callOpInfo;
-    
     if (handleInlining(callTarget, result, callLinkStatus, registerOffset, virtualRegisterForArgument(0, registerOffset), VirtualRegister(), 0, argumentCountIncludingThis, nextOffset, op, kind, prediction)) {
         if (m_graph.compilation())
             m_graph.compilation()->noticeInlinedCall();
         return NonTerminal;
     }
     
-    Node* callNode = addCall(result, op, callOpInfo, callTarget, argumentCountIncludingThis, registerOffset, prediction);
+    Node* callNode = addCall(result, op, OpInfo(), callTarget, argumentCountIncludingThis, registerOffset, prediction);
     if (callNode->op() == TailCall)
         return Terminal;
     ASSERT(callNode->op() != TailCallVarargs && callNode->op() != TailCallForwardVarargs);
index 3b50b58..f929337 100644 (file)
@@ -515,8 +515,11 @@ void clobberize(Graph& graph, Node* node, const ReadFunctor& read, const WriteFu
     case ArrayPush:
     case ArrayPop:
     case Call:
+    case DirectCall:
     case TailCallInlinedCaller:
+    case DirectTailCallInlinedCaller:
     case Construct:
+    case DirectConstruct:
     case CallVarargs:
     case CallForwardVarargs:
     case TailCallVarargsInlinedCaller:
@@ -544,6 +547,7 @@ void clobberize(Graph& graph, Node* node, const ReadFunctor& read, const WriteFu
         return;
 
     case TailCall:
+    case DirectTailCall:
     case TailCallVarargs:
     case TailCallForwardVarargs:
         read(World);
index fb83332..3f13842 100644 (file)
@@ -145,8 +145,11 @@ bool doesGC(Graph& graph, Node* node)
     case CompareStrictEq:
     case CompareEqPtr:
     case Call:
+    case DirectCall:
     case TailCallInlinedCaller:
+    case DirectTailCallInlinedCaller:
     case Construct:
+    case DirectConstruct:
     case CallVarargs:
     case CallEval:
     case TailCallVarargsInlinedCaller:
@@ -183,6 +186,7 @@ bool doesGC(Graph& graph, Node* node)
     case Switch:
     case Return:
     case TailCall:
+    case DirectTailCall:
     case TailCallVarargs:
     case Throw:
     case CountExecution:
index 6347c79..33c0cf1 100644 (file)
@@ -1736,9 +1736,12 @@ private:
         case GetGlobalLexicalVariable:
         case NotifyWrite:
         case Call:
+        case DirectCall:
         case CheckTypeInfoFlags:
         case TailCallInlinedCaller:
+        case DirectTailCallInlinedCaller:
         case Construct:
+        case DirectConstruct:
         case CallVarargs:
         case CallEval:
         case TailCallVarargsInlinedCaller:
@@ -1766,6 +1769,7 @@ private:
         case Jump:
         case Return:
         case TailCall:
+        case DirectTailCall:
         case TailCallVarargs:
         case Throw:
         case ThrowStaticError:
index 9c0d427..a0f535f 100644 (file)
@@ -1159,6 +1159,13 @@ BitVector Graph::localsLiveInBytecode(CodeOrigin codeOrigin)
     return result;
 }
 
+unsigned Graph::parameterSlotsForArgCount(unsigned argCount)
+{
+    size_t frameSize = CallFrame::headerSizeInRegisters + argCount;
+    size_t alignedFrameSize = WTF::roundUpToMultipleOf(stackAlignmentRegisters(), frameSize);
+    return alignedFrameSize - CallerFrameAndPC::sizeInRegisters;
+}
+
 unsigned Graph::frameRegisterCount()
 {
     unsigned result = m_nextMachineLocal + std::max(m_parameterSlots, static_cast<unsigned>(maxFrameExtentForSlowPathCallInRegisters));
index 923f615..76b2bae 100644 (file)
@@ -790,6 +790,8 @@ public:
     BytecodeKills& killsFor(CodeBlock*);
     BytecodeKills& killsFor(InlineCallFrame*);
     
+    static unsigned parameterSlotsForArgCount(unsigned);
+    
     unsigned frameRegisterCount();
     unsigned stackPointerOffset();
     unsigned requiredRegisterCountForExit();
index d0fc06d..f75d8b4 100644 (file)
@@ -364,6 +364,7 @@ inline bool InPlaceAbstractState::mergeToSuccessors(BasicBlock* basicBlock)
         
     case Return:
     case TailCall:
+    case DirectTailCall:
     case TailCallVarargs:
     case TailCallForwardVarargs:
     case Unreachable:
index 42e0746..d35fdc5 100644 (file)
@@ -276,11 +276,29 @@ void JITCompiler::link(LinkBuffer& linkBuffer)
     
     for (unsigned i = 0; i < m_jsCalls.size(); ++i) {
         JSCallRecord& record = m_jsCalls[i];
-        CallLinkInfo& info = *record.m_info;
-        linkBuffer.link(record.m_slowCall, FunctionPtr(m_vm->getCTIStub(linkCallThunkGenerator).code().executableAddress()));
-        info.setCallLocations(linkBuffer.locationOfNearCall(record.m_slowCall),
-            linkBuffer.locationOf(record.m_targetToCheck),
-            linkBuffer.locationOfNearCall(record.m_fastCall));
+        CallLinkInfo& info = *record.info;
+        linkBuffer.link(record.slowCall, FunctionPtr(m_vm->getCTIStub(linkCallThunkGenerator).code().executableAddress()));
+        info.setCallLocations(
+            CodeLocationLabel(linkBuffer.locationOfNearCall(record.slowCall)),
+            CodeLocationLabel(linkBuffer.locationOf(record.targetToCheck)),
+            linkBuffer.locationOfNearCall(record.fastCall));
+    }
+    
+    for (JSDirectCallRecord& record : m_jsDirectCalls) {
+        CallLinkInfo& info = *record.info;
+        linkBuffer.link(record.call, linkBuffer.locationOf(record.slowPath));
+        info.setCallLocations(
+            CodeLocationLabel(),
+            linkBuffer.locationOf(record.slowPath),
+            linkBuffer.locationOfNearCall(record.call));
+    }
+    
+    for (JSDirectTailCallRecord& record : m_jsDirectTailCalls) {
+        CallLinkInfo& info = *record.info;
+        info.setCallLocations(
+            linkBuffer.locationOf(record.patchableJump),
+            linkBuffer.locationOf(record.slowPath),
+            linkBuffer.locationOfNearCall(record.call));
     }
     
     MacroAssemblerCodeRef osrExitThunk = vm()->getCTIStub(osrExitGenerationThunkGenerator);
index f233488..4f8e817 100644 (file)
@@ -207,16 +207,21 @@ public:
         m_ins.append(record);
     }
     
-    unsigned currentJSCallIndex() const
-    {
-        return m_jsCalls.size();
-    }
-
     void addJSCall(Call fastCall, Call slowCall, DataLabelPtr targetToCheck, CallLinkInfo* info)
     {
         m_jsCalls.append(JSCallRecord(fastCall, slowCall, targetToCheck, info));
     }
     
+    void addJSDirectCall(Call call, Label slowPath, CallLinkInfo* info)
+    {
+        m_jsDirectCalls.append(JSDirectCallRecord(call, slowPath, info));
+    }
+    
+    void addJSDirectTailCall(PatchableJump patchableJump, Call call, Label slowPath, CallLinkInfo* info)
+    {
+        m_jsDirectTailCalls.append(JSDirectTailCallRecord(patchableJump, call, slowPath, info));
+    }
+    
     void addWeakReference(JSCell* target)
     {
         m_graph.m_plan.weakReferences.addLazily(target);
@@ -292,23 +297,53 @@ private:
 
     struct JSCallRecord {
         JSCallRecord(Call fastCall, Call slowCall, DataLabelPtr targetToCheck, CallLinkInfo* info)
-            : m_fastCall(fastCall)
-            , m_slowCall(slowCall)
-            , m_targetToCheck(targetToCheck)
-            , m_info(info)
+            : fastCall(fastCall)
+            , slowCall(slowCall)
+            , targetToCheck(targetToCheck)
+            , info(info)
+        {
+        }
+        
+        Call fastCall;
+        Call slowCall;
+        DataLabelPtr targetToCheck;
+        CallLinkInfo* info;
+    };
+    
+    struct JSDirectCallRecord {
+        JSDirectCallRecord(Call call, Label slowPath, CallLinkInfo* info)
+            : call(call)
+            , slowPath(slowPath)
+            , info(info)
+        {
+        }
+        
+        Call call;
+        Label slowPath;
+        CallLinkInfo* info;
+    };
+    
+    struct JSDirectTailCallRecord {
+        JSDirectTailCallRecord(PatchableJump patchableJump, Call call, Label slowPath, CallLinkInfo* info)
+            : patchableJump(patchableJump)
+            , call(call)
+            , slowPath(slowPath)
+            , info(info)
         {
         }
         
-        Call m_fastCall;
-        Call m_slowCall;
-        DataLabelPtr m_targetToCheck;
-        CallLinkInfo* m_info;
+        PatchableJump patchableJump;
+        Call call;
+        Label slowPath;
+        CallLinkInfo* info;
     };
     
     Vector<InlineCacheWrapper<JITGetByIdGenerator>, 4> m_getByIds;
     Vector<InlineCacheWrapper<JITPutByIdGenerator>, 4> m_putByIds;
     Vector<InRecord, 4> m_ins;
     Vector<JSCallRecord, 4> m_jsCalls;
+    Vector<JSDirectCallRecord, 4> m_jsDirectCalls;
+    Vector<JSDirectTailCallRecord, 4> m_jsDirectTailCalls;
     SegmentedVector<OSRExitCompilationInfo, 4> m_exitCompilationInfo;
     Vector<Vector<Label>> m_exitSiteLabels;
     
index d4d9c81..f37117b 100644 (file)
@@ -195,6 +195,31 @@ void Node::convertToPutClosureVarHint()
         child1().node(), child2().node());
 }
 
+void Node::convertToDirectCall(FrozenValue* executable)
+{
+    NodeType newOp = LastNodeType;
+    switch (op()) {
+    case Call:
+        newOp = DirectCall;
+        break;
+    case Construct:
+        newOp = DirectConstruct;
+        break;
+    case TailCallInlinedCaller:
+        newOp = DirectTailCallInlinedCaller;
+        break;
+    case TailCall:
+        newOp = DirectTailCall;
+        break;
+    default:
+        RELEASE_ASSERT_NOT_REACHED();
+        break;
+    }
+    
+    m_op = newOp;
+    m_opInfo = executable;
+}
+
 String Node::tryGetString(Graph& graph)
 {
     if (hasConstant())
index b159f86..c9411d5 100644 (file)
@@ -643,6 +643,8 @@ public:
         m_op = ArithNegate;
     }
     
+    void convertToDirectCall(FrozenValue*);
+    
     JSValue asJSValue()
     {
         return constant()->value();
@@ -1254,6 +1256,7 @@ public:
         case Switch:
         case Return:
         case TailCall:
+        case DirectTailCall:
         case TailCallVarargs:
         case TailCallForwardVarargs:
         case Unreachable:
@@ -1425,8 +1428,11 @@ public:
         case GetByVal:
         case GetByValWithThis:
         case Call:
+        case DirectCall:
         case TailCallInlinedCaller:
+        case DirectTailCallInlinedCaller:
         case Construct:
+        case DirectConstruct:
         case CallVarargs:
         case CallEval:
         case TailCallVarargsInlinedCaller:
@@ -1477,6 +1483,10 @@ public:
         case MaterializeCreateActivation:
         case NewRegexp:
         case CompareEqPtr:
+        case DirectCall:
+        case DirectTailCall:
+        case DirectConstruct:
+        case DirectTailCallInlinedCaller:
             return true;
         default:
             return false;
@@ -2405,8 +2415,7 @@ private:
     unsigned m_refCount;
     // The prediction ascribed to this node after propagation.
     SpeculatedType m_prediction { SpecNone };
-    // Immediate values, accesses type-checked via accessors above. The first one is
-    // big enough to store a pointer.
+    // Immediate values, accesses type-checked via accessors above.
     struct OpInfoWrapper {
         OpInfoWrapper()
         {
index 749fbbd..19b2825 100644 (file)
@@ -270,12 +270,15 @@ namespace JSC { namespace DFG {
     \
     /* Calls. */\
     macro(Call, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(DirectCall, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
     macro(Construct, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(DirectConstruct, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
     macro(CallVarargs, NodeResultJS | NodeMustGenerate) \
     macro(CallForwardVarargs, NodeResultJS | NodeMustGenerate) \
     macro(ConstructVarargs, NodeResultJS | NodeMustGenerate) \
     macro(ConstructForwardVarargs, NodeResultJS | NodeMustGenerate) \
     macro(TailCallInlinedCaller, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
+    macro(DirectTailCallInlinedCaller, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
     macro(TailCallVarargsInlinedCaller, NodeResultJS | NodeMustGenerate) \
     macro(TailCallForwardVarargsInlinedCaller, NodeResultJS | NodeMustGenerate) \
     macro(CallEval, NodeResultJS | NodeMustGenerate | NodeHasVarArgs) \
@@ -358,6 +361,7 @@ namespace JSC { namespace DFG {
     macro(Switch, NodeMustGenerate) \
     macro(Return, NodeMustGenerate) \
     macro(TailCall, NodeMustGenerate | NodeHasVarArgs) \
+    macro(DirectTailCall, NodeMustGenerate | NodeHasVarArgs) \
     macro(TailCallVarargs, NodeMustGenerate) \
     macro(TailCallForwardVarargs, NodeMustGenerate) \
     macro(Unreachable, NodeMustGenerate) \
index abd9563..e11428c 100644 (file)
@@ -691,8 +691,11 @@ private:
         case MultiGetByOffset:
         case GetDirectPname:
         case Call:
+        case DirectCall:
         case TailCallInlinedCaller:
+        case DirectTailCallInlinedCaller:
         case Construct:
+        case DirectConstruct:
         case CallVarargs:
         case CallEval:
         case TailCallVarargsInlinedCaller:
@@ -1043,6 +1046,7 @@ private:
         case PutToArguments:
         case Return:
         case TailCall:
+        case DirectTailCall:
         case TailCallVarargs:
         case TailCallForwardVarargs:
         case Throw:
index 9c62615..69bbe11 100644 (file)
@@ -245,8 +245,11 @@ bool safeToExecute(AbstractStateType& state, Graph& graph, Node* node)
     case CompareStrictEq:
     case CompareEqPtr:
     case Call:
+    case DirectCall:
     case TailCallInlinedCaller:
+    case DirectTailCallInlinedCaller:
     case Construct:
+    case DirectConstruct:
     case CallVarargs:
     case CallEval:
     case TailCallVarargsInlinedCaller:
@@ -301,6 +304,7 @@ bool safeToExecute(AbstractStateType& state, Graph& graph, Node* node)
     case Switch:
     case Return:
     case TailCall:
+    case DirectTailCall:
     case TailCallVarargs:
     case TailCallForwardVarargs:
     case Throw:
index 8afefa9..b4045b2 100644 (file)
@@ -1155,6 +1155,12 @@ public:
         return appendCall(operation);
     }
 
+    JITCompiler::Call callOperation(V_JITOperation_ECliJsf operation, CallLinkInfo* callLinkInfo, GPRReg arg1)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImmPtr(callLinkInfo), arg1);
+        return appendCall(operation);
+    }
+
     JITCompiler::Call callOperation(V_JITOperation_EC operation, JSCell* arg1)
     {
         m_jit.setupArgumentsWithExecState(TrustedImmPtr(arg1));
index bcd8b1a..458a292 100644 (file)
@@ -657,6 +657,7 @@ void SpeculativeJIT::emitCall(Node* node)
     bool isVarargs = false;
     bool isForwardVarargs = false;
     bool isTail = false;
+    bool isDirect = false;
     bool isEmulatedTail = false;
     switch (node->op()) {
     case Call:
@@ -710,6 +711,24 @@ void SpeculativeJIT::emitCall(Node* node)
         callType = CallLinkInfo::ConstructVarargs;
         isForwardVarargs = true;
         break;
+    case DirectCall:
+        callType = CallLinkInfo::DirectCall;
+        isDirect = true;
+        break;
+    case DirectConstruct:
+        callType = CallLinkInfo::DirectConstruct;
+        isDirect = true;
+        break;
+    case DirectTailCall:
+        callType = CallLinkInfo::DirectTailCall;
+        isTail = true;
+        isDirect = true;
+        break;
+    case DirectTailCallInlinedCaller:
+        callType = CallLinkInfo::DirectCall;
+        isEmulatedTail = true;
+        isDirect = true;
+        break;
     default:
         DFG_CRASH(m_jit.graph(), node, "bad node type");
         break;
@@ -720,8 +739,19 @@ void SpeculativeJIT::emitCall(Node* node)
     GPRReg calleePayloadGPR = InvalidGPRReg;
     CallFrameShuffleData shuffleData;
     
+    ExecutableBase* executable = nullptr;
+    FunctionExecutable* functionExecutable = nullptr;
+    if (isDirect) {
+        executable = node->castOperand<ExecutableBase*>();
+        functionExecutable = jsDynamicCast<FunctionExecutable*>(executable);
+    }
+    
+    unsigned numPassedArgs = 0;
+    unsigned numAllocatedArgs = 0;
+    
     // Gotta load the arguments somehow. Varargs is trickier.
     if (isVarargs || isForwardVarargs) {
+        RELEASE_ASSERT(!isDirect);
         CallVarargsData* data = node->callVarargsData();
 
         GPRReg resultGPR;
@@ -817,28 +847,53 @@ void SpeculativeJIT::emitCall(Node* node)
     } else {        
         // The call instruction's first child is either the function (normal call) or the
         // receiver (method call). subsequent children are the arguments.
-        int numPassedArgs = node->numChildren() - 1;
+        numPassedArgs = node->numChildren() - 1;
+        numAllocatedArgs = numPassedArgs;
+        
+        if (functionExecutable) {
+            // Allocate more args if this would let us avoid arity checks. This is throttled by
+            // CallLinkInfo's limit. It's probably good to throttle it - if the callee wants a
+            // ginormous amount of argument space then it's better for them to do it so that when we
+            // make calls to other things, we don't waste space.
+            unsigned desiredNumAllocatedArgs = static_cast<unsigned>(functionExecutable->parameterCount()) + 1;
+            if (desiredNumAllocatedArgs <= Options::maximumDirectCallStackSize()) {
+                numAllocatedArgs = std::max(numAllocatedArgs, desiredNumAllocatedArgs);
+                
+                // Whoever converts to DirectCall should do this adjustment. It's too late for us to
+                // do this adjustment now since we will have already emitted code that relied on the
+                // value of m_parameterSlots.
+                DFG_ASSERT(
+                    m_jit.graph(), node,
+                    Graph::parameterSlotsForArgCount(numAllocatedArgs)
+                    <= m_jit.graph().m_parameterSlots);
+            }
+        }
 
-        if (node->op() == TailCall) {
+        if (isTail) {
             JSValueOperand callee(this, calleeEdge);
             calleeTagGPR = callee.tagGPR();
             calleePayloadGPR = callee.payloadGPR();
-            use(calleeEdge);
+            if (!isDirect)
+                use(calleeEdge);
 
             shuffleData.numLocals = m_jit.graph().frameRegisterCount();
             shuffleData.callee = ValueRecovery::inPair(calleeTagGPR, calleePayloadGPR);
-            shuffleData.args.resize(numPassedArgs);
+            shuffleData.args.resize(numAllocatedArgs);
 
-            for (int i = 0; i < numPassedArgs; ++i) {
+            for (unsigned i = 0; i < numPassedArgs; ++i) {
                 Edge argEdge = m_jit.graph().varArgChild(node, i + 1);
                 GenerationInfo& info = generationInfo(argEdge.node());
-                use(argEdge);
+                if (!isDirect)
+                    use(argEdge);
                 shuffleData.args[i] = info.recovery(argEdge->virtualRegister());
             }
+            
+            for (unsigned i = numPassedArgs; i < numAllocatedArgs; ++i)
+                shuffleData.args[i] = ValueRecovery::constant(jsUndefined());
         } else {
             m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), m_jit.calleeFramePayloadSlot(CallFrameSlot::argumentCount));
         
-            for (int i = 0; i < numPassedArgs; i++) {
+            for (unsigned i = 0; i < numPassedArgs; i++) {
                 Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i];
                 JSValueOperand arg(this, argEdge);
                 GPRReg argTagGPR = arg.tagGPR();
@@ -848,10 +903,13 @@ void SpeculativeJIT::emitCall(Node* node)
                 m_jit.store32(argTagGPR, m_jit.calleeArgumentTagSlot(i));
                 m_jit.store32(argPayloadGPR, m_jit.calleeArgumentPayloadSlot(i));
             }
+            
+            for (unsigned i = numPassedArgs; i < numAllocatedArgs; ++i)
+                m_jit.storeTrustedValue(jsUndefined(), JITCompiler::calleeArgumentSlot(i));
         }
     }
 
-    if (node->op() != TailCall) {
+    if (!isTail || isVarargs || isForwardVarargs) {
         JSValueOperand callee(this, calleeEdge);
         calleeTagGPR = callee.tagGPR();
         calleePayloadGPR = callee.payloadGPR();
@@ -863,11 +921,6 @@ void SpeculativeJIT::emitCall(Node* node)
             flushRegisters();
     }
 
-    GPRFlushedCallResult resultPayload(this);
-    GPRFlushedCallResult2 resultTag(this);
-    GPRReg resultPayloadGPR = resultPayload.gpr();
-    GPRReg resultTagGPR = resultTag.gpr();
-
     JITCompiler::DataLabelPtr targetToCheck;
     JITCompiler::JumpList slowPath;
 
@@ -877,12 +930,16 @@ void SpeculativeJIT::emitCall(Node* node)
     CodeOrigin dynamicOrigin =
         isEmulatedTail ? *staticOrigin.inlineCallFrame->getCallerSkippingTailCalls() : staticOrigin;
     CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(dynamicOrigin, m_stream->size());
-    m_jit.emitStoreCallSiteIndex(callSite);
     
     CallLinkInfo* info = m_jit.codeBlock()->addCallLinkInfo();
     info->setUpCall(callType, node->origin.semantic, calleePayloadGPR);
     
     auto setResultAndResetStack = [&] () {
+        GPRFlushedCallResult resultPayload(this);
+        GPRFlushedCallResult2 resultTag(this);
+        GPRReg resultPayloadGPR = resultPayload.gpr();
+        GPRReg resultTagGPR = resultTag.gpr();
+        
         m_jit.setupResults(resultPayloadGPR, resultTagGPR);
 
         jsValueResult(resultTagGPR, resultPayloadGPR, node, DataFormatJS, UseChildrenCalledExplicitly);
@@ -898,6 +955,7 @@ void SpeculativeJIT::emitCall(Node* node)
         // subtract stack to make room for the call. Lucky for us, at this point we have the whole
         // register file to ourselves.
         
+        m_jit.emitStoreCallSiteIndex(callSite);
         m_jit.addPtr(TrustedImm32(-static_cast<ptrdiff_t>(sizeof(CallerFrameAndPC))), JITCompiler::stackPointerRegister, GPRInfo::regT0);
         m_jit.storePtr(GPRInfo::callFrameRegister, JITCompiler::Address(GPRInfo::regT0, CallFrame::callerFrameOffset()));
         
@@ -924,6 +982,63 @@ void SpeculativeJIT::emitCall(Node* node)
         return;
     }
 
+    if (isDirect) {
+        info->setExecutableDuringCompilation(executable);
+        info->setMaxNumArguments(numAllocatedArgs);
+
+        if (isTail) {
+            RELEASE_ASSERT(node->op() == DirectTailCall);
+            
+            JITCompiler::PatchableJump patchableJump = m_jit.patchableJump();
+            JITCompiler::Label mainPath = m_jit.label();
+            
+            m_jit.emitStoreCallSiteIndex(callSite);
+            
+            info->setFrameShuffleData(shuffleData);
+            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+            
+            JITCompiler::Call call = m_jit.nearTailCall();
+            
+            JITCompiler::Label slowPath = m_jit.label();
+            patchableJump.m_jump.linkTo(slowPath, &m_jit);
+            
+            silentSpillAllRegisters(InvalidGPRReg);
+            callOperation(operationLinkDirectCall, info, calleePayloadGPR);
+            silentFillAllRegisters(InvalidGPRReg);
+            m_jit.exceptionCheck();
+            m_jit.jump().linkTo(mainPath, &m_jit);
+            
+            useChildren(node);
+            
+            m_jit.addJSDirectTailCall(patchableJump, call, slowPath, info);
+            return;
+        }
+        
+        JITCompiler::Label mainPath = m_jit.label();
+        
+        m_jit.emitStoreCallSiteIndex(callSite);
+        
+        JITCompiler::Call call = m_jit.nearCall();
+        JITCompiler::Jump done = m_jit.jump();
+        
+        JITCompiler::Label slowPath = m_jit.label();
+        if (isX86())
+            m_jit.pop(JITCompiler::selectScratchGPR(calleePayloadGPR));
+
+        callOperation(operationLinkDirectCall, info, calleePayloadGPR);
+        m_jit.exceptionCheck();
+        m_jit.jump().linkTo(mainPath, &m_jit);
+        
+        done.link(&m_jit);
+        
+        setResultAndResetStack();
+        
+        m_jit.addJSDirectCall(call, slowPath, info);
+        return;
+    }
+    
+    m_jit.emitStoreCallSiteIndex(callSite);
+    
     slowPath.append(m_jit.branchIfNotCell(JSValueRegs(calleeTagGPR, calleePayloadGPR)));
     slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck));
 
@@ -4780,6 +4895,10 @@ void SpeculativeJIT::compile(Node* node)
     case TailCallForwardVarargsInlinedCaller:
     case ConstructForwardVarargs:
     case CallEval:
+    case DirectCall:
+    case DirectConstruct:
+    case DirectTailCall:
+    case DirectTailCallInlinedCaller:
         emitCall(node);
         break;
 
index b9dd99f..d185b9d 100644 (file)
@@ -636,6 +636,7 @@ void SpeculativeJIT::emitCall(Node* node)
     bool isForwardVarargs = false;
     bool isTail = false;
     bool isEmulatedTail = false;
+    bool isDirect = false;
     switch (node->op()) {
     case Call:
     case CallEval:
@@ -688,6 +689,24 @@ void SpeculativeJIT::emitCall(Node* node)
         isEmulatedTail = true;
         isForwardVarargs = true;
         break;
+    case DirectCall:
+        callType = CallLinkInfo::DirectCall;
+        isDirect = true;
+        break;
+    case DirectConstruct:
+        callType = CallLinkInfo::DirectConstruct;
+        isDirect = true;
+        break;
+    case DirectTailCall:
+        callType = CallLinkInfo::DirectTailCall;
+        isTail = true;
+        isDirect = true;
+        break;
+    case DirectTailCallInlinedCaller:
+        callType = CallLinkInfo::DirectCall;
+        isEmulatedTail = true;
+        isDirect = true;
+        break;
     default:
         DFG_CRASH(m_jit.graph(), node, "bad node type");
         break;
@@ -696,8 +715,19 @@ void SpeculativeJIT::emitCall(Node* node)
     GPRReg calleeGPR = InvalidGPRReg;
     CallFrameShuffleData shuffleData;
     
+    ExecutableBase* executable = nullptr;
+    FunctionExecutable* functionExecutable = nullptr;
+    if (isDirect) {
+        executable = node->castOperand<ExecutableBase*>();
+        functionExecutable = jsDynamicCast<FunctionExecutable*>(executable);
+    }
+    
+    unsigned numPassedArgs = 0;
+    unsigned numAllocatedArgs = 0;
+    
     // Gotta load the arguments somehow. Varargs is trickier.
     if (isVarargs || isForwardVarargs) {
+        RELEASE_ASSERT(!isDirect);
         CallVarargsData* data = node->callVarargsData();
 
         GPRReg resultGPR;
@@ -788,31 +818,56 @@ void SpeculativeJIT::emitCall(Node* node)
     } else {
         // The call instruction's first child is the function; the subsequent children are the
         // arguments.
-        int numPassedArgs = node->numChildren() - 1;
+        numPassedArgs = node->numChildren() - 1;
+        numAllocatedArgs = numPassedArgs;
+        
+        if (functionExecutable) {
+            // Allocate more args if this would let us avoid arity checks. This is throttled by
+            // CallLinkInfo's limit. It's probably good to throttle it - if the callee wants a
+            // ginormous amount of argument space then it's better for them to do it so that when we
+            // make calls to other things, we don't waste space.
+            unsigned desiredNumAllocatedArgs = static_cast<unsigned>(functionExecutable->parameterCount()) + 1;
+            if (desiredNumAllocatedArgs <= Options::maximumDirectCallStackSize()) {
+                numAllocatedArgs = std::max(numAllocatedArgs, desiredNumAllocatedArgs);
+                
+                // Whoever converts to DirectCall should do this adjustment. It's too late for us to
+                // do this adjustment now since we will have already emitted code that relied on the
+                // value of m_parameterSlots.
+                DFG_ASSERT(
+                    m_jit.graph(), node,
+                    Graph::parameterSlotsForArgCount(numAllocatedArgs)
+                    <= m_jit.graph().m_parameterSlots);
+            }
+        }
 
-        if (node->op() == TailCall) {
+        if (isTail) {
             Edge calleeEdge = m_jit.graph().child(node, 0);
             JSValueOperand callee(this, calleeEdge);
             calleeGPR = callee.gpr();
-            callee.use();
+            if (!isDirect)
+                callee.use();
 
             shuffleData.tagTypeNumber = GPRInfo::tagTypeNumberRegister;
             shuffleData.numLocals = m_jit.graph().frameRegisterCount();
             shuffleData.callee = ValueRecovery::inGPR(calleeGPR, DataFormatJS);
-            shuffleData.args.resize(numPassedArgs);
+            shuffleData.args.resize(numAllocatedArgs);
 
-            for (int i = 0; i < numPassedArgs; ++i) {
+            for (unsigned i = 0; i < numPassedArgs; ++i) {
                 Edge argEdge = m_jit.graph().varArgChild(node, i + 1);
                 GenerationInfo& info = generationInfo(argEdge.node());
-                use(argEdge);
+                if (!isDirect)
+                    use(argEdge);
                 shuffleData.args[i] = info.recovery(argEdge->virtualRegister());
             }
+            
+            for (unsigned i = numPassedArgs; i < numAllocatedArgs; ++i)
+                shuffleData.args[i] = ValueRecovery::constant(jsUndefined());
 
             shuffleData.setupCalleeSaveRegisters(m_jit.codeBlock());
         } else {
             m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), JITCompiler::calleeFramePayloadSlot(CallFrameSlot::argumentCount));
 
-            for (int i = 0; i < numPassedArgs; i++) {
+            for (unsigned i = 0; i < numPassedArgs; i++) {
                 Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i];
                 JSValueOperand arg(this, argEdge);
                 GPRReg argGPR = arg.gpr();
@@ -820,10 +875,13 @@ void SpeculativeJIT::emitCall(Node* node)
                 
                 m_jit.store64(argGPR, JITCompiler::calleeArgumentSlot(i));
             }
+            
+            for (unsigned i = numPassedArgs; i < numAllocatedArgs; ++i)
+                m_jit.storeTrustedValue(jsUndefined(), JITCompiler::calleeArgumentSlot(i));
         }
     }
-
-    if (node->op() != TailCall) {
+    
+    if (!isTail || isVarargs || isForwardVarargs) {
         Edge calleeEdge = m_jit.graph().child(node, 0);
         JSValueOperand callee(this, calleeEdge);
         calleeGPR = callee.gpr();
@@ -840,7 +898,6 @@ void SpeculativeJIT::emitCall(Node* node)
         isEmulatedTail ? *staticOrigin.inlineCallFrame->getCallerSkippingTailCalls() : staticOrigin;
 
     CallSiteIndex callSite = m_jit.recordCallSiteAndGenerateExceptionHandlingOSRExitIfNeeded(dynamicOrigin, m_stream->size());
-    m_jit.emitStoreCallSiteIndex(callSite);
     
     auto setResultAndResetStack = [&] () {
         GPRFlushedCallResult result(this);
@@ -864,6 +921,7 @@ void SpeculativeJIT::emitCall(Node* node)
         // subtract stack to make room for the call. Lucky for us, at this point we have the whole
         // register file to ourselves.
         
+        m_jit.emitStoreCallSiteIndex(callSite);
         m_jit.addPtr(TrustedImm32(-static_cast<ptrdiff_t>(sizeof(CallerFrameAndPC))), JITCompiler::stackPointerRegister, GPRInfo::regT0);
         m_jit.storePtr(GPRInfo::callFrameRegister, JITCompiler::Address(GPRInfo::regT0, CallFrame::callerFrameOffset()));
         
@@ -889,6 +947,63 @@ void SpeculativeJIT::emitCall(Node* node)
         return;
     }
     
+    if (isDirect) {
+        callLinkInfo->setExecutableDuringCompilation(executable);
+        callLinkInfo->setMaxNumArguments(numAllocatedArgs);
+
+        if (isTail) {
+            RELEASE_ASSERT(node->op() == DirectTailCall);
+            
+            JITCompiler::PatchableJump patchableJump = m_jit.patchableJump();
+            JITCompiler::Label mainPath = m_jit.label();
+            
+            m_jit.emitStoreCallSiteIndex(callSite);
+            
+            callLinkInfo->setFrameShuffleData(shuffleData);
+            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+            
+            JITCompiler::Call call = m_jit.nearTailCall();
+            
+            JITCompiler::Label slowPath = m_jit.label();
+            patchableJump.m_jump.linkTo(slowPath, &m_jit);
+            
+            silentSpillAllRegisters(InvalidGPRReg);
+            callOperation(operationLinkDirectCall, callLinkInfo, calleeGPR);
+            silentFillAllRegisters(InvalidGPRReg);
+            m_jit.exceptionCheck();
+            m_jit.jump().linkTo(mainPath, &m_jit);
+            
+            useChildren(node);
+            
+            m_jit.addJSDirectTailCall(patchableJump, call, slowPath, callLinkInfo);
+            return;
+        }
+        
+        JITCompiler::Label mainPath = m_jit.label();
+        
+        m_jit.emitStoreCallSiteIndex(callSite);
+        
+        JITCompiler::Call call = m_jit.nearCall();
+        JITCompiler::Jump done = m_jit.jump();
+        
+        JITCompiler::Label slowPath = m_jit.label();
+        if (isX86())
+            m_jit.pop(JITCompiler::selectScratchGPR(calleeGPR));
+
+        callOperation(operationLinkDirectCall, callLinkInfo, calleeGPR);
+        m_jit.exceptionCheck();
+        m_jit.jump().linkTo(mainPath, &m_jit);
+        
+        done.link(&m_jit);
+        
+        setResultAndResetStack();
+        
+        m_jit.addJSDirectCall(call, slowPath, callLinkInfo);
+        return;
+    }
+    
+    m_jit.emitStoreCallSiteIndex(callSite);
+    
     JITCompiler::DataLabelPtr targetToCheck;
     JITCompiler::Jump slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, MacroAssembler::TrustedImmPtr(0));
 
@@ -4955,6 +5070,10 @@ void SpeculativeJIT::compile(Node* node)
     case TailCallForwardVarargs:
     case TailCallForwardVarargsInlinedCaller:
     case CallEval:
+    case DirectCall:
+    case DirectConstruct:
+    case DirectTailCall:
+    case DirectTailCallInlinedCaller:
         emitCall(node);
         break;
 
index c8b3ff8..10406fa 100644 (file)
@@ -760,6 +760,38 @@ private:
             m_node->origin = origin;
             break;
         }
+            
+        case Call:
+        case Construct:
+        case TailCallInlinedCaller:
+        case TailCall: {
+            ExecutableBase* executable = nullptr;
+            Edge callee = m_graph.varArgChild(m_node, 0);
+            if (JSFunction* function = callee->dynamicCastConstant<JSFunction*>())
+                executable = function->executable();
+            else if (callee->isFunctionAllocation())
+                executable = callee->castOperand<FunctionExecutable*>();
+            
+            if (!executable)
+                break;
+            
+            if (FunctionExecutable* functionExecutable = jsDynamicCast<FunctionExecutable*>(executable)) {
+                // We need to update m_parameterSlots before we get to the backend, but we don't
+                // want to do too much of this.
+                unsigned numAllocatedArgs =
+                    static_cast<unsigned>(functionExecutable->parameterCount()) + 1;
+                
+                if (numAllocatedArgs <= Options::maximumDirectCallStackSize()) {
+                    m_graph.m_parameterSlots = std::max(
+                        m_graph.m_parameterSlots,
+                        Graph::parameterSlotsForArgCount(numAllocatedArgs));
+                }
+            }
+            
+            m_node->convertToDirectCall(m_graph.freeze(executable));
+            m_changed = true;
+            break;
+        }
 
         default:
             break;
index 74568a2..ef5f3f1 100644 (file)
@@ -140,9 +140,13 @@ inline CapabilityLevel canCompile(Node* node)
     case StoreBarrier:
     case FencedStoreBarrier:
     case Call:
+    case DirectCall:
     case TailCall:
+    case DirectTailCall:
     case TailCallInlinedCaller:
+    case DirectTailCallInlinedCaller:
     case Construct:
+    case DirectConstruct:
     case CallVarargs:
     case CallEval:
     case TailCallVarargs:
index 8f3550d..a657e06 100644 (file)
@@ -852,6 +852,12 @@ private:
         case Construct:
             compileCallOrConstruct();
             break;
+        case DirectCall:
+        case DirectTailCallInlinedCaller:
+        case DirectConstruct:
+        case DirectTailCall:
+            compileDirectCallOrConstruct();
+            break;
         case TailCall:
             compileTailCall();
             break;
@@ -5555,14 +5561,214 @@ private:
                         linkBuffer.link(slowCall, FunctionPtr(linkCall.executableAddress()));
 
                         callLinkInfo->setCallLocations(
-                            linkBuffer.locationOfNearCall(slowCall),
-                            linkBuffer.locationOf(targetToCheck),
+                            CodeLocationLabel(linkBuffer.locationOfNearCall(slowCall)),
+                            CodeLocationLabel(linkBuffer.locationOf(targetToCheck)),
                             linkBuffer.locationOfNearCall(fastCall));
                     });
             });
 
         setJSValue(patchpoint);
     }
+    
+    void compileDirectCallOrConstruct()
+    {
+        Node* node = m_node;
+        bool isTail = node->op() == DirectTailCall;
+        bool isConstruct = node->op() == DirectConstruct;
+        
+        ExecutableBase* executable = node->castOperand<ExecutableBase*>();
+        FunctionExecutable* functionExecutable = jsDynamicCast<FunctionExecutable*>(executable);
+        
+        unsigned numPassedArgs = node->numChildren() - 1;
+        unsigned numAllocatedArgs = numPassedArgs;
+        
+        if (functionExecutable) {
+            numAllocatedArgs = std::max(
+                numAllocatedArgs,
+                std::min(
+                    static_cast<unsigned>(functionExecutable->parameterCount()) + 1,
+                    Options::maximumDirectCallStackSize()));
+        }
+        
+        LValue jsCallee = lowJSValue(m_graph.varArgChild(node, 0));
+        
+        if (!isTail) {
+            unsigned frameSize = (CallFrame::headerSizeInRegisters + numAllocatedArgs) * sizeof(EncodedJSValue);
+            unsigned alignedFrameSize = WTF::roundUpToMultipleOf(stackAlignmentBytes(), frameSize);
+            
+            m_proc.requestCallArgAreaSizeInBytes(alignedFrameSize);
+        }
+        
+        Vector<ConstrainedValue> arguments;
+        
+        arguments.append(ConstrainedValue(jsCallee, ValueRep::SomeRegister));
+        if (!isTail) {
+            auto addArgument = [&] (LValue value, VirtualRegister reg, int offset) {
+                intptr_t offsetFromSP =
+                    (reg.offset() - CallerFrameAndPC::sizeInRegisters) * sizeof(EncodedJSValue) + offset;
+                arguments.append(ConstrainedValue(value, ValueRep::stackArgument(offsetFromSP)));
+            };
+            
+            addArgument(jsCallee, VirtualRegister(CallFrameSlot::callee), 0);
+            addArgument(m_out.constInt32(numPassedArgs), VirtualRegister(CallFrameSlot::argumentCount), PayloadOffset);
+            for (unsigned i = 0; i < numPassedArgs; ++i)
+                addArgument(lowJSValue(m_graph.varArgChild(node, 1 + i)), virtualRegisterForArgument(i), 0);
+            for (unsigned i = numPassedArgs; i < numAllocatedArgs; ++i)
+                addArgument(m_out.constInt64(JSValue::encode(jsUndefined())), virtualRegisterForArgument(i), 0);
+        } else {
+            for (unsigned i = 0; i < numPassedArgs; ++i)
+                arguments.append(ConstrainedValue(lowJSValue(m_graph.varArgChild(node, 1 + i)), ValueRep::WarmAny));
+        }
+        
+        PatchpointValue* patchpoint = m_out.patchpoint(isTail ? Void : Int64);
+        patchpoint->appendVector(arguments);
+        
+        RefPtr<PatchpointExceptionHandle> exceptionHandle = preparePatchpointForExceptions(patchpoint);
+        
+        if (isTail) {
+            // The shuffler needs tags.
+            patchpoint->append(m_tagMask, ValueRep::reg(GPRInfo::tagMaskRegister));
+            patchpoint->append(m_tagTypeNumber, ValueRep::reg(GPRInfo::tagTypeNumberRegister));
+        }
+        
+        patchpoint->clobber(RegisterSet::macroScratchRegisters());
+        if (!isTail) {
+            patchpoint->clobberLate(RegisterSet::volatileRegistersForJSCall());
+            patchpoint->resultConstraint = ValueRep::reg(GPRInfo::returnValueGPR);
+        }
+        
+        CodeOrigin codeOrigin = codeOriginDescriptionOfCallSite();
+        State* state = &m_ftlState;
+        patchpoint->setGenerator(
+            [=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
+                AllowMacroScratchRegisterUsage allowScratch(jit);
+                CallSiteIndex callSiteIndex = state->jitCode->common.addUniqueCallSiteIndex(codeOrigin);
+                
+                GPRReg calleeGPR = params[!isTail].gpr();
+
+                exceptionHandle->scheduleExitCreationForUnwind(params, callSiteIndex);
+                
+                Box<CCallHelpers::JumpList> exceptions =
+                    exceptionHandle->scheduleExitCreation(params)->jumps(jit);
+                
+                if (isTail) {
+                    CallFrameShuffleData shuffleData;
+                    shuffleData.numLocals = state->jitCode->common.frameRegisterCount;
+                    
+                    RegisterSet toSave = params.unavailableRegisters();
+                    shuffleData.callee = ValueRecovery::inGPR(calleeGPR, DataFormatCell);
+                    toSave.set(calleeGPR);
+                    for (unsigned i = 0; i < numPassedArgs; ++i) {
+                        ValueRecovery recovery = params[1 + i].recoveryForJSValue();
+                        shuffleData.args.append(recovery);
+                        recovery.forEachReg(
+                            [&] (Reg reg) {
+                                toSave.set(reg);
+                            });
+                    }
+                    for (unsigned i = numPassedArgs; i < numAllocatedArgs; ++i)
+                        shuffleData.args.append(ValueRecovery::constant(jsUndefined()));
+                    shuffleData.setupCalleeSaveRegisters(jit.codeBlock());
+                    
+                    CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo();
+                    
+                    CCallHelpers::PatchableJump patchableJump = jit.patchableJump();
+                    CCallHelpers::Label mainPath = jit.label();
+                    
+                    jit.store32(
+                        CCallHelpers::TrustedImm32(callSiteIndex.bits()),
+                        CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCount)));
+                
+                    callLinkInfo->setFrameShuffleData(shuffleData);
+                    CallFrameShuffler(jit, shuffleData).prepareForTailCall();
+                    
+                    CCallHelpers::Call call = jit.nearTailCall();
+                    
+                    jit.abortWithReason(JITDidReturnFromTailCall);
+                    
+                    CCallHelpers::Label slowPath = jit.label();
+                    patchableJump.m_jump.linkTo(slowPath, &jit);
+                    callOperation(
+                        *state, toSave, jit,
+                        node->origin.semantic, exceptions.get(), operationLinkDirectCall,
+                        InvalidGPRReg, CCallHelpers::TrustedImmPtr(callLinkInfo), calleeGPR).call();
+                    jit.jump().linkTo(mainPath, &jit);
+                    
+                    callLinkInfo->setUpCall(
+                        CallLinkInfo::DirectTailCall, node->origin.semantic, InvalidGPRReg);
+                    callLinkInfo->setExecutableDuringCompilation(executable);
+                    if (numAllocatedArgs > numPassedArgs)
+                        callLinkInfo->setMaxNumArguments(numAllocatedArgs);
+                    
+                    jit.addLinkTask(
+                        [=] (LinkBuffer& linkBuffer) {
+                            CodeLocationLabel patchableJumpLocation = linkBuffer.locationOf(patchableJump);
+                            CodeLocationNearCall callLocation = linkBuffer.locationOfNearCall(call);
+                            CodeLocationLabel slowPathLocation = linkBuffer.locationOf(slowPath);
+                            
+                            callLinkInfo->setCallLocations(
+                                patchableJumpLocation,
+                                slowPathLocation,
+                                callLocation);
+                        });
+                    return;
+                }
+                
+                CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo();
+                
+                CCallHelpers::Label mainPath = jit.label();
+
+                jit.store32(
+                    CCallHelpers::TrustedImm32(callSiteIndex.bits()),
+                    CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCount)));
+                
+                CCallHelpers::Call call = jit.nearCall();
+                jit.addPtr(
+                    CCallHelpers::TrustedImm32(-params.proc().frameSize()),
+                    GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
+                
+                callLinkInfo->setUpCall(
+                    isConstruct ? CallLinkInfo::DirectConstruct : CallLinkInfo::DirectCall,
+                    node->origin.semantic, InvalidGPRReg);
+                callLinkInfo->setExecutableDuringCompilation(executable);
+                if (numAllocatedArgs > numPassedArgs)
+                    callLinkInfo->setMaxNumArguments(numAllocatedArgs);
+                
+                params.addLatePath(
+                    [=] (CCallHelpers& jit) {
+                        AllowMacroScratchRegisterUsage allowScratch(jit);
+                        
+                        CCallHelpers::Label slowPath = jit.label();
+                        if (isX86())
+                            jit.pop(CCallHelpers::selectScratchGPR(calleeGPR));
+                        
+                        callOperation(
+                            *state, params.unavailableRegisters(), jit,
+                            node->origin.semantic, exceptions.get(), operationLinkDirectCall,
+                            InvalidGPRReg, CCallHelpers::TrustedImmPtr(callLinkInfo),
+                            calleeGPR).call();
+                        jit.jump().linkTo(mainPath, &jit);
+                        
+                        jit.addLinkTask(
+                            [=] (LinkBuffer& linkBuffer) {
+                                CodeLocationNearCall callLocation = linkBuffer.locationOfNearCall(call);
+                                CodeLocationLabel slowPathLocation = linkBuffer.locationOf(slowPath);
+                                
+                                linkBuffer.link(call, slowPathLocation);
+                                
+                                callLinkInfo->setCallLocations(
+                                    CodeLocationLabel(),
+                                    slowPathLocation,
+                                    callLocation);
+                            });
+                    });
+            });
+        
+        if (isTail)
+            patchpoint->effects.terminal = true;
+        else
+            setJSValue(patchpoint);
+    }
 
     void compileTailCall()
     {
@@ -5601,6 +5807,8 @@ private:
 
         // Prevent any of the arguments from using the scratch register.
         patchpoint->clobberEarly(RegisterSet::macroScratchRegisters());
+        
+        patchpoint->effects.terminal = true;
 
         // We don't have to tell the patchpoint that we will clobber registers, since we won't return
         // anyway.
@@ -5660,13 +5868,11 @@ private:
                         linkBuffer.link(slowCall, FunctionPtr(linkCall.executableAddress()));
 
                         callLinkInfo->setCallLocations(
-                            linkBuffer.locationOfNearCall(slowCall),
-                            linkBuffer.locationOf(targetToCheck),
+                            CodeLocationLabel(linkBuffer.locationOfNearCall(slowCall)),
+                            CodeLocationLabel(linkBuffer.locationOf(targetToCheck)),
                             linkBuffer.locationOfNearCall(fastCall));
                     });
             });
-        
-        m_out.unreachable();
     }
     
     void compileCallOrConstructVarargs()
@@ -5929,8 +6135,8 @@ private:
                         linkBuffer.link(slowCall, FunctionPtr(linkCall.executableAddress()));
                         
                         callLinkInfo->setCallLocations(
-                            linkBuffer.locationOfNearCall(slowCall),
-                            linkBuffer.locationOf(targetToCheck),
+                            CodeLocationLabel(linkBuffer.locationOfNearCall(slowCall)),
+                            CodeLocationLabel(linkBuffer.locationOf(targetToCheck)),
                             linkBuffer.locationOfNearCall(fastCall));
                     });
             });
@@ -12010,7 +12216,8 @@ private:
         CodeOrigin codeOrigin = m_node->origin.semantic;
         if (m_node->op() == TailCallInlinedCaller
             || m_node->op() == TailCallVarargsInlinedCaller
-            || m_node->op() == TailCallForwardVarargsInlinedCaller) {
+            || m_node->op() == TailCallForwardVarargsInlinedCaller
+            || m_node->op() == DirectTailCallInlinedCaller) {
             // This case arises when you have a situation like this:
             // foo makes a call to bar, bar is inlined in foo. bar makes a call
             // to baz and baz is inlined in bar. And then baz makes a tail-call to jaz,
index 6c2a1ec..91920dc 100644 (file)
@@ -870,7 +870,7 @@ failedJSONP:
     ProgramCodeBlock* codeBlock;
     {
         CodeBlock* tempCodeBlock;
-        JSObject* error = program->prepareForExecution<ProgramExecutable>(callFrame, nullptr, scope, CodeForCall, tempCodeBlock);
+        JSObject* error = program->prepareForExecution<ProgramExecutable>(vm, nullptr, scope, CodeForCall, tempCodeBlock);
         ASSERT(!throwScope.exception() || throwScope.exception() == jsDynamicCast<Exception*>(error));
         if (error)
             return checkedReturn(throwException(callFrame, throwScope, error));
@@ -925,7 +925,7 @@ JSValue Interpreter::executeCall(CallFrame* callFrame, JSObject* function, CallT
 
     if (isJSCall) {
         // Compile the callee:
-        JSObject* compileError = callData.js.functionExecutable->prepareForExecution<FunctionExecutable>(callFrame, jsCast<JSFunction*>(function), scope, CodeForCall, newCodeBlock);
+        JSObject* compileError = callData.js.functionExecutable->prepareForExecution<FunctionExecutable>(vm, jsCast<JSFunction*>(function), scope, CodeForCall, newCodeBlock);
         ASSERT(throwScope.exception() == reinterpret_cast<Exception*>(compileError));
         if (UNLIKELY(!!compileError))
             return checkedReturn(throwException(callFrame, throwScope, compileError));
@@ -989,7 +989,7 @@ JSObject* Interpreter::executeConstruct(CallFrame* callFrame, JSObject* construc
 
     if (isJSConstruct) {
         // Compile the callee:
-        JSObject* compileError = constructData.js.functionExecutable->prepareForExecution<FunctionExecutable>(callFrame, jsCast<JSFunction*>(constructor), scope, CodeForConstruct, newCodeBlock);
+        JSObject* compileError = constructData.js.functionExecutable->prepareForExecution<FunctionExecutable>(vm, jsCast<JSFunction*>(constructor), scope, CodeForConstruct, newCodeBlock);
         if (UNLIKELY(!!compileError))
             return checkedReturn(throwException(callFrame, throwScope, compileError));
 
@@ -1033,7 +1033,7 @@ CallFrameClosure Interpreter::prepareForRepeatCall(FunctionExecutable* functionE
 
     // Compile the callee:
     CodeBlock* newCodeBlock;
-    JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(callFrame, function, scope, CodeForCall, newCodeBlock);
+    JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(vm, function, scope, CodeForCall, newCodeBlock);
     if (error) {
         throwException(callFrame, throwScope, error);
         return CallFrameClosure();
@@ -1112,7 +1112,7 @@ JSValue Interpreter::execute(EvalExecutable* eval, CallFrame* callFrame, JSValue
     EvalCodeBlock* codeBlock;
     {
         CodeBlock* tempCodeBlock;
-        JSObject* compileError = eval->prepareForExecution<EvalExecutable>(callFrame, nullptr, scope, CodeForCall, tempCodeBlock);
+        JSObject* compileError = eval->prepareForExecution<EvalExecutable>(vm, nullptr, scope, CodeForCall, tempCodeBlock);
         if (UNLIKELY(!!compileError))
             return checkedReturn(throwException(callFrame, throwScope, compileError));
         codeBlock = jsCast<EvalCodeBlock*>(tempCodeBlock);
@@ -1194,7 +1194,7 @@ JSValue Interpreter::execute(ModuleProgramExecutable* executable, CallFrame* cal
     ModuleProgramCodeBlock* codeBlock;
     {
         CodeBlock* tempCodeBlock;
-        JSObject* compileError = executable->prepareForExecution<ModuleProgramExecutable>(callFrame, nullptr, scope, CodeForCall, tempCodeBlock);
+        JSObject* compileError = executable->prepareForExecution<ModuleProgramExecutable>(vm, nullptr, scope, CodeForCall, tempCodeBlock);
         if (UNLIKELY(!!compileError))
             return checkedReturn(throwException(callFrame, throwScope, compileError));
         codeBlock = jsCast<ModuleProgramCodeBlock*>(tempCodeBlock);
index 237d19a..542a471 100644 (file)
@@ -760,8 +760,9 @@ CompilationResult JIT::link()
     for (unsigned i = 0; i < m_callCompilationInfo.size(); ++i) {
         CallCompilationInfo& compilationInfo = m_callCompilationInfo[i];
         CallLinkInfo& info = *compilationInfo.callLinkInfo;
-        info.setCallLocations(patchBuffer.locationOfNearCall(compilationInfo.callReturnLocation),
-            patchBuffer.locationOf(compilationInfo.hotPathBegin),
+        info.setCallLocations(
+            CodeLocationLabel(patchBuffer.locationOfNearCall(compilationInfo.callReturnLocation)),
+            CodeLocationLabel(patchBuffer.locationOf(compilationInfo.hotPathBegin)),
             patchBuffer.locationOfNearCall(compilationInfo.hotPathOther));
     }
 
index 4b39e85..64ed087 100644 (file)
@@ -81,12 +81,9 @@ void JIT::compileSetupVarargsFrame(OpcodeID opcode, Instruction* instruction, Ca
 
     // Profile the argument count.
     load32(Address(regT1, CallFrameSlot::argumentCount * static_cast<int>(sizeof(Register)) + PayloadOffset), regT2);
-    load8(info->addressOfMaxNumArguments(), regT0);
+    load32(info->addressOfMaxNumArguments(), regT0);
     Jump notBiggest = branch32(Above, regT0, regT2);
-    Jump notSaturated = branch32(BelowOrEqual, regT2, TrustedImm32(255));
-    move(TrustedImm32(255), regT2);
-    notSaturated.link(this);
-    store8(regT2, info->addressOfMaxNumArguments());
+    store32(regT2, info->addressOfMaxNumArguments());
     notBiggest.link(this);
     
     // Initialize 'this'.
index c7b439c..573b062 100644 (file)
@@ -171,12 +171,9 @@ void JIT::compileSetupVarargsFrame(OpcodeID opcode, Instruction* instruction, Ca
 
     // Profile the argument count.
     load32(Address(regT1, CallFrameSlot::argumentCount * static_cast<int>(sizeof(Register)) + PayloadOffset), regT2);
-    load8(info->addressOfMaxNumArguments(), regT0);
+    load32(info->addressOfMaxNumArguments(), regT0);
     Jump notBiggest = branch32(Above, regT0, regT2);
-    Jump notSaturated = branch32(BelowOrEqual, regT2, TrustedImm32(255));
-    move(TrustedImm32(255), regT2);
-    notSaturated.link(this);
-    store8(regT2, info->addressOfMaxNumArguments());
+    store32(regT2, info->addressOfMaxNumArguments());
     notBiggest.link(this);
     
     // Initialize 'this'.
index 8b5269f..8f8d4ad 100644 (file)
@@ -861,6 +861,8 @@ SlowPathReturnType JIT_OPERATION operationLinkCall(ExecState* execCallee, CallLi
     CodeSpecializationKind kind = callLinkInfo->specializationKind();
     NativeCallFrameTracer tracer(vm, exec);
     
+    RELEASE_ASSERT(!callLinkInfo->isDirect());
+    
     JSValue calleeAsValue = execCallee->calleeAsValue();
     JSCell* calleeAsFunctionCell = getJSFunction(calleeAsValue);
     if (!calleeAsFunctionCell) {
@@ -902,7 +904,7 @@ SlowPathReturnType JIT_OPERATION operationLinkCall(ExecState* execCallee, CallLi
         }
 
         CodeBlock** codeBlockSlot = execCallee->addressOfCodeBlock();
-        JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(execCallee, callee, scope, kind, *codeBlockSlot);
+        JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(*vm, callee, scope, kind, *codeBlockSlot);
         ASSERT(throwScope.exception() == reinterpret_cast<Exception*>(error));
         if (error) {
             throwException(exec, throwScope, error);
@@ -926,6 +928,60 @@ SlowPathReturnType JIT_OPERATION operationLinkCall(ExecState* execCallee, CallLi
     return encodeResult(codePtr.executableAddress(), reinterpret_cast<void*>(callLinkInfo->callMode() == CallMode::Tail ? ReuseTheFrame : KeepTheFrame));
 }
 
+void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee)
+{
+    VM* vm = &exec->vm();
+    auto throwScope = DECLARE_THROW_SCOPE(*vm);
+
+    CodeSpecializationKind kind = callLinkInfo->specializationKind();
+    NativeCallFrameTracer tracer(vm, exec);
+    
+    RELEASE_ASSERT(callLinkInfo->isDirect());
+    
+    // This would happen if the executable died during GC but the CodeBlock did not die. That should
+    // not happen because the CodeBlock should have a weak reference to any executable it uses for
+    // this purpose.
+    RELEASE_ASSERT(callLinkInfo->executable());
+    
+    // Having a CodeBlock indicates that this is linked. We shouldn't be taking this path if it's
+    // linked.
+    RELEASE_ASSERT(!callLinkInfo->codeBlock());
+    
+    // We just don't support this yet.
+    RELEASE_ASSERT(!callLinkInfo->isVarargs());
+    
+    ExecutableBase* executable = callLinkInfo->executable();
+    RELEASE_ASSERT(callee->executable() == callLinkInfo->executable());
+
+    JSScope* scope = callee->scopeUnchecked();
+
+    MacroAssemblerCodePtr codePtr;
+    CodeBlock* codeBlock = nullptr;
+    if (executable->isHostFunction())
+        codePtr = executable->entrypointFor(kind, MustCheckArity);
+    else {
+        FunctionExecutable* functionExecutable = static_cast<FunctionExecutable*>(executable);
+
+        RELEASE_ASSERT(isCall(kind) || functionExecutable->constructAbility() != ConstructAbility::CannotConstruct);
+        
+        JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(*vm, callee, scope, kind, codeBlock);
+        ASSERT(throwScope.exception() == reinterpret_cast<Exception*>(error));
+        if (error) {
+            throwException(exec, throwScope, error);
+            return;
+        }
+        ArityCheckMode arity;
+        unsigned argumentStackSlots = callLinkInfo->maxNumArguments();
+        if (argumentStackSlots < static_cast<size_t>(codeBlock->numParameters()))
+            arity = MustCheckArity;
+        else
+            arity = ArityCheckNotRequired;
+        codePtr = functionExecutable->entrypointFor(kind, arity);
+    }
+    
+    linkDirectFor(exec, *callLinkInfo, codeBlock, codePtr);
+}
+
 inline SlowPathReturnType virtualForWithFunction(
     ExecState* execCallee, CallLinkInfo* callLinkInfo, JSCell*& calleeAsFunctionCell)
 {
@@ -960,7 +1016,7 @@ inline SlowPathReturnType virtualForWithFunction(
             }
 
             CodeBlock** codeBlockSlot = execCallee->addressOfCodeBlock();
-            JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(execCallee, function, scope, kind, *codeBlockSlot);
+            JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(*vm, function, scope, kind, *codeBlockSlot);
             if (error) {
                 throwException(exec, throwScope, error);
                 return encodeResult(
index 338a40a..adfef78 100644 (file)
@@ -255,6 +255,7 @@ typedef void (JIT_OPERATION *V_JITOperation_ECJJ)(ExecState*, JSCell*, EncodedJS
 typedef void (JIT_OPERATION *V_JITOperation_ECPSPS)(ExecState*, JSCell*, void*, size_t, void*, size_t);
 typedef void (JIT_OPERATION *V_JITOperation_ECZ)(ExecState*, JSCell*, int32_t);
 typedef void (JIT_OPERATION *V_JITOperation_ECC)(ExecState*, JSCell*, JSCell*);
+typedef void (JIT_OPERATION *V_JITOperation_ECliJsf)(ExecState*, CallLinkInfo*, JSFunction*);
 typedef void (JIT_OPERATION *V_JITOperation_EZSymtabJ)(ExecState*, int32_t, SymbolTable*, EncodedJSValue);
 typedef void (JIT_OPERATION *V_JITOperation_EJ)(ExecState*, EncodedJSValue);
 typedef void (JIT_OPERATION *V_JITOperation_EJCI)(ExecState*, EncodedJSValue, JSCell*, UniquedStringImpl*);
@@ -353,6 +354,7 @@ void JIT_OPERATION operationPutByValGeneric(ExecState*, EncodedJSValue, EncodedJ
 void JIT_OPERATION operationDirectPutByValGeneric(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, ByValInfo*) WTF_INTERNAL;
 EncodedJSValue JIT_OPERATION operationCallEval(ExecState*, ExecState*) WTF_INTERNAL;
 SlowPathReturnType JIT_OPERATION operationLinkCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
+void JIT_OPERATION operationLinkDirectCall(ExecState*, CallLinkInfo*, JSFunction*) WTF_INTERNAL;
 SlowPathReturnType JIT_OPERATION operationLinkPolymorphicCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 SlowPathReturnType JIT_OPERATION operationVirtualCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 
index 1020d35..90691f2 100644 (file)
@@ -583,6 +583,28 @@ void linkFor(
     linkSlowFor(vm, callLinkInfo);
 }
 
+void linkDirectFor(
+    ExecState* exec, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock,
+    MacroAssemblerCodePtr codePtr)
+{
+    ASSERT(!callLinkInfo.stub());
+    
+    CodeBlock* callerCodeBlock = exec->codeBlock();
+
+    VM* vm = callerCodeBlock->vm();
+    
+    ASSERT(!callLinkInfo.isLinked());
+    callLinkInfo.setCodeBlock(*vm, callerCodeBlock, jsCast<FunctionCodeBlock*>(calleeCodeBlock));
+    if (shouldDumpDisassemblyFor(callerCodeBlock))
+        dataLog("Linking call in ", *callerCodeBlock, " at ", callLinkInfo.codeOrigin(), " to ", pointerDump(calleeCodeBlock), ", entrypoint at ", codePtr, "\n");
+    if (callLinkInfo.callType() == CallLinkInfo::DirectTailCall)
+        MacroAssembler::repatchJumpToNop(callLinkInfo.patchableJump());
+    MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), CodeLocationLabel(codePtr));
+    
+    if (calleeCodeBlock)
+        calleeCodeBlock->linkIncomingCall(exec, &callLinkInfo);
+}
+
 void linkSlowFor(
     ExecState* exec, CallLinkInfo& callLinkInfo)
 {
@@ -594,12 +616,20 @@ void linkSlowFor(
 
 static void revertCall(VM* vm, CallLinkInfo& callLinkInfo, MacroAssemblerCodeRef codeRef)
 {
-    MacroAssembler::revertJumpReplacementToBranchPtrWithPatch(
-        MacroAssembler::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin()),
-        static_cast<MacroAssembler::RegisterID>(callLinkInfo.calleeGPR()), 0);
-    linkSlowFor(vm, callLinkInfo, codeRef);
+    if (callLinkInfo.isDirect()) {
+        callLinkInfo.clearCodeBlock();
+        if (callLinkInfo.callType() == CallLinkInfo::DirectTailCall)
+            MacroAssembler::repatchJump(callLinkInfo.patchableJump(), callLinkInfo.slowPathStart());
+        else
+            MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), callLinkInfo.slowPathStart());
+    } else {
+        MacroAssembler::revertJumpReplacementToBranchPtrWithPatch(
+            MacroAssembler::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin()),
+            static_cast<MacroAssembler::RegisterID>(callLinkInfo.calleeGPR()), 0);
+        linkSlowFor(vm, callLinkInfo, codeRef);
+        callLinkInfo.clearCallee();
+    }
     callLinkInfo.clearSeen();
-    callLinkInfo.clearCallee();
     callLinkInfo.clearStub();
     callLinkInfo.clearSlowStub();
     if (callLinkInfo.isOnList())
@@ -609,7 +639,7 @@ static void revertCall(VM* vm, CallLinkInfo& callLinkInfo, MacroAssemblerCodeRef
 void unlinkFor(VM& vm, CallLinkInfo& callLinkInfo)
 {
     if (Options::dumpDisassembly())
-        dataLog("Unlinking call from ", callLinkInfo.callReturnLocation(), "\n");
+        dataLog("Unlinking call at ", callLinkInfo.hotPathOther(), "\n");
     
     revertCall(&vm, callLinkInfo, vm.getCTIStub(linkCallThunkGenerator));
 }
index f46fc67..e176768 100644 (file)
@@ -46,6 +46,7 @@ void repatchPutByID(ExecState*, JSValue, Structure*, const Identifier&, const Pu
 void buildPutByIdList(ExecState*, JSValue, Structure*, const Identifier&, const PutPropertySlot&, StructureStubInfo&, PutKind);
 void repatchIn(ExecState*, JSCell*, const Identifier&, bool wasFound, const PropertySlot&, StructureStubInfo&);
 void linkFor(ExecState*, CallLinkInfo&, CodeBlock*, JSFunction* callee, MacroAssemblerCodePtr);
+void linkDirectFor(ExecState*, CallLinkInfo&, CodeBlock*, MacroAssemblerCodePtr);
 void linkSlowFor(ExecState*, CallLinkInfo&);
 void unlinkFor(VM&, CallLinkInfo&);
 void linkVirtualFor(ExecState*, CallLinkInfo&);
index 00012ac..8758251 100644 (file)
@@ -1275,7 +1275,7 @@ inline SlowPathReturnType setUpCall(ExecState* execCallee, Instruction* pc, Code
             LLINT_CALL_THROW(exec, createNotAConstructorError(exec, callee));
 
         CodeBlock** codeBlockSlot = execCallee->addressOfCodeBlock();
-        JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(execCallee, callee, scope, kind, *codeBlockSlot);
+        JSObject* error = functionExecutable->prepareForExecution<FunctionExecutable>(vm, callee, scope, kind, *codeBlockSlot);
         if (error)
             LLINT_CALL_THROW(exec, error);
         codeBlock = *codeBlockSlot;
index d1438ba..347cfde 100644 (file)
@@ -398,13 +398,12 @@ static void setupJIT(VM& vm, CodeBlock* codeBlock)
 }
 
 JSObject* ScriptExecutable::prepareForExecutionImpl(
-    ExecState* exec, JSFunction* function, JSScope* scope, CodeSpecializationKind kind, CodeBlock*& resultCodeBlock)
+    VM& vm, JSFunction* function, JSScope* scope, CodeSpecializationKind kind, CodeBlock*& resultCodeBlock)
 {
-    VM& vm = exec->vm();
     DeferGCForAWhile deferGC(vm.heap);
 
     if (vm.getAndClearFailNextNewCodeBlock())
-        return createError(exec->callerFrame(), ASCIILiteral("Forced Failure"));
+        return createError(scope->globalObject()->globalExec(), ASCIILiteral("Forced Failure"));
 
     JSObject* exception = 0;
     CodeBlock* codeBlock = newCodeBlockFor(kind, function, scope, exception);
@@ -422,7 +421,7 @@ JSObject* ScriptExecutable::prepareForExecutionImpl(
     else
         setupJIT(vm, codeBlock);
     
-    installCode(*codeBlock->vm(), codeBlock, codeBlock->codeType(), codeBlock->specializationKind());
+    installCode(vm, codeBlock, codeBlock->codeType(), codeBlock->specializationKind());
     return nullptr;
 }
 
index 8191ac1..9c35596 100644 (file)
@@ -371,13 +371,13 @@ public:
     // to point to it. This forces callers to have a CodeBlock* in a register or on the stack that will be marked
     // by conservative GC if a GC happens after we create the CodeBlock.
     template <typename ExecutableType>
-    JSObject* prepareForExecution(ExecState*, JSFunction*, JSScope*, CodeSpecializationKind, CodeBlock*& resultCodeBlock);
+    JSObject* prepareForExecution(VM&, JSFunction*, JSScope*, CodeSpecializationKind, CodeBlock*& resultCodeBlock);
 
     template <typename Functor> void forEachCodeBlock(Functor&&);
 
 private:
     friend class ExecutableBase;
-    JSObject* prepareForExecutionImpl(ExecState*, JSFunction*, JSScope*, CodeSpecializationKind, CodeBlock*&);
+    JSObject* prepareForExecutionImpl(VM&, JSFunction*, JSScope*, CodeSpecializationKind, CodeBlock*&);
 
 protected:
     ScriptExecutable(Structure*, VM&, const SourceCode&, bool isInStrictContext, DerivedContextType, bool isInArrowFunctionContext, EvalContextType, Intrinsic);
@@ -735,7 +735,7 @@ public:
 
     DECLARE_INFO;
 
-    void prepareForExecution(ExecState*);
+    void prepareForExecution(VM&);
 
     WebAssemblyCodeBlock* codeBlockForCall()
     {
@@ -757,7 +757,7 @@ private:
 #endif
 
 template <typename ExecutableType>
-JSObject* ScriptExecutable::prepareForExecution(ExecState* exec, JSFunction* function, JSScope* scope, CodeSpecializationKind kind, CodeBlock*& resultCodeBlock)
+JSObject* ScriptExecutable::prepareForExecution(VM& vm, JSFunction* function, JSScope* scope, CodeSpecializationKind kind, CodeBlock*& resultCodeBlock)
 {
     if (hasJITCodeFor(kind)) {
         if (std::is_same<ExecutableType, EvalExecutable>::value)
@@ -772,7 +772,7 @@ JSObject* ScriptExecutable::prepareForExecution(ExecState* exec, JSFunction* fun
             RELEASE_ASSERT_NOT_REACHED();
         return nullptr;
     }
-    return prepareForExecutionImpl(exec, function, scope, kind, resultCodeBlock);
+    return prepareForExecutionImpl(vm, function, scope, kind, resultCodeBlock);
 }
 
 } // namespace JSC
index 530bf1c..201bc0b 100644 (file)
@@ -301,6 +301,8 @@ typedef const char* optionString;
     v(double, structureCheckVoteRatioForHoisting, 1, Normal, nullptr) \
     v(double, checkArrayVoteRatioForHoisting, 1, Normal, nullptr) \
     \
+    v(unsigned, maximumDirectCallStackSize, 200, Normal, nullptr) \
+    \
     v(unsigned, minimumNumberOfScansBetweenRebalance, 100, Normal, nullptr) \
     v(unsigned, numberOfGCMarkers, computeNumberOfGCMarkers(7), Normal, nullptr) \
     v(unsigned, opaqueRootMergeThreshold, 1000, Normal, nullptr) \