DFG should have fast virtual calls
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 12 Jul 2012 00:12:03 +0000 (00:12 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 12 Jul 2012 00:12:03 +0000 (00:12 +0000)
https://bugs.webkit.org/show_bug.cgi?id=90924

Source/JavaScriptCore:

Reviewed by Gavin Barraclough.

Implements virtual call support in the style of the old JIT, with the
caveat that we still use the same slow path for both InternalFunction
calls and JSFunction calls. Also rationalized the way that our
CodeOrigin indices tie into exception checks (previously it was a
strange one-to-one mapping with fairly limited assertions; now it's a
one-to-many mapping for CodeOrigins to exception checks, respectively).
I also took the opportunity to clean up
CallLinkInfo::callReturnLocation, which previously was either a Call or
a NearCall. Now it's just a NearCall. As well, exceptions during slow
path call resolution are now handled by returning an exception throwing
thunk rather than returning null. And finally, I made a few things
public that were previously private-with-lots-of-friends, because I
truly despise the thought of listing each thunk generating function as
a friend of JSValue and friends.

* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::unlink):
* bytecode/CallLinkInfo.h:
(CallLinkInfo):
* bytecode/CodeOrigin.h:
(JSC::CodeOrigin::CodeOrigin):
(JSC::CodeOrigin::isSet):
* dfg/DFGAssemblyHelpers.h:
(JSC::DFG::AssemblyHelpers::AssemblyHelpers):
* dfg/DFGCCallHelpers.h:
(JSC::DFG::CCallHelpers::CCallHelpers):
* dfg/DFGGPRInfo.h:
(GPRInfo):
* dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::link):
(JSC::DFG::JITCompiler::compileFunction):
* dfg/DFGJITCompiler.h:
(JSC::DFG::CallBeginToken::CallBeginToken):
(JSC::DFG::CallBeginToken::~CallBeginToken):
(CallBeginToken):
(JSC::DFG::CallBeginToken::set):
(JSC::DFG::CallBeginToken::registerWithExceptionCheck):
(JSC::DFG::CallBeginToken::codeOrigin):
(JSC::DFG::CallExceptionRecord::CallExceptionRecord):
(CallExceptionRecord):
(JSC::DFG::JITCompiler::currentCodeOriginIndex):
(JITCompiler):
(JSC::DFG::JITCompiler::beginCall):
(JSC::DFG::JITCompiler::notifyCall):
(JSC::DFG::JITCompiler::prepareForExceptionCheck):
(JSC::DFG::JITCompiler::addExceptionCheck):
(JSC::DFG::JITCompiler::addFastExceptionCheck):
* dfg/DFGOperations.cpp:
* dfg/DFGRepatch.cpp:
(JSC::DFG::dfgLinkFor):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::appendCallWithExceptionCheck):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
* dfg/DFGThunks.cpp:
(JSC::DFG::emitPointerValidation):
(DFG):
(JSC::DFG::throwExceptionFromCallSlowPathGenerator):
(JSC::DFG::slowPathFor):
(JSC::DFG::linkForThunkGenerator):
(JSC::DFG::linkCallThunkGenerator):
(JSC::DFG::linkConstructThunkGenerator):
(JSC::DFG::virtualForThunkGenerator):
(JSC::DFG::virtualCallThunkGenerator):
(JSC::DFG::virtualConstructThunkGenerator):
* dfg/DFGThunks.h:
(DFG):
* jit/JIT.cpp:
(JSC::JIT::privateCompile):
(JSC::JIT::linkFor):
* runtime/Executable.h:
(ExecutableBase):
(JSC::ExecutableBase::offsetOfJITCodeFor):
(JSC::ExecutableBase::offsetOfNumParametersFor):
* runtime/JSValue.h:
(JSValue):

LayoutTests:

Rubber stamped by Oliver Hunt.

This changes which piece of code appears on top of the stack at the point of a stack
overflow. As far as Oliver and I can tell, it doesn't matter, so I just rebased the
test.

* fast/js/stack-trace-expected.txt:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@122392 268f45cc-cd09-0410-ab3c-d52691b4dbfc

21 files changed:
LayoutTests/ChangeLog
LayoutTests/fast/js/stack-trace-expected.txt
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/bytecode/CallLinkInfo.cpp
Source/JavaScriptCore/bytecode/CallLinkInfo.h
Source/JavaScriptCore/bytecode/CodeOrigin.h
Source/JavaScriptCore/dfg/DFGAssemblyHelpers.h
Source/JavaScriptCore/dfg/DFGCCallHelpers.h
Source/JavaScriptCore/dfg/DFGGPRInfo.h
Source/JavaScriptCore/dfg/DFGJITCompiler.cpp
Source/JavaScriptCore/dfg/DFGJITCompiler.h
Source/JavaScriptCore/dfg/DFGOperations.cpp
Source/JavaScriptCore/dfg/DFGRepatch.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp
Source/JavaScriptCore/dfg/DFGThunks.cpp
Source/JavaScriptCore/dfg/DFGThunks.h
Source/JavaScriptCore/jit/JIT.cpp
Source/JavaScriptCore/runtime/Executable.h
Source/JavaScriptCore/runtime/JSValue.h

index d00842e..b476ba8 100644 (file)
@@ -1,3 +1,16 @@
+2012-07-11  Filip Pizlo  <fpizlo@apple.com>
+
+        DFG should have fast virtual calls
+        https://bugs.webkit.org/show_bug.cgi?id=90924
+
+        Rubber stamped by Oliver Hunt.
+
+        This changes which piece of code appears on top of the stack at the point of a stack
+        overflow. As far as Oliver and I can tell, it doesn't matter, so I just rebased the
+        test.
+
+        * fast/js/stack-trace-expected.txt:
+
 2012-07-11  Adam Barth  <abarth@webkit.org>
 
         Unreviewed.  Remove empty directories.
index f5bf2e7..7bfeec3 100644 (file)
@@ -151,7 +151,7 @@ On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE
     99   selfRecursive1 at stack-trace.js:52
 
 --> Stack Trace:
-    0   selfRecursive2 at stack-trace.js:58
+    0   selfRecursive2 at stack-trace.js:62
     1   selfRecursive2 at stack-trace.js:62
     2   selfRecursive2 at stack-trace.js:62
     3   selfRecursive2 at stack-trace.js:62
index 3bdc73e..d67827d 100644 (file)
@@ -1,5 +1,91 @@
 2012-07-11  Filip Pizlo  <fpizlo@apple.com>
 
+        DFG should have fast virtual calls
+        https://bugs.webkit.org/show_bug.cgi?id=90924
+
+        Reviewed by Gavin Barraclough.
+        
+        Implements virtual call support in the style of the old JIT, with the
+        caveat that we still use the same slow path for both InternalFunction
+        calls and JSFunction calls. Also rationalized the way that our
+        CodeOrigin indices tie into exception checks (previously it was a
+        strange one-to-one mapping with fairly limited assertions; now it's a
+        one-to-many mapping for CodeOrigins to exception checks, respectively).
+        I also took the opportunity to clean up
+        CallLinkInfo::callReturnLocation, which previously was either a Call or
+        a NearCall. Now it's just a NearCall. As well, exceptions during slow
+        path call resolution are now handled by returning an exception throwing
+        thunk rather than returning null. And finally, I made a few things
+        public that were previously private-with-lots-of-friends, because I
+        truly despise the thought of listing each thunk generating function as
+        a friend of JSValue and friends.
+        
+        * bytecode/CallLinkInfo.cpp:
+        (JSC::CallLinkInfo::unlink):
+        * bytecode/CallLinkInfo.h:
+        (CallLinkInfo):
+        * bytecode/CodeOrigin.h:
+        (JSC::CodeOrigin::CodeOrigin):
+        (JSC::CodeOrigin::isSet):
+        * dfg/DFGAssemblyHelpers.h:
+        (JSC::DFG::AssemblyHelpers::AssemblyHelpers):
+        * dfg/DFGCCallHelpers.h:
+        (JSC::DFG::CCallHelpers::CCallHelpers):
+        * dfg/DFGGPRInfo.h:
+        (GPRInfo):
+        * dfg/DFGJITCompiler.cpp:
+        (JSC::DFG::JITCompiler::link):
+        (JSC::DFG::JITCompiler::compileFunction):
+        * dfg/DFGJITCompiler.h:
+        (JSC::DFG::CallBeginToken::CallBeginToken):
+        (JSC::DFG::CallBeginToken::~CallBeginToken):
+        (CallBeginToken):
+        (JSC::DFG::CallBeginToken::set):
+        (JSC::DFG::CallBeginToken::registerWithExceptionCheck):
+        (JSC::DFG::CallBeginToken::codeOrigin):
+        (JSC::DFG::CallExceptionRecord::CallExceptionRecord):
+        (CallExceptionRecord):
+        (JSC::DFG::JITCompiler::currentCodeOriginIndex):
+        (JITCompiler):
+        (JSC::DFG::JITCompiler::beginCall):
+        (JSC::DFG::JITCompiler::notifyCall):
+        (JSC::DFG::JITCompiler::prepareForExceptionCheck):
+        (JSC::DFG::JITCompiler::addExceptionCheck):
+        (JSC::DFG::JITCompiler::addFastExceptionCheck):
+        * dfg/DFGOperations.cpp:
+        * dfg/DFGRepatch.cpp:
+        (JSC::DFG::dfgLinkFor):
+        * dfg/DFGSpeculativeJIT.h:
+        (JSC::DFG::SpeculativeJIT::appendCallWithExceptionCheck):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        * dfg/DFGThunks.cpp:
+        (JSC::DFG::emitPointerValidation):
+        (DFG):
+        (JSC::DFG::throwExceptionFromCallSlowPathGenerator):
+        (JSC::DFG::slowPathFor):
+        (JSC::DFG::linkForThunkGenerator):
+        (JSC::DFG::linkCallThunkGenerator):
+        (JSC::DFG::linkConstructThunkGenerator):
+        (JSC::DFG::virtualForThunkGenerator):
+        (JSC::DFG::virtualCallThunkGenerator):
+        (JSC::DFG::virtualConstructThunkGenerator):
+        * dfg/DFGThunks.h:
+        (DFG):
+        * jit/JIT.cpp:
+        (JSC::JIT::privateCompile):
+        (JSC::JIT::linkFor):
+        * runtime/Executable.h:
+        (ExecutableBase):
+        (JSC::ExecutableBase::offsetOfJITCodeFor):
+        (JSC::ExecutableBase::offsetOfNumParametersFor):
+        * runtime/JSValue.h:
+        (JSValue):
+
+2012-07-11  Filip Pizlo  <fpizlo@apple.com>
+
         Accidentally used the wrong license (3-clause instead of 2-clause) in some
         files I just committed.
 
index 4c108ec..4933a49 100644 (file)
@@ -27,6 +27,7 @@
 #include "CallLinkInfo.h"
 
 #include "DFGOperations.h"
+#include "DFGThunks.h"
 #include "RepatchBuffer.h"
 
 #if ENABLE(JIT)
@@ -38,12 +39,12 @@ void CallLinkInfo::unlink(JSGlobalData& globalData, RepatchBuffer& repatchBuffer
     
     if (isDFG) {
 #if ENABLE(DFG_JIT)
-        repatchBuffer.relink(CodeLocationCall(callReturnLocation), callType == Construct ? DFG::operationLinkConstruct : DFG::operationLinkCall);
+        repatchBuffer.relink(callReturnLocation, (callType == Construct ? globalData.getCTIStub(DFG::linkConstructThunkGenerator) : globalData.getCTIStub(DFG::linkCallThunkGenerator)).code());
 #else
         ASSERT_NOT_REACHED();
 #endif
     } else
-        repatchBuffer.relink(CodeLocationNearCall(callReturnLocation), callType == Construct ? globalData.jitStubs->ctiVirtualConstructLink() : globalData.jitStubs->ctiVirtualCallLink());
+        repatchBuffer.relink(callReturnLocation, callType == Construct ? globalData.jitStubs->ctiVirtualConstructLink() : globalData.jitStubs->ctiVirtualCallLink());
     hasSeenShouldRepatch = false;
     callee.clear();
 
index 44d50a9..4a78e5d 100644 (file)
@@ -65,7 +65,7 @@ struct CallLinkInfo : public BasicRawSentinelNode<CallLinkInfo> {
             remove();
     }
 
-    CodeLocationLabel callReturnLocation; // it's a near call in the old JIT, or a normal call in DFG
+    CodeLocationNearCall callReturnLocation;
     CodeLocationDataLabelPtr hotPathBegin;
     CodeLocationNearCall hotPathOther;
     JITWriteBarrier<JSFunction> callee;
index 034e48f..c9c0f70 100644 (file)
@@ -39,6 +39,8 @@ class ExecutableBase;
 class JSFunction;
 
 struct CodeOrigin {
+    static const unsigned maximumBytecodeIndex = (1u << 29) - 1;
+    
     // Bytecode offset that you'd use to re-execute this instruction.
     unsigned bytecodeIndex : 29;
     // Bytecode offset corresponding to the opcode that gives the result (needed to handle
@@ -48,7 +50,7 @@ struct CodeOrigin {
     InlineCallFrame* inlineCallFrame;
     
     CodeOrigin()
-        : bytecodeIndex(std::numeric_limits<uint32_t>::max())
+        : bytecodeIndex(maximumBytecodeIndex)
         , valueProfileOffset(0)
         , inlineCallFrame(0)
     {
@@ -59,11 +61,11 @@ struct CodeOrigin {
         , valueProfileOffset(valueProfileOffset)
         , inlineCallFrame(inlineCallFrame)
     {
-        ASSERT(bytecodeIndex < (1u << 29));
+        ASSERT(bytecodeIndex <= maximumBytecodeIndex);
         ASSERT(valueProfileOffset < (1u << 3));
     }
     
-    bool isSet() const { return bytecodeIndex != std::numeric_limits<uint32_t>::max(); }
+    bool isSet() const { return bytecodeIndex != maximumBytecodeIndex; }
     
     unsigned bytecodeIndexForValueProfile() const
     {
index 4bea292..57f758c 100644 (file)
@@ -46,12 +46,13 @@ public:
     AssemblyHelpers(JSGlobalData* globalData, CodeBlock* codeBlock)
         : m_globalData(globalData)
         , m_codeBlock(codeBlock)
-        , m_baselineCodeBlock(codeBlock->baselineVersion())
+        , m_baselineCodeBlock(codeBlock ? codeBlock->baselineVersion() : 0)
     {
-        ASSERT(m_codeBlock);
-        ASSERT(m_baselineCodeBlock);
-        ASSERT(!m_baselineCodeBlock->alternative());
-        ASSERT(m_baselineCodeBlock->getJITType() == JITCode::BaselineJIT);
+        if (m_codeBlock) {
+            ASSERT(m_baselineCodeBlock);
+            ASSERT(!m_baselineCodeBlock->alternative());
+            ASSERT(m_baselineCodeBlock->getJITType() == JITCode::BaselineJIT);
+        }
     }
     
     CodeBlock* codeBlock() { return m_codeBlock; }
index 9c1718b..5985b25 100644 (file)
@@ -37,7 +37,7 @@ namespace JSC { namespace DFG {
 
 class CCallHelpers : public AssemblyHelpers {
 public:
-    CCallHelpers(JSGlobalData* globalData, CodeBlock* codeBlock)
+    CCallHelpers(JSGlobalData* globalData, CodeBlock* codeBlock = 0)
         : AssemblyHelpers(globalData, codeBlock)
     {
     }
index 89faef9..23f1697 100644 (file)
@@ -273,6 +273,7 @@ public:
     static const GPRReg argumentGPR1 = X86Registers::edx; // regT1
     static const GPRReg nonArgGPR0 = X86Registers::eax; // regT0
     static const GPRReg nonArgGPR1 = X86Registers::ebx; // regT3
+    static const GPRReg nonArgGPR2 = X86Registers::esi; // regT4
     static const GPRReg returnValueGPR = X86Registers::eax; // regT0
     static const GPRReg returnValueGPR2 = X86Registers::edx; // regT1
     static const GPRReg nonPreservedNonReturnGPR = X86Registers::ecx;
@@ -344,6 +345,7 @@ public:
     static const GPRReg argumentGPR5 = X86Registers::r9;  // regT7
     static const GPRReg nonArgGPR0 = X86Registers::eax; // regT0
     static const GPRReg nonArgGPR1 = X86Registers::ebx; // regT3
+    static const GPRReg nonArgGPR2 = X86Registers::r10; // regT8
     static const GPRReg returnValueGPR = X86Registers::eax; // regT0
     static const GPRReg returnValueGPR2 = X86Registers::edx; // regT1
     static const GPRReg nonPreservedNonReturnGPR = X86Registers::esi;
@@ -416,6 +418,7 @@ public:
     static const GPRReg argumentGPR3 = ARMRegisters::r3; // FIXME!
     static const GPRReg nonArgGPR0 = ARMRegisters::r4; // regT3
     static const GPRReg nonArgGPR1 = ARMRegisters::r8; // regT4
+    static const GPRReg nonArgGPR2 = ARMRegisters::r9; // regT5
     static const GPRReg returnValueGPR = ARMRegisters::r0; // regT0
     static const GPRReg returnValueGPR2 = ARMRegisters::r1; // regT1
     static const GPRReg nonPreservedNonReturnGPR = ARMRegisters::r2;
index 497fc34..2ebee13 100644 (file)
@@ -153,7 +153,6 @@ void JITCompiler::link(LinkBuffer& linkBuffer)
         unsigned returnAddressOffset = linkBuffer.returnAddressOffset(m_exceptionChecks[i].m_call);
         codeOrigins[i].codeOrigin = record.m_codeOrigin;
         codeOrigins[i].callReturnOffset = returnAddressOffset;
-        record.m_token.assertCodeOriginIndex(i);
     }
     
     m_codeBlock->setNumberOfStructureStubInfos(m_propertyAccesses.size());
@@ -189,7 +188,8 @@ void JITCompiler::link(LinkBuffer& linkBuffer)
         CallLinkInfo& info = m_codeBlock->callLinkInfo(i);
         info.callType = m_jsCalls[i].m_callType;
         info.isDFG = true;
-        info.callReturnLocation = CodeLocationLabel(linkBuffer.locationOf(m_jsCalls[i].m_slowCall));
+        linkBuffer.link(m_jsCalls[i].m_slowCall, FunctionPtr((m_globalData->getCTIStub(info.callType == CallLinkInfo::Construct ? linkConstructThunkGenerator : linkCallThunkGenerator)).code().executableAddress()));
+        info.callReturnLocation = linkBuffer.locationOfNearCall(m_jsCalls[i].m_slowCall);
         info.hotPathBegin = linkBuffer.locationOf(m_jsCalls[i].m_targetToCheck);
         info.hotPathOther = linkBuffer.locationOfNearCall(m_jsCalls[i].m_fastCall);
     }
@@ -280,7 +280,8 @@ bool JITCompiler::compileFunction(JITCode& entry, MacroAssemblerCodePtr& entryWi
     move(stackPointerRegister, GPRInfo::argumentGPR0);
     poke(GPRInfo::callFrameRegister, OBJECT_OFFSETOF(struct JITStackFrame, callFrame) / sizeof(void*));
 
-    CallBeginToken token = beginCall();
+    CallBeginToken token;
+    beginCall(CodeOrigin(0), token);
     Call callRegisterFileCheck = call();
     notifyCall(callRegisterFileCheck, CodeOrigin(0), token);
     jump(fromRegisterFileCheck);
@@ -297,7 +298,7 @@ bool JITCompiler::compileFunction(JITCode& entry, MacroAssemblerCodePtr& entryWi
     branch32(AboveOrEqual, GPRInfo::regT1, TrustedImm32(m_codeBlock->numParameters())).linkTo(fromArityCheck, this);
     move(stackPointerRegister, GPRInfo::argumentGPR0);
     poke(GPRInfo::callFrameRegister, OBJECT_OFFSETOF(struct JITStackFrame, callFrame) / sizeof(void*));
-    token = beginCall();
+    beginCall(CodeOrigin(0), token);
     Call callArityCheck = call();
     notifyCall(callArityCheck, CodeOrigin(0), token);
     move(GPRInfo::regT0, GPRInfo::callFrameRegister);
index 24dbbdc..d6374b7 100644 (file)
@@ -76,28 +76,58 @@ class CallBeginToken {
 public:
     CallBeginToken()
 #if !ASSERT_DISABLED
-        : m_codeOriginIndex(UINT_MAX)
+        : m_registered(false)
+        , m_exceptionCheckIndex(std::numeric_limits<unsigned>::max())
 #endif
     {
     }
     
-    explicit CallBeginToken(unsigned codeOriginIndex)
+    ~CallBeginToken()
+    {
+        ASSERT(m_registered || !m_codeOrigin.isSet());
+        ASSERT(m_codeOrigin.isSet() == (m_exceptionCheckIndex != std::numeric_limits<unsigned>::max()));
+    }
+    
+    void set(CodeOrigin codeOrigin, unsigned index)
+    {
 #if !ASSERT_DISABLED
-        : m_codeOriginIndex(codeOriginIndex)
+        ASSERT(m_registered || !m_codeOrigin.isSet());
+        ASSERT(m_codeOrigin.isSet() == (m_exceptionCheckIndex != std::numeric_limits<unsigned>::max()));
+        m_codeOrigin = codeOrigin;
+        m_registered = false;
+        m_exceptionCheckIndex = index;
+#else
+        UNUSED_PARAM(codeOrigin);
+        UNUSED_PARAM(index);
 #endif
-    {
-        UNUSED_PARAM(codeOriginIndex);
     }
     
-    void assertCodeOriginIndex(unsigned codeOriginIndex) const
+    void registerWithExceptionCheck(CodeOrigin codeOrigin, unsigned index)
     {
-        ASSERT_UNUSED(codeOriginIndex, codeOriginIndex < UINT_MAX);
-        ASSERT_UNUSED(codeOriginIndex, codeOriginIndex == m_codeOriginIndex);
+#if !ASSERT_DISABLED
+        ASSERT(m_codeOrigin == codeOrigin);
+        if (m_registered)
+            return;
+        ASSERT(m_exceptionCheckIndex == index);
+        m_registered = true;
+#else
+        UNUSED_PARAM(codeOrigin);
+        UNUSED_PARAM(index);
+#endif
     }
 
+#if !ASSERT_DISABLED
+    const CodeOrigin& codeOrigin() const
+    {
+        return m_codeOrigin;
+    }
+#endif
+    
 private:
 #if !ASSERT_DISABLED
-    unsigned m_codeOriginIndex;
+    CodeOrigin m_codeOrigin;
+    bool m_registered;
+    unsigned m_exceptionCheckIndex;
 #endif
 };
 
@@ -107,25 +137,22 @@ private:
 // Calls that might throw an exception also record the Jump taken on exception
 // (unset if not present) and code origin used to recover handler/source info.
 struct CallExceptionRecord {
-    CallExceptionRecord(MacroAssembler::Call call, CodeOrigin codeOrigin, CallBeginToken token)
+    CallExceptionRecord(MacroAssembler::Call call, CodeOrigin codeOrigin)
         : m_call(call)
         , m_codeOrigin(codeOrigin)
-        , m_token(token)
     {
     }
 
-    CallExceptionRecord(MacroAssembler::Call call, MacroAssembler::Jump exceptionCheck, CodeOrigin codeOrigin, CallBeginToken token)
+    CallExceptionRecord(MacroAssembler::Call call, MacroAssembler::Jump exceptionCheck, CodeOrigin codeOrigin)
         : m_call(call)
         , m_exceptionCheck(exceptionCheck)
         , m_codeOrigin(codeOrigin)
-        , m_token(token)
     {
     }
 
     MacroAssembler::Call m_call;
     MacroAssembler::Jump m_exceptionCheck;
     CodeOrigin m_codeOrigin;
-    CallBeginToken m_token;
 };
 
 struct PropertyAccessRecord {
@@ -257,19 +284,27 @@ public:
         m_disassembler->setEndOfCode(labelIgnoringWatchpoints());
     }
     
+    unsigned currentCodeOriginIndex() const
+    {
+        return m_currentCodeOriginIndex;
+    }
+    
     // Get a token for beginning a call, and set the current code origin index in
-    // the call frame.
-    CallBeginToken beginCall()
+    // the call frame. For each beginCall() there must be at least one exception
+    // check, and all of the exception checks must have the same CodeOrigin as the
+    // beginCall().
+    void beginCall(CodeOrigin codeOrigin, CallBeginToken& token)
     {
-        unsigned codeOriginIndex = m_currentCodeOriginIndex++;
-        store32(TrustedImm32(codeOriginIndex), tagFor(static_cast<VirtualRegister>(RegisterFile::ArgumentCount)));
-        return CallBeginToken(codeOriginIndex);
+        unsigned index = m_exceptionChecks.size();
+        store32(TrustedImm32(index), tagFor(static_cast<VirtualRegister>(RegisterFile::ArgumentCount)));
+        token.set(codeOrigin, index);
     }
 
     // Notify the JIT of a call that does not require linking.
-    void notifyCall(Call functionCall, CodeOrigin codeOrigin, CallBeginToken token)
+    void notifyCall(Call functionCall, CodeOrigin codeOrigin, CallBeginToken& token)
     {
-        m_exceptionChecks.append(CallExceptionRecord(functionCall, codeOrigin, token));
+        token.registerWithExceptionCheck(codeOrigin, m_exceptionChecks.size());
+        m_exceptionChecks.append(CallExceptionRecord(functionCall, codeOrigin));
     }
 
     // Add a call out from JIT code, without an exception check.
@@ -279,20 +314,27 @@ public:
         m_calls.append(CallLinkRecord(functionCall, function));
         return functionCall;
     }
+    
+    void prepareForExceptionCheck()
+    {
+        move(TrustedImm32(m_exceptionChecks.size()), GPRInfo::nonPreservedNonReturnGPR);
+    }
 
     // Add a call out from JIT code, with an exception check.
-    void addExceptionCheck(Call functionCall, CodeOrigin codeOrigin, CallBeginToken token)
+    void addExceptionCheck(Call functionCall, CodeOrigin codeOrigin, CallBeginToken& token)
     {
-        move(TrustedImm32(m_exceptionChecks.size()), GPRInfo::nonPreservedNonReturnGPR);
-        m_exceptionChecks.append(CallExceptionRecord(functionCall, emitExceptionCheck(), codeOrigin, token));
+        prepareForExceptionCheck();
+        token.registerWithExceptionCheck(codeOrigin, m_exceptionChecks.size());
+        m_exceptionChecks.append(CallExceptionRecord(functionCall, emitExceptionCheck(), codeOrigin));
     }
     
     // Add a call out from JIT code, with a fast exception check that tests if the return value is zero.
-    void addFastExceptionCheck(Call functionCall, CodeOrigin codeOrigin, CallBeginToken token)
+    void addFastExceptionCheck(Call functionCall, CodeOrigin codeOrigin, CallBeginToken& token)
     {
-        move(TrustedImm32(m_exceptionChecks.size()), GPRInfo::nonPreservedNonReturnGPR);
+        prepareForExceptionCheck();
         Jump exceptionCheck = branchTestPtr(Zero, GPRInfo::returnValueGPR);
-        m_exceptionChecks.append(CallExceptionRecord(functionCall, exceptionCheck, codeOrigin, token));
+        token.registerWithExceptionCheck(codeOrigin, m_exceptionChecks.size());
+        m_exceptionChecks.append(CallExceptionRecord(functionCall, exceptionCheck, codeOrigin));
     }
     
     // Helper methods to get predictions
index 5d6575a..bbe55d3 100644 (file)
@@ -30,6 +30,7 @@
 #include "CodeBlock.h"
 #include "DFGOSRExit.h"
 #include "DFGRepatch.h"
+#include "DFGThunks.h"
 #include "HostCallReturnValue.h"
 #include "GetterSetter.h"
 #include "Interpreter.h"
@@ -849,7 +850,6 @@ static void* handleHostCall(ExecState* execCallee, JSValue callee, CodeSpecializ
 
     execCallee->setScopeChain(exec->scopeChain());
     execCallee->setCodeBlock(0);
-    execCallee->clearReturnPC();
 
     if (kind == CodeForCall) {
         CallData callData;
@@ -862,14 +862,14 @@ static void* handleHostCall(ExecState* execCallee, JSValue callee, CodeSpecializ
             execCallee->setCallee(asObject(callee));
             globalData->hostCallReturnValue = JSValue::decode(callData.native.function(execCallee));
             if (globalData->exception)
-                return 0;
+                return globalData->getCTIStub(throwExceptionFromCallSlowPathGenerator).code().executableAddress();
 
             return reinterpret_cast<void*>(getHostCallReturnValue);
         }
     
         ASSERT(callType == CallTypeNone);
         exec->globalData().exception = createNotAFunctionError(exec, callee);
-        return 0;
+        return globalData->getCTIStub(throwExceptionFromCallSlowPathGenerator).code().executableAddress();
     }
 
     ASSERT(kind == CodeForConstruct);
@@ -884,17 +884,17 @@ static void* handleHostCall(ExecState* execCallee, JSValue callee, CodeSpecializ
         execCallee->setCallee(asObject(callee));
         globalData->hostCallReturnValue = JSValue::decode(constructData.native.function(execCallee));
         if (globalData->exception)
-            return 0;
+            return globalData->getCTIStub(throwExceptionFromCallSlowPathGenerator).code().executableAddress();
 
         return reinterpret_cast<void*>(getHostCallReturnValue);
     }
     
     ASSERT(constructType == ConstructTypeNone);
     exec->globalData().exception = createNotAConstructorError(exec, callee);
-    return 0;
+    return globalData->getCTIStub(throwExceptionFromCallSlowPathGenerator).code().executableAddress();
 }
 
-inline void* linkFor(ExecState* execCallee, ReturnAddressPtr returnAddress, CodeSpecializationKind kind)
+inline void* linkFor(ExecState* execCallee, CodeSpecializationKind kind)
 {
     ExecState* exec = execCallee->callerFrame();
     JSGlobalData* globalData = &exec->globalData();
@@ -918,7 +918,7 @@ inline void* linkFor(ExecState* execCallee, ReturnAddressPtr returnAddress, Code
         JSObject* error = functionExecutable->compileFor(execCallee, callee->scope(), kind);
         if (error) {
             globalData->exception = createStackOverflowError(exec);
-            return 0;
+            return globalData->getCTIStub(throwExceptionFromCallSlowPathGenerator).code().executableAddress();
         }
         codeBlock = &functionExecutable->generatedBytecodeFor(kind);
         if (execCallee->argumentCountIncludingThis() < static_cast<size_t>(codeBlock->numParameters()))
@@ -926,7 +926,7 @@ inline void* linkFor(ExecState* execCallee, ReturnAddressPtr returnAddress, Code
         else
             codePtr = functionExecutable->generatedJITCodeFor(kind).addressForCall();
     }
-    CallLinkInfo& callLinkInfo = exec->codeBlock()->getCallLinkInfo(returnAddress);
+    CallLinkInfo& callLinkInfo = exec->codeBlock()->getCallLinkInfo(execCallee->returnPC());
     if (!callLinkInfo.seenOnce())
         callLinkInfo.setSeen();
     else
@@ -934,16 +934,14 @@ inline void* linkFor(ExecState* execCallee, ReturnAddressPtr returnAddress, Code
     return codePtr.executableAddress();
 }
 
-P_FUNCTION_WRAPPER_WITH_RETURN_ADDRESS_E(operationLinkCall);
-void* DFG_OPERATION operationLinkCallWithReturnAddress(ExecState* execCallee, ReturnAddressPtr returnAddress)
+void* DFG_OPERATION operationLinkCall(ExecState* execCallee)
 {
-    return linkFor(execCallee, returnAddress, CodeForCall);
+    return linkFor(execCallee, CodeForCall);
 }
 
-P_FUNCTION_WRAPPER_WITH_RETURN_ADDRESS_E(operationLinkConstruct);
-void* DFG_OPERATION operationLinkConstructWithReturnAddress(ExecState* execCallee, ReturnAddressPtr returnAddress)
+void* DFG_OPERATION operationLinkConstruct(ExecState* execCallee)
 {
-    return linkFor(execCallee, returnAddress, CodeForConstruct);
+    return linkFor(execCallee, CodeForConstruct);
 }
 
 inline void* virtualFor(ExecState* execCallee, CodeSpecializationKind kind)
@@ -965,7 +963,7 @@ inline void* virtualFor(ExecState* execCallee, CodeSpecializationKind kind)
         JSObject* error = functionExecutable->compileFor(execCallee, function->scope(), kind);
         if (error) {
             exec->globalData().exception = error;
-            return 0;
+            return globalData->getCTIStub(throwExceptionFromCallSlowPathGenerator).code().executableAddress();
         }
     }
     return executable->generatedJITCodeWithArityCheckFor(kind).executableAddress();
index 752316f..4438d54 100644 (file)
@@ -30,6 +30,7 @@
 
 #include "DFGCCallHelpers.h"
 #include "DFGSpeculativeJIT.h"
+#include "DFGThunks.h"
 #include "GCAwareJITStubRoutine.h"
 #include "LinkBuffer.h"
 #include "Operations.h"
@@ -916,6 +917,7 @@ void dfgBuildPutByIdList(ExecState* exec, JSValue baseValue, const Identifier& p
 void dfgLinkFor(ExecState* exec, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock, JSFunction* callee, MacroAssemblerCodePtr codePtr, CodeSpecializationKind kind)
 {
     CodeBlock* callerCodeBlock = exec->callerFrame()->codeBlock();
+    JSGlobalData* globalData = callerCodeBlock->globalData();
     
     RepatchBuffer repatchBuffer(callerCodeBlock);
     
@@ -928,11 +930,11 @@ void dfgLinkFor(ExecState* exec, CallLinkInfo& callLinkInfo, CodeBlock* calleeCo
         calleeCodeBlock->linkIncomingCall(&callLinkInfo);
     
     if (kind == CodeForCall) {
-        repatchBuffer.relink(CodeLocationCall(callLinkInfo.callReturnLocation), operationVirtualCall);
+        repatchBuffer.relink(callLinkInfo.callReturnLocation, globalData->getCTIStub(virtualCallThunkGenerator).code());
         return;
     }
     ASSERT(kind == CodeForConstruct);
-    repatchBuffer.relink(CodeLocationCall(callLinkInfo.callReturnLocation), operationVirtualConstruct);
+    repatchBuffer.relink(callLinkInfo.callReturnLocation, globalData->getCTIStub(virtualConstructThunkGenerator).code());
 }
 
 void dfgResetGetByID(RepatchBuffer& repatchBuffer, StructureStubInfo& stubInfo)
index 3b2df7e..28d8033 100644 (file)
@@ -1738,7 +1738,8 @@ public:
     {
         prepareForExternalCall();
         CodeOrigin codeOrigin = at(m_compileIndex).codeOrigin;
-        CallBeginToken token = m_jit.beginCall();
+        CallBeginToken token;
+        m_jit.beginCall(codeOrigin, token);
         JITCompiler::Call call = m_jit.appendCall(function);
         m_jit.addExceptionCheck(call, codeOrigin, token);
         return call;
index bbbf3c4..e09b557 100644 (file)
@@ -1007,16 +1007,18 @@ void SpeculativeJIT::emitCall(Node& node)
     JITCompiler::DataLabelPtr targetToCheck;
     JITCompiler::JumpList slowPath;
 
+    CallBeginToken token;
+    m_jit.beginCall(node.codeOrigin, token);
+    
+    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister);
+    
     slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck));
     slowPath.append(m_jit.branch32(MacroAssembler::NotEqual, calleeTagGPR, TrustedImm32(JSValue::CellTag)));
     m_jit.loadPtr(MacroAssembler::Address(calleePayloadGPR, OBJECT_OFFSETOF(JSFunction, m_scopeChain)), resultPayloadGPR);
-    m_jit.storePtr(resultPayloadGPR, callFramePayloadSlot(RegisterFile::ScopeChain));
-    m_jit.store32(MacroAssembler::TrustedImm32(JSValue::CellTag), callFrameTagSlot(RegisterFile::ScopeChain));
-
-    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister);
+    m_jit.storePtr(resultPayloadGPR, MacroAssembler::Address(GPRInfo::callFrameRegister, static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ScopeChain + OBJECT_OFFSETOF(EncodedValueDescriptor, asBits.payload)));
+    m_jit.store32(MacroAssembler::TrustedImm32(JSValue::CellTag), MacroAssembler::Address(GPRInfo::callFrameRegister, static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ScopeChain + OBJECT_OFFSETOF(EncodedValueDescriptor, asBits.tag)));
 
     CodeOrigin codeOrigin = at(m_compileIndex).codeOrigin;
-    CallBeginToken token = m_jit.beginCall();
     JITCompiler::Call fastCall = m_jit.nearCall();
     m_jit.notifyCall(fastCall, codeOrigin, token);
 
@@ -1024,15 +1026,20 @@ void SpeculativeJIT::emitCall(Node& node)
 
     slowPath.link(&m_jit);
 
-    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister, GPRInfo::argumentGPR0);
-    m_jit.poke(GPRInfo::argumentGPR0);
-    token = m_jit.beginCall();
-    JITCompiler::Call slowCall = m_jit.appendCall(slowCallFunction);
-    m_jit.addFastExceptionCheck(slowCall, codeOrigin, token);
-    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister);
-    token = m_jit.beginCall();
-    JITCompiler::Call theCall = m_jit.call(GPRInfo::returnValueGPR);
-    m_jit.notifyCall(theCall, codeOrigin, token);
+    if (calleeTagGPR == GPRInfo::nonArgGPR0) {
+        if (calleePayloadGPR == GPRInfo::nonArgGPR1)
+            m_jit.swap(GPRInfo::nonArgGPR1, GPRInfo::nonArgGPR0);
+        else {
+            m_jit.move(calleeTagGPR, GPRInfo::nonArgGPR1);
+            m_jit.move(calleePayloadGPR, GPRInfo::nonArgGPR0);
+        }
+    } else {
+        m_jit.move(calleePayloadGPR, GPRInfo::nonArgGPR0);
+        m_jit.move(calleeTagGPR, GPRInfo::nonArgGPR1);
+    }
+    m_jit.prepareForExceptionCheck();
+    JITCompiler::Call slowCall = m_jit.nearCall();
+    m_jit.notifyCall(slowCall, codeOrigin, token);
 
     done.link(&m_jit);
 
index 27eb28f..0e040fb 100644 (file)
@@ -998,14 +998,16 @@ void SpeculativeJIT::emitCall(Node& node)
     JITCompiler::DataLabelPtr targetToCheck;
     JITCompiler::Jump slowPath;
 
+    CallBeginToken token;
+    m_jit.beginCall(node.codeOrigin, token);
+    
+    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister);
+    
     slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, MacroAssembler::TrustedImmPtr(JSValue::encode(JSValue())));
     m_jit.loadPtr(MacroAssembler::Address(calleeGPR, OBJECT_OFFSETOF(JSFunction, m_scopeChain)), resultGPR);
-    m_jit.storePtr(resultGPR, callFrameSlot(RegisterFile::ScopeChain));
+    m_jit.storePtr(resultGPR, MacroAssembler::Address(GPRInfo::callFrameRegister, static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ScopeChain));
 
-    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister);
-    
     CodeOrigin codeOrigin = at(m_compileIndex).codeOrigin;
-    CallBeginToken token = m_jit.beginCall();
     JITCompiler::Call fastCall = m_jit.nearCall();
     m_jit.notifyCall(fastCall, codeOrigin, token);
     
@@ -1013,14 +1015,10 @@ void SpeculativeJIT::emitCall(Node& node)
     
     slowPath.link(&m_jit);
     
-    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister, GPRInfo::argumentGPR0);
-    token = m_jit.beginCall();
-    JITCompiler::Call slowCall = m_jit.appendCall(slowCallFunction);
-    m_jit.addFastExceptionCheck(slowCall, codeOrigin, token);
-    m_jit.addPtr(TrustedImm32(m_jit.codeBlock()->m_numCalleeRegisters * sizeof(Register)), GPRInfo::callFrameRegister);
-    token = m_jit.beginCall();
-    JITCompiler::Call theCall = m_jit.call(GPRInfo::returnValueGPR);
-    m_jit.notifyCall(theCall, codeOrigin, token);
+    m_jit.move(calleeGPR, GPRInfo::nonArgGPR0);
+    m_jit.prepareForExceptionCheck();
+    JITCompiler::Call slowCall = m_jit.nearCall();
+    m_jit.notifyCall(slowCall, codeOrigin, token);
     
     done.link(&m_jit);
     
index 08ca6ea..b056de7 100644 (file)
@@ -28,6 +28,7 @@
 
 #if ENABLE(DFG_JIT)
 
+#include "DFGCCallHelpers.h"
 #include "DFGFPRInfo.h"
 #include "DFGGPRInfo.h"
 #include "DFGOSRExitCompiler.h"
@@ -82,6 +83,217 @@ MacroAssemblerCodeRef osrExitGenerationThunkGenerator(JSGlobalData* globalData)
     return FINALIZE_CODE(patchBuffer, ("DFG OSR exit generation thunk"));
 }
 
+inline void emitPointerValidation(CCallHelpers& jit, GPRReg pointerGPR)
+{
+#if !ASSERT_DISABLED
+    CCallHelpers::Jump isNonZero = jit.branchTestPtr(CCallHelpers::NonZero, pointerGPR);
+    jit.breakpoint();
+    isNonZero.link(&jit);
+    jit.push(pointerGPR);
+    jit.load8(pointerGPR, pointerGPR);
+    jit.pop(pointerGPR);
+#else
+    UNUSED_PARAM(jit);
+    UNUSED_PARAM(pointerGPR);
+#endif
+}
+
+MacroAssemblerCodeRef throwExceptionFromCallSlowPathGenerator(JSGlobalData* globalData)
+{
+    CCallHelpers jit(globalData);
+    
+    // We will jump to here if the JIT code thinks it's making a call, but the
+    // linking helper (C++ code) decided to throw an exception instead. We will
+    // have saved the callReturnIndex in the first arguments of JITStackFrame.
+    // Note that the return address will be on the stack at this point, so we
+    // need to remove it and drop it on the floor, since we don't care about it.
+    // Finally note that the call frame register points at the callee frame, so
+    // we need to pop it.
+    jit.preserveReturnAddressAfterCall(GPRInfo::nonPreservedNonReturnGPR);
+    jit.loadPtr(
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::CallerFrame),
+        GPRInfo::callFrameRegister);
+    jit.peek(GPRInfo::nonPreservedNonReturnGPR, JITSTACKFRAME_ARGS_INDEX);
+    jit.setupArgumentsWithExecState(GPRInfo::nonPreservedNonReturnGPR);
+    jit.move(CCallHelpers::TrustedImmPtr(bitwise_cast<void*>(lookupExceptionHandler)), GPRInfo::nonArgGPR0);
+    emitPointerValidation(jit, GPRInfo::nonArgGPR0);
+    jit.call(GPRInfo::nonArgGPR0);
+    emitPointerValidation(jit, GPRInfo::returnValueGPR2);
+    jit.jump(GPRInfo::returnValueGPR2);
+    
+    LinkBuffer patchBuffer(*globalData, &jit, GLOBAL_THUNK_ID);
+    return FINALIZE_CODE(patchBuffer, ("DFG throw exception from call slow path thunk"));
+}
+
+static void slowPathFor(
+    CCallHelpers& jit, JSGlobalData* globalData, P_DFGOperation_E slowPathFunction)
+{
+    jit.preserveReturnAddressAfterCall(GPRInfo::nonArgGPR2);
+    emitPointerValidation(jit, GPRInfo::nonArgGPR2);
+    jit.storePtr(
+        GPRInfo::nonArgGPR2,
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ReturnPC));
+    jit.storePtr(GPRInfo::callFrameRegister, &globalData->topCallFrame);
+    jit.poke(GPRInfo::nonPreservedNonReturnGPR, JITSTACKFRAME_ARGS_INDEX);
+    jit.setupArgumentsExecState();
+    jit.move(CCallHelpers::TrustedImmPtr(bitwise_cast<void*>(slowPathFunction)), GPRInfo::nonArgGPR0);
+    emitPointerValidation(jit, GPRInfo::nonArgGPR0);
+    jit.call(GPRInfo::nonArgGPR0);
+    
+    // This slow call will return the address of one of the following:
+    // 1) Exception throwing thunk.
+    // 2) Host call return value returner thingy.
+    // 3) The function to call.
+    jit.loadPtr(
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ReturnPC),
+        GPRInfo::nonPreservedNonReturnGPR);
+    jit.storePtr(
+        CCallHelpers::TrustedImmPtr(0),
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ReturnPC));
+    emitPointerValidation(jit, GPRInfo::nonPreservedNonReturnGPR);
+    jit.restoreReturnAddressBeforeReturn(GPRInfo::nonPreservedNonReturnGPR);
+    emitPointerValidation(jit, GPRInfo::returnValueGPR);
+    jit.jump(GPRInfo::returnValueGPR);
+}
+
+static MacroAssemblerCodeRef linkForThunkGenerator(
+    JSGlobalData* globalData, CodeSpecializationKind kind)
+{
+    // The return address is on the stack or in the link register. We will hence
+    // save the return address to the call frame while we make a C++ function call
+    // to perform linking and lazy compilation if necessary. We expect the callee
+    // to be in nonArgGPR0/nonArgGPR1 (payload/tag), the call frame to have already
+    // been adjusted, nonPreservedNonReturnGPR holds the exception handler index,
+    // and all other registers to be available for use. We use JITStackFrame::args
+    // to save important information across calls.
+    
+    CCallHelpers jit(globalData);
+    
+    slowPathFor(jit, globalData, kind == CodeForCall ? operationLinkCall : operationLinkConstruct);
+    
+    LinkBuffer patchBuffer(*globalData, &jit, GLOBAL_THUNK_ID);
+    return FINALIZE_CODE(
+        patchBuffer,
+        ("DFG link %s slow path thunk", kind == CodeForCall ? "call" : "construct"));
+}
+
+MacroAssemblerCodeRef linkCallThunkGenerator(JSGlobalData* globalData)
+{
+    return linkForThunkGenerator(globalData, CodeForCall);
+}
+
+MacroAssemblerCodeRef linkConstructThunkGenerator(JSGlobalData* globalData)
+{
+    return linkForThunkGenerator(globalData, CodeForConstruct);
+}
+
+static MacroAssemblerCodeRef virtualForThunkGenerator(
+    JSGlobalData* globalData, CodeSpecializationKind kind)
+{
+    // The return address is on the stack, or in the link register. We will hence
+    // jump to the callee, or save the return address to the call frame while we
+    // make a C++ function call to the appropriate DFG operation.
+
+    CCallHelpers jit(globalData);
+    
+    CCallHelpers::JumpList slowCase;
+
+    // FIXME: we should have a story for eliminating these checks. In many cases,
+    // the DFG knows that the value is definitely a cell, or definitely a function.
+    
+#if USE(JSVALUE64)
+    slowCase.append(
+        jit.branchTestPtr(
+            CCallHelpers::NonZero, GPRInfo::nonArgGPR0, GPRInfo::tagMaskRegister));
+#else
+    slowCase.append(
+        jit.branch32(
+            CCallHelpers::NotEqual, GPRInfo::nonArgGPR1,
+            CCallHelpers::TrustedImm32(JSValue::CellTag)));
+#endif
+    slowCase.append(
+        jit.branchPtr(
+            CCallHelpers::NotEqual,
+            CCallHelpers::Address(GPRInfo::nonArgGPR0, JSCell::classInfoOffset()),
+            CCallHelpers::TrustedImmPtr(&JSFunction::s_info)));
+    
+    // Now we know we have a JSFunction.
+    
+    jit.loadPtr(
+        CCallHelpers::Address(GPRInfo::nonArgGPR0, JSFunction::offsetOfExecutable()),
+        GPRInfo::nonArgGPR2);
+    slowCase.append(
+        jit.branch32(
+            CCallHelpers::LessThan,
+            CCallHelpers::Address(
+                GPRInfo::nonArgGPR2, ExecutableBase::offsetOfNumParametersFor(kind)),
+            CCallHelpers::TrustedImm32(0)));
+    
+    // Now we know that we have a CodeBlock, and we're committed to making a fast
+    // call.
+    
+    jit.loadPtr(
+        CCallHelpers::Address(GPRInfo::nonArgGPR0, JSFunction::offsetOfScopeChain()),
+        GPRInfo::nonArgGPR1);
+#if USE(JSVALUE64)
+    jit.storePtr(
+        GPRInfo::nonArgGPR1,
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ScopeChain));
+#else
+    jit.storePtr(
+        GPRInfo::nonArgGPR1,
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ScopeChain +
+            OBJECT_OFFSETOF(EncodedValueDescriptor, asBits.payload)));
+    jit.store32(
+        CCallHelpers::TrustedImm32(JSValue::CellTag),
+        CCallHelpers::Address(
+            GPRInfo::callFrameRegister,
+            static_cast<ptrdiff_t>(sizeof(Register)) * RegisterFile::ScopeChain +
+            OBJECT_OFFSETOF(EncodedValueDescriptor, asBits.tag)));
+#endif
+    
+    jit.loadPtr(
+        CCallHelpers::Address(GPRInfo::nonArgGPR2, ExecutableBase::offsetOfJITCodeWithArityCheckFor(kind)),
+        GPRInfo::regT0);
+    
+    // Make a tail call. This will return back to DFG code.
+    emitPointerValidation(jit, GPRInfo::regT0);
+    jit.jump(GPRInfo::regT0);
+
+    slowCase.link(&jit);
+    
+    // Here we don't know anything, so revert to the full slow path.
+    
+    slowPathFor(jit, globalData, kind == CodeForCall ? operationVirtualCall : operationVirtualConstruct);
+    
+    LinkBuffer patchBuffer(*globalData, &jit, GLOBAL_THUNK_ID);
+    return FINALIZE_CODE(
+        patchBuffer,
+        ("DFG virtual %s slow path thunk", kind == CodeForCall ? "call" : "construct"));
+}
+
+MacroAssemblerCodeRef virtualCallThunkGenerator(JSGlobalData* globalData)
+{
+    return virtualForThunkGenerator(globalData, CodeForCall);
+}
+
+MacroAssemblerCodeRef virtualConstructThunkGenerator(JSGlobalData* globalData)
+{
+    return virtualForThunkGenerator(globalData, CodeForConstruct);
+}
+
 } } // namespace JSC::DFG
 
 #endif // ENABLE(DFG_JIT)
index 3db6244..11a06d1 100644 (file)
@@ -40,6 +40,14 @@ namespace DFG {
 
 MacroAssemblerCodeRef osrExitGenerationThunkGenerator(JSGlobalData*);
 
+MacroAssemblerCodeRef throwExceptionFromCallSlowPathGenerator(JSGlobalData*);
+
+MacroAssemblerCodeRef linkCallThunkGenerator(JSGlobalData*);
+MacroAssemblerCodeRef linkConstructThunkGenerator(JSGlobalData*);
+
+MacroAssemblerCodeRef virtualCallThunkGenerator(JSGlobalData*);
+MacroAssemblerCodeRef virtualConstructThunkGenerator(JSGlobalData*);
+
 } } // namespace JSC::DFG
 
 #endif // ENABLE(DFG_JIT)
index 285355f..52a7158 100644 (file)
@@ -739,7 +739,7 @@ JITCode JIT::privateCompile(CodePtr* functionEntryArityCheck, JITCompilationEffo
         CallLinkInfo& info = m_codeBlock->callLinkInfo(i);
         info.callType = m_callStructureStubCompilationInfo[i].callType;
         info.bytecodeIndex = m_callStructureStubCompilationInfo[i].bytecodeIndex;
-        info.callReturnLocation = CodeLocationLabel(patchBuffer.locationOfNearCall(m_callStructureStubCompilationInfo[i].callReturnLocation));
+        info.callReturnLocation = patchBuffer.locationOfNearCall(m_callStructureStubCompilationInfo[i].callReturnLocation);
         info.hotPathBegin = patchBuffer.locationOf(m_callStructureStubCompilationInfo[i].hotPathBegin);
         info.hotPathOther = patchBuffer.locationOfNearCall(m_callStructureStubCompilationInfo[i].hotPathOther);
     }
@@ -802,12 +802,12 @@ void JIT::linkFor(JSFunction* callee, CodeBlock* callerCodeBlock, CodeBlock* cal
 
     // Patch the slow patch so we do not continue to try to link.
     if (kind == CodeForCall) {
-        repatchBuffer.relink(CodeLocationNearCall(callLinkInfo->callReturnLocation), globalData->jitStubs->ctiVirtualCall());
+        repatchBuffer.relink(callLinkInfo->callReturnLocation, globalData->jitStubs->ctiVirtualCall());
         return;
     }
 
     ASSERT(kind == CodeForConstruct);
-    repatchBuffer.relink(CodeLocationNearCall(callLinkInfo->callReturnLocation), globalData->jitStubs->ctiVirtualConstruct());
+    repatchBuffer.relink(callLinkInfo->callReturnLocation, globalData->jitStubs->ctiVirtualConstruct());
 }
 
 } // namespace JSC
index e5f6de4..2f6c6a2 100644 (file)
@@ -176,6 +176,30 @@ namespace JSC {
                 return intrinsic();
             return NoIntrinsic;
         }
+        
+        static ptrdiff_t offsetOfJITCodeFor(CodeSpecializationKind kind)
+        {
+            if (kind == CodeForCall)
+                return OBJECT_OFFSETOF(ExecutableBase, m_jitCodeForCall);
+            ASSERT(kind == CodeForConstruct);
+            return OBJECT_OFFSETOF(ExecutableBase, m_jitCodeForConstruct);
+        }
+        
+        static ptrdiff_t offsetOfJITCodeWithArityCheckFor(CodeSpecializationKind kind)
+        {
+            if (kind == CodeForCall)
+                return OBJECT_OFFSETOF(ExecutableBase, m_jitCodeForCallWithArityCheck);
+            ASSERT(kind == CodeForConstruct);
+            return OBJECT_OFFSETOF(ExecutableBase, m_jitCodeForConstructWithArityCheck);
+        }
+        
+        static ptrdiff_t offsetOfNumParametersFor(CodeSpecializationKind kind)
+        {
+            if (kind == CodeForCall)
+                return OBJECT_OFFSETOF(ExecutableBase, m_numParametersForCall);
+            ASSERT(kind == CodeForConstruct);
+            return OBJECT_OFFSETOF(ExecutableBase, m_numParametersForConstruct);
+        }
 #endif
 
     protected:
index f74bfad..6f98bd6 100644 (file)
@@ -124,6 +124,18 @@ namespace JSC {
         friend class LLInt::Data;
 
     public:
+#if USE(JSVALUE32_64)
+        enum { Int32Tag =        0xffffffff };
+        enum { BooleanTag =      0xfffffffe };
+        enum { NullTag =         0xfffffffd };
+        enum { UndefinedTag =    0xfffffffc };
+        enum { CellTag =         0xfffffffb };
+        enum { EmptyValueTag =   0xfffffffa };
+        enum { DeletedValueTag = 0xfffffff9 };
+
+        enum { LowestTag =  DeletedValueTag };
+#endif
+
         static EncodedJSValue encode(JSValue);
         static JSValue decode(EncodedJSValue);
 
@@ -278,16 +290,6 @@ namespace JSC {
          * cell, integer and bool values the lower 32 bits (the 'payload') contain the pointer
          * integer or boolean value; in the case of all other tags the payload is 0.
          */
-        enum { Int32Tag =        0xffffffff };
-        enum { BooleanTag =      0xfffffffe };
-        enum { NullTag =         0xfffffffd };
-        enum { UndefinedTag =    0xfffffffc };
-        enum { CellTag =         0xfffffffb };
-        enum { EmptyValueTag =   0xfffffffa };
-        enum { DeletedValueTag = 0xfffffff9 };
-
-        enum { LowestTag =  DeletedValueTag };
-
         uint32_t tag() const;
         int32_t payload() const;
 #elif USE(JSVALUE64)