FTL should be able to do simple OSR exits using llvm.webkit.stackmap
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 10 Oct 2013 04:24:57 +0000 (04:24 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 10 Oct 2013 04:24:57 +0000 (04:24 +0000)
https://bugs.webkit.org/show_bug.cgi?id=122538

Reviewed by Oliver Hunt.

This gives the FTL the ability to OSR exit using the llvm.webkit.stackmap intrinsic.

- The FTL compiles all OSR exit calls as calls to llvm.webkit.stackmap with a unique
  ID, passing a requested size that is big enough for own jump replacement.

- After LLVM compilation, we parse the new LLVM stackmap section.

- For all llvm.webkit.stackmaps that we used for OSR exits, we do a jumpReplacement,
  which targets exit thunks that we generate.

- If an exit thunk fires, it causes JSC to compile an exit off-ramp that uses a
  combination of the JSC-internal OSR exit accounting (FTL::ExitValue and friends) and
  LLVM stackmap's accounting of where data actually ended up (register, indirect,
  constant) to reconstruct bytecode state.

This still has shortcomings; for example it cannot handle XMM or YMM registers. Handling
YMM registers will require adding some basic YMM support to our assemblers - really we
just need the ability to move a YMM's value into a GPR.

This patch preserves all of the old, intrinsic-less, FTL OSR exit support. Hence it
manages to pass all existing FTL tests even despite its incompleteness. I think that's
the right way to go since this is already a big patch, and anyway it would be great to
keep the intrinsic-less FTL OSR exit support so long as the LLVM side of this hasn't
landed.

* JavaScriptCore.xcodeproj/project.pbxproj:
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::firstRegister):
(JSC::AbstractMacroAssembler::lastRegister):
* assembler/MacroAssembler.h:
(JSC::MacroAssembler::isStackRelated):
(JSC::MacroAssembler::firstRealRegister):
(JSC::MacroAssembler::nextRegister):
(JSC::MacroAssembler::secondRealRegister):
* assembler/MacroAssemblerX86Common.h:
* assembler/X86Assembler.h:
(JSC::X86Assembler::firstRegister):
(JSC::X86Assembler::lastRegister):
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* ftl/FTLCArgumentGetter.cpp:
(JSC::FTL::CArgumentGetter::loadNextAndBox):
* ftl/FTLCArgumentGetter.h:
(JSC::FTL::CArgumentGetter::loadNextDoubleIntoGPR):
* ftl/FTLCompile.cpp:
(JSC::FTL::mmAllocateCodeSection):
(JSC::FTL::mmAllocateDataSection):
(JSC::FTL::dumpDataSection):
(JSC::FTL::fixFunctionBasedOnStackMaps):
(JSC::FTL::compile):
* ftl/FTLExitThunkGenerator.cpp:
(JSC::FTL::ExitThunkGenerator::emitThunk):
(JSC::FTL::ExitThunkGenerator::emitThunks):
* ftl/FTLExitThunkGenerator.h:
* ftl/FTLExitValue.h:
(JSC::FTL::ExitValue::isInJSStackSomehow):
(JSC::FTL::ExitValue::valueFormat):
* ftl/FTLFail.cpp:
(JSC::FTL::fail):
* ftl/FTLIntrinsicRepository.h:
* ftl/FTLJITCode.h:
* ftl/FTLLowerDFGToLLVM.cpp:
(JSC::FTL::generateExitThunks):
(JSC::FTL::LowerDFGToLLVM::LowerDFGToLLVM):
(JSC::FTL::LowerDFGToLLVM::appendOSRExit):
(JSC::FTL::LowerDFGToLLVM::emitOSRExitCall):
(JSC::FTL::LowerDFGToLLVM::linkOSRExitsAndCompleteInitializationBlocks):
* ftl/FTLOSRExit.h:
* ftl/FTLOSRExitCompilationInfo.h:
(JSC::FTL::OSRExitCompilationInfo::OSRExitCompilationInfo):
* ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::compileStubWithOSRExitStackmap):
(JSC::FTL::compileStubWithoutOSRExitStackmap):
(JSC::FTL::compileFTLOSRExit):
* ftl/FTLSaveRestore.cpp: Added.
(JSC::FTL::bytesForGPRs):
(JSC::FTL::requiredScratchMemorySizeInBytes):
(JSC::FTL::offsetOfGPR):
(JSC::FTL::saveAllRegisters):
(JSC::FTL::restoreAllRegisters):
* ftl/FTLSaveRestore.h: Added.
* ftl/FTLStackMaps.cpp: Added.
(JSC::FTL::readObject):
(JSC::FTL::StackMaps::Constant::parse):
(JSC::FTL::StackMaps::Constant::dump):
(JSC::FTL::StackMaps::Location::parse):
(JSC::FTL::StackMaps::Location::dump):
(JSC::FTL::StackMaps::Location::involvesGPR):
(JSC::FTL::StackMaps::Location::isGPR):
(JSC::FTL::StackMaps::Location::gpr):
(JSC::FTL::StackMaps::Location::restoreInto):
(JSC::FTL::StackMaps::Record::parse):
(JSC::FTL::StackMaps::Record::dump):
(JSC::FTL::StackMaps::parse):
(JSC::FTL::StackMaps::dump):
(JSC::FTL::StackMaps::dumpMultiline):
(JSC::FTL::StackMaps::getRecordMap):
(WTF::printInternal):
* ftl/FTLStackMaps.h: Added.
* ftl/FTLState.h:
* ftl/FTLThunks.cpp:
(JSC::FTL::osrExitGenerationThunkGenerator):
* ftl/FTLValueFormat.cpp:
(JSC::FTL::reboxAccordingToFormat):
* ftl/FTLValueFormat.h:
* runtime/DataView.cpp:
(JSC::DataView::create):
* runtime/DataView.h:
(JSC::DataView::read):
* runtime/Options.h:

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@157209 268f45cc-cd09-0410-ab3c-d52691b4dbfc

31 files changed:
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/assembler/AbstractMacroAssembler.h
Source/JavaScriptCore/assembler/MacroAssembler.h
Source/JavaScriptCore/assembler/MacroAssemblerX86Common.h
Source/JavaScriptCore/assembler/X86Assembler.h
Source/JavaScriptCore/dfg/DFGPlan.cpp
Source/JavaScriptCore/ftl/FTLCArgumentGetter.cpp
Source/JavaScriptCore/ftl/FTLCArgumentGetter.h
Source/JavaScriptCore/ftl/FTLCompile.cpp
Source/JavaScriptCore/ftl/FTLExitThunkGenerator.cpp
Source/JavaScriptCore/ftl/FTLExitThunkGenerator.h
Source/JavaScriptCore/ftl/FTLExitValue.h
Source/JavaScriptCore/ftl/FTLFail.cpp
Source/JavaScriptCore/ftl/FTLIntrinsicRepository.h
Source/JavaScriptCore/ftl/FTLJITCode.h
Source/JavaScriptCore/ftl/FTLLowerDFGToLLVM.cpp
Source/JavaScriptCore/ftl/FTLOSRExit.h
Source/JavaScriptCore/ftl/FTLOSRExitCompilationInfo.h
Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp
Source/JavaScriptCore/ftl/FTLSaveRestore.cpp [new file with mode: 0644]
Source/JavaScriptCore/ftl/FTLSaveRestore.h [new file with mode: 0644]
Source/JavaScriptCore/ftl/FTLStackMaps.cpp [new file with mode: 0644]
Source/JavaScriptCore/ftl/FTLStackMaps.h [new file with mode: 0644]
Source/JavaScriptCore/ftl/FTLState.h
Source/JavaScriptCore/ftl/FTLThunks.cpp
Source/JavaScriptCore/ftl/FTLValueFormat.cpp
Source/JavaScriptCore/ftl/FTLValueFormat.h
Source/JavaScriptCore/runtime/DataView.cpp
Source/JavaScriptCore/runtime/DataView.h
Source/JavaScriptCore/runtime/Options.h

index 226e92a..0cce4c2 100644 (file)
@@ -1,3 +1,121 @@
+2013-10-08  Filip Pizlo  <fpizlo@apple.com>
+
+        FTL should be able to do simple OSR exits using llvm.webkit.stackmap
+        https://bugs.webkit.org/show_bug.cgi?id=122538
+
+        Reviewed by Oliver Hunt.
+        
+        This gives the FTL the ability to OSR exit using the llvm.webkit.stackmap intrinsic.
+        
+        - The FTL compiles all OSR exit calls as calls to llvm.webkit.stackmap with a unique
+          ID, passing a requested size that is big enough for own jump replacement.
+        
+        - After LLVM compilation, we parse the new LLVM stackmap section.
+        
+        - For all llvm.webkit.stackmaps that we used for OSR exits, we do a jumpReplacement,
+          which targets exit thunks that we generate.
+        
+        - If an exit thunk fires, it causes JSC to compile an exit off-ramp that uses a
+          combination of the JSC-internal OSR exit accounting (FTL::ExitValue and friends) and
+          LLVM stackmap's accounting of where data actually ended up (register, indirect,
+          constant) to reconstruct bytecode state.
+        
+        This still has shortcomings; for example it cannot handle XMM or YMM registers. Handling
+        YMM registers will require adding some basic YMM support to our assemblers - really we
+        just need the ability to move a YMM's value into a GPR.
+        
+        This patch preserves all of the old, intrinsic-less, FTL OSR exit support. Hence it
+        manages to pass all existing FTL tests even despite its incompleteness. I think that's
+        the right way to go since this is already a big patch, and anyway it would be great to
+        keep the intrinsic-less FTL OSR exit support so long as the LLVM side of this hasn't
+        landed.
+
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::AbstractMacroAssembler::firstRegister):
+        (JSC::AbstractMacroAssembler::lastRegister):
+        * assembler/MacroAssembler.h:
+        (JSC::MacroAssembler::isStackRelated):
+        (JSC::MacroAssembler::firstRealRegister):
+        (JSC::MacroAssembler::nextRegister):
+        (JSC::MacroAssembler::secondRealRegister):
+        * assembler/MacroAssemblerX86Common.h:
+        * assembler/X86Assembler.h:
+        (JSC::X86Assembler::firstRegister):
+        (JSC::X86Assembler::lastRegister):
+        * dfg/DFGPlan.cpp:
+        (JSC::DFG::Plan::compileInThreadImpl):
+        * ftl/FTLCArgumentGetter.cpp:
+        (JSC::FTL::CArgumentGetter::loadNextAndBox):
+        * ftl/FTLCArgumentGetter.h:
+        (JSC::FTL::CArgumentGetter::loadNextDoubleIntoGPR):
+        * ftl/FTLCompile.cpp:
+        (JSC::FTL::mmAllocateCodeSection):
+        (JSC::FTL::mmAllocateDataSection):
+        (JSC::FTL::dumpDataSection):
+        (JSC::FTL::fixFunctionBasedOnStackMaps):
+        (JSC::FTL::compile):
+        * ftl/FTLExitThunkGenerator.cpp:
+        (JSC::FTL::ExitThunkGenerator::emitThunk):
+        (JSC::FTL::ExitThunkGenerator::emitThunks):
+        * ftl/FTLExitThunkGenerator.h:
+        * ftl/FTLExitValue.h:
+        (JSC::FTL::ExitValue::isInJSStackSomehow):
+        (JSC::FTL::ExitValue::valueFormat):
+        * ftl/FTLFail.cpp:
+        (JSC::FTL::fail):
+        * ftl/FTLIntrinsicRepository.h:
+        * ftl/FTLJITCode.h:
+        * ftl/FTLLowerDFGToLLVM.cpp:
+        (JSC::FTL::generateExitThunks):
+        (JSC::FTL::LowerDFGToLLVM::LowerDFGToLLVM):
+        (JSC::FTL::LowerDFGToLLVM::appendOSRExit):
+        (JSC::FTL::LowerDFGToLLVM::emitOSRExitCall):
+        (JSC::FTL::LowerDFGToLLVM::linkOSRExitsAndCompleteInitializationBlocks):
+        * ftl/FTLOSRExit.h:
+        * ftl/FTLOSRExitCompilationInfo.h:
+        (JSC::FTL::OSRExitCompilationInfo::OSRExitCompilationInfo):
+        * ftl/FTLOSRExitCompiler.cpp:
+        (JSC::FTL::compileStubWithOSRExitStackmap):
+        (JSC::FTL::compileStubWithoutOSRExitStackmap):
+        (JSC::FTL::compileFTLOSRExit):
+        * ftl/FTLSaveRestore.cpp: Added.
+        (JSC::FTL::bytesForGPRs):
+        (JSC::FTL::requiredScratchMemorySizeInBytes):
+        (JSC::FTL::offsetOfGPR):
+        (JSC::FTL::saveAllRegisters):
+        (JSC::FTL::restoreAllRegisters):
+        * ftl/FTLSaveRestore.h: Added.
+        * ftl/FTLStackMaps.cpp: Added.
+        (JSC::FTL::readObject):
+        (JSC::FTL::StackMaps::Constant::parse):
+        (JSC::FTL::StackMaps::Constant::dump):
+        (JSC::FTL::StackMaps::Location::parse):
+        (JSC::FTL::StackMaps::Location::dump):
+        (JSC::FTL::StackMaps::Location::involvesGPR):
+        (JSC::FTL::StackMaps::Location::isGPR):
+        (JSC::FTL::StackMaps::Location::gpr):
+        (JSC::FTL::StackMaps::Location::restoreInto):
+        (JSC::FTL::StackMaps::Record::parse):
+        (JSC::FTL::StackMaps::Record::dump):
+        (JSC::FTL::StackMaps::parse):
+        (JSC::FTL::StackMaps::dump):
+        (JSC::FTL::StackMaps::dumpMultiline):
+        (JSC::FTL::StackMaps::getRecordMap):
+        (WTF::printInternal):
+        * ftl/FTLStackMaps.h: Added.
+        * ftl/FTLState.h:
+        * ftl/FTLThunks.cpp:
+        (JSC::FTL::osrExitGenerationThunkGenerator):
+        * ftl/FTLValueFormat.cpp:
+        (JSC::FTL::reboxAccordingToFormat):
+        * ftl/FTLValueFormat.h:
+        * runtime/DataView.cpp:
+        (JSC::DataView::create):
+        * runtime/DataView.h:
+        (JSC::DataView::read):
+        * runtime/Options.h:
+
 2013-10-09  Filip Pizlo  <fpizlo@apple.com>
 
         Minor clean-ups in the JSC Xcode project.
index a3bbcd1..063a3ed 100644 (file)
                0F9D3370165DBB90005AD387 /* Disassembler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F9D336E165DBB8D005AD387 /* Disassembler.cpp */; };
                0F9D339617FFC4E60073C2BC /* DFGFlushedAt.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F9D339417FFC4E60073C2BC /* DFGFlushedAt.cpp */; };
                0F9D339717FFC4E60073C2BC /* DFGFlushedAt.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F9D339517FFC4E60073C2BC /* DFGFlushedAt.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F9D339A1803ADB70073C2BC /* FTLStackMaps.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F9D33981803ADB70073C2BC /* FTLStackMaps.cpp */; };
+               0F9D339B1803ADB70073C2BC /* FTLStackMaps.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F9D33991803ADB70073C2BC /* FTLStackMaps.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F9FB4F417FCB91700CB67F8 /* DFGStackLayoutPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F9FB4F217FCB91700CB67F8 /* DFGStackLayoutPhase.cpp */; };
                0F9FB4F517FCB91700CB67F8 /* DFGStackLayoutPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F9FB4F317FCB91700CB67F8 /* DFGStackLayoutPhase.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F9FC8C314E1B5FE00D52AE0 /* PolymorphicPutByIdList.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F9FC8BF14E1B5FB00D52AE0 /* PolymorphicPutByIdList.cpp */; };
                0FC815151405119B00CFA603 /* VTableSpectrum.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FC815141405118D00CFA603 /* VTableSpectrum.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FC81516140511B500CFA603 /* VTableSpectrum.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FC815121405118600CFA603 /* VTableSpectrum.cpp */; };
                0FCCAE4516D0CF7400D0C65B /* ParserError.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FCCAE4316D0CF6E00D0C65B /* ParserError.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0FCEFAAB1804C13E00472CE4 /* FTLSaveRestore.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FCEFAA91804C13E00472CE4 /* FTLSaveRestore.cpp */; };
+               0FCEFAAC1804C13E00472CE4 /* FTLSaveRestore.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FCEFAAA1804C13E00472CE4 /* FTLSaveRestore.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FD2C92416D01EE900C7803F /* StructureInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FD2C92316D01EE900C7803F /* StructureInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FD3C82614115D4000FD81CB /* DFGDriver.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FD3C82014115CF800FD81CB /* DFGDriver.cpp */; };
                0FD3C82814115D4F00FD81CB /* DFGDriver.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FD3C82214115D0E00FD81CB /* DFGDriver.h */; };
                0F9D336E165DBB8D005AD387 /* Disassembler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = Disassembler.cpp; path = disassembler/Disassembler.cpp; sourceTree = "<group>"; };
                0F9D339417FFC4E60073C2BC /* DFGFlushedAt.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGFlushedAt.cpp; path = dfg/DFGFlushedAt.cpp; sourceTree = "<group>"; };
                0F9D339517FFC4E60073C2BC /* DFGFlushedAt.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGFlushedAt.h; path = dfg/DFGFlushedAt.h; sourceTree = "<group>"; };
+               0F9D33981803ADB70073C2BC /* FTLStackMaps.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = FTLStackMaps.cpp; path = ftl/FTLStackMaps.cpp; sourceTree = "<group>"; };
+               0F9D33991803ADB70073C2BC /* FTLStackMaps.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = FTLStackMaps.h; path = ftl/FTLStackMaps.h; sourceTree = "<group>"; };
                0F9FB4F217FCB91700CB67F8 /* DFGStackLayoutPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGStackLayoutPhase.cpp; path = dfg/DFGStackLayoutPhase.cpp; sourceTree = "<group>"; };
                0F9FB4F317FCB91700CB67F8 /* DFGStackLayoutPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStackLayoutPhase.h; path = dfg/DFGStackLayoutPhase.h; sourceTree = "<group>"; };
                0F9FC8BF14E1B5FB00D52AE0 /* PolymorphicPutByIdList.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = PolymorphicPutByIdList.cpp; sourceTree = "<group>"; };
                0FC815141405118D00CFA603 /* VTableSpectrum.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VTableSpectrum.h; sourceTree = "<group>"; };
                0FCB408515C0A3C30048932B /* SlotVisitorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SlotVisitorInlines.h; sourceTree = "<group>"; };
                0FCCAE4316D0CF6E00D0C65B /* ParserError.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ParserError.h; sourceTree = "<group>"; };
+               0FCEFAA91804C13E00472CE4 /* FTLSaveRestore.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = FTLSaveRestore.cpp; path = ftl/FTLSaveRestore.cpp; sourceTree = "<group>"; };
+               0FCEFAAA1804C13E00472CE4 /* FTLSaveRestore.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = FTLSaveRestore.h; path = ftl/FTLSaveRestore.h; sourceTree = "<group>"; };
                0FD2C92316D01EE900C7803F /* StructureInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StructureInlines.h; sourceTree = "<group>"; };
                0FD3C82014115CF800FD81CB /* DFGDriver.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGDriver.cpp; path = dfg/DFGDriver.cpp; sourceTree = "<group>"; };
                0FD3C82214115D0E00FD81CB /* DFGDriver.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGDriver.h; path = dfg/DFGDriver.h; sourceTree = "<group>"; };
                                0F235BCA17178E1C00690C7F /* FTLOSRExitCompiler.h */,
                                0FEA0A291709629600BB722C /* FTLOutput.cpp */,
                                0FEA0A06170513DB00BB722C /* FTLOutput.h */,
+                               0FCEFAA91804C13E00472CE4 /* FTLSaveRestore.cpp */,
+                               0FCEFAAA1804C13E00472CE4 /* FTLSaveRestore.h */,
+                               0F9D33981803ADB70073C2BC /* FTLStackMaps.cpp */,
+                               0F9D33991803ADB70073C2BC /* FTLStackMaps.h */,
                                0FEA0A151706BB9000BB722C /* FTLState.cpp */,
                                0FEA0A07170513DB00BB722C /* FTLState.h */,
                                A7FCC26C17A0B6AA00786D1A /* FTLSwitchCase.h */,
                                BC18C42A0E16F5CD00B34460 /* JSType.h in Headers */,
                                0F2B66FB17B6B5AB00A7AE3F /* JSTypedArrayConstructors.h in Headers */,
                                0F2B66FD17B6B5AB00A7AE3F /* JSTypedArrayPrototypes.h in Headers */,
+                               0FCEFAAC1804C13E00472CE4 /* FTLSaveRestore.h in Headers */,
                                0F2B66FF17B6B5AB00A7AE3F /* JSTypedArrays.h in Headers */,
                                6507D29E0E871E5E00D7D896 /* JSTypeInfo.h in Headers */,
                                0F2B670217B6B5AB00A7AE3F /* JSUint16Array.h in Headers */,
                                BC9041480EB9250900FE26FA /* StructureTransitionTable.h in Headers */,
                                C2DF44301707AC0100A5CA96 /* SuperRegion.h in Headers */,
                                BC18C46B0E16F5CD00B34460 /* SymbolTable.h in Headers */,
+                               0F9D339B1803ADB70073C2BC /* FTLStackMaps.h in Headers */,
                                A784A26411D16622005776AC /* SyntaxChecker.h in Headers */,
                                0F572D4F16879FDD00E57FBD /* ThunkGenerator.h in Headers */,
                                0FD8A32A17D51F5700CA2C40 /* DFGToFTLDeferredCompilationCallback.h in Headers */,
                                969A079A0ED1D3AE00F1F681 /* Opcode.cpp in Sources */,
                                14280850107EC0D70013E7B2 /* Operations.cpp in Sources */,
                                A7BDAEC817F4EA1400F6140C /* ArrayIteratorPrototype.cpp in Sources */,
+                               0FCEFAAB1804C13E00472CE4 /* FTLSaveRestore.cpp in Sources */,
                                0FE228EE1436AB2C00196C48 /* Options.cpp in Sources */,
                                148F21BC107EC54D0042EC2C /* Parser.cpp in Sources */,
                                93052C340FB792190048FDC3 /* ParserArena.cpp in Sources */,
                                86704B8412DBA33700A9FE7B /* YarrInterpreter.cpp in Sources */,
                                86704B8612DBA33700A9FE7B /* YarrJIT.cpp in Sources */,
                                86704B8912DBA33700A9FE7B /* YarrPattern.cpp in Sources */,
+                               0F9D339A1803ADB70073C2BC /* FTLStackMaps.cpp in Sources */,
                                86704B4212DB8A8100A9FE7B /* YarrSyntaxChecker.cpp in Sources */,
                        );
                        runOnlyForDeploymentPostprocessing = 0;
index f74a2c1..ff1f3dd 100644 (file)
@@ -74,6 +74,9 @@ public:
     class Jump;
 
     typedef typename AssemblerType::RegisterID RegisterID;
+    
+    static RegisterID firstRegister() { return AssemblerType::firstRegister(); }
+    static RegisterID lastRegister() { return AssemblerType::lastRegister(); }
 
     // Section 1: MacroAssembler operand types
     //
index d775f82..98058e7 100644 (file)
@@ -67,6 +67,32 @@ namespace JSC {
 class MacroAssembler : public MacroAssemblerBase {
 public:
 
+    static bool isStackRelated(RegisterID reg)
+    {
+        return reg == stackPointerRegister || reg == framePointerRegister;
+    }
+    
+    static RegisterID firstRealRegister()
+    {
+        RegisterID firstRegister = MacroAssembler::firstRegister();
+        while (MacroAssembler::isStackRelated(firstRegister))
+            firstRegister = static_cast<RegisterID>(firstRegister + 1);
+        return firstRegister;
+    }
+    
+    static RegisterID nextRegister(RegisterID reg)
+    {
+        RegisterID result = static_cast<RegisterID>(reg + 1);
+        while (MacroAssembler::isStackRelated(result))
+            result = static_cast<RegisterID>(result + 1);
+        return result;
+    }
+    
+    static RegisterID secondRealRegister()
+    {
+        return nextRegister(firstRealRegister());
+    }
+
     using MacroAssemblerBase::pop;
     using MacroAssemblerBase::jump;
     using MacroAssemblerBase::branch32;
index 46379dd..64c77cc 100644 (file)
@@ -97,7 +97,7 @@ public:
 
     static const RegisterID stackPointerRegister = X86Registers::esp;
     static const RegisterID framePointerRegister = X86Registers::ebp;
-
+    
     static bool shouldBlindForSpecificArch(uint32_t value) { return value >= 0x00ffffff; }
 #if CPU(X86_64)
     static bool shouldBlindForSpecificArch(uint64_t value) { return value >= 0x00ffffff; }
index 3a859ea..a1d2f25 100644 (file)
@@ -126,6 +126,16 @@ namespace X86Registers {
 class X86Assembler {
 public:
     typedef X86Registers::RegisterID RegisterID;
+    static RegisterID firstRegister() { return X86Registers::eax; }
+    static RegisterID lastRegister()
+    {
+#if CPU(X86_64)
+        return X86Registers::r15;
+#else
+        return X86Registers::edi;
+#endif
+    }
+    
     typedef X86Registers::XMMRegisterID XMMRegisterID;
     typedef XMMRegisterID FPRegisterID;
 
index 398d768..579d71a 100644 (file)
@@ -264,12 +264,18 @@ Plan::CompilationPath Plan::compileInThreadImpl(LongLivedState& longLivedState)
         if (Options::reportCompileTimes())
             beforeFTL = currentTimeMS();
         
-        if (Options::llvmAlwaysFails()) {
+        if (Options::llvmAlwaysFailsBeforeCompile()) {
             FTL::fail(state);
             return FTLPath;
         }
         
         FTL::compile(state);
+
+        if (Options::llvmAlwaysFailsBeforeLink()) {
+            FTL::fail(state);
+            return FTLPath;
+        }
+        
         FTL::link(state);
         return FTLPath;
 #else
index ac5858a..7e14147 100644 (file)
@@ -52,60 +52,31 @@ void CArgumentGetter::loadNextAndBox(
     }
     
     switch (format) {
-    case ValueFormatInt32: {
+    case ValueFormatInt32:
+    case ValueFormatUInt32:
         loadNext32(destination);
-        m_jit.or64(GPRInfo::tagTypeNumberRegister, destination);
         break;
-    }
-            
-    case ValueFormatUInt32: {
-        loadNext32(destination);
-        m_jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch2);
-        m_jit.boxInt52(destination, destination, scratch1, FPRInfo::fpRegT0);
-        m_jit.move64ToDouble(scratch2, FPRInfo::fpRegT0);
-        break;
-    }
         
-    case ValueFormatInt52: {
+    case ValueFormatInt52:
+    case ValueFormatStrictInt52:
+    case ValueFormatJSValue:
         loadNext64(destination);
-        m_jit.rshift64(AssemblyHelpers::TrustedImm32(JSValue::int52ShiftAmount), destination);
-        m_jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch2);
-        m_jit.boxInt52(destination, destination, scratch1, FPRInfo::fpRegT0);
-        m_jit.move64ToDouble(scratch2, FPRInfo::fpRegT0);
         break;
-    }
             
-    case ValueFormatStrictInt52: {
-        loadNext64(destination);
-        m_jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch2);
-        m_jit.boxInt52(destination, destination, scratch1, FPRInfo::fpRegT0);
-        m_jit.move64ToDouble(scratch2, FPRInfo::fpRegT0);
-        break;
-    }
-            
-    case ValueFormatBoolean: {
+    case ValueFormatBoolean:
         loadNext8(destination);
-        m_jit.or32(MacroAssembler::TrustedImm32(ValueFalse), destination);
-        break;
-    }
-            
-    case ValueFormatJSValue: {
-        loadNext64(destination);
         break;
-    }
             
-    case ValueFormatDouble: {
-        m_jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch1);
-        loadNextDouble(FPRInfo::fpRegT0);
-        m_jit.boxDouble(FPRInfo::fpRegT0, destination);
-        m_jit.move64ToDouble(scratch1, FPRInfo::fpRegT0);
+    case ValueFormatDouble:
+        loadNextDoubleIntoGPR(destination);
         break;
-    }
             
     default:
         RELEASE_ASSERT_NOT_REACHED();
         break;
     }
+    
+    reboxAccordingToFormat(format, m_jit, destination, scratch1, scratch2);
 }
 
 } } // namespace JSC::FTL
index 73b015b..0944fa5 100644 (file)
@@ -100,6 +100,16 @@ public:
         loadNext64(destination);
     }
     
+    void loadNextDoubleIntoGPR(GPRReg destination)
+    {
+        if (m_fprArgumentIndex < FPRInfo::numberOfArgumentRegisters) {
+            m_jit.moveDoubleTo64(FPRInfo::toArgumentRegister(m_fprArgumentIndex++), destination);
+            return;
+        }
+        
+        m_jit.load64(nextAddress(), destination);
+    }
+    
     void loadNextDouble(FPRReg destination)
     {
         ASSERT(
index a376763..ca62094 100644 (file)
 #include "CodeBlockWithJITType.h"
 #include "CCallHelpers.h"
 #include "DFGCommon.h"
+#include "DataView.h"
 #include "Disassembler.h"
+#include "FTLExitThunkGenerator.h"
 #include "FTLJITCode.h"
+#include "FTLThunks.h"
 #include "JITStubs.h"
 #include "LinkBuffer.h"
+#include "RepatchBuffer.h"
 #include <wtf/LLVMHeaders.h>
 
 namespace JSC { namespace FTL {
@@ -42,11 +46,8 @@ namespace JSC { namespace FTL {
 using namespace DFG;
 
 static uint8_t* mmAllocateCodeSection(
-    void* opaqueState, uintptr_t size, unsigned alignment, unsigned sectionID,
-    const char* sectionName)
+    void* opaqueState, uintptr_t size, unsigned alignment, unsigned, const char* sectionName)
 {
-    UNUSED_PARAM(sectionID);
-    UNUSED_PARAM(sectionName);
     
     State& state = *static_cast<State*>(opaqueState);
     
@@ -57,6 +58,7 @@ static uint8_t* mmAllocateCodeSection(
             state.graph.m_vm, size, state.graph.m_codeBlock, JITCompilationMustSucceed);
     
     state.jitCode->addHandle(result);
+    state.codeSectionNames.append(sectionName);
     
     return static_cast<uint8_t*>(result->start());
 }
@@ -66,7 +68,6 @@ static uint8_t* mmAllocateDataSection(
     const char* sectionName, LLVMBool isReadOnly)
 {
     UNUSED_PARAM(sectionID);
-    UNUSED_PARAM(sectionName);
     UNUSED_PARAM(isReadOnly);
 
     State& state = *static_cast<State*>(opaqueState);
@@ -76,7 +77,12 @@ static uint8_t* mmAllocateDataSection(
     RefCountedArray<LSectionWord> section(
         (size + sizeof(LSectionWord) - 1) / sizeof(LSectionWord));
     
-    state.jitCode->addDataSection(section);
+    if (!strcmp(sectionName, "__js_stackmaps"))
+        state.stackmapsSection = section;
+    else {
+        state.jitCode->addDataSection(section);
+        state.dataSectionNames.append(sectionName);
+    }
     
     return bitwise_cast<uint8_t*>(section.data());
 }
@@ -90,6 +96,60 @@ static void mmDestroy(void*)
 {
 }
 
+static void dumpDataSection(RefCountedArray<LSectionWord> section, const char* prefix)
+{
+    for (unsigned j = 0; j < section.size(); ++j) {
+        char buf[32];
+        snprintf(buf, sizeof(buf), "0x%lx", static_cast<unsigned long>(bitwise_cast<uintptr_t>(section.data() + j)));
+        dataLogF("%s%16s: 0x%016llx\n", prefix, buf, static_cast<long long>(section[j]));
+    }
+}
+
+static void fixFunctionBasedOnStackMaps(
+    State& state, CodeBlock* codeBlock, JITCode* jitCode, GeneratedFunction generatedFunction,
+    StackMaps::RecordMap& recordMap)
+{
+    VM& vm = state.graph.m_vm;
+    MacroAssemblerCodeRef osrExitThunk =
+        vm.getCTIStub(osrExitGenerationThunkGenerator);
+    CodeLocationLabel target = CodeLocationLabel(osrExitThunk.code());
+
+    ExitThunkGenerator exitThunkGenerator(state);
+    exitThunkGenerator.emitThunks();
+    if (exitThunkGenerator.didThings()) {
+        OwnPtr<LinkBuffer> linkBuffer = adoptPtr(new LinkBuffer(
+            vm, &exitThunkGenerator, codeBlock, JITCompilationMustSucceed));
+        
+        ASSERT(state.osrExit.size() == state.jitCode->osrExit.size());
+        
+        for (unsigned i = 0; i < state.osrExit.size(); ++i) {
+            OSRExitCompilationInfo& info = state.osrExit[i];
+            OSRExit& exit = jitCode->osrExit[i];
+            
+            linkBuffer->link(info.m_thunkJump, target);
+            
+            info.m_thunkAddress = linkBuffer->locationOf(info.m_thunkLabel);
+            
+            exit.m_patchableCodeOffset = linkBuffer->offsetOf(info.m_thunkJump);
+        }
+        
+        state.finalizer->initializeExitThunksLinkBuffer(linkBuffer.release());
+    }
+
+    RepatchBuffer repatchBuffer(codeBlock);
+    
+    for (unsigned exitIndex = jitCode->osrExit.size(); exitIndex--;) {
+        OSRExitCompilationInfo& info = state.osrExit[exitIndex];
+        OSRExit& exit = jitCode->osrExit[exitIndex];
+        StackMaps::Record& record = recordMap.find(exit.m_stackmapID)->value;
+        
+        repatchBuffer.replaceWithJump(
+            CodeLocationLabel(
+                bitwise_cast<char*>(generatedFunction) + record.instructionOffset),
+            info.m_thunkAddress);
+    }
+}
+
 void compile(State& state)
 {
     char* error = 0;
@@ -143,15 +203,12 @@ void compile(State& state)
     LLVMDisposeExecutionEngine(engine);
 
     if (shouldShowDisassembly()) {
-        // FIXME: fourthTier: FTL memory allocator should be able to tell us which of
-        // these things is actually code or data.
-        // https://bugs.webkit.org/show_bug.cgi?id=116189
         for (unsigned i = 0; i < state.jitCode->handles().size(); ++i) {
             ExecutableMemoryHandle* handle = state.jitCode->handles()[i].get();
             dataLog(
                 "Generated LLVM code for ",
                 CodeBlockWithJITType(state.graph.m_codeBlock, JITCode::DFGJIT),
-                " #", i, ":\n");
+                " #", i, ", ", state.codeSectionNames[i], ":\n");
             disassemble(
                 MacroAssemblerCodePtr(handle->start()), handle->sizeInBytes(),
                 "    ", WTF::dataFile(), LLVMSubset);
@@ -162,14 +219,52 @@ void compile(State& state)
             dataLog(
                 "Generated LLVM data section for ",
                 CodeBlockWithJITType(state.graph.m_codeBlock, JITCode::DFGJIT),
-                " #", i, ":\n");
-            for (unsigned j = 0; j < section.size(); ++j) {
-                char buf[32];
-                snprintf(buf, sizeof(buf), "0x%lx", static_cast<unsigned long>(bitwise_cast<uintptr_t>(section.data() + j)));
-                dataLogF("    %16s: 0x%016llx\n", buf, static_cast<long long>(section[j]));
+                " #", i, ", ", state.dataSectionNames[i], ":\n");
+            dumpDataSection(section, "    ");
+        }
+    }
+    
+    if (state.stackmapsSection.size()) {
+        if (shouldShowDisassembly()) {
+            dataLog(
+                "Generated LLVM stackmaps section for ",
+                CodeBlockWithJITType(state.graph.m_codeBlock, JITCode::DFGJIT), ":\n");
+            dataLog("    Raw data:\n");
+            dumpDataSection(state.stackmapsSection, "    ");
+        }
+        
+        RefPtr<DataView> stackmapsData = DataView::create(
+            ArrayBuffer::create(state.stackmapsSection.data(), state.stackmapsSection.byteSize()));
+        state.jitCode->stackmaps.parse(stackmapsData.get());
+    
+        if (shouldShowDisassembly()) {
+            dataLog("    Structured data:\n");
+            state.jitCode->stackmaps.dumpMultiline(WTF::dataFile(), "        ");
+        }
+        
+        StackMaps::RecordMap recordMap = state.jitCode->stackmaps.getRecordMap();
+        fixFunctionBasedOnStackMaps(
+            state, state.graph.m_codeBlock, state.jitCode.get(), state.generatedFunction,
+            recordMap);
+        
+        if (shouldShowDisassembly()) {
+            for (unsigned i = 0; i < state.jitCode->handles().size(); ++i) {
+                if (state.codeSectionNames[i] != "__text")
+                    continue;
+                
+                ExecutableMemoryHandle* handle = state.jitCode->handles()[i].get();
+                dataLog(
+                    "Generated LLVM code after stackmap-based fix-up for ",
+                    CodeBlockWithJITType(state.graph.m_codeBlock, JITCode::DFGJIT),
+                    " #", i, ", ", state.codeSectionNames[i], ":\n");
+                disassemble(
+                    MacroAssemblerCodePtr(handle->start()), handle->sizeInBytes(),
+                    "    ", WTF::dataFile(), LLVMSubset);
             }
         }
     }
+    
+    state.module = 0; // We no longer own the module.
 }
 
 } } // namespace JSC::FTL
index 4bf88cf..115013e 100644 (file)
@@ -51,12 +51,21 @@ void ExitThunkGenerator::emitThunk(unsigned index)
     OSRExitCompilationInfo& info = m_state.osrExit[index];
     
     info.m_thunkLabel = label();
-    move(TrustedImm32(index), GPRInfo::nonArgGPR0);
+    if (Options::ftlOSRExitUsesStackmap())
+        push(TrustedImm32(index));
+    else
+        move(TrustedImm32(index), GPRInfo::nonArgGPR0);
     info.m_thunkJump = patchableJump();
     
     m_didThings = true;
 }
 
+void ExitThunkGenerator::emitThunks()
+{
+    for (unsigned i = 0; i < m_state.osrExit.size(); ++i)
+        emitThunk(i);
+}
+
 } } // namespace JSC::FTL
 
 #endif // ENABLE(FTL_JIT)
index 435162e..56936b5 100644 (file)
@@ -43,6 +43,7 @@ public:
     ~ExitThunkGenerator();
     
     void emitThunk(unsigned index);
+    void emitThunks();
     
     bool didThings() const { return m_didThings; }
 
index 38b5016..7aaaedd 100644 (file)
@@ -126,6 +126,7 @@ public:
         switch (kind()) {
         case ExitValueInJSStack:
         case ExitValueInJSStackAsInt32:
+        case ExitValueInJSStackAsInt52:
         case ExitValueInJSStackAsDouble:
             return true;
         default:
@@ -152,7 +153,40 @@ public:
         ASSERT(isInJSStackSomehow());
         return VirtualRegister(u.virtualRegister);
     }
-    
+
+    // If it's in the JSStack somehow, this will tell you what format it's in, in a manner
+    // that is compatible with exitArgument().format(). If it's a constant or it's dead, it
+    // will claim to be a JSValue. If it's an argument then it will tell you the argument's
+    // format.
+    ValueFormat valueFormat() const
+    {
+        switch (kind()) {
+        case InvalidExitValue:
+            RELEASE_ASSERT_NOT_REACHED();
+            return InvalidValueFormat;
+            
+        case ExitValueDead:
+        case ExitValueConstant:
+        case ExitValueInJSStack:
+            return ValueFormatJSValue;
+            
+        case ExitValueArgument:
+            return exitArgument().format();
+            
+        case ExitValueInJSStackAsInt32:
+            return ValueFormatInt32;
+            
+        case ExitValueInJSStackAsInt52:
+            return ValueFormatInt52;
+            
+        case ExitValueInJSStackAsDouble:
+            return ValueFormatDouble;
+        }
+        
+        RELEASE_ASSERT_NOT_REACHED();
+        return InvalidValueFormat;
+    }
+
     void dump(PrintStream&) const;
     void dumpInContext(PrintStream&, DumpContext*) const;
     
index 9dd7c0a..52bd5d0 100644 (file)
@@ -40,7 +40,8 @@ void fail(State& state)
 {
     state.graph.m_plan.finalizer = adoptPtr(new FailedFinalizer(state.graph.m_plan));
     
-    LLVMDisposeModule(state.module);
+    if (state.module)
+        LLVMDisposeModule(state.module);
 }
 
 } } // namespace JSC::FTL
index 6f3681d..a617d05 100644 (file)
@@ -44,6 +44,7 @@ namespace JSC { namespace FTL {
     macro(mulWithOverflow64, "llvm.smul.with.overflow.i64", functionType(structType(m_context, int64, boolean), int64, int64)) \
     macro(subWithOverflow32, "llvm.ssub.with.overflow.i32", functionType(structType(m_context, int32, boolean), int32, int32)) \
     macro(subWithOverflow64, "llvm.ssub.with.overflow.i64", functionType(structType(m_context, int64, boolean), int64, int64)) \
+    macro(webkitStackmap, "llvm.webkit.stackmap", functionType(voidType, int32, int32, Variadic)) \
     macro(trap, "llvm.trap", functionType(voidType)) \
     macro(osrExit, "webkit_osr_exit", functionType(voidType, boolean, int32, Variadic))
 
index d3b8e4f..f3d010c 100644 (file)
@@ -32,6 +32,7 @@
 
 #include "DFGCommonData.h"
 #include "FTLOSRExit.h"
+#include "FTLStackMaps.h"
 #include "JITCode.h"
 #include <wtf/LLVMHeaders.h>
 #include <wtf/RefCountedArray.h>
@@ -67,6 +68,7 @@ public:
     
     DFG::CommonData common;
     SegmentedVector<OSRExit, 8> osrExit;
+    StackMaps stackmaps;
     
 private:
     Vector<RefCountedArray<LSectionWord>> m_dataSections;
index 873abb6..10e9434 100644 (file)
@@ -52,6 +52,11 @@ using namespace DFG;
 
 static int compileCounter;
 
+static bool generateExitThunks()
+{
+    return !Options::useLLVMOSRExitIntrinsic() && !Options::ftlOSRExitUsesStackmap();
+}
+
 // Using this instead of typeCheck() helps to reduce the load on LLVM, by creating
 // significantly less dead code.
 #define FTL_TYPE_CHECK(lowValue, highValue, typesPassedThrough, failCondition) do { \
@@ -75,6 +80,7 @@ public:
         , m_exitThunkGenerator(state)
         , m_state(state.graph)
         , m_interpreter(state.graph, m_state)
+        , m_stackmapIDs(0)
     {
     }
     
@@ -3256,8 +3262,10 @@ private:
             
             m_out.branch(failCondition, failCase, continuation);
 
-            m_out.appendTo(m_prologue);
-            info.m_thunkAddress = buildAlloca(m_out.m_builder, m_out.intPtr);
+            if (generateExitThunks()) {
+                m_out.appendTo(m_prologue);
+                info.m_thunkAddressValue = buildAlloca(m_out.m_builder, m_out.intPtr);
+            }
         
             lastNext = m_out.appendTo(failCase, continuation);
         }
@@ -3265,7 +3273,7 @@ private:
         if (Options::ftlOSRExitOmitsMarshalling()) {
             m_out.call(
                 m_out.intToPtr(
-                    m_out.get(info.m_thunkAddress),
+                    m_out.get(info.m_thunkAddressValue),
                     pointerType(functionType(m_out.voidType))));
         } else
             emitOSRExitCall(failCondition, index, exit, info, lowValue, direction, recovery);
@@ -3275,7 +3283,8 @@ private:
             
             m_out.appendTo(continuation, lastNext);
         
-            m_exitThunkGenerator.emitThunk(index);
+            if (generateExitThunks())
+                m_exitThunkGenerator.emitThunk(index);
         }
     }
     
@@ -3341,9 +3350,18 @@ private:
             return;
         }
         
+        if (Options::ftlOSRExitUsesStackmap()) {
+            exit.m_stackmapID = m_stackmapIDs++;
+            arguments.insert(0, m_out.constInt32(MacroAssembler::maxJumpReplacementSize()));
+            arguments.insert(0, m_out.constInt32(exit.m_stackmapID));
+        
+            m_out.call(m_out.webkitStackmapIntrinsic(), arguments);
+            return;
+        }
+        
         m_out.call(
             m_out.intToPtr(
-                m_out.get(info.m_thunkAddress),
+                m_out.get(info.m_thunkAddressValue),
                 pointerType(functionType(m_out.voidType, argumentTypes))),
             arguments);
     }
@@ -3499,7 +3517,7 @@ private:
                 m_out.set(
                     m_out.constIntPtr(
                         linkBuffer->locationOf(info.m_thunkLabel).executableAddress()),
-                    info.m_thunkAddress);
+                    info.m_thunkAddressValue);
             
                 exit.m_patchableCodeOffset = linkBuffer->offsetOf(info.m_thunkJump);
             }
@@ -3696,6 +3714,8 @@ private:
     unsigned m_nodeIndex;
     Node* m_node;
     SpeculationDirection m_direction;
+    
+    uint32_t m_stackmapIDs;
 };
 
 void lowerDFGToLLVM(State& state)
index 824fe7d..45d2fef 100644 (file)
@@ -165,6 +165,8 @@ struct OSRExit : public DFG::OSRExitBase {
     
     Operands<ExitValue> m_values;
     
+    uint32_t m_stackmapID;
+    
     CodeLocationJump codeLocationForRepatch(CodeBlock* ftlCodeBlock) const;
     
     void convertToForward(
index 8e61b52..1e7266f 100644 (file)
@@ -37,13 +37,14 @@ namespace JSC { namespace FTL {
 
 struct OSRExitCompilationInfo {
     OSRExitCompilationInfo()
-        : m_thunkAddress(0)
+        : m_thunkAddressValue(0)
     {
     }
     
     MacroAssembler::Label m_thunkLabel;
     MacroAssembler::PatchableJump m_thunkJump;
-    LValue m_thunkAddress;
+    CodeLocationLabel m_thunkAddress;
+    LValue m_thunkAddressValue;
 };
 
 } } // namespace JSC::FTL
index b9a5686..c596832 100644 (file)
@@ -34,6 +34,7 @@
 #include "FTLExitArgumentForOperand.h"
 #include "FTLJITCode.h"
 #include "FTLOSRExit.h"
+#include "FTLSaveRestore.h"
 #include "Operations.h"
 #include "RepatchBuffer.h"
 
@@ -41,7 +42,135 @@ namespace JSC { namespace FTL {
 
 using namespace DFG;
 
-static void compileStub(
+// This implements two flavors of OSR exit: one that involves having LLVM intrinsics to help
+// OSR exit, and one that doesn't. The one that doesn't will get killed off, so we don't attempt
+// to share code between the two.
+
+static void compileStubWithOSRExitStackmap(
+    unsigned exitID, JITCode* jitCode, OSRExit& exit, VM* vm, CodeBlock* codeBlock)
+{
+    StackMaps::Record* record;
+    
+    for (unsigned i = jitCode->stackmaps.records.size(); i--;) {
+        record = &jitCode->stackmaps.records[i];
+        if (record->patchpointID == exit.m_stackmapID)
+            break;
+    }
+    
+    RELEASE_ASSERT(record->patchpointID == exit.m_stackmapID);
+    
+    CCallHelpers jit(vm, codeBlock);
+    
+    // We need scratch space to save all registers and to build up the JSStack.
+    // Use a scratch buffer to transfer all values.
+    ScratchBuffer* scratchBuffer = vm->scratchBufferForSize(sizeof(EncodedJSValue) * exit.m_values.size() + requiredScratchMemorySizeInBytes());
+    EncodedJSValue* scratch = scratchBuffer ? static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer()) : 0;
+    char* registerScratch = bitwise_cast<char*>(scratch + exit.m_values.size());
+    
+    saveAllRegisters(jit, registerScratch);
+    
+    // Bring the stack back into a sane form.
+    jit.pop(GPRInfo::regT0);
+    
+    // Get the call frame and tag thingies.
+    record->locations[0].restoreInto(jit, jitCode->stackmaps, registerScratch, GPRInfo::callFrameRegister);
+    jit.move(MacroAssembler::TrustedImm64(TagTypeNumber), GPRInfo::tagTypeNumberRegister);
+    jit.move(MacroAssembler::TrustedImm64(TagMask), GPRInfo::tagMaskRegister);
+    
+    // Do some value profiling.
+    if (exit.m_profileValueFormat != InvalidValueFormat) {
+        record->locations[1].restoreInto(jit, jitCode->stackmaps, registerScratch, GPRInfo::regT0);
+        reboxAccordingToFormat(
+            exit.m_profileValueFormat, jit, GPRInfo::regT0, GPRInfo::regT1, GPRInfo::regT2);
+        
+        if (exit.m_kind == BadCache || exit.m_kind == BadIndexingType) {
+            CodeOrigin codeOrigin = exit.m_codeOriginForExitProfile;
+            if (ArrayProfile* arrayProfile = jit.baselineCodeBlockFor(codeOrigin)->getArrayProfile(codeOrigin.bytecodeIndex)) {
+                jit.loadPtr(MacroAssembler::Address(GPRInfo::regT0, JSCell::structureOffset()), GPRInfo::regT1);
+                jit.storePtr(GPRInfo::regT1, arrayProfile->addressOfLastSeenStructure());
+                jit.load8(MacroAssembler::Address(GPRInfo::regT1, Structure::indexingTypeOffset()), GPRInfo::regT1);
+                jit.move(MacroAssembler::TrustedImm32(1), GPRInfo::regT2);
+                jit.lshift32(GPRInfo::regT1, GPRInfo::regT2);
+                jit.or32(GPRInfo::regT2, MacroAssembler::AbsoluteAddress(arrayProfile->addressOfArrayModes()));
+            }
+        }
+        
+        if (!!exit.m_valueProfile)
+            jit.store64(GPRInfo::regT0, exit.m_valueProfile.getSpecFailBucket(0));
+    }
+
+    // Save all state from wherever the exit data tells us it was, into the appropriate place in
+    // the scratch buffer. This doesn't rebox any values yet.
+    
+    for (unsigned index = exit.m_values.size(); index--;) {
+        ExitValue value = exit.m_values[index];
+        
+        switch (value.kind()) {
+        case ExitValueDead:
+            jit.move(MacroAssembler::TrustedImm64(JSValue::encode(jsUndefined())), GPRInfo::regT0);
+            break;
+            
+        case ExitValueConstant:
+            jit.move(MacroAssembler::TrustedImm64(JSValue::encode(value.constant())), GPRInfo::regT0);
+            break;
+            
+        case ExitValueArgument:
+            record->locations[value.exitArgument().argument()].restoreInto(
+                jit, jitCode->stackmaps, registerScratch, GPRInfo::regT0);
+            break;
+            
+        case ExitValueInJSStack:
+        case ExitValueInJSStackAsInt32:
+        case ExitValueInJSStackAsInt52:
+        case ExitValueInJSStackAsDouble:
+            jit.load64(AssemblyHelpers::addressFor(value.virtualRegister()), GPRInfo::regT0);
+            break;
+            
+        default:
+            RELEASE_ASSERT_NOT_REACHED();
+            break;
+        }
+        
+        jit.store64(GPRInfo::regT0, scratch + index);
+    }
+    
+    // Now get state out of the scratch buffer and place it back into the stack. This part does
+    // all reboxing.
+    for (unsigned index = exit.m_values.size(); index--;) {
+        int operand = exit.m_values.operandForIndex(index);
+        ExitValue value = exit.m_values[index];
+        
+        jit.load64(scratch + index, GPRInfo::regT0);
+        reboxAccordingToFormat(
+            value.valueFormat(), jit, GPRInfo::regT0, GPRInfo::regT1, GPRInfo::regT2);
+        jit.store64(GPRInfo::regT0, AssemblyHelpers::addressFor(operand));
+    }
+    
+    handleExitCounts(jit, exit);
+    reifyInlinedCallFrames(jit, exit);
+    
+    jit.move(MacroAssembler::framePointerRegister, MacroAssembler::stackPointerRegister);
+    jit.pop(MacroAssembler::framePointerRegister);
+    jit.pop(GPRInfo::nonArgGPR0); // ignore the result.
+    
+    if (exit.m_lastSetOperand.isValid()) {
+        jit.load64(
+            AssemblyHelpers::addressFor(exit.m_lastSetOperand), GPRInfo::cachedResultRegister);
+    }
+    
+    adjustAndJumpToTarget(jit, exit);
+    
+    LinkBuffer patchBuffer(*vm, &jit, codeBlock);
+    exit.m_code = FINALIZE_CODE_IF(
+        shouldShowDisassembly(),
+        patchBuffer,
+        ("FTL OSR exit #%u (bc#%u, %s) from %s, with operands = %s",
+            exitID, exit.m_codeOrigin.bytecodeIndex,
+            exitKindToString(exit.m_kind), toCString(*codeBlock).data(),
+            toCString(ignoringContext<DumpContext>(exit.m_values)).data()));
+}
+
+static void compileStubWithoutOSRExitStackmap(
     unsigned exitID, OSRExit& exit, VM* vm, CodeBlock* codeBlock)
 {
     CCallHelpers jit(vm, codeBlock);
@@ -219,11 +348,15 @@ extern "C" void* compileFTLOSRExit(ExecState* exec, unsigned exitID)
     // really be profitable.
     DeferGCForAWhile deferGC(vm->heap);
 
-    OSRExit& exit = codeBlock->jitCode()->ftl()->osrExit[exitID];
+    JITCode* jitCode = codeBlock->jitCode()->ftl();
+    OSRExit& exit = jitCode->osrExit[exitID];
     
     prepareCodeOriginForOSRExit(exec, exit.m_codeOrigin);
     
-    compileStub(exitID, exit, vm, codeBlock);
+    if (Options::ftlOSRExitUsesStackmap())
+        compileStubWithOSRExitStackmap(exitID, jitCode, exit, vm, codeBlock);
+    else
+        compileStubWithoutOSRExitStackmap(exitID, exit, vm, codeBlock);
     
     RepatchBuffer repatchBuffer(codeBlock);
     repatchBuffer.relink(
diff --git a/Source/JavaScriptCore/ftl/FTLSaveRestore.cpp b/Source/JavaScriptCore/ftl/FTLSaveRestore.cpp
new file mode 100644 (file)
index 0000000..df45394
--- /dev/null
@@ -0,0 +1,97 @@
+/*
+ * Copyright (C) 2013 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+ */
+
+#include "config.h"
+#include "FTLSaveRestore.h"
+
+#if ENABLE(FTL_JIT)
+
+#include "FPRInfo.h"
+#include "GPRInfo.h"
+#include "MacroAssembler.h"
+
+namespace JSC { namespace FTL {
+
+static size_t bytesForGPRs()
+{
+    return (MacroAssembler::lastRegister() - MacroAssembler::firstRegister() + 1) * sizeof(int64_t);
+}
+
+size_t requiredScratchMemorySizeInBytes()
+{
+    return bytesForGPRs() + FPRInfo::numberOfArgumentRegisters * sizeof(double);
+}
+
+size_t offsetOfGPR(GPRReg reg)
+{
+    return (reg - MacroAssembler::firstRegister()) * sizeof(int64_t);
+}
+
+void saveAllRegisters(MacroAssembler& jit, char* scratchMemory)
+{
+    // Get the first register out of the way, so that we can use it as a pointer.
+    jit.poke64(MacroAssembler::firstRealRegister(), 0);
+    jit.move(MacroAssembler::TrustedImmPtr(scratchMemory), MacroAssembler::firstRealRegister());
+    
+    // Get all of the other GPRs out of the way.
+    for (MacroAssembler::RegisterID reg = MacroAssembler::secondRealRegister(); reg <= MacroAssembler::lastRegister(); reg = MacroAssembler::nextRegister(reg))
+        jit.store64(reg, MacroAssembler::Address(MacroAssembler::firstRealRegister(), offsetOfGPR(reg)));
+    
+    // Restore the first register into the second one and save it.
+    jit.peek64(MacroAssembler::secondRealRegister(), 0);
+    jit.store64(MacroAssembler::secondRealRegister(), MacroAssembler::Address(MacroAssembler::firstRealRegister(), offsetOfGPR(MacroAssembler::firstRealRegister())));
+    
+    // Save all FP argument registers.
+    // FIXME: We should actually be saving all FP registers.
+    // https://bugs.webkit.org/show_bug.cgi?id=122518
+    for (unsigned i = 0; i < FPRInfo::numberOfArgumentRegisters; ++i) {
+        jit.move(MacroAssembler::TrustedImmPtr(scratchMemory + bytesForGPRs() + sizeof(double) * i), GPRInfo::regT0);
+        jit.storeDouble(FPRInfo::toArgumentRegister(i), GPRInfo::regT0);
+    }
+}
+
+void restoreAllRegisters(MacroAssembler& jit, char* scratchMemory)
+{
+    // Restore all FP argument registers.
+    // FIXME: We should actually be restoring all FP registers.
+    // https://bugs.webkit.org/show_bug.cgi?id=122518
+    for (unsigned i = 0; i < FPRInfo::numberOfArgumentRegisters; ++i) {
+        jit.move(MacroAssembler::TrustedImmPtr(scratchMemory + bytesForGPRs() + sizeof(double) * i), GPRInfo::regT0);
+        jit.loadDouble(GPRInfo::regT0, FPRInfo::toArgumentRegister(i));
+    }
+    
+    // Now, restore all GPRs.
+    jit.move(MacroAssembler::TrustedImmPtr(scratchMemory), MacroAssembler::firstRealRegister());
+    
+    for (MacroAssembler::RegisterID reg = MacroAssembler::secondRealRegister(); reg <= MacroAssembler::lastRegister(); reg = MacroAssembler::nextRegister(reg))
+        jit.load64(MacroAssembler::Address(MacroAssembler::firstRealRegister(), offsetOfGPR(reg)), reg);
+    
+    jit.load64(MacroAssembler::Address(MacroAssembler::firstRealRegister(), offsetOfGPR(MacroAssembler::firstRealRegister())), MacroAssembler::firstRealRegister());
+}
+
+} } // namespace JSC::FTL
+
+#endif // ENABLE(FTL_JIT)
+
diff --git a/Source/JavaScriptCore/ftl/FTLSaveRestore.h b/Source/JavaScriptCore/ftl/FTLSaveRestore.h
new file mode 100644 (file)
index 0000000..1cd0b8b
--- /dev/null
@@ -0,0 +1,57 @@
+/*
+ * Copyright (C) 2013 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+ */
+
+#ifndef FTLSaveRestore_h
+#define FTLSaveRestore_h
+
+#include <wtf/Platform.h>
+
+#if ENABLE(FTL_JIT)
+
+#include "GPRInfo.h"
+
+namespace JSC {
+
+class MacroAssembler;
+
+namespace FTL {
+
+size_t requiredScratchMemorySizeInBytes();
+
+size_t offsetOfGPR(GPRReg);
+
+// Assumes that top-of-stack can be used as a pointer-sized scratchpad. Saves all of
+// the registers into the scratch buffer such that RegisterID * sizeof(int64_t) is the
+// offset of every register.
+void saveAllRegisters(MacroAssembler& jit, char* scratchMemory);
+
+void restoreAllRegisters(MacroAssembler& jit, char* scratchMemory);
+
+} } // namespace JSC::FTL
+
+#endif // ENABLE(FTL_JIT)
+
+#endif // FTLSaveRestore_h
+
diff --git a/Source/JavaScriptCore/ftl/FTLStackMaps.cpp b/Source/JavaScriptCore/ftl/FTLStackMaps.cpp
new file mode 100644 (file)
index 0000000..32a8b30
--- /dev/null
@@ -0,0 +1,256 @@
+/*
+ * Copyright (C) 2013 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+ */
+
+#include "config.h"
+#include "FTLStackMaps.h"
+
+#if ENABLE(FTL_JIT)
+
+#include "FTLSaveRestore.h"
+#include <wtf/CommaPrinter.h>
+#include <wtf/DataLog.h>
+#include <wtf/ListDump.h>
+
+namespace JSC { namespace FTL {
+
+template<typename T>
+T readObject(DataView* view, unsigned& offset)
+{
+    T result;
+    result.parse(view, offset);
+    return result;
+}
+
+void StackMaps::Constant::parse(DataView* view, unsigned& offset)
+{
+    integer = view->read<int64_t>(offset, true);
+}
+
+void StackMaps::Constant::dump(PrintStream& out) const
+{
+    out.printf("0x%016llx", integer);
+}
+
+void StackMaps::Location::parse(DataView* view, unsigned& offset)
+{
+    uint16_t taggedReg = view->read<uint16_t>(offset, true);
+    if (static_cast<int16_t>(taggedReg) < 0) {
+        dataLog(
+            "Detected a negative tagged register ", static_cast<int16_t>(taggedReg),
+            " at offset ", offset, "\n");
+        RELEASE_ASSERT_NOT_REACHED();
+    }
+    dwarfRegNum = taggedReg & ((1 << 12) - 1);
+    kind = static_cast<Kind>(taggedReg >> 12);
+
+    this->offset = view->read<int16_t>(offset, true);
+}
+
+void StackMaps::Location::dump(PrintStream& out) const
+{
+    out.print("(", kind, ", reg", dwarfRegNum, ", ", offset, ")");
+}
+
+bool StackMaps::Location::involvesGPR() const
+{
+    return isGPR() || kind == Indirect;
+}
+
+#if CPU(X86_64) // CPU cases for StackMaps::Location
+// This decodes Dwarf flavour 0 for x86-64.
+bool StackMaps::Location::isGPR() const
+{
+    return kind == Register && dwarfRegNum < 16;
+}
+
+GPRReg StackMaps::Location::gpr() const
+{
+    // Stupidly, Dwarf doesn't number the registers in the same way as the architecture;
+    // for example, the architecture encodes CX as 1 and DX as 2 while Dwarf does the
+    // opposite. Hence we need the switch.
+    
+    switch (dwarfRegNum) {
+    case 0:
+        return X86Registers::eax;
+    case 1:
+        return X86Registers::edx;
+    case 2:
+        return X86Registers::ecx;
+    case 3:
+        return X86Registers::ebx;
+    case 4:
+        return X86Registers::esi;
+    case 5:
+        return X86Registers::edi;
+    case 6:
+        return X86Registers::ebp;
+    case 7:
+        return X86Registers::esp;
+    default:
+        RELEASE_ASSERT(dwarfRegNum < 16);
+        // Registers r8..r15 are numbered sensibly.
+        return static_cast<GPRReg>(dwarfRegNum);
+    }
+}
+
+void StackMaps::Location::restoreInto(
+    MacroAssembler& jit, const StackMaps& stackmaps, char* savedRegisters, GPRReg result)
+{
+    if (isGPR()) {
+        jit.load64(savedRegisters + offsetOfGPR(gpr()), result);
+        return;
+    }
+    
+    switch (kind) {
+    case Register:
+        // FIXME: We need to handle the other registers.
+        // https://bugs.webkit.org/show_bug.cgi?id=122518
+        RELEASE_ASSERT_NOT_REACHED();
+        return;
+        
+    case Indirect:
+        jit.load64(savedRegisters + offsetOfGPR(gpr()), result);
+        jit.load64(MacroAssembler::Address(result, offset), result);
+        return;
+        
+    case Constant:
+        jit.move(MacroAssembler::TrustedImm32(offset), result);
+        return;
+        
+    case ConstantIndex:
+        jit.move(MacroAssembler::TrustedImm64(stackmaps.constants[offset].integer), result);
+        return;
+        
+    case Unprocessed:
+        // Should never see this - it's an enumeration entry on LLVM's side that means that
+        // it hasn't processed this location.
+        RELEASE_ASSERT_NOT_REACHED();
+        return;
+    }
+    
+    RELEASE_ASSERT_NOT_REACHED();
+}
+#else // CPU cases for StackMaps::Location
+#error "CPU architecture not supported."
+#endif // CPU cases for StackMaps::Location
+
+bool StackMaps::Record::parse(DataView* view, unsigned& offset)
+{
+    patchpointID = view->read<uint32_t>(offset, true);
+    if (static_cast<int32_t>(patchpointID) < 0)
+        return false;
+    
+    instructionOffset = view->read<uint32_t>(offset, true);
+    flags = view->read<uint16_t>(offset, true);
+    
+    unsigned length = view->read<uint16_t>(offset, true);
+    while (length--)
+        locations.append(readObject<Location>(view, offset));
+    
+    return true;
+}
+
+void StackMaps::Record::dump(PrintStream& out) const
+{
+    out.print(
+        "(#", patchpointID, ", offset = ", instructionOffset, ", flags = ", flags,
+        ", [", listDump(locations), "])");
+}
+
+bool StackMaps::parse(DataView* view)
+{
+    unsigned offset = 0;
+    
+    uint32_t numConstants = view->read<uint32_t>(offset, true);
+    while (numConstants--)
+        constants.append(readObject<Constant>(view, offset));
+    
+    uint32_t numRecords = view->read<uint32_t>(offset, true);
+    while (numRecords--) {
+        Record record;
+        if (!record.parse(view, offset))
+            return false;
+        records.append(record);
+    }
+    
+    return true;
+}
+
+void StackMaps::dump(PrintStream& out) const
+{
+    out.print("Constants:[", listDump(constants), "], Records:[", listDump(records), "]");
+}
+
+void StackMaps::dumpMultiline(PrintStream& out, const char* prefix) const
+{
+    out.print(prefix, "Constants:\n");
+    for (unsigned i = 0; i < constants.size(); ++i)
+        out.print(prefix, "    ", constants[i], "\n");
+    out.print(prefix, "Records:\n");
+    for (unsigned i = 0; i < records.size(); ++i)
+        out.print(prefix, "    ", records[i], "\n");
+}
+
+StackMaps::RecordMap StackMaps::getRecordMap() const
+{
+    RecordMap result;
+    for (unsigned i = records.size(); i--;)
+        result.add(records[i].patchpointID, records[i]);
+    return result;
+}
+
+} } // namespace JSC::FTL
+
+namespace WTF {
+
+using namespace JSC::FTL;
+
+void printInternal(PrintStream& out, StackMaps::Location::Kind kind)
+{
+    switch (kind) {
+    case StackMaps::Location::Unprocessed:
+        out.print("Unprocessed");
+        return;
+    case StackMaps::Location::Register:
+        out.print("Register");
+        return;
+    case StackMaps::Location::Indirect:
+        out.print("Indirect");
+        return;
+    case StackMaps::Location::Constant:
+        out.print("Constant");
+        return;
+    case StackMaps::Location::ConstantIndex:
+        out.print("ConstantIndex");
+        return;
+    }
+    dataLog("Unrecognized kind: ", static_cast<int>(kind), "\n");
+    RELEASE_ASSERT_NOT_REACHED();
+}
+
+} // namespace WTF
+
+#endif // ENABLE(FTL_JIT)
+
diff --git a/Source/JavaScriptCore/ftl/FTLStackMaps.h b/Source/JavaScriptCore/ftl/FTLStackMaps.h
new file mode 100644 (file)
index 0000000..4e4c427
--- /dev/null
@@ -0,0 +1,115 @@
+/*
+ * Copyright (C) 2013 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+ */
+
+#ifndef FTLStackMaps_h
+#define FTLStackMaps_h
+
+#include <wtf/Platform.h>
+
+#if ENABLE(FTL_JIT)
+
+#include "DataView.h"
+#include "GPRInfo.h"
+#include "JSCJSValue.h"
+#include <wtf/HashMap.h>
+
+namespace JSC {
+
+class MacroAssembler;
+
+namespace FTL {
+
+struct StackMaps {
+    union Constant {
+        EncodedJSValue value;
+        int64_t integer;
+        
+        void parse(DataView*, unsigned& offset);
+        void dump(PrintStream& out) const;
+    };
+    
+    struct Location {
+        enum Kind {
+            Unprocessed,
+            Register,
+            Indirect,
+            Constant,
+            ConstantIndex
+        };
+        
+        uint16_t dwarfRegNum; // Represented as a 12-bit int in the section.
+        Kind kind;
+        int16_t offset;
+        
+        void parse(DataView*, unsigned& offset);
+        void dump(PrintStream& out) const;
+        
+        bool isGPR() const;
+        bool involvesGPR() const;
+        GPRReg gpr() const;
+        
+        // Assuming that all registers are saved to the savedRegisters buffer according
+        // to FTLSaveRestore convention, this loads the value into the given register.
+        // The code that this generates isn't exactly super fast.
+        void restoreInto(
+            MacroAssembler&, const StackMaps& stackmaps, char* savedRegisters,
+            GPRReg result);
+    };
+    
+    struct Record {
+        uint32_t patchpointID;
+        uint32_t instructionOffset;
+        uint16_t flags;
+        
+        Vector<Location> locations;
+        
+        bool parse(DataView*, unsigned& offset);
+        void dump(PrintStream&) const;
+    };
+    
+    Vector<Constant> constants;
+    Vector<Record> records;
+    
+    bool parse(DataView*); // Returns true on parse success, false on failure. Failure means that LLVM is signaling compile failure to us.
+    void dump(PrintStream&) const;
+    void dumpMultiline(PrintStream&, const char* prefix) const;
+    
+    typedef HashMap<uint32_t, Record, WTF::IntHash<uint32_t>, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> RecordMap;
+    
+    RecordMap getRecordMap() const;
+};
+
+} } // namespace JSC::FTL
+
+namespace WTF {
+
+void printInternal(PrintStream&, JSC::FTL::StackMaps::Location::Kind);
+
+} // namespace WTF
+
+#endif // ENABLE(FTL_JIT)
+
+#endif // FTLStackMaps_h
+
index cef0fe3..db21a58 100644 (file)
@@ -36,6 +36,7 @@
 #include "FTLJITCode.h"
 #include "FTLJITFinalizer.h"
 #include "FTLOSRExitCompilationInfo.h"
+#include "FTLStackMaps.h"
 #include <wtf/Noncopyable.h>
 
 namespace JSC { namespace FTL {
@@ -57,6 +58,9 @@ public:
     Vector<OSRExitCompilationInfo> osrExit;
     GeneratedFunction generatedFunction;
     JITFinalizer* finalizer;
+    Vector<CString> codeSectionNames;
+    Vector<CString> dataSectionNames;
+    RefCountedArray<LSectionWord> stackmapsSection;
     
     void dumpState(const char* when);
 };
index 6c846f5..594332e 100644 (file)
 
 #if ENABLE(FTL_JIT)
 
+#include "AssemblyHelpers.h"
 #include "FPRInfo.h"
 #include "FTLOSRExitCompiler.h"
+#include "FTLSaveRestore.h"
 #include "GPRInfo.h"
 #include "LinkBuffer.h"
-#include "MacroAssembler.h"
 
 namespace JSC { namespace FTL {
 
@@ -40,7 +41,10 @@ using namespace DFG;
 
 MacroAssemblerCodeRef osrExitGenerationThunkGenerator(VM* vm)
 {
-    MacroAssembler jit;
+    AssemblyHelpers jit(vm, 0);
+    
+    // Note that in the ftlOSRExitUsesStackmap() case, the "return address" will be the
+    // OSR exit ID.
     
     // Pretend that we're a C call frame.
     jit.push(MacroAssembler::framePointerRegister);
@@ -48,44 +52,46 @@ MacroAssemblerCodeRef osrExitGenerationThunkGenerator(VM* vm)
     jit.push(GPRInfo::regT0);
     jit.push(GPRInfo::regT0);
     
-    size_t scratchSize = sizeof(EncodedJSValue) * (GPRInfo::numberOfArgumentRegisters + FPRInfo::numberOfArgumentRegisters);
-    ScratchBuffer* scratchBuffer = vm->scratchBufferForSize(scratchSize);
-    EncodedJSValue* buffer = static_cast<EncodedJSValue*>(scratchBuffer->dataBuffer());
+    if (!Options::ftlOSRExitUsesStackmap())
+        jit.poke(GPRInfo::nonArgGPR0, 1);
+    
+    ScratchBuffer* scratchBuffer = vm->scratchBufferForSize(requiredScratchMemorySizeInBytes());
+    char* buffer = static_cast<char*>(scratchBuffer->dataBuffer());
     
-    for (unsigned i = 0; i < GPRInfo::numberOfArgumentRegisters; ++i)
-        jit.store64(GPRInfo::toArgumentRegister(i), buffer + i);
-    for (unsigned i = 0; i < FPRInfo::numberOfArgumentRegisters; ++i) {
-        jit.move(MacroAssembler::TrustedImmPtr(buffer + GPRInfo::numberOfArgumentRegisters + i), GPRInfo::nonArgGPR1);
-        jit.storeDouble(FPRInfo::toArgumentRegister(i), GPRInfo::nonArgGPR1);
-    }
+    saveAllRegisters(jit, buffer);
     
     // Tell GC mark phase how much of the scratch buffer is active during call.
     jit.move(MacroAssembler::TrustedImmPtr(scratchBuffer->activeLengthPtr()), GPRInfo::nonArgGPR1);
-    jit.storePtr(MacroAssembler::TrustedImmPtr(scratchSize), GPRInfo::nonArgGPR1);
+    jit.storePtr(MacroAssembler::TrustedImmPtr(requiredScratchMemorySizeInBytes()), GPRInfo::nonArgGPR1);
 
     // argument 0 is already the call frame.
-    jit.move(GPRInfo::nonArgGPR0, GPRInfo::argumentGPR1);
+    if (Options::ftlOSRExitUsesStackmap())
+        jit.peek(GPRInfo::argumentGPR1, 3);
+    else
+        jit.peek(GPRInfo::argumentGPR1, 1);
     MacroAssembler::Call functionCall = jit.call();
     
-    // Make sure that we're not using the return register if it's an argument register.
-    jit.move(GPRInfo::returnValueGPR, GPRInfo::nonArgGPR0);
+    // At this point we want to make a tail call to what was returned to us in the
+    // returnValueGPR. But at the same time as we do this, we must restore all registers.
+    // The way we will accomplish this is by arranging to have the tail call target in the
+    // return address "slot" (be it a register or the stack).
+    
+    jit.move(GPRInfo::returnValueGPR, GPRInfo::regT0);
     
     // Prepare for tail call.
-    jit.pop(GPRInfo::nonArgGPR1);
-    jit.pop(GPRInfo::nonArgGPR1);
+    jit.pop(GPRInfo::regT1);
+    jit.pop(GPRInfo::regT1);
     jit.pop(MacroAssembler::framePointerRegister);
     
-    jit.move(MacroAssembler::TrustedImmPtr(scratchBuffer->activeLengthPtr()), GPRInfo::nonArgGPR1);
-    jit.storePtr(MacroAssembler::TrustedImmPtr(0), GPRInfo::nonArgGPR1);
+    // At this point we're sitting on the return address - so if we did a jump right now, the
+    // tail-callee would be happy. Instead we'll stash the callee in the return address and then
+    // restore all registers.
     
-    for (unsigned i = 0; i < FPRInfo::numberOfArgumentRegisters; ++i) {
-        jit.move(MacroAssembler::TrustedImmPtr(buffer + GPRInfo::numberOfArgumentRegisters + i), GPRInfo::nonArgGPR1);
-        jit.loadDouble(GPRInfo::nonArgGPR1, FPRInfo::toArgumentRegister(i));
-    }
-    for (unsigned i = 0; i < GPRInfo::numberOfArgumentRegisters; ++i)
-        jit.load64(buffer + i, GPRInfo::toArgumentRegister(i));
+    jit.restoreReturnAddressBeforeReturn(GPRInfo::regT0);
     
-    jit.jump(GPRInfo::nonArgGPR0);
+    restoreAllRegisters(jit, buffer);
+
+    jit.ret();
     
     LinkBuffer patchBuffer(*vm, &jit, GLOBAL_THUNK_ID);
     patchBuffer.link(functionCall, compileFTLOSRExit);
index fa200d7..2e2bec7 100644 (file)
 
 #if ENABLE(FTL_JIT)
 
+#include "AssemblyHelpers.h"
+
+namespace JSC { namespace FTL {
+
+void reboxAccordingToFormat(
+    ValueFormat format, AssemblyHelpers& jit, GPRReg value, GPRReg scratch1, GPRReg scratch2)
+{
+    switch (format) {
+    case ValueFormatInt32: {
+        jit.or64(GPRInfo::tagTypeNumberRegister, value);
+        break;
+    }
+    
+    case ValueFormatUInt32: {
+        jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch2);
+        jit.boxInt52(value, value, scratch1, FPRInfo::fpRegT0);
+        jit.move64ToDouble(scratch2, FPRInfo::fpRegT0);
+        break;
+    }
+    
+    case ValueFormatInt52: {
+        jit.rshift64(AssemblyHelpers::TrustedImm32(JSValue::int52ShiftAmount), value);
+        jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch2);
+        jit.boxInt52(value, value, scratch1, FPRInfo::fpRegT0);
+        jit.move64ToDouble(scratch2, FPRInfo::fpRegT0);
+        break;
+    }
+    
+    case ValueFormatStrictInt52: {
+        jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch2);
+        jit.boxInt52(value, value, scratch1, FPRInfo::fpRegT0);
+        jit.move64ToDouble(scratch2, FPRInfo::fpRegT0);
+        break;
+    }
+    
+    case ValueFormatBoolean: {
+        jit.or32(MacroAssembler::TrustedImm32(ValueFalse), value);
+        break;
+    }
+    
+    case ValueFormatJSValue: {
+        // Done already!
+        break;
+    }
+    
+    case ValueFormatDouble: {
+        jit.moveDoubleTo64(FPRInfo::fpRegT0, scratch1);
+        jit.move64ToDouble(value, FPRInfo::fpRegT0);
+        jit.boxDouble(FPRInfo::fpRegT0, value);
+        jit.move64ToDouble(scratch1, FPRInfo::fpRegT0);
+        break;
+    }
+    
+    default:
+        RELEASE_ASSERT_NOT_REACHED();
+        break;
+    }
+}
+
+} } // namespace JSC::FTL
+
 namespace WTF {
 
 using namespace JSC::FTL;
index f14e8c7..40ac775 100644 (file)
 
 #if ENABLE(FTL_JIT)
 
+#include "GPRInfo.h"
 #include <wtf/PrintStream.h>
 
-namespace JSC { namespace FTL {
+namespace JSC {
+
+class AssemblyHelpers;
+
+namespace FTL {
 
 // Note that this is awkwardly similar to DataFormat in other parts of JSC, except that
 // unlike DataFormat and like ValueRecovery, it distinguishes between UInt32 and Int32.
@@ -48,6 +53,9 @@ enum ValueFormat {
     ValueFormatDouble
 };
 
+void reboxAccordingToFormat(
+    ValueFormat, AssemblyHelpers&, GPRReg value, GPRReg scratch1, GPRReg scratch2);
+
 } } // namespace JSC::FTL
 
 namespace WTF {
index d52df78..0c83d6e 100644 (file)
@@ -43,6 +43,12 @@ PassRefPtr<DataView> DataView::create(
     return adoptRef(new DataView(buffer, byteOffset, byteLength));
 }
 
+PassRefPtr<DataView> DataView::create(PassRefPtr<ArrayBuffer> passedBuffer)
+{
+    RefPtr<ArrayBuffer> buffer = passedBuffer;
+    return create(buffer, 0, buffer->byteLength());
+}
+
 JSArrayBufferView* DataView::wrap(ExecState* exec, JSGlobalObject* globalObject)
 {
     return JSDataView::create(
index 8ce6332..7521ded 100644 (file)
@@ -38,6 +38,7 @@ protected:
     
 public:
     JS_EXPORT_PRIVATE static PassRefPtr<DataView> create(PassRefPtr<ArrayBuffer>, unsigned byteOffset, unsigned length);
+    static PassRefPtr<DataView> create(PassRefPtr<ArrayBuffer>);
     
     virtual unsigned byteLength() const OVERRIDE
     {
@@ -68,6 +69,15 @@ public:
     }
     
     template<typename T>
+    T read(unsigned& offset, bool littleEndian, bool* status = 0)
+    {
+        T result = this->template get<T>(offset, littleEndian, status);
+        if (!status || *status)
+            offset += sizeof(T);
+        return result;
+    }
+    
+    template<typename T>
     void set(unsigned offset, T value, bool littleEndian, bool* status = 0)
     {
         if (status) {
index d5be789..6e520c1 100644 (file)
@@ -123,9 +123,11 @@ typedef OptionRange optionRange;
     v(bool, useLLVMSmallCodeModel, false) \
     v(bool, ftlTrapsOnOSRExit, false) \
     v(bool, ftlOSRExitOmitsMarshalling, false) \
+    v(bool, ftlOSRExitUsesStackmap, false) \
     v(bool, useLLVMOSRExitIntrinsic, false) \
     v(bool, dumpLLVMIR, false) \
-    v(bool, llvmAlwaysFails, false) \
+    v(bool, llvmAlwaysFailsBeforeCompile, false) \
+    v(bool, llvmAlwaysFailsBeforeLink, false) \
     v(unsigned, llvmBackendOptimizationLevel, 2) \
     v(unsigned, llvmOptimizationLevel, 2) \
     v(unsigned, llvmSizeLevel, 0) \