Change WebAssembly calling conventions
authortzagallo@apple.com <tzagallo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 17 Sep 2019 23:52:28 +0000 (23:52 +0000)
committertzagallo@apple.com <tzagallo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 17 Sep 2019 23:52:28 +0000 (23:52 +0000)
https://bugs.webkit.org/show_bug.cgi?id=201799

Reviewed by Saam Barati.

Currently, the Wasm::Callee writes itself to CallFrameSlot::callee. However, this won't work when
we have the Wasm interpreter, since we need the callee in order to know which function are we executing.
This patch changes the calling conventions in preparation for the interpreter, so that the caller
becomes responsible for writing the callee into the call frame.
However, there are exceptions to this rule: stubs can still write to the callee slot, since they are individually
generated and will still be present in the interpreter. We keep this design to avoid emitting unnecessary
code when we know statically who is the callee:
- Caller writes to call frame: intra-module direct wasm calls, indirect wasm calls, JS-to-wasm stub (new frame), JS-to-wasm IC.
- Callee writes to call frame: inter-module wasm-to-wasm stub, JS-to-wasm stub (callee frame), wasm-to-JS stub, OMG osr entry

Additionally, this patch also changes it so that the callee keeps track of its callers, instead of having a global mapping
of calls in the Wasm::CodeBlock. This makes it easier to repatch all callers of a given Callee when it tiers up.

* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* wasm/WasmAirIRGenerator.cpp:
(JSC::Wasm::AirIRGenerator::AirIRGenerator):
(JSC::Wasm::AirIRGenerator::addCall):
(JSC::Wasm::AirIRGenerator::addCallIndirect):
(JSC::Wasm::parseAndCompileAir):
* wasm/WasmAirIRGenerator.h:
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::addCall):
(JSC::Wasm::B3IRGenerator::addCallIndirect):
(JSC::Wasm::parseAndCompile):
* wasm/WasmB3IRGenerator.h:
* wasm/WasmBBQPlan.cpp:
(JSC::Wasm::BBQPlan::BBQPlan):
(JSC::Wasm::BBQPlan::prepare):
(JSC::Wasm::BBQPlan::compileFunctions):
(JSC::Wasm::BBQPlan::complete):
* wasm/WasmBBQPlan.h:
* wasm/WasmBBQPlanInlines.h:
(JSC::Wasm::BBQPlan::initializeCallees):
* wasm/WasmBinding.cpp:
(JSC::Wasm::wasmToWasm):
* wasm/WasmCallee.cpp:
(JSC::Wasm::Callee::Callee):
(JSC::Wasm::repatchMove):
(JSC::Wasm::repatchCall):
(JSC::Wasm::BBQCallee::addCaller):
(JSC::Wasm::BBQCallee::addAndLinkCaller):
(JSC::Wasm::BBQCallee::repatchCallers):
* wasm/WasmCallee.h:
(JSC::Wasm::Callee::entrypoint):
(JSC::Wasm::Callee::code const):
(JSC::Wasm::Callee::calleeSaveRegisters):
* wasm/WasmCallingConvention.h:
(JSC::Wasm::CallingConvention::setupFrameInPrologue const):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::CodeBlock):
* wasm/WasmCodeBlock.h:
(JSC::Wasm::CodeBlock::embedderEntrypointCalleeFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::wasmBBQCalleeFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::entrypointLoadLocationFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::boxedCalleeLoadLocationFromFunctionIndexSpace):
* wasm/WasmEmbedder.h:
* wasm/WasmFormat.h:
(JSC::Wasm::WasmToWasmImportableFunction::offsetOfBoxedCalleeLoadLocation):
* wasm/WasmInstance.h:
(JSC::Wasm::Instance::offsetOfBoxedCalleeLoadLocation):
* wasm/WasmOMGForOSREntryPlan.cpp:
(JSC::Wasm::OMGForOSREntryPlan::OMGForOSREntryPlan):
(JSC::Wasm::OMGForOSREntryPlan::work):
* wasm/WasmOMGForOSREntryPlan.h:
* wasm/WasmOMGPlan.cpp:
(JSC::Wasm::OMGPlan::OMGPlan):
(JSC::Wasm::OMGPlan::work):
* wasm/WasmOMGPlan.h:
* wasm/WasmOperations.cpp:
(JSC::Wasm::triggerOMGReplacementCompile):
(JSC::Wasm::doOSREntry):
(JSC::Wasm::triggerOSREntryNow):
* wasm/js/JSToWasm.cpp:
(JSC::Wasm::createJSToWasmWrapper):
* wasm/js/JSToWasm.h:
* wasm/js/WebAssemblyFunction.cpp:
(JSC::WebAssemblyFunction::jsCallEntrypointSlow):
(JSC::WebAssemblyFunction::create):
(JSC::WebAssemblyFunction::WebAssemblyFunction):
* wasm/js/WebAssemblyFunction.h:
* wasm/js/WebAssemblyModuleRecord.cpp:
(JSC::WebAssemblyModuleRecord::link):
(JSC::WebAssemblyModuleRecord::evaluate):
* wasm/js/WebAssemblyWrapperFunction.cpp:
(JSC::WebAssemblyWrapperFunction::create):

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@250002 268f45cc-cd09-0410-ab3c-d52691b4dbfc

30 files changed:
Source/JavaScriptCore/CMakeLists.txt
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/wasm/WasmAirIRGenerator.cpp
Source/JavaScriptCore/wasm/WasmAirIRGenerator.h
Source/JavaScriptCore/wasm/WasmB3IRGenerator.cpp
Source/JavaScriptCore/wasm/WasmB3IRGenerator.h
Source/JavaScriptCore/wasm/WasmBBQPlan.cpp
Source/JavaScriptCore/wasm/WasmBBQPlan.h
Source/JavaScriptCore/wasm/WasmBBQPlanInlines.h
Source/JavaScriptCore/wasm/WasmBinding.cpp
Source/JavaScriptCore/wasm/WasmCallee.cpp
Source/JavaScriptCore/wasm/WasmCallee.h
Source/JavaScriptCore/wasm/WasmCallingConvention.h
Source/JavaScriptCore/wasm/WasmCodeBlock.cpp
Source/JavaScriptCore/wasm/WasmCodeBlock.h
Source/JavaScriptCore/wasm/WasmEmbedder.h
Source/JavaScriptCore/wasm/WasmFormat.h
Source/JavaScriptCore/wasm/WasmInstance.h
Source/JavaScriptCore/wasm/WasmOMGForOSREntryPlan.cpp
Source/JavaScriptCore/wasm/WasmOMGForOSREntryPlan.h
Source/JavaScriptCore/wasm/WasmOMGPlan.cpp
Source/JavaScriptCore/wasm/WasmOMGPlan.h
Source/JavaScriptCore/wasm/WasmOperations.cpp
Source/JavaScriptCore/wasm/js/JSToWasm.cpp
Source/JavaScriptCore/wasm/js/JSToWasm.h
Source/JavaScriptCore/wasm/js/WebAssemblyFunction.cpp
Source/JavaScriptCore/wasm/js/WebAssemblyFunction.h
Source/JavaScriptCore/wasm/js/WebAssemblyModuleRecord.cpp
Source/JavaScriptCore/wasm/js/WebAssemblyWrapperFunction.cpp

index a6c0f02..0c57125 100644 (file)
@@ -393,6 +393,7 @@ set(JavaScriptCore_PUBLIC_FRAMEWORK_HEADERS
 set(JavaScriptCore_PRIVATE_FRAMEWORK_HEADERS
     ${JavaScriptCore_DERIVED_SOURCES_DIR}/Bytecodes.h
     ${JavaScriptCore_DERIVED_SOURCES_DIR}/JSCBuiltins.h
+    ${JavaScriptCore_DERIVED_SOURCES_DIR}/WasmOps.h
 
     ${JavaScriptCore_DERIVED_SOURCES_DIR}/inspector/InspectorBackendDispatchers.h
     ${JavaScriptCore_DERIVED_SOURCES_DIR}/inspector/InspectorFrontendDispatchers.h
@@ -463,6 +464,10 @@ set(JavaScriptCore_PRIVATE_FRAMEWORK_HEADERS
     assembler/X86Registers.h
     assembler/X86_64Registers.h
 
+    b3/B3Common.h
+    b3/B3Compilation.h
+    b3/B3Type.h
+
     bindings/ScriptFunctionCall.h
     bindings/ScriptObject.h
     bindings/ScriptValue.h
@@ -1001,6 +1006,7 @@ set(JavaScriptCore_PRIVATE_FRAMEWORK_HEADERS
     runtime/WriteBarrier.h
     runtime/WriteBarrierInlines.h
 
+    wasm/WasmCallee.h
     wasm/WasmCapabilities.h
     wasm/WasmCodeBlock.h
     wasm/WasmCompilationMode.h
@@ -1008,13 +1014,16 @@ set(JavaScriptCore_PRIVATE_FRAMEWORK_HEADERS
     wasm/WasmEmbedder.h
     wasm/WasmExceptionType.h
     wasm/WasmFaultSignalHandler.h
+    wasm/WasmFormat.h
     wasm/WasmIndexOrName.h
     wasm/WasmMemory.h
+    wasm/WasmMemoryInformation.h
     wasm/WasmMemoryMode.h
     wasm/WasmModule.h
     wasm/WasmName.h
     wasm/WasmNameSection.h
     wasm/WasmPageCount.h
+    wasm/WasmSignature.h
     wasm/WasmTierUpCount.h
 
     wasm/js/JSWebAssembly.h
index 25d8bd3..ba81939 100644 (file)
@@ -1,3 +1,98 @@
+2019-09-17  Tadeu Zagallo  <tzagallo@apple.com>
+
+        Change WebAssembly calling conventions
+        https://bugs.webkit.org/show_bug.cgi?id=201799
+
+        Reviewed by Saam Barati.
+
+        Currently, the Wasm::Callee writes itself to CallFrameSlot::callee. However, this won't work when
+        we have the Wasm interpreter, since we need the callee in order to know which function are we executing.
+        This patch changes the calling conventions in preparation for the interpreter, so that the caller
+        becomes responsible for writing the callee into the call frame.
+        However, there are exceptions to this rule: stubs can still write to the callee slot, since they are individually
+        generated and will still be present in the interpreter. We keep this design to avoid emitting unnecessary
+        code when we know statically who is the callee:
+        - Caller writes to call frame: intra-module direct wasm calls, indirect wasm calls, JS-to-wasm stub (new frame), JS-to-wasm IC.
+        - Callee writes to call frame: inter-module wasm-to-wasm stub, JS-to-wasm stub (callee frame), wasm-to-JS stub, OMG osr entry
+
+        Additionally, this patch also changes it so that the callee keeps track of its callers, instead of having a global mapping
+        of calls in the Wasm::CodeBlock. This makes it easier to repatch all callers of a given Callee when it tiers up.
+
+        * CMakeLists.txt:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * wasm/WasmAirIRGenerator.cpp:
+        (JSC::Wasm::AirIRGenerator::AirIRGenerator):
+        (JSC::Wasm::AirIRGenerator::addCall):
+        (JSC::Wasm::AirIRGenerator::addCallIndirect):
+        (JSC::Wasm::parseAndCompileAir):
+        * wasm/WasmAirIRGenerator.h:
+        * wasm/WasmB3IRGenerator.cpp:
+        (JSC::Wasm::B3IRGenerator::B3IRGenerator):
+        (JSC::Wasm::B3IRGenerator::addCall):
+        (JSC::Wasm::B3IRGenerator::addCallIndirect):
+        (JSC::Wasm::parseAndCompile):
+        * wasm/WasmB3IRGenerator.h:
+        * wasm/WasmBBQPlan.cpp:
+        (JSC::Wasm::BBQPlan::BBQPlan):
+        (JSC::Wasm::BBQPlan::prepare):
+        (JSC::Wasm::BBQPlan::compileFunctions):
+        (JSC::Wasm::BBQPlan::complete):
+        * wasm/WasmBBQPlan.h:
+        * wasm/WasmBBQPlanInlines.h:
+        (JSC::Wasm::BBQPlan::initializeCallees):
+        * wasm/WasmBinding.cpp:
+        (JSC::Wasm::wasmToWasm):
+        * wasm/WasmCallee.cpp:
+        (JSC::Wasm::Callee::Callee):
+        (JSC::Wasm::repatchMove):
+        (JSC::Wasm::repatchCall):
+        (JSC::Wasm::BBQCallee::addCaller):
+        (JSC::Wasm::BBQCallee::addAndLinkCaller):
+        (JSC::Wasm::BBQCallee::repatchCallers):
+        * wasm/WasmCallee.h:
+        (JSC::Wasm::Callee::entrypoint):
+        (JSC::Wasm::Callee::code const):
+        (JSC::Wasm::Callee::calleeSaveRegisters):
+        * wasm/WasmCallingConvention.h:
+        (JSC::Wasm::CallingConvention::setupFrameInPrologue const):
+        * wasm/WasmCodeBlock.cpp:
+        (JSC::Wasm::CodeBlock::CodeBlock):
+        * wasm/WasmCodeBlock.h:
+        (JSC::Wasm::CodeBlock::embedderEntrypointCalleeFromFunctionIndexSpace):
+        (JSC::Wasm::CodeBlock::wasmBBQCalleeFromFunctionIndexSpace):
+        (JSC::Wasm::CodeBlock::entrypointLoadLocationFromFunctionIndexSpace):
+        (JSC::Wasm::CodeBlock::boxedCalleeLoadLocationFromFunctionIndexSpace):
+        * wasm/WasmEmbedder.h:
+        * wasm/WasmFormat.h:
+        (JSC::Wasm::WasmToWasmImportableFunction::offsetOfBoxedCalleeLoadLocation):
+        * wasm/WasmInstance.h:
+        (JSC::Wasm::Instance::offsetOfBoxedCalleeLoadLocation):
+        * wasm/WasmOMGForOSREntryPlan.cpp:
+        (JSC::Wasm::OMGForOSREntryPlan::OMGForOSREntryPlan):
+        (JSC::Wasm::OMGForOSREntryPlan::work):
+        * wasm/WasmOMGForOSREntryPlan.h:
+        * wasm/WasmOMGPlan.cpp:
+        (JSC::Wasm::OMGPlan::OMGPlan):
+        (JSC::Wasm::OMGPlan::work):
+        * wasm/WasmOMGPlan.h:
+        * wasm/WasmOperations.cpp:
+        (JSC::Wasm::triggerOMGReplacementCompile):
+        (JSC::Wasm::doOSREntry):
+        (JSC::Wasm::triggerOSREntryNow):
+        * wasm/js/JSToWasm.cpp:
+        (JSC::Wasm::createJSToWasmWrapper):
+        * wasm/js/JSToWasm.h:
+        * wasm/js/WebAssemblyFunction.cpp:
+        (JSC::WebAssemblyFunction::jsCallEntrypointSlow):
+        (JSC::WebAssemblyFunction::create):
+        (JSC::WebAssemblyFunction::WebAssemblyFunction):
+        * wasm/js/WebAssemblyFunction.h:
+        * wasm/js/WebAssemblyModuleRecord.cpp:
+        (JSC::WebAssemblyModuleRecord::link):
+        (JSC::WebAssemblyModuleRecord::evaluate):
+        * wasm/js/WebAssemblyWrapperFunction.cpp:
+        (JSC::WebAssemblyWrapperFunction::create):
+
 2019-09-17  Yusuke Suzuki  <ysuzuki@apple.com>
 
         [JSC] CheckArray+NonArray is not filtering out Array in AI
index e22c14f..62edca9 100644 (file)
                0F338DF61BE93D550013C88F /* B3ConstrainedValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338DF41BE93D550013C88F /* B3ConstrainedValue.h */; };
                0F338DFA1BE96AA80013C88F /* B3CCallValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338DF81BE96AA80013C88F /* B3CCallValue.h */; };
                0F338DFE1BED51270013C88F /* AirSimplifyCFG.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338DFC1BED51270013C88F /* AirSimplifyCFG.h */; };
-               0F338E0C1BF0276C0013C88F /* B3Compilation.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338E001BF0276C0013C88F /* B3Compilation.h */; };
+               0F338E0C1BF0276C0013C88F /* B3Compilation.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338E001BF0276C0013C88F /* B3Compilation.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F338E0E1BF0276C0013C88F /* B3DataSection.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338E021BF0276C0013C88F /* B3DataSection.h */; };
                0F338E101BF0276C0013C88F /* B3MoveConstants.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338E041BF0276C0013C88F /* B3MoveConstants.h */; };
                0F338E111BF0276C0013C88F /* B3OpaqueByproduct.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F338E051BF0276C0013C88F /* B3OpaqueByproduct.h */; };
                0FEC85041BDACDAC0080FF74 /* B3BlockWorklist.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84BA1BDACDAC0080FF74 /* B3BlockWorklist.h */; };
                0FEC85061BDACDAC0080FF74 /* B3CheckSpecial.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84BC1BDACDAC0080FF74 /* B3CheckSpecial.h */; };
                0FEC85081BDACDAC0080FF74 /* B3CheckValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84BE1BDACDAC0080FF74 /* B3CheckValue.h */; };
-               0FEC850A1BDACDAC0080FF74 /* B3Common.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84C01BDACDAC0080FF74 /* B3Common.h */; };
+               0FEC850A1BDACDAC0080FF74 /* B3Common.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84C01BDACDAC0080FF74 /* B3Common.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FEC850C1BDACDAC0080FF74 /* B3Commutativity.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84C21BDACDAC0080FF74 /* B3Commutativity.h */; };
                0FEC850E1BDACDAC0080FF74 /* B3Const32Value.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84C41BDACDAC0080FF74 /* B3Const32Value.h */; };
                0FEC85101BDACDAC0080FF74 /* B3Const64Value.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84C61BDACDAC0080FF74 /* B3Const64Value.h */; };
                0FEC85361BDACDAC0080FF74 /* B3SuccessorCollection.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84EC1BDACDAC0080FF74 /* B3SuccessorCollection.h */; };
                0FEC85381BDACDAC0080FF74 /* B3SwitchCase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84EE1BDACDAC0080FF74 /* B3SwitchCase.h */; };
                0FEC853A1BDACDAC0080FF74 /* B3SwitchValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84F01BDACDAC0080FF74 /* B3SwitchValue.h */; };
-               0FEC853C1BDACDAC0080FF74 /* B3Type.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84F21BDACDAC0080FF74 /* B3Type.h */; };
+               0FEC853C1BDACDAC0080FF74 /* B3Type.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84F21BDACDAC0080FF74 /* B3Type.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FEC853E1BDACDAC0080FF74 /* B3UpsilonValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84F41BDACDAC0080FF74 /* B3UpsilonValue.h */; };
                0FEC85401BDACDAC0080FF74 /* B3UseCounts.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84F61BDACDAC0080FF74 /* B3UseCounts.h */; };
                0FEC85421BDACDAC0080FF74 /* B3Validate.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84F81BDACDAC0080FF74 /* B3Validate.h */; };
                                868921C11F9C0CB7001159F6 /* JSBigInt.h */,
                                86FA9E8F142BBB2D001773B7 /* JSBoundFunction.cpp */,
                                86FA9E90142BBB2E001773B7 /* JSBoundFunction.h */,
-                               FE48BD4323245E8700F136D0 /* JSCConfig.cpp */,
-                               FE48BD4223245E8700F136D0 /* JSCConfig.h */,
                                657CF45619BF6662004ACBF2 /* JSCallee.cpp */,
                                657CF45719BF6662004ACBF2 /* JSCallee.h */,
                                53B601EB2034B8C5006BE667 /* JSCast.h */,
+                               FE48BD4323245E8700F136D0 /* JSCConfig.cpp */,
+                               FE48BD4223245E8700F136D0 /* JSCConfig.h */,
                                BC7F8FBA0E19D1EF008632C0 /* JSCell.cpp */,
                                BC1167D80E19BCC9008066DD /* JSCell.h */,
                                0F97496F1687ADE200A4FF6A /* JSCellInlines.h */,
                                0FEA0A10170513DB00BB722C /* FTLLowerDFGToB3.h in Headers */,
                                A7D89D0217A0B90400773AD8 /* FTLLoweredNodeValue.h in Headers */,
                                0F2B9CF919D0BAC100B1D1B5 /* FTLOperations.h in Headers */,
-                               FE3842322324D51B009DD445 /* OptionEntry.h in Headers */,
                                0FD8A31C17D51F2200CA2C40 /* FTLOSREntry.h in Headers */,
                                0F235BDD17178E1C00690C7F /* FTLOSRExit.h in Headers */,
                                0F235BE017178E1C00690C7F /* FTLOSRExitCompiler.h in Headers */,
                                A1D793011B43864B004516F5 /* IntlNumberFormatPrototype.h in Headers */,
                                A125846F1B45A36000CC7F6C /* IntlNumberFormatPrototype.lut.h in Headers */,
                                A12BBFF21B044A8B00664B69 /* IntlObject.h in Headers */,
-                               FE48BD4423245E9300F136D0 /* JSCConfig.h in Headers */,
                                708EBE241CE8F35800453146 /* IntlObjectInlines.h in Headers */,
                                860BD801148EA6F200112B2F /* Intrinsic.h in Headers */,
                                534E03541E53BD2900213F64 /* IntrinsicGetterAccessCase.h in Headers */,
                                657CF45919BF6662004ACBF2 /* JSCallee.h in Headers */,
                                53B601EC2034B8C5006BE667 /* JSCast.h in Headers */,
                                A7D801A91880D6A80026C39B /* JSCBuiltins.h in Headers */,
+                               FE48BD4423245E9300F136D0 /* JSCConfig.h in Headers */,
                                BC1167DA0E19BCC9008066DD /* JSCell.h in Headers */,
                                0F9749711687ADE400A4FF6A /* JSCellInlines.h in Headers */,
                                0F1DD84A18A945BE0026F3FA /* JSCInlines.h in Headers */,
                                0F2BDC2C151FDE9100CD8910 /* Operands.h in Headers */,
                                A70447EA17A0BD4600F5898E /* OperandsInlines.h in Headers */,
                                BC18C4480E16F5CD00B34460 /* Operations.h in Headers */,
+                               FE3842322324D51B009DD445 /* OptionEntry.h in Headers */,
                                0FE228ED1436AB2700196C48 /* Options.h in Headers */,
+                               FE3842332324D51B009DD445 /* OptionsList.h in Headers */,
                                E356987222841187008CDCCB /* PackedCellPtr.h in Headers */,
                                0F9DAA0A1FD1C3D30079C5B2 /* ParallelSourceAdapter.h in Headers */,
                                E34E657520668EAA00FB81AC /* ParseHash.h in Headers */,
                                0FE050281AA9095600D33B33 /* ScopedArguments.h in Headers */,
                                0FE050291AA9095600D33B33 /* ScopedArgumentsTable.h in Headers */,
                                0FE0502B1AA9095600D33B33 /* ScopeOffset.h in Headers */,
-                               FE3842332324D51B009DD445 /* OptionsList.h in Headers */,
                                0F24E55217EE274900ABB217 /* ScratchRegisterAllocator.h in Headers */,
                                A5FD0068189AFE9C00633231 /* ScriptArguments.h in Headers */,
                                A503FA21188EFF6800110F14 /* ScriptBreakpoint.h in Headers */,
index 319aa5b..c671563 100644 (file)
@@ -229,7 +229,7 @@ public:
             return fail(__VA_ARGS__);             \
     } while (0)
 
-    AirIRGenerator(const ModuleInformation&, B3::Procedure&, InternalFunction*, Vector<UnlinkedWasmToWasmCall>&, MemoryMode, unsigned functionIndex, TierUpCount*, ThrowWasmException, const Signature&);
+    AirIRGenerator(const ModuleInformation&, B3::Procedure&, Vector<UnlinkedWasmToWasmCall>&, const Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>&, MemoryMode, unsigned functionIndex, TierUpCount*, ThrowWasmException, const Signature&);
 
     PartialResult WARN_UNUSED_RETURN addArguments(const Signature&);
     PartialResult WARN_UNUSED_RETURN addLocal(Type, uint32_t);
@@ -639,7 +639,8 @@ private:
     BasicBlock* m_currentBlock { nullptr };
     BasicBlock* m_rootBlock { nullptr };
     Vector<TypedTmp> m_locals;
-    Vector<UnlinkedWasmToWasmCall>& m_unlinkedWasmToWasmCalls; // List each call site and the function index whose address it should be patched with.
+    Vector<UnlinkedWasmToWasmCall>& m_outgoingCalls;
+    const Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>& m_wasmToWasmExitStubs;
     GPRReg m_memoryBaseGPR { InvalidGPRReg };
     GPRReg m_memorySizeGPR { InvalidGPRReg };
     GPRReg m_wasmContextInstanceGPR { InvalidGPRReg };
@@ -708,14 +709,15 @@ void AirIRGenerator::restoreWasmContextInstance(BasicBlock* block, TypedTmp inst
     emitPatchpoint(block, patchpoint, Tmp(), instance);
 }
 
-AirIRGenerator::AirIRGenerator(const ModuleInformation& info, B3::Procedure& procedure, InternalFunction* compilation, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, MemoryMode mode, unsigned functionIndex, TierUpCount* tierUp, ThrowWasmException throwWasmException, const Signature& signature)
+AirIRGenerator::AirIRGenerator(const ModuleInformation& info, B3::Procedure& procedure, Vector<UnlinkedWasmToWasmCall>& outgoingCalls, const Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>& wasmToWasmExitStubs, MemoryMode mode, unsigned functionIndex, TierUpCount* tierUp, ThrowWasmException throwWasmException, const Signature& signature)
     : m_info(info)
     , m_mode(mode)
     , m_functionIndex(functionIndex)
     , m_tierUp(tierUp)
     , m_proc(procedure)
     , m_code(m_proc.code())
-    , m_unlinkedWasmToWasmCalls(unlinkedWasmToWasmCalls)
+    , m_outgoingCalls(outgoingCalls)
+    , m_wasmToWasmExitStubs(wasmToWasmExitStubs)
     , m_numImportFunctions(info.importFunctionCount())
 {
     m_currentBlock = m_code.addBlock();
@@ -761,16 +763,7 @@ AirIRGenerator::AirIRGenerator(const ModuleInformation& info, B3::Procedure& pro
     Ref<B3::Air::PrologueGenerator> prologueGenerator = createSharedTask<B3::Air::PrologueGeneratorFunction>([=] (CCallHelpers& jit, B3::Air::Code& code) {
         AllowMacroScratchRegisterUsage allowScratch(jit);
         code.emitDefaultPrologue(jit);
-
-        {
-            GPRReg calleeGPR = wasmCallingConventionAir().prologueScratch(0);
-            auto moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), calleeGPR);
-            jit.addLinkTask([compilation, moveLocation] (LinkBuffer& linkBuffer) {
-                compilation->calleeMoveLocation = linkBuffer.locationOf<WasmEntryPtrTag>(moveLocation);
-            });
-            jit.emitPutToCallFrameHeader(calleeGPR, CallFrameSlot::callee);
-            jit.emitPutToCallFrameHeader(nullptr, CallFrameSlot::codeBlock);
-        }
+        jit.emitPutToCallFrameHeader(nullptr, CallFrameSlot::codeBlock);
 
         {
             const Checked<int32_t> wasmFrameSize = m_code.frameSize();
@@ -1936,8 +1929,6 @@ auto AirIRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
     if (returnType != Type::Void)
         result = tmpForType(returnType);
 
-    Vector<UnlinkedWasmToWasmCall>* unlinkedWasmToWasmCalls = &m_unlinkedWasmToWasmCalls;
-
     if (m_info.isImportedFunctionFromFunctionIndexSpace(functionIndex)) {
         m_maxNumJSCallArguments = std::max(m_maxNumJSCallArguments, static_cast<uint32_t>(args.size()));
 
@@ -1973,11 +1964,12 @@ auto AirIRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
                 patchArgs.append({ tmp, rep });
             });
 
-            patchpoint->setGenerator([unlinkedWasmToWasmCalls, functionIndex] (CCallHelpers& jit, const B3::StackmapGenerationParams&) {
+            MacroAssemblerCodeRef<WasmEntryPtrTag> wasmToWasmExitStub = m_wasmToWasmExitStubs[functionIndex];
+            patchpoint->setGenerator([wasmToWasmExitStub] (CCallHelpers& jit, const B3::StackmapGenerationParams&) {
                 AllowMacroScratchRegisterUsage allowScratch(jit);
-                CCallHelpers::Call call = jit.threadSafePatchableNearCall();
-                jit.addLinkTask([unlinkedWasmToWasmCalls, call, functionIndex] (LinkBuffer& linkBuffer) {
-                    unlinkedWasmToWasmCalls->append({ linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call), functionIndex });
+                MacroAssembler::Call call = jit.call(WasmEntryPtrTag);
+                jit.addLinkTask([call, wasmToWasmExitStub] (LinkBuffer& linkBuffer) {
+                    linkBuffer.link(call, CodeLocationLabel<WasmEntryPtrTag>(wasmToWasmExitStub.code()));
                 });
             });
 
@@ -2024,18 +2016,20 @@ auto AirIRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
         auto* patchpoint = addPatchpoint(toB3Type(returnType));
         patchpoint->effects.writesPinned = true;
         patchpoint->effects.readsPinned = true;
+        patchpoint->numGPScratchRegisters++;
 
         Vector<ConstrainedTmp> patchArgs;
         wasmCallingConventionAir().setupCall(m_code, returnType, patchpoint, toTmpVector(args), [&] (Tmp tmp, B3::ValueRep rep) {
             patchArgs.append({ tmp, rep });
         });
 
-        patchpoint->setGenerator([unlinkedWasmToWasmCalls, functionIndex] (CCallHelpers& jit, const B3::StackmapGenerationParams&) {
-            AllowMacroScratchRegisterUsage allowScratch(jit);
-            CCallHelpers::Call call = jit.threadSafePatchableNearCall();
-            jit.addLinkTask([unlinkedWasmToWasmCalls, call, functionIndex] (LinkBuffer& linkBuffer) {
-                unlinkedWasmToWasmCalls->append({ linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call), functionIndex });
-            });
+        Vector<UnlinkedWasmToWasmCall>* outgoingCalls = &m_outgoingCalls;
+        patchpoint->setGenerator([outgoingCalls, functionIndex] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
+            GPRReg calleeGPR = params.gpScratch(0);
+            auto moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), calleeGPR);
+            jit.storePtr(calleeGPR, MacroAssembler::Address(MacroAssembler::stackPointerRegister, CallFrameSlot::callee * sizeof(Register) - sizeof(CallerFrameAndPC)));
+            CCallHelpers::Call callLocation = jit.threadSafePatchableNearCall();
+            outgoingCalls->append({ functionIndex, { moveLocation, callLocation }});
         });
 
         emitPatchpoint(m_currentBlock, patchpoint, result, WTFMove(patchArgs));
@@ -2089,6 +2083,7 @@ auto AirIRGenerator::addCallIndirect(unsigned tableIndex, const Signature& signa
     });
 
     ExpressionType calleeCode = g64();
+    ExpressionType callee = g64();
     {
         ExpressionType calleeSignatureIndex = g64();
         // Compute the offset in the table index space we are looking for.
@@ -2097,6 +2092,7 @@ auto AirIRGenerator::addCallIndirect(unsigned tableIndex, const Signature& signa
         append(Add64, callableFunctionBuffer, calleeSignatureIndex);
         
         append(Move, Arg::addr(calleeSignatureIndex, WasmToWasmImportableFunction::offsetOfEntrypointLoadLocation()), calleeCode); // Pointer to callee code.
+        append(Move, Arg::addr(calleeSignatureIndex, WasmToWasmImportableFunction::offsetOfBoxedCalleeLoadLocation()), callee); // Pointer to boxed callee
 
         // Check that the WasmToWasmImportableFunction is initialized. We trap if it isn't. An "invalid" SignatureIndex indicates it's not initialized.
         // FIXME: when we have trap handlers, we can just let the call fail because Signature::invalidIndex is 0. https://bugs.webkit.org/show_bug.cgi?id=177210
@@ -2189,6 +2185,10 @@ auto AirIRGenerator::addCallIndirect(unsigned tableIndex, const Signature& signa
 
     Vector<ConstrainedTmp> emitArgs;
     emitArgs.append(calleeCode);
+
+    append(Move, Arg::addr(callee), callee);
+    emitArgs.append({ callee, B3::ValueRep::stackArgument(CallFrameSlot::callee * sizeof(Register) - sizeof(CallerFrameAndPC)) });
+
     wasmCallingConventionAir().setupCall(m_code, returnType, patch, toTmpVector(args), [&] (Tmp tmp, B3::ValueRep rep) {
         emitArgs.append({ tmp, rep });
     });
@@ -2229,10 +2229,8 @@ auto AirIRGenerator::origin() -> B3::Origin
     return B3::Origin();
 }
 
-Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileAir(CompilationContext& compilationContext, const FunctionData& function, const Signature& signature, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, const ModuleInformation& info, MemoryMode mode, uint32_t functionIndex, TierUpCount* tierUp, ThrowWasmException throwWasmException)
+Expected<void, String> parseAndCompileAir(CompilationContext& compilationContext, const FunctionData& function, const Signature& signature, Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>& wasmToWasmExitStubs, const ModuleInformation& info, MemoryMode mode, uint32_t functionIndex, TierUpCount* tierUp, ThrowWasmException throwWasmException)
 {
-    auto result = makeUnique<InternalFunction>();
-
     compilationContext.embedderEntrypointJIT = makeUnique<CCallHelpers>();
     compilationContext.wasmEntrypointJIT = makeUnique<CCallHelpers>();
 
@@ -2252,7 +2250,7 @@ Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileAir(Compilati
     
     procedure.setOptLevel(Options::webAssemblyBBQAirOptimizationLevel());
 
-    AirIRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, mode, functionIndex, tierUp, throwWasmException, signature);
+    AirIRGenerator irGenerator(info, procedure, compilationContext.outgoingCalls, wasmToWasmExitStubs, mode, functionIndex, tierUp, throwWasmException, signature);
     FunctionParser<AirIRGenerator> parser(irGenerator, function.data.data(), function.data.size(), signature, info);
     WASM_FAIL_IF_HELPER_FAILS(parser.parse());
 
@@ -2272,10 +2270,10 @@ Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileAir(Compilati
         B3::Air::prepareForGeneration(code);
         B3::Air::generate(code, *compilationContext.wasmEntrypointJIT);
         compilationContext.wasmEntrypointByproducts = procedure.releaseByproducts();
-        result->entrypoint.calleeSaveRegisters = code.calleeSaveRegisterAtOffsetList();
+        compilationContext.calleeSaveRegisters = code.calleeSaveRegisterAtOffsetList();
     }
 
-    return result;
+    return { };
 }
 
 template <typename IntType>
index 331b4b0..8272ce3 100644 (file)
@@ -31,7 +31,7 @@
 
 namespace JSC { namespace Wasm {
 
-Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileAir(CompilationContext&, const FunctionData&, const Signature&, Vector<UnlinkedWasmToWasmCall>&, const ModuleInformation&, MemoryMode, uint32_t functionIndex, TierUpCount* = nullptr, ThrowWasmException = nullptr);
+Expected<void, String> parseAndCompileAir(CompilationContext&, const FunctionData&, const Signature&, Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>&, const ModuleInformation&, MemoryMode, uint32_t functionIndex, TierUpCount* = nullptr, ThrowWasmException = nullptr);
 
 } } // namespace JSC::Wasm
 
index 3d8df8e..e0003ec 100644 (file)
@@ -172,7 +172,6 @@ public:
 
     typedef String ErrorType;
     typedef Unexpected<ErrorType> UnexpectedResult;
-    typedef Expected<std::unique_ptr<InternalFunction>, ErrorType> Result;
     typedef Expected<void, ErrorType> PartialResult;
     template <typename ...Args>
     NEVER_INLINE UnexpectedResult WARN_UNUSED_RETURN fail(Args... args) const
@@ -185,7 +184,7 @@ public:
             return fail(__VA_ARGS__);             \
     } while (0)
 
-    B3IRGenerator(const ModuleInformation&, Procedure&, InternalFunction*, Vector<UnlinkedWasmToWasmCall>&, unsigned& osrEntryScratchBufferSize, MemoryMode, CompilationMode, unsigned functionIndex, unsigned loopIndexForOSREntry, TierUpCount*, ThrowWasmException);
+    B3IRGenerator(const ModuleInformation&, Procedure&, Vector<UnlinkedWasmToWasmCall>&, Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>&, unsigned& osrEntryScratchBufferSize, MemoryMode, CompilationMode, unsigned functionIndex, unsigned loopIndexForOSREntry, TierUpCount*, ThrowWasmException);
 
     PartialResult WARN_UNUSED_RETURN addArguments(const Signature&);
     PartialResult WARN_UNUSED_RETURN addLocal(Type, uint32_t);
@@ -283,6 +282,7 @@ private:
 
     FunctionParser<B3IRGenerator>* m_parser { nullptr };
     const ModuleInformation& m_info;
+    Vector<UnlinkedWasmToWasmCall>& m_outgoingCalls;
     const MemoryMode m_mode { MemoryMode::BoundsChecking };
     const CompilationMode m_compilationMode { CompilationMode::BBQMode };
     const unsigned m_functionIndex { UINT_MAX };
@@ -294,7 +294,7 @@ private:
     BasicBlock* m_currentBlock { nullptr };
     Vector<uint32_t> m_outerLoops;
     Vector<Variable*> m_locals;
-    Vector<UnlinkedWasmToWasmCall>& m_unlinkedWasmToWasmCalls; // List each call site and the function index whose address it should be patched with.
+    Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>& m_wasmToWasmExitStubs; // List each call site and the function index whose address it should be patched with.
     unsigned& m_osrEntryScratchBufferSize;
     HashMap<ValueKey, Value*> m_constantPool;
     InsertionSet m_constantInsertionValues;
@@ -355,15 +355,16 @@ void B3IRGenerator::restoreWasmContextInstance(Procedure& proc, BasicBlock* bloc
     });
 }
 
-B3IRGenerator::B3IRGenerator(const ModuleInformation& info, Procedure& procedure, InternalFunction* compilation, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, unsigned& osrEntryScratchBufferSize, MemoryMode mode, CompilationMode compilationMode, unsigned functionIndex, unsigned loopIndexForOSREntry, TierUpCount* tierUp, ThrowWasmException throwWasmException)
+B3IRGenerator::B3IRGenerator(const ModuleInformation& info, Procedure& procedure, Vector<UnlinkedWasmToWasmCall>& outgoingCalls, Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>& wasmToWasmExitStubs, unsigned& osrEntryScratchBufferSize, MemoryMode mode, CompilationMode compilationMode, unsigned functionIndex, unsigned loopIndexForOSREntry, TierUpCount* tierUp, ThrowWasmException throwWasmException)
     : m_info(info)
+    , m_outgoingCalls(outgoingCalls)
     , m_mode(mode)
     , m_compilationMode(compilationMode)
     , m_functionIndex(functionIndex)
     , m_loopIndexForOSREntry(loopIndexForOSREntry)
     , m_tierUp(tierUp)
     , m_proc(procedure)
-    , m_unlinkedWasmToWasmCalls(unlinkedWasmToWasmCalls)
+    , m_wasmToWasmExitStubs(wasmToWasmExitStubs)
     , m_osrEntryScratchBufferSize(osrEntryScratchBufferSize)
     , m_constantInsertionValues(m_proc)
     , m_numImportFunctions(info.importFunctionCount())
@@ -417,7 +418,7 @@ B3IRGenerator::B3IRGenerator(const ModuleInformation& info, Procedure& procedure
         }
     }
 
-    wasmCallingConvention().setupFrameInPrologue(&compilation->calleeMoveLocation, m_proc, Origin(), m_currentBlock);
+    wasmCallingConvention().setupFrameInPrologue(m_proc, Origin(), m_currentBlock);
 
     {
         B3::Value* framePointer = m_currentBlock->appendNew<B3::Value>(m_proc, B3::FramePointer, Origin());
@@ -1402,8 +1403,6 @@ auto B3IRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
     m_makesCalls = true;
 
     Type returnType = signature.returnType();
-    Vector<UnlinkedWasmToWasmCall>* unlinkedWasmToWasmCalls = &m_unlinkedWasmToWasmCalls;
-
     if (m_info.isImportedFunctionFromFunctionIndexSpace(functionIndex)) {
         m_maxNumJSCallArguments = std::max(m_maxNumJSCallArguments, static_cast<uint32_t>(args.size()));
 
@@ -1417,6 +1416,7 @@ auto B3IRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
         BasicBlock* continuation = m_proc.addBlock();
         m_currentBlock->appendNewControlValue(m_proc, B3::Branch, origin(), isWasmCall, FrequentedBlock(isWasmBlock), FrequentedBlock(isEmbedderBlock));
 
+        MacroAssemblerCodeRef<WasmEntryPtrTag> wasmToWasmExitStub = m_wasmToWasmExitStubs[functionIndex];
         Value* wasmCallResult = wasmCallingConvention().setupCall(m_proc, isWasmBlock, origin(), args, toB3Type(returnType),
             [=] (PatchpointValue* patchpoint) {
                 patchpoint->effects.writesPinned = true;
@@ -1425,11 +1425,11 @@ auto B3IRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
                 // We pessimistically assume we could be calling to something that is bounds checking.
                 // FIXME: We shouldn't have to do this: https://bugs.webkit.org/show_bug.cgi?id=172181
                 patchpoint->clobberLate(PinnedRegisterInfo::get().toSave(MemoryMode::BoundsChecking));
-                patchpoint->setGenerator([unlinkedWasmToWasmCalls, functionIndex] (CCallHelpers& jit, const B3::StackmapGenerationParams&) {
+                patchpoint->setGenerator([wasmToWasmExitStub] (CCallHelpers& jit, const B3::StackmapGenerationParams&) {
                     AllowMacroScratchRegisterUsage allowScratch(jit);
-                    CCallHelpers::Call call = jit.threadSafePatchableNearCall();
-                    jit.addLinkTask([unlinkedWasmToWasmCalls, call, functionIndex] (LinkBuffer& linkBuffer) {
-                        unlinkedWasmToWasmCalls->append({ linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call), functionIndex });
+                    MacroAssembler::Call call = jit.call(WasmEntryPtrTag);
+                    jit.addLinkTask([call, wasmToWasmExitStub] (LinkBuffer& linkBuffer) {
+                        linkBuffer.link(call, CodeLocationLabel<WasmEntryPtrTag>(wasmToWasmExitStub.code()));
                     });
                 });
             });
@@ -1473,17 +1473,18 @@ auto B3IRGenerator::addCall(uint32_t functionIndex, const Signature& signature,
         // The call could have been to another WebAssembly instance, and / or could have modified our Memory.
         restoreWebAssemblyGlobalState(RestoreCachedStackLimit::Yes, m_info.memory, instanceValue(), m_proc, continuation);
     } else {
+        Vector<UnlinkedWasmToWasmCall>* outgoingCalls = &m_outgoingCalls;
         result = wasmCallingConvention().setupCall(m_proc, m_currentBlock, origin(), args, toB3Type(returnType),
             [=] (PatchpointValue* patchpoint) {
                 patchpoint->effects.writesPinned = true;
                 patchpoint->effects.readsPinned = true;
-
-                patchpoint->setGenerator([unlinkedWasmToWasmCalls, functionIndex] (CCallHelpers& jit, const B3::StackmapGenerationParams&) {
-                    AllowMacroScratchRegisterUsage allowScratch(jit);
-                    CCallHelpers::Call call = jit.threadSafePatchableNearCall();
-                    jit.addLinkTask([unlinkedWasmToWasmCalls, call, functionIndex] (LinkBuffer& linkBuffer) {
-                        unlinkedWasmToWasmCalls->append({ linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call), functionIndex });
-                    });
+                patchpoint->numGPScratchRegisters++;
+                patchpoint->setGenerator([=] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
+                    GPRReg calleeGPR = params.gpScratch(0);
+                    auto moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), calleeGPR);
+                    jit.storePtr(calleeGPR, MacroAssembler::Address(MacroAssembler::stackPointerRegister, CallFrameSlot::callee * sizeof(Register) - sizeof(CallerFrameAndPC)));
+                    CCallHelpers::Call callLocation = jit.threadSafePatchableNearCall();
+                    outgoingCalls->append({ functionIndex, { moveLocation, callLocation }});
                 });
             });
     }
@@ -1624,6 +1625,10 @@ auto B3IRGenerator::addCallIndirect(unsigned tableIndex, const Signature& signat
         m_currentBlock->appendNew<MemoryValue>(m_proc, Load, pointerType(), origin(), callableFunction,
             safeCast<int32_t>(WasmToWasmImportableFunction::offsetOfEntrypointLoadLocation())));
 
+    ExpressionType callee = m_currentBlock->appendNew<MemoryValue>(m_proc, Load, pointerType(), origin(),
+        m_currentBlock->appendNew<MemoryValue>(m_proc, Load, pointerType(), origin(), callableFunction,
+            safeCast<int32_t>(WasmToWasmImportableFunction::offsetOfBoxedCalleeLoadLocation())));
+
     Type returnType = signature.returnType();
     result = wasmCallingConvention().setupCall(m_proc, m_currentBlock, origin(), args, toB3Type(returnType),
         [=] (PatchpointValue* patchpoint) {
@@ -1635,8 +1640,8 @@ auto B3IRGenerator::addCallIndirect(unsigned tableIndex, const Signature& signat
             // FIXME: We should not have to do this, but the wasm->wasm stub assumes it can
             // use all the pinned registers as scratch: https://bugs.webkit.org/show_bug.cgi?id=172181
             patchpoint->clobberLate(PinnedRegisterInfo::get().toSave(MemoryMode::BoundsChecking));
-
             patchpoint->append(calleeCode, ValueRep::SomeRegister);
+            patchpoint->append(callee, B3::ValueRep::stackArgument(CallFrameSlot::callee * sizeof(Register) - sizeof(CallerFrameAndPC)));
             patchpoint->setGenerator([=] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
                 AllowMacroScratchRegisterUsage allowScratch(jit);
                 jit.call(params[returnType == Void ? 0 : 1].gpr(), WasmEntryPtrTag);
@@ -1697,10 +1702,8 @@ auto B3IRGenerator::origin() -> Origin
     return bitwise_cast<Origin>(origin);
 }
 
-Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext& compilationContext, const FunctionData& function, const Signature& signature, Vector<UnlinkedWasmToWasmCall>& unlinkedWasmToWasmCalls, unsigned& osrEntryScratchBufferSize, const ModuleInformation& info, MemoryMode mode, CompilationMode compilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* tierUp, ThrowWasmException throwWasmException)
+Expected<void, String> parseAndCompile(CompilationContext& compilationContext, const FunctionData& function, const Signature& signature, Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>& wasmToWasmExitStubs, unsigned& osrEntryScratchBufferSize, const ModuleInformation& info, MemoryMode mode, CompilationMode compilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* tierUp, ThrowWasmException throwWasmException)
 {
-    auto result = makeUnique<InternalFunction>();
-
     compilationContext.embedderEntrypointJIT = makeUnique<CCallHelpers>();
     compilationContext.wasmEntrypointJIT = makeUnique<CCallHelpers>();
 
@@ -1721,7 +1724,7 @@ Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationC
         ? Options::webAssemblyBBQB3OptimizationLevel()
         : Options::webAssemblyOMGOptimizationLevel());
 
-    B3IRGenerator irGenerator(info, procedure, result.get(), unlinkedWasmToWasmCalls, osrEntryScratchBufferSize, mode, compilationMode, functionIndex, loopIndexForOSREntry, tierUp, throwWasmException);
+    B3IRGenerator irGenerator(info, procedure, compilationContext.outgoingCalls, wasmToWasmExitStubs, osrEntryScratchBufferSize, mode, compilationMode, functionIndex, loopIndexForOSREntry, tierUp, throwWasmException);
     FunctionParser<B3IRGenerator> parser(irGenerator, function.data.data(), function.data.size(), signature, info);
     WASM_FAIL_IF_HELPER_FAILS(parser.parse());
 
@@ -1739,10 +1742,10 @@ Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationC
         B3::prepareForGeneration(procedure);
         B3::generate(procedure, *compilationContext.wasmEntrypointJIT);
         compilationContext.wasmEntrypointByproducts = procedure.releaseByproducts();
-        result->entrypoint.calleeSaveRegisters = procedure.calleeSaveRegisterAtOffsetList();
+        compilationContext.calleeSaveRegisters = procedure.calleeSaveRegisterAtOffsetList();
     }
 
-    return result;
+    return { };
 }
 
 // Custom wasm ops. These are the ones too messy to do in wasm.json.
index ea55024..348b547 100644 (file)
@@ -49,9 +49,12 @@ struct CompilationContext {
     std::unique_ptr<B3::OpaqueByproducts> embedderEntrypointByproducts;
     std::unique_ptr<CCallHelpers> wasmEntrypointJIT;
     std::unique_ptr<B3::OpaqueByproducts> wasmEntrypointByproducts;
+    Vector<UnlinkedWasmToWasmCall> outgoingCalls;
+    Optional<UnlinkedMoveAndCall> embedderMoveAndCall;
+    RegisterAtOffsetList calleeSaveRegisters;
 };
 
-Expected<std::unique_ptr<InternalFunction>, String> parseAndCompile(CompilationContext&, const FunctionData&, const Signature&, Vector<UnlinkedWasmToWasmCall>&, unsigned& osrEntryScratchBufferSize, const ModuleInformation&, MemoryMode, CompilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* = nullptr, ThrowWasmException = nullptr);
+Expected<void, String> parseAndCompile(CompilationContext&, const FunctionData&, const Signature&, Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>>&, unsigned& osrEntryScratchBufferSize, const ModuleInformation&, MemoryMode, CompilationMode, uint32_t functionIndex, uint32_t loopIndexForOSREntry, TierUpCount* = nullptr, ThrowWasmException = nullptr);
 
 } } // namespace JSC::Wasm
 
index 0e83d53..6c6afe3 100644 (file)
@@ -52,9 +52,10 @@ namespace WasmBBQPlanInternal {
 static const bool verbose = false;
 }
 
-BBQPlan::BBQPlan(Context* context, Ref<ModuleInformation> info, AsyncWork work, CompletionTask&& task, CreateEmbedderWrapper&& createEmbedderWrapper, ThrowWasmException throwWasmException)
+BBQPlan::BBQPlan(Context* context, const Vector<Ref<BBQCallee>>& callees, Ref<ModuleInformation> info, AsyncWork work, CompletionTask&& task, CreateEmbedderWrapper&& createEmbedderWrapper, ThrowWasmException throwWasmException)
     : Base(context, WTFMove(info), WTFMove(task), WTFMove(createEmbedderWrapper), throwWasmException)
     , m_streamingParser(m_moduleInformation.get(), *this)
+    , m_callees(&callees)
     , m_state(State::Validated)
     , m_asyncWork(work)
 {
@@ -159,16 +160,10 @@ void BBQPlan::prepare()
 
     const auto& functions = m_moduleInformation->functions;
     if (!tryReserveCapacity(m_wasmToWasmExitStubs, m_moduleInformation->importFunctionSignatureIndices.size(), " WebAssembly to JavaScript stubs")
-        || !tryReserveCapacity(m_unlinkedWasmToWasmCalls, functions.size(), " unlinked WebAssembly to WebAssembly calls")
-        || !tryReserveCapacity(m_wasmInternalFunctions, functions.size(), " WebAssembly functions")
-        || !tryReserveCapacity(m_compilationContexts, functions.size(), " compilation contexts")
-        || !tryReserveCapacity(m_tierUpCounts, functions.size(), " tier-up counts"))
+        || !tryReserveCapacity(m_compilationContexts, functions.size(), " compilation contexts"))
         return;
 
-    m_unlinkedWasmToWasmCalls.resize(functions.size());
-    m_wasmInternalFunctions.resize(functions.size());
     m_compilationContexts.resize(functions.size());
-    m_tierUpCounts.resize(functions.size());
 
     for (unsigned importIndex = 0; importIndex < m_moduleInformation->imports.size(); ++importIndex) {
         Import* import = &m_moduleInformation->imports[importIndex];
@@ -230,6 +225,7 @@ public:
 
 void BBQPlan::compileFunctions(CompilationEffort effort)
 {
+    RELEASE_ASSERT(m_callees);
     ASSERT(m_state >= State::Prepared);
     dataLogLnIf(WasmBBQPlanInternal::verbose, "Starting compilation");
 
@@ -266,13 +262,7 @@ void BBQPlan::compileFunctions(CompilationEffort effort)
         ASSERT_UNUSED(functionIndexSpace, m_moduleInformation->signatureIndexFromFunctionIndexSpace(functionIndexSpace) == signatureIndex);
         ASSERT(validateFunction(function, signature, m_moduleInformation.get()));
 
-        m_unlinkedWasmToWasmCalls[functionIndex] = Vector<UnlinkedWasmToWasmCall>();
-        if (Options::useBBQTierUpChecks())
-            m_tierUpCounts[functionIndex] = makeUnique<TierUpCount>();
-        else
-            m_tierUpCounts[functionIndex] = nullptr;
-        TierUpCount* tierUp = m_tierUpCounts[functionIndex].get();
-        Expected<std::unique_ptr<InternalFunction>, String> parseAndCompileResult;
+        Expected<void, String> parseAndCompileResult;
         unsigned osrEntryScratchBufferSize = 0;
 
         // FIXME: Some webpages use very large Wasm module, and it exhausts all executable memory in ARM64 devices since the size of executable memory region is only limited to 128MB.
@@ -282,10 +272,11 @@ void BBQPlan::compileFunctions(CompilationEffort effort)
         if (Options::webAssemblyBBQAirModeThreshold() && m_moduleInformation->codeSectionSize >= Options::webAssemblyBBQAirModeThreshold())
             forceUsingB3 = true;
 
+        TierUpCount* tierUp = m_callees->at(functionIndex)->tierUpCount();
         if (!forceUsingB3 && Options::wasmBBQUsesAir())
-            parseAndCompileResult = parseAndCompileAir(m_compilationContexts[functionIndex], function, signature, m_unlinkedWasmToWasmCalls[functionIndex], m_moduleInformation.get(), m_mode, functionIndex, tierUp, m_throwWasmException);
+            parseAndCompileResult = parseAndCompileAir(m_compilationContexts[functionIndex], function, signature, m_wasmToWasmExitStubs, m_moduleInformation.get(), m_mode, functionIndex, tierUp, m_throwWasmException);
         else
-            parseAndCompileResult = parseAndCompile(m_compilationContexts[functionIndex], function, signature, m_unlinkedWasmToWasmCalls[functionIndex], osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::BBQMode, functionIndex, UINT32_MAX, tierUp, m_throwWasmException);
+            parseAndCompileResult = parseAndCompile(m_compilationContexts[functionIndex], function, signature, m_wasmToWasmExitStubs, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::BBQMode, functionIndex, UINT32_MAX, tierUp, m_throwWasmException);
 
         if (UNLIKELY(!parseAndCompileResult)) {
             auto locker = holdLock(m_lock);
@@ -294,11 +285,11 @@ void BBQPlan::compileFunctions(CompilationEffort effort)
             return;
         }
 
-        m_wasmInternalFunctions[functionIndex] = WTFMove(*parseAndCompileResult);
-
         if (m_exportedFunctionIndices.contains(functionIndex) || m_moduleInformation->referencedFunctions().contains(functionIndex)) {
             auto locker = holdLock(m_lock);
-            auto result = m_embedderToWasmInternalFunctions.add(functionIndex, m_createEmbedderWrapper(m_compilationContexts[functionIndex], signature, &m_unlinkedWasmToWasmCalls[functionIndex], m_moduleInformation.get(), m_mode, functionIndex));
+            RefPtr<EmbedderEntrypointCallee> embedderEntrypointCallee = Wasm::EmbedderEntrypointCallee::create();
+            m_createEmbedderWrapper(m_compilationContexts[functionIndex], signature, embedderEntrypointCallee.get(), m_moduleInformation.get(), m_mode);
+            auto result = m_embedderToWasmInternalFunctions.add(functionIndex, WTFMove(embedderEntrypointCallee));
             ASSERT_UNUSED(result, result.isNewEntry);
         }
 
@@ -316,7 +307,9 @@ void BBQPlan::complete(const AbstractLocker& locker)
             CompilationContext& context = m_compilationContexts[functionIndex];
             SignatureIndex signatureIndex = m_moduleInformation->internalFunctionSignatureIndices[functionIndex];
             const Signature& signature = SignatureInformation::get(signatureIndex);
-            const uint32_t functionIndexSpace = functionIndex + m_moduleInformation->importFunctionCount();
+            const uint32_t importFunctionCount = m_moduleInformation->importFunctionCount();
+            const uint32_t functionIndexSpace = functionIndex + importFunctionCount;
+            BBQCallee& callee = m_callees->at(functionIndex).get();
             ASSERT(functionIndexSpace < m_moduleInformation->functionIndexSpaceSize());
             {
                 LinkBuffer linkBuffer(*context.wasmEntrypointJIT, nullptr, JITCompilationCanFail);
@@ -325,9 +318,16 @@ void BBQPlan::complete(const AbstractLocker& locker)
                     return;
                 }
 
-                m_wasmInternalFunctions[functionIndex]->entrypoint.compilation = makeUnique<B3::Compilation>(
+                callee.entrypoint().calleeSaveRegisters = WTFMove(context.calleeSaveRegisters);
+                callee.entrypoint().compilation = makeUnique<B3::Compilation>(
                     FINALIZE_CODE(linkBuffer, B3CompilationPtrTag, "WebAssembly BBQ function[%i] %s name %s", functionIndex, signature.toString().ascii().data(), makeString(IndexOrName(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace))).ascii().data()),
                     WTFMove(context.wasmEntrypointByproducts));
+
+                for (auto& call : context.outgoingCalls) {
+                    ASSERT(call.targetFunctionIndexSpace >= importFunctionCount);
+                    const uint32_t targetFunctionIndex = call.targetFunctionIndexSpace - m_moduleInformation->importFunctionCount();
+                    m_callees->at(targetFunctionIndex)->addCaller(locker, linkBuffer, call.unlinkedMoveAndCall);
+                }
             }
 
             if (auto embedderToWasmInternalFunction = m_embedderToWasmInternalFunctions.get(functionIndex)) {
@@ -337,23 +337,18 @@ void BBQPlan::complete(const AbstractLocker& locker)
                     return;
                 }
 
-                embedderToWasmInternalFunction->entrypoint.compilation = makeUnique<B3::Compilation>(
+                embedderToWasmInternalFunction->entrypoint().compilation = makeUnique<B3::Compilation>(
                     FINALIZE_CODE(linkBuffer, B3CompilationPtrTag, "Embedder->WebAssembly entrypoint[%i] %s name %s", functionIndex, signature.toString().ascii().data(), makeString(IndexOrName(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace))).ascii().data()),
                     WTFMove(context.embedderEntrypointByproducts));
-            }
-        }
 
-        for (auto& unlinked : m_unlinkedWasmToWasmCalls) {
-            for (auto& call : unlinked) {
-                MacroAssemblerCodePtr<WasmEntryPtrTag> executableAddress;
-                if (m_moduleInformation->isImportedFunctionFromFunctionIndexSpace(call.functionIndexSpace)) {
-                    // FIXME imports could have been linked in B3, instead of generating a patchpoint. This condition should be replaced by a RELEASE_ASSERT. https://bugs.webkit.org/show_bug.cgi?id=166462
-                    executableAddress = m_wasmToWasmExitStubs.at(call.functionIndexSpace).code();
-                } else
-                    executableAddress = m_wasmInternalFunctions.at(call.functionIndexSpace - m_moduleInformation->importFunctionCount())->entrypoint.compilation->code().retagged<WasmEntryPtrTag>();
-                MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(executableAddress));
+                if (context.embedderMoveAndCall)
+                    callee.addCaller(locker, linkBuffer, *context.embedderMoveAndCall);
             }
+
         }
+
+        for (auto& callee : *m_callees)
+            callee->repatchCallers(locker, callee.get());
     }
     
     if (!isComplete()) {
index 883617f..e217462 100644 (file)
@@ -51,7 +51,7 @@ public:
     enum AsyncWork : uint8_t { FullCompile, Validation };
 
     // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
-    BBQPlan(Context*, Ref<ModuleInformation>, AsyncWork, CompletionTask&&, CreateEmbedderWrapper&&, ThrowWasmException);
+    BBQPlan(Context*, const Vector<Ref<BBQCallee>>&, Ref<ModuleInformation>, AsyncWork, CompletionTask&&, CreateEmbedderWrapper&&, ThrowWasmException);
     JS_EXPORT_PRIVATE BBQPlan(Context*, Vector<uint8_t>&&, AsyncWork, CompletionTask&&, CreateEmbedderWrapper&&, ThrowWasmException);
     BBQPlan(Context*, AsyncWork, CompletionTask&&);
 
@@ -98,12 +98,6 @@ public:
         return WTFMove(m_wasmToWasmExitStubs);
     }
 
-    Vector<Vector<UnlinkedWasmToWasmCall>> takeWasmToWasmCallsites()
-    {
-        RELEASE_ASSERT(!failed() && !hasWork());
-        return WTFMove(m_unlinkedWasmToWasmCalls);
-    }
-
     enum class State : uint8_t {
         Initial,
         Validated,
@@ -139,14 +133,13 @@ private:
     Vector<uint8_t> m_source;
     Bag<CallLinkInfo> m_callLinkInfos;
     Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>> m_wasmToWasmExitStubs;
-    Vector<std::unique_ptr<InternalFunction>> m_wasmInternalFunctions;
     HashSet<uint32_t, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_exportedFunctionIndices;
-    HashMap<uint32_t, std::unique_ptr<InternalFunction>, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_embedderToWasmInternalFunctions;
+    HashMap<uint32_t, RefPtr<EmbedderEntrypointCallee>, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_embedderToWasmInternalFunctions;
     Vector<CompilationContext> m_compilationContexts;
     Vector<std::unique_ptr<TierUpCount>> m_tierUpCounts;
 
-    Vector<Vector<UnlinkedWasmToWasmCall>> m_unlinkedWasmToWasmCalls;
     StreamingParser m_streamingParser;
+    const Vector<Ref<BBQCallee>>* m_callees { nullptr };
     State m_state;
 
     const AsyncWork m_asyncWork;
index 86601ac..9bbc6f3 100644 (file)
@@ -38,21 +38,8 @@ template<typename Functor>
 void BBQPlan::initializeCallees(const Functor& callback)
 {
     ASSERT(!failed());
-    for (unsigned internalFunctionIndex = 0; internalFunctionIndex < m_wasmInternalFunctions.size(); ++internalFunctionIndex) {
-
-        RefPtr<Wasm::Callee> embedderEntrypointCallee;
-        if (auto embedderToWasmFunction = m_embedderToWasmInternalFunctions.get(internalFunctionIndex)) {
-            embedderEntrypointCallee = Wasm::EmbedderEntrypointCallee::create(WTFMove(embedderToWasmFunction->entrypoint));
-            MacroAssembler::repatchPointer(embedderToWasmFunction->calleeMoveLocation, CalleeBits::boxWasm(embedderEntrypointCallee.get()));
-        }
-
-        InternalFunction* function = m_wasmInternalFunctions[internalFunctionIndex].get();
-        size_t functionIndexSpace = internalFunctionIndex + m_moduleInformation->importFunctionCount();
-        Ref<Wasm::Callee> wasmEntrypointCallee = Wasm::BBQCallee::create(WTFMove(function->entrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(m_tierUpCounts[internalFunctionIndex]));
-        MacroAssembler::repatchPointer(function->calleeMoveLocation, CalleeBits::boxWasm(wasmEntrypointCallee.ptr()));
-
-        callback(internalFunctionIndex, WTFMove(embedderEntrypointCallee), WTFMove(wasmEntrypointCallee));
-    }
+    for (unsigned internalFunctionIndex = 0; internalFunctionIndex < m_moduleInformation->functions.size(); ++internalFunctionIndex)
+        callback(internalFunctionIndex, m_embedderToWasmInternalFunctions.get(internalFunctionIndex));
 }
 
 } } // namespace JSC::Wasm
index ce61076..eefb972 100644 (file)
@@ -47,6 +47,7 @@ Expected<MacroAssemblerCodeRef<WasmEntryPtrTag>, BindingFailure> wasmToWasm(unsi
     JIT jit;
 
     GPRReg scratch = wasmCallingConventionAir().prologueScratch(0);
+    GPRReg callee = wasmCallingConventionAir().prologueScratch(1);
     GPRReg baseMemory = pinnedRegs.baseMemoryPointer;
     ASSERT(baseMemory != scratch);
     ASSERT(pinnedRegs.sizeRegister != baseMemory);
@@ -57,13 +58,20 @@ Expected<MacroAssemblerCodeRef<WasmEntryPtrTag>, BindingFailure> wasmToWasm(unsi
     jit.loadWasmContextInstance(sizeRegAsScratch); // Old Instance*
     // Get the callee's Wasm::Instance and set it as WasmContext's instance. The caller will take care of restoring its own Instance.
     jit.loadPtr(JIT::Address(sizeRegAsScratch, Instance::offsetOfTargetInstance(importIndex)), baseMemory); // Instance*.
-    // While we're accessing that cacheline, also get the wasm entrypoint so we can tail call to it below.
+    // While we're accessing that cacheline, also get the wasm entrypoint and callee so we can tail call to it below.
     jit.loadPtr(JIT::Address(sizeRegAsScratch, Instance::offsetOfWasmEntrypointLoadLocation(importIndex)), scratch);
+    jit.loadPtr(JIT::Address(sizeRegAsScratch, Instance::offsetOfBoxedCalleeLoadLocation(importIndex)), callee);
     jit.storeWasmContextInstance(baseMemory);
 
     jit.loadPtr(JIT::Address(sizeRegAsScratch, Instance::offsetOfCachedStackLimit()), sizeRegAsScratch);
     jit.storePtr(sizeRegAsScratch, JIT::Address(baseMemory, Instance::offsetOfCachedStackLimit()));
 
+    // Set the callee slot in the call frame
+    jit.loadPtr(callee, callee);
+    // At this point, we have been called, so ReturnPC is on the stack, but we have not yet pushed the frame pointer,
+    // so we have to subtract the space for CallerFrame from the callee slot
+    jit.storePtr(callee, JIT::Address(JIT::stackPointerRegister, CallFrameSlot::callee * sizeof(Register) - sizeof(CPURegister)));
+
     // FIXME the following code assumes that all Wasm::Instance have the same pinned registers. https://bugs.webkit.org/show_bug.cgi?id=162952
     // Set up the callee's baseMemory register as well as the memory size registers.
     {
index e8ddf15..93314ab 100644 (file)
 
 namespace JSC { namespace Wasm {
 
-Callee::Callee(Wasm::CompilationMode compilationMode, Entrypoint&& entrypoint)
+Callee::Callee(Wasm::CompilationMode compilationMode)
     : m_compilationMode(compilationMode)
-    , m_entrypoint(WTFMove(entrypoint))
 {
     CalleeRegistry::singleton().registerCallee(this);
 }
 
-Callee::Callee(Wasm::CompilationMode compilationMode, Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name)
+Callee::Callee(Wasm::CompilationMode compilationMode, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name)
     : m_compilationMode(compilationMode)
-    , m_entrypoint(WTFMove(entrypoint))
     , m_indexOrName(index, WTFMove(name))
 {
     CalleeRegistry::singleton().registerCallee(this);
@@ -52,6 +50,48 @@ Callee::~Callee()
     CalleeRegistry::singleton().unregisterCallee(this);
 }
 
+
+inline void repatchMove(const AbstractLocker&, CodeLocationDataLabelPtr<WasmEntryPtrTag> moveLocation, Callee& targetCallee)
+{
+    MacroAssembler::repatchPointer(moveLocation, CalleeBits::boxWasm(&targetCallee));
+}
+
+inline void repatchCall(const AbstractLocker&, CodeLocationNearCall<WasmEntryPtrTag> callLocation, Callee& targetCallee)
+{
+    MacroAssembler::repatchNearCall(callLocation, CodeLocationLabel<WasmEntryPtrTag>(targetCallee.code()));
+}
+
+void BBQCallee::addCaller(const AbstractLocker&, LinkBuffer& linkBuffer, UnlinkedMoveAndCall call)
+{
+    auto moveLocation = linkBuffer.locationOf<WasmEntryPtrTag>(call.moveLocation);
+    auto callLocation = linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call.callLocation);
+    m_moveLocations.append(moveLocation);
+    m_callLocations.append(callLocation);
+}
+
+void BBQCallee::addAndLinkCaller(const AbstractLocker& locker, LinkBuffer& linkBuffer, UnlinkedMoveAndCall call)
+{
+    RELEASE_ASSERT(entrypoint().compilation);
+
+    auto moveLocation = linkBuffer.locationOf<WasmEntryPtrTag>(call.moveLocation);
+    auto callLocation = linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call.callLocation);
+    m_moveLocations.append(moveLocation);
+    m_callLocations.append(callLocation);
+
+    Callee& targetCallee = m_replacement ? static_cast<Callee&>(*m_replacement) : *this;
+    repatchMove(locker, moveLocation, targetCallee);
+    repatchCall(locker, callLocation, targetCallee);
+}
+
+void BBQCallee::repatchCallers(const AbstractLocker& locker, Callee& targetCallee)
+{
+    RELEASE_ASSERT(targetCallee.entrypoint().compilation);
+    for (auto moveLocation : m_moveLocations)
+        repatchMove(locker, moveLocation, targetCallee);
+    for (auto callLocation : m_callLocations)
+        repatchCall(locker, callLocation, targetCallee);
+}
+
 } } // namespace JSC::Wasm
 
 #endif // ENABLE(WEBASSEMBLY)
index f7ea4d0..7eeed52 100644 (file)
@@ -40,12 +40,13 @@ namespace JSC { namespace Wasm {
 class Callee : public ThreadSafeRefCounted<Callee> {
     WTF_MAKE_FAST_ALLOCATED;
 public:
-    MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint() const { return m_entrypoint.compilation->code().retagged<WasmEntryPtrTag>(); }
-
-    RegisterAtOffsetList* calleeSaveRegisters() { return &m_entrypoint.calleeSaveRegisters; }
+    Entrypoint& entrypoint() { return m_entrypoint; }
+    MacroAssemblerCodePtr<WasmEntryPtrTag> code() const { return m_entrypoint.compilation->code().retagged<WasmEntryPtrTag>(); }
     IndexOrName indexOrName() const { return m_indexOrName; }
     CompilationMode compilationMode() const { return m_compilationMode; }
 
+    RegisterAtOffsetList* calleeSaveRegisters() { return &m_entrypoint.calleeSaveRegisters; }
+
     std::tuple<void*, void*> range() const
     {
         void* start = m_entrypoint.compilation->codeRef().executableMemory()->start().untaggedPtr();
@@ -56,78 +57,70 @@ public:
     JS_EXPORT_PRIVATE virtual ~Callee();
 
 protected:
-    JS_EXPORT_PRIVATE Callee(Wasm::CompilationMode, Wasm::Entrypoint&&);
-    JS_EXPORT_PRIVATE Callee(Wasm::CompilationMode, Wasm::Entrypoint&&, size_t, std::pair<const Name*, RefPtr<NameSection>>&&);
+    JS_EXPORT_PRIVATE Callee(CompilationMode);
+    JS_EXPORT_PRIVATE Callee(CompilationMode, size_t, std::pair<const Name*, RefPtr<NameSection>>&&);
 
 private:
     CompilationMode m_compilationMode;
-    Wasm::Entrypoint m_entrypoint;
+    Entrypoint m_entrypoint;
     IndexOrName m_indexOrName;
 };
 
 class OMGCallee final : public Callee {
 public:
-    static Ref<OMGCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
+    static Ref<OMGCallee> create(size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name)
     {
-        return adoptRef(*new OMGCallee(WTFMove(entrypoint), index, WTFMove(name), WTFMove(unlinkedCalls)));
+        return adoptRef(*new OMGCallee(index, WTFMove(name)));
     }
 
-    Vector<UnlinkedWasmToWasmCall>& wasmToWasmCallsites() { return m_wasmToWasmCallsites; }
-
 private:
-    OMGCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
-        : Callee(Wasm::CompilationMode::OMGMode, WTFMove(entrypoint), index, WTFMove(name))
-        , m_wasmToWasmCallsites(WTFMove(unlinkedCalls))
+    OMGCallee(size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name)
+        : Callee(CompilationMode::OMGMode, index, WTFMove(name))
     {
     }
-
-    Vector<UnlinkedWasmToWasmCall> m_wasmToWasmCallsites;
 };
 
 class OMGForOSREntryCallee final : public Callee {
 public:
-    static Ref<OMGForOSREntryCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, unsigned osrEntryScratchBufferSize, uint32_t loopIndex, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
+    static Ref<OMGForOSREntryCallee> create(size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, unsigned osrEntryScratchBufferSize, uint32_t loopIndex)
     {
-        return adoptRef(*new OMGForOSREntryCallee(WTFMove(entrypoint), index, WTFMove(name), osrEntryScratchBufferSize, loopIndex, WTFMove(unlinkedCalls)));
+        return adoptRef(*new OMGForOSREntryCallee(index, WTFMove(name), osrEntryScratchBufferSize, loopIndex));
     }
 
     unsigned osrEntryScratchBufferSize() const { return m_osrEntryScratchBufferSize; }
     uint32_t loopIndex() const { return m_loopIndex; }
-    Vector<UnlinkedWasmToWasmCall>& wasmToWasmCallsites() { return m_wasmToWasmCallsites; }
 
 private:
-    OMGForOSREntryCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, unsigned osrEntryScratchBufferSize, uint32_t loopIndex, Vector<UnlinkedWasmToWasmCall>&& unlinkedCalls)
-        : Callee(Wasm::CompilationMode::OMGForOSREntryMode, WTFMove(entrypoint), index, WTFMove(name))
-        , m_wasmToWasmCallsites(WTFMove(unlinkedCalls))
+    OMGForOSREntryCallee(size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, unsigned osrEntryScratchBufferSize, uint32_t loopIndex)
+        : Callee(CompilationMode::OMGForOSREntryMode, index, WTFMove(name))
         , m_osrEntryScratchBufferSize(osrEntryScratchBufferSize)
         , m_loopIndex(loopIndex)
     {
     }
 
-    Vector<UnlinkedWasmToWasmCall> m_wasmToWasmCallsites;
     unsigned m_osrEntryScratchBufferSize;
     uint32_t m_loopIndex;
 };
 
 class EmbedderEntrypointCallee final : public Callee {
 public:
-    static Ref<EmbedderEntrypointCallee> create(Wasm::Entrypoint&& entrypoint)
+    static Ref<EmbedderEntrypointCallee> create()
     {
-        return adoptRef(*new EmbedderEntrypointCallee(WTFMove(entrypoint)));
+        return adoptRef(*new EmbedderEntrypointCallee());
     }
 
 private:
-    EmbedderEntrypointCallee(Wasm::Entrypoint&& entrypoint)
-        : Callee(Wasm::CompilationMode::EmbedderEntrypointMode, WTFMove(entrypoint))
+    EmbedderEntrypointCallee()
+        : Callee(CompilationMode::EmbedderEntrypointMode)
     {
     }
 };
 
 class BBQCallee final : public Callee {
 public:
-    static Ref<BBQCallee> create(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount)
+    static Ref<BBQCallee> create(size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount)
     {
-        return adoptRef(*new BBQCallee(WTFMove(entrypoint), index, WTFMove(name), WTFMove(tierUpCount)));
+        return adoptRef(*new BBQCallee(index, WTFMove(name), WTFMove(tierUpCount)));
     }
 
     OMGForOSREntryCallee* osrEntryCallee() { return m_osrEntryCallee.get(); }
@@ -140,20 +133,27 @@ public:
     void setDidStartCompilingOSREntryCallee(bool value) { m_didStartCompilingOSREntryCallee = value; }
 
     OMGCallee* replacement() { return m_replacement.get(); }
-    void setReplacement(Ref<OMGCallee>&& replacement)
+    void setReplacement(const AbstractLocker&, Ref<OMGCallee>&& replacement)
     {
         m_replacement = WTFMove(replacement);
     }
 
     TierUpCount* tierUpCount() { return m_tierUpCount.get(); }
 
+    void addCaller(const AbstractLocker&, LinkBuffer&, UnlinkedMoveAndCall);
+    void addAndLinkCaller(const AbstractLocker&, LinkBuffer&, UnlinkedMoveAndCall);
+
+    void repatchCallers(const AbstractLocker&, Callee&);
+
 private:
-    BBQCallee(Wasm::Entrypoint&& entrypoint, size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount)
-        : Callee(Wasm::CompilationMode::BBQMode, WTFMove(entrypoint), index, WTFMove(name))
+    BBQCallee(size_t index, std::pair<const Name*, RefPtr<NameSection>>&& name, std::unique_ptr<TierUpCount>&& tierUpCount)
+        : Callee(CompilationMode::BBQMode, index, WTFMove(name))
         , m_tierUpCount(WTFMove(tierUpCount))
     {
     }
 
+    Vector<CodeLocationDataLabelPtr<WasmEntryPtrTag>> m_moveLocations;
+    Vector<CodeLocationNearCall<WasmEntryPtrTag>> m_callLocations;
     RefPtr<OMGForOSREntryCallee> m_osrEntryCallee;
     RefPtr<OMGCallee> m_replacement;
     std::unique_ptr<TierUpCount> m_tierUpCount;
index b442b61..6fa4369 100644 (file)
@@ -88,34 +88,16 @@ private:
 
 public:
     static unsigned headerSizeInBytes() { return headerSize; }
-    void setupFrameInPrologue(CodeLocationDataLabelPtr<WasmEntryPtrTag>* calleeMoveLocation, B3::Procedure& proc, B3::Origin origin, B3::BasicBlock* block) const
+    void setupFrameInPrologue(B3::Procedure& proc, B3::Origin origin, B3::BasicBlock* block) const
     {
-        static_assert(CallFrameSlot::callee * sizeof(Register) < headerSize, "We rely on this here for now.");
         static_assert(CallFrameSlot::codeBlock * sizeof(Register) < headerSize, "We rely on this here for now.");
 
-        B3::PatchpointValue* getCalleePatchpoint = block->appendNew<B3::PatchpointValue>(proc, B3::Int64, origin);
-        getCalleePatchpoint->resultConstraints = { B3::ValueRep::SomeRegister };
-        getCalleePatchpoint->effects = B3::Effects::none();
-        getCalleePatchpoint->setGenerator(
-            [=] (CCallHelpers& jit, const B3::StackmapGenerationParams& params) {
-                GPRReg result = params[0].gpr();
-                MacroAssembler::DataLabelPtr moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), result);
-                jit.addLinkTask([calleeMoveLocation, moveLocation] (LinkBuffer& linkBuffer) {
-                    *calleeMoveLocation = linkBuffer.locationOf<WasmEntryPtrTag>(moveLocation);
-                });
-            });
-
-        B3::Value* framePointer = block->appendNew<B3::Value>(proc, B3::FramePointer, origin);
-        B3::Value* offsetOfCallee = block->appendNew<B3::Const64Value>(proc, origin, CallFrameSlot::callee * sizeof(Register));
-        block->appendNew<B3::MemoryValue>(proc, B3::Store, origin,
-            getCalleePatchpoint,
-            block->appendNew<B3::Value>(proc, B3::Add, origin, framePointer, offsetOfCallee));
-
         // FIXME: We shouldn't have to store zero into the CodeBlock* spot in the call frame,
         // but there are places that interpret non-null CodeBlock slot to mean a valid CodeBlock.
         // When doing unwinding, we'll need to verify that the entire runtime is OK with a non-null
         // CodeBlock not implying that the CodeBlock is valid.
         // https://bugs.webkit.org/show_bug.cgi?id=165321
+        B3::Value* framePointer = block->appendNew<B3::Value>(proc, B3::FramePointer, origin);
         B3::Value* offsetOfCodeBlock = block->appendNew<B3::Const64Value>(proc, origin, CallFrameSlot::codeBlock * sizeof(Register));
         block->appendNew<B3::MemoryValue>(proc, B3::Store, origin,
             block->appendNew<B3::Const64Value>(proc, origin, 0),
index d4a9cd6..3a87f61 100644 (file)
@@ -29,7 +29,6 @@
 #if ENABLE(WEBASSEMBLY)
 
 #include "WasmBBQPlanInlines.h"
-#include "WasmCallee.h"
 #include "WasmFormat.h"
 #include "WasmWorklist.h"
 
@@ -47,7 +46,15 @@ CodeBlock::CodeBlock(Context* context, MemoryMode mode, ModuleInformation& modul
 {
     RefPtr<CodeBlock> protectedThis = this;
 
-    m_plan = adoptRef(*new BBQPlan(context, makeRef(moduleInformation), BBQPlan::FullCompile, createSharedTask<Plan::CallbackType>([this, protectedThis = WTFMove(protectedThis)] (Plan&) {
+    m_callees.resize(m_calleeCount);
+    m_boxedCallees.resize(m_calleeCount);
+    for (uint32_t i = 0; i < m_calleeCount; i++) {
+        uint32_t functionIndexSpace = moduleInformation.importFunctionCount() + i;
+        m_callees[i] = BBQCallee::create(functionIndexSpace, moduleInformation.nameSection->get(functionIndexSpace), makeUnique<TierUpCount>());
+        m_boxedCallees[i] = CalleeBits::boxWasm(m_callees[i].ptr());
+    }
+
+    m_plan = adoptRef(*new BBQPlan(context, m_callees, makeRef(moduleInformation), BBQPlan::FullCompile, createSharedTask<Plan::CallbackType>([this, protectedThis = WTFMove(protectedThis)] (Plan&) {
         auto locker = holdLock(m_lock);
         if (m_plan->failed()) {
             m_errorMessage = m_plan->errorMessage();
@@ -56,21 +63,18 @@ CodeBlock::CodeBlock(Context* context, MemoryMode mode, ModuleInformation& modul
         }
 
         // FIXME: we should eventually collect the BBQ code.
-        m_callees.resize(m_calleeCount);
         m_optimizedCallees.resize(m_calleeCount);
         m_wasmIndirectCallEntryPoints.resize(m_calleeCount);
 
-        m_plan->initializeCallees([&] (unsigned calleeIndex, RefPtr<Wasm::Callee>&& embedderEntrypointCallee, Ref<Wasm::Callee>&& wasmEntrypointCallee) {
+        m_plan->initializeCallees([&] (unsigned calleeIndex, RefPtr<EmbedderEntrypointCallee>&& embedderEntrypointCallee) {
             if (embedderEntrypointCallee) {
                 auto result = m_embedderCallees.set(calleeIndex, WTFMove(embedderEntrypointCallee));
                 ASSERT_UNUSED(result, result.isNewEntry);
             }
-            m_callees[calleeIndex] = WTFMove(wasmEntrypointCallee);
-            m_wasmIndirectCallEntryPoints[calleeIndex] = m_callees[calleeIndex]->entrypoint();
+            m_wasmIndirectCallEntryPoints[calleeIndex] = m_callees[calleeIndex]->code();
         });
 
         m_wasmToWasmExitStubs = m_plan->takeWasmToWasmExitStubs();
-        m_wasmToWasmCallsites = m_plan->takeWasmToWasmCallsites();
 
         setCompilationFinished();
     }), WTFMove(createEmbedderWrapper), throwWasmException));
index dea2ffa..10ef24c 100644 (file)
@@ -28,6 +28,7 @@
 #if ENABLE(WEBASSEMBLY)
 
 #include "MacroAssemblerCodeRef.h"
+#include "WasmCallee.h"
 #include "WasmEmbedder.h"
 #include "WasmTierUpCount.h"
 #include <wtf/Lock.h>
@@ -40,9 +41,8 @@ namespace JSC {
 
 namespace Wasm {
 
-class Callee;
-struct Context;
 class BBQPlan;
+struct Context;
 class OMGPlan;
 struct ModuleInformation;
 struct UnlinkedWasmToWasmCall;
@@ -76,7 +76,7 @@ public:
 
     // These two callee getters are only valid once the callees have been populated.
 
-    Callee& embedderEntrypointCalleeFromFunctionIndexSpace(unsigned functionIndexSpace)
+    EmbedderEntrypointCallee& embedderEntrypointCalleeFromFunctionIndexSpace(unsigned functionIndexSpace)
     {
         ASSERT(runnable());
         RELEASE_ASSERT(functionIndexSpace >= functionImportCount());
@@ -86,29 +86,27 @@ public:
         RELEASE_ASSERT(callee);
         return *callee;
     }
-    Callee& wasmEntrypointCalleeFromFunctionIndexSpace(unsigned functionIndexSpace)
+
+    BBQCallee& wasmBBQCalleeFromFunctionIndexSpace(unsigned functionIndexSpace)
     {
         ASSERT(runnable());
         RELEASE_ASSERT(functionIndexSpace >= functionImportCount());
         unsigned calleeIndex = functionIndexSpace - functionImportCount();
-        if (m_optimizedCallees[calleeIndex])
-            return *m_optimizedCallees[calleeIndex].get();
-        return *m_callees[calleeIndex].get();
+        return m_callees[calleeIndex].get();
     }
 
-    Callee& wasmBBQCalleeFromFunctionIndexSpace(unsigned functionIndexSpace)
+    MacroAssemblerCodePtr<WasmEntryPtrTag>* entrypointLoadLocationFromFunctionIndexSpace(unsigned functionIndexSpace)
     {
-        ASSERT(runnable());
         RELEASE_ASSERT(functionIndexSpace >= functionImportCount());
         unsigned calleeIndex = functionIndexSpace - functionImportCount();
-        return *m_callees[calleeIndex].get();
+        return &m_wasmIndirectCallEntryPoints[calleeIndex];
     }
 
-    MacroAssemblerCodePtr<WasmEntryPtrTag>* entrypointLoadLocationFromFunctionIndexSpace(unsigned functionIndexSpace)
+    void** boxedCalleeLoadLocationFromFunctionIndexSpace(unsigned functionIndexSpace)
     {
         RELEASE_ASSERT(functionIndexSpace >= functionImportCount());
         unsigned calleeIndex = functionIndexSpace - functionImportCount();
-        return &m_wasmIndirectCallEntryPoints[calleeIndex];
+        return &m_boxedCallees[calleeIndex];
     }
 
     bool isSafeToRun(MemoryMode);
@@ -124,11 +122,11 @@ private:
     void setCompilationFinished();
     unsigned m_calleeCount;
     MemoryMode m_mode;
-    Vector<RefPtr<Callee>> m_callees;
-    Vector<RefPtr<Callee>> m_optimizedCallees;
-    HashMap<uint32_t, RefPtr<Callee>, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_embedderCallees;
+    Vector<Ref<BBQCallee>> m_callees;
+    Vector<RefPtr<OMGCallee>> m_optimizedCallees;
+    Vector<void*> m_boxedCallees;
+    HashMap<uint32_t, RefPtr<EmbedderEntrypointCallee>, typename DefaultHash<uint32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint32_t>> m_embedderCallees;
     Vector<MacroAssemblerCodePtr<WasmEntryPtrTag>> m_wasmIndirectCallEntryPoints;
-    Vector<Vector<UnlinkedWasmToWasmCall>> m_wasmToWasmCallsites;
     Vector<MacroAssemblerCodeRef<WasmEntryPtrTag>> m_wasmToWasmExitStubs;
     RefPtr<BBQPlan> m_plan;
     std::atomic<bool> m_compilationFinished { false };
index a2379fd..c78b443 100644 (file)
@@ -41,6 +41,7 @@ class ExecState;
 namespace Wasm {
 
 struct CompilationContext;
+class EmbedderEntrypointCallee;
 class Instance;
 struct InternalFunction;
 struct ModuleInformation;
@@ -48,7 +49,7 @@ class Signature;
 struct UnlinkedWasmToWasmCall;
 
 // Create wrapper code to call from embedder -> WebAssembly.
-using CreateEmbedderWrapper = WTF::Function<std::unique_ptr<InternalFunction>(CompilationContext&, const Signature&, Vector<UnlinkedWasmToWasmCall>*, const ModuleInformation&, MemoryMode, uint32_t)>;
+using CreateEmbedderWrapper = WTF::Function<void(CompilationContext&, const Signature&, EmbedderEntrypointCallee*, const ModuleInformation&, MemoryMode)>;
 
 // Called as soon as an exception is detected. The return value is the PC to continue at.
 using ThrowWasmException = void* (*)(ExecState*, Wasm::ExceptionType, Instance*);
index 0669bb4..ea0e0b6 100644 (file)
@@ -287,10 +287,15 @@ inline bool isValidNameType(Int val)
     return false;
 }
 
+struct UnlinkedMoveAndCall {
+    MacroAssembler::DataLabelPtr moveLocation;
+    MacroAssembler::Call callLocation;
+};
+
 struct UnlinkedWasmToWasmCall {
     WTF_MAKE_STRUCT_FAST_ALLOCATED;
-    CodeLocationNearCall<WasmEntryPtrTag> callLocation;
-    size_t functionIndexSpace;
+    size_t targetFunctionIndexSpace;
+    UnlinkedMoveAndCall unlinkedMoveAndCall;
 };
 
 struct Entrypoint {
@@ -313,10 +318,12 @@ struct WasmToWasmImportableFunction {
     using LoadLocation = MacroAssemblerCodePtr<WasmEntryPtrTag>*;
     static ptrdiff_t offsetOfSignatureIndex() { return OBJECT_OFFSETOF(WasmToWasmImportableFunction, signatureIndex); }
     static ptrdiff_t offsetOfEntrypointLoadLocation() { return OBJECT_OFFSETOF(WasmToWasmImportableFunction, entrypointLoadLocation); }
+    static ptrdiff_t offsetOfBoxedCalleeLoadLocation() { return OBJECT_OFFSETOF(WasmToWasmImportableFunction, boxedCalleeLoadLocation); }
 
     // FIXME: Pack signature index and code pointer into one 64-bit value. See <https://bugs.webkit.org/show_bug.cgi?id=165511>.
     SignatureIndex signatureIndex { Signature::invalidIndex };
     LoadLocation entrypointLoadLocation;
+    void** boxedCalleeLoadLocation;
 };
 using FunctionIndexSpace = Vector<WasmToWasmImportableFunction>;
 
index f4f3508..7e2124f 100644 (file)
@@ -130,6 +130,7 @@ public:
         WasmToWasmImportableFunction::LoadLocation wasmEntrypointLoadLocation { nullptr };
         MacroAssemblerCodePtr<WasmEntryPtrTag> wasmToEmbedderStub;
         void* importFunction { nullptr }; // In a JS embedding, this is a WriteBarrier<JSObject>.
+        void** boxedCalleeLoadLocation { nullptr };
     };
     unsigned numImportFunctions() const { return m_numImportFunctions; }
     ImportFunctionInfo* importFunctionInfo(size_t importFunctionNum)
@@ -141,6 +142,7 @@ public:
     static size_t offsetOfWasmEntrypointLoadLocation(size_t importFunctionNum) { return offsetOfTail() + importFunctionNum * sizeof(ImportFunctionInfo) + OBJECT_OFFSETOF(ImportFunctionInfo, wasmEntrypointLoadLocation); }
     static size_t offsetOfWasmToEmbedderStub(size_t importFunctionNum) { return offsetOfTail() + importFunctionNum * sizeof(ImportFunctionInfo) + OBJECT_OFFSETOF(ImportFunctionInfo, wasmToEmbedderStub); }
     static size_t offsetOfImportFunction(size_t importFunctionNum) { return offsetOfTail() + importFunctionNum * sizeof(ImportFunctionInfo) + OBJECT_OFFSETOF(ImportFunctionInfo, importFunction); }
+    static size_t offsetOfBoxedCalleeLoadLocation(size_t importFunctionNum) { return offsetOfTail() + importFunctionNum * sizeof(ImportFunctionInfo) + OBJECT_OFFSETOF(ImportFunctionInfo, boxedCalleeLoadLocation); }
     template<typename T> T* importFunction(unsigned importFunctionNum) { return reinterpret_cast<T*>(&importFunctionInfo(importFunctionNum)->importFunction); }
 
     static_assert(sizeof(ImportFunctionInfo) == WTF::roundUpToMultipleOf<sizeof(uint64_t)>(sizeof(ImportFunctionInfo)), "We rely on this for the alignment to be correct");
index a0bb6cf..cfaeaf2 100644 (file)
@@ -54,15 +54,15 @@ namespace WasmOMGForOSREntryPlanInternal {
 static const bool verbose = false;
 }
 
-OMGForOSREntryPlan::OMGForOSREntryPlan(Context* context, Ref<Module>&& module, Ref<BBQCallee>&& callee, uint32_t functionIndex, uint32_t loopIndex, MemoryMode mode, CompletionTask&& task)
+OMGForOSREntryPlan::OMGForOSREntryPlan(Context* context, Ref<Module>&& module, Ref<BBQCallee>&& callee, uint32_t functionIndex, uint32_t loopIndex, CodeBlock& codeBlock, CompletionTask&& task)
     : Base(context, makeRef(const_cast<ModuleInformation&>(module->moduleInformation())), WTFMove(task))
     , m_module(WTFMove(module))
-    , m_codeBlock(*m_module->codeBlockFor(mode))
+    , m_codeBlock(codeBlock)
     , m_callee(WTFMove(callee))
     , m_functionIndex(functionIndex)
     , m_loopIndex(loopIndex)
 {
-    setMode(mode);
+    setMode(codeBlock.mode());
     ASSERT(m_codeBlock->runnable());
     ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(m_mode));
     dataLogLnIf(WasmOMGForOSREntryPlanInternal::verbose, "Starting OMGForOSREntry plan for ", functionIndex, " of module: ", RawPointer(&m_module.get()));
@@ -81,45 +81,33 @@ void OMGForOSREntryPlan::work(CompilationEffort)
     const Signature& signature = SignatureInformation::get(signatureIndex);
     ASSERT(validateFunction(function, signature, m_moduleInformation.get()));
 
-    Vector<UnlinkedWasmToWasmCall> unlinkedCalls;
     CompilationContext context;
     unsigned osrEntryScratchBufferSize = 0;
-    auto parseAndCompileResult = parseAndCompile(context, function, signature, unlinkedCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGForOSREntryMode, m_functionIndex, m_loopIndex);
+    auto parseAndCompileResult = parseAndCompile(context, function, signature, m_codeBlock->m_wasmToWasmExitStubs, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGForOSREntryMode, m_functionIndex, m_loopIndex);
 
     if (UNLIKELY(!parseAndCompileResult)) {
         fail(holdLock(m_lock), makeString(parseAndCompileResult.error(), "when trying to tier up ", String::number(m_functionIndex)));
         return;
     }
 
-    Entrypoint omgEntrypoint;
     LinkBuffer linkBuffer(*context.wasmEntrypointJIT, nullptr, JITCompilationCanFail);
     if (UNLIKELY(linkBuffer.didFailToAllocate())) {
         Base::fail(holdLock(m_lock), makeString("Out of executable memory while tiering up function at index ", String::number(m_functionIndex)));
         return;
     }
 
-    omgEntrypoint.compilation = makeUnique<B3::Compilation>(
+    Ref<OMGForOSREntryCallee> callee = OMGForOSREntryCallee::create(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), osrEntryScratchBufferSize, m_loopIndex);
+    callee->entrypoint().calleeSaveRegisters = WTFMove(context.calleeSaveRegisters);
+    callee->entrypoint().compilation = makeUnique<B3::Compilation>(
         FINALIZE_CODE(linkBuffer, B3CompilationPtrTag, "WebAssembly OMGForOSREntry function[%i] %s name %s", m_functionIndex, signature.toString().ascii().data(), makeString(IndexOrName(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace))).ascii().data()),
         WTFMove(context.wasmEntrypointByproducts));
 
-    omgEntrypoint.calleeSaveRegisters = WTFMove(parseAndCompileResult.value()->entrypoint.calleeSaveRegisters);
-
-    MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
     ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(mode()));
-    Ref<OMGForOSREntryCallee> callee = OMGForOSREntryCallee::create(WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), osrEntryScratchBufferSize, m_loopIndex, WTFMove(unlinkedCalls));
     {
-        MacroAssembler::repatchPointer(parseAndCompileResult.value()->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
-        entrypoint = callee->entrypoint();
-
         auto locker = holdLock(m_codeBlock->m_lock);
-        for (auto& call : callee->wasmToWasmCallsites()) {
-            MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
-            if (call.functionIndexSpace < m_module->moduleInformation().importFunctionCount())
-                entrypoint = m_codeBlock->m_wasmToWasmExitStubs[call.functionIndexSpace].code();
-            else
-                entrypoint = m_codeBlock->wasmEntrypointCalleeFromFunctionIndexSpace(call.functionIndexSpace).entrypoint().retagged<WasmEntryPtrTag>();
-
-            MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
+        for (auto& call : context.outgoingCalls) {
+            BBQCallee& targetCallee = m_codeBlock->wasmBBQCalleeFromFunctionIndexSpace(call.targetFunctionIndexSpace);
+            targetCallee.addAndLinkCaller(locker, linkBuffer, call.unlinkedMoveAndCall);
         }
     }
     resetInstructionCacheOnAllThreads();
@@ -132,19 +120,6 @@ void OMGForOSREntryPlan::work(CompilationEffort)
             m_callee->tierUpCount()->osrEntryTriggers()[m_loopIndex] = TierUpCount::TriggerReason::CompilationDone;
             m_callee->tierUpCount()->m_compilationStatusForOMGForOSREntry = TierUpCount::CompilationStatus::Compiled;
         }
-        WTF::storeStoreFence();
-        // It is possible that a new OMG callee is added while we release m_codeBlock->lock.
-        // Until we add OMGForOSREntry callee to BBQCallee's m_osrEntryCallee, this new OMG function linking does not happen for this OMGForOSREntry callee.
-        // We re-link this OMGForOSREntry callee again not to miss that chance.
-        for (auto& call : callee->wasmToWasmCallsites()) {
-            MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
-            if (call.functionIndexSpace < m_module->moduleInformation().importFunctionCount())
-                entrypoint = m_codeBlock->m_wasmToWasmExitStubs[call.functionIndexSpace].code();
-            else
-                entrypoint = m_codeBlock->wasmEntrypointCalleeFromFunctionIndexSpace(call.functionIndexSpace).entrypoint().retagged<WasmEntryPtrTag>();
-
-            MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
-        }
     }
     dataLogLnIf(WasmOMGForOSREntryPlanInternal::verbose, "Finished OMGForOSREntry ", m_functionIndex, " with tier up count at: ", m_callee->tierUpCount()->count());
     complete(holdLock(m_lock));
index 0151da6..d56ea42 100644 (file)
@@ -48,7 +48,7 @@ public:
     bool multiThreaded() const override { return false; }
 
     // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
-    OMGForOSREntryPlan(Context*, Ref<Module>&&, Ref<BBQCallee>&&, uint32_t functionIndex, uint32_t loopIndex, MemoryMode, CompletionTask&&);
+    OMGForOSREntryPlan(Context*, Ref<Module>&&, Ref<BBQCallee>&&, uint32_t functionIndex, uint32_t loopIndex, CodeBlock&, CompletionTask&&);
 
 private:
     // For some reason friendship doesn't extend to parent classes...
index a805d01..414690f 100644 (file)
@@ -54,13 +54,13 @@ namespace WasmOMGPlanInternal {
 static const bool verbose = false;
 }
 
-OMGPlan::OMGPlan(Context* context, Ref<Module>&& module, uint32_t functionIndex, MemoryMode mode, CompletionTask&& task)
+OMGPlan::OMGPlan(Context* context, Ref<Module>&& module, uint32_t functionIndex, CodeBlock& codeBlock, CompletionTask&& task)
     : Base(context, makeRef(const_cast<ModuleInformation&>(module->moduleInformation())), WTFMove(task))
     , m_module(WTFMove(module))
-    , m_codeBlock(*m_module->codeBlockFor(mode))
+    , m_codeBlock(codeBlock)
     , m_functionIndex(functionIndex)
 {
-    setMode(mode);
+    setMode(codeBlock.mode());
     ASSERT(m_codeBlock->runnable());
     ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(m_mode));
     dataLogLnIf(WasmOMGPlanInternal::verbose, "Starting OMG plan for ", functionIndex, " of module: ", RawPointer(&m_module.get()));
@@ -79,57 +79,48 @@ void OMGPlan::work(CompilationEffort)
     const Signature& signature = SignatureInformation::get(signatureIndex);
     ASSERT(validateFunction(function, signature, m_moduleInformation.get()));
 
-    Vector<UnlinkedWasmToWasmCall> unlinkedCalls;
     unsigned osrEntryScratchBufferSize;
     CompilationContext context;
-    auto parseAndCompileResult = parseAndCompile(context, function, signature, unlinkedCalls, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex, UINT32_MAX);
+    auto parseAndCompileResult = parseAndCompile(context, function, signature, m_codeBlock->m_wasmToWasmExitStubs, osrEntryScratchBufferSize, m_moduleInformation.get(), m_mode, CompilationMode::OMGMode, m_functionIndex, UINT32_MAX);
 
     if (UNLIKELY(!parseAndCompileResult)) {
         fail(holdLock(m_lock), makeString(parseAndCompileResult.error(), "when trying to tier up ", String::number(m_functionIndex)));
         return;
     }
 
-    Entrypoint omgEntrypoint;
     LinkBuffer linkBuffer(*context.wasmEntrypointJIT, nullptr, JITCompilationCanFail);
     if (UNLIKELY(linkBuffer.didFailToAllocate())) {
         Base::fail(holdLock(m_lock), makeString("Out of executable memory while tiering up function at index ", String::number(m_functionIndex)));
         return;
     }
 
-    omgEntrypoint.compilation = makeUnique<B3::Compilation>(
+    Ref<OMGCallee> omgCallee = OMGCallee::create(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace));
+    omgCallee->entrypoint().compilation = makeUnique<B3::Compilation>(
         FINALIZE_CODE(linkBuffer, B3CompilationPtrTag, "WebAssembly OMG function[%i] %s name %s", m_functionIndex, signature.toString().ascii().data(), makeString(IndexOrName(functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace))).ascii().data()),
         WTFMove(context.wasmEntrypointByproducts));
 
-    omgEntrypoint.calleeSaveRegisters = WTFMove(parseAndCompileResult.value()->entrypoint.calleeSaveRegisters);
+    omgCallee->entrypoint().calleeSaveRegisters = WTFMove(context.calleeSaveRegisters);
 
-    MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
+    BBQCallee& bbqCallee = m_codeBlock->m_callees[m_functionIndex].get();
     {
         ASSERT(m_codeBlock.ptr() == m_module->codeBlockFor(mode()));
-        Ref<OMGCallee> callee = OMGCallee::create(WTFMove(omgEntrypoint), functionIndexSpace, m_moduleInformation->nameSection->get(functionIndexSpace), WTFMove(unlinkedCalls));
-        MacroAssembler::repatchPointer(parseAndCompileResult.value()->calleeMoveLocation, CalleeBits::boxWasm(callee.ptr()));
         ASSERT(!m_codeBlock->m_optimizedCallees[m_functionIndex]);
-        entrypoint = callee->entrypoint();
 
         // We want to make sure we publish our callee at the same time as we link our callsites. This enables us to ensure we
         // always call the fastest code. Any function linked after us will see our new code and the new callsites, which they
         // will update. It's also ok if they publish their code before we reset the instruction caches because after we release
         // the lock our code is ready to be published too.
         LockHolder holder(m_codeBlock->m_lock);
-        m_codeBlock->m_optimizedCallees[m_functionIndex] = callee.copyRef();
+        m_codeBlock->m_optimizedCallees[m_functionIndex] = omgCallee.copyRef();
         {
-            BBQCallee& bbqCallee = *static_cast<BBQCallee*>(m_codeBlock->m_callees[m_functionIndex].get());
             auto locker = holdLock(bbqCallee.tierUpCount()->getLock());
-            bbqCallee.setReplacement(callee.copyRef());
+            bbqCallee.setReplacement(holder, omgCallee.copyRef());
             bbqCallee.tierUpCount()->m_compilationStatusForOMG = TierUpCount::CompilationStatus::Compiled;
         }
-        for (auto& call : callee->wasmToWasmCallsites()) {
-            MacroAssemblerCodePtr<WasmEntryPtrTag> entrypoint;
-            if (call.functionIndexSpace < m_module->moduleInformation().importFunctionCount())
-                entrypoint = m_codeBlock->m_wasmToWasmExitStubs[call.functionIndexSpace].code();
-            else
-                entrypoint = m_codeBlock->wasmEntrypointCalleeFromFunctionIndexSpace(call.functionIndexSpace).entrypoint().retagged<WasmEntryPtrTag>();
-
-            MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
+
+        for (auto& call : context.outgoingCalls) {
+            BBQCallee& targetCallee = m_codeBlock->wasmBBQCalleeFromFunctionIndexSpace(call.targetFunctionIndexSpace);
+            targetCallee.addAndLinkCaller(holder, linkBuffer, call.unlinkedMoveAndCall);
         }
     }
 
@@ -139,30 +130,14 @@ void OMGPlan::work(CompilationEffort)
     resetInstructionCacheOnAllThreads();
     WTF::storeStoreFence(); // This probably isn't necessary but it's good to be paranoid.
 
-    m_codeBlock->m_wasmIndirectCallEntryPoints[m_functionIndex] = entrypoint;
+    m_codeBlock->m_wasmIndirectCallEntryPoints[m_functionIndex] = omgCallee->code();
+    m_codeBlock->m_boxedCallees[m_functionIndex] = CalleeBits::boxWasm(omgCallee.ptr());
     {
         LockHolder holder(m_codeBlock->m_lock);
-
-        auto repatchCalls = [&] (const Vector<UnlinkedWasmToWasmCall>& callsites) {
-            for (auto& call : callsites) {
-                dataLogLnIf(WasmOMGPlanInternal::verbose, "Considering repatching call at: ", RawPointer(call.callLocation.dataLocation()), " that targets ", call.functionIndexSpace);
-                if (call.functionIndexSpace == functionIndexSpace) {
-                    dataLogLnIf(WasmOMGPlanInternal::verbose, "Repatching call at: ", RawPointer(call.callLocation.dataLocation()), " to ", RawPointer(entrypoint.executableAddress()));
-                    MacroAssembler::repatchNearCall(call.callLocation, CodeLocationLabel<WasmEntryPtrTag>(entrypoint));
-                }
-            }
-        };
-
-        for (unsigned i = 0; i < m_codeBlock->m_wasmToWasmCallsites.size(); ++i) {
-            repatchCalls(m_codeBlock->m_wasmToWasmCallsites[i]);
-            if (OMGCallee* replacementCallee = static_cast<BBQCallee*>(m_codeBlock->m_callees[i].get())->replacement())
-                repatchCalls(replacementCallee->wasmToWasmCallsites());
-            if (OMGForOSREntryCallee* osrEntryCallee = static_cast<BBQCallee*>(m_codeBlock->m_callees[i].get())->osrEntryCallee())
-                repatchCalls(osrEntryCallee->wasmToWasmCallsites());
-        }
+        bbqCallee.repatchCallers(holder, omgCallee.get());
     }
 
-    dataLogLnIf(WasmOMGPlanInternal::verbose, "Finished OMG ", m_functionIndex, " with tier up count at: ", static_cast<BBQCallee*>(m_codeBlock->m_callees[m_functionIndex].get())->tierUpCount()->count());
+    dataLogLnIf(WasmOMGPlanInternal::verbose, "Finished OMG ", m_functionIndex, " with tier up count at: ", m_codeBlock->m_callees[m_functionIndex]->tierUpCount()->count());
     complete(holdLock(m_lock));
 }
 
index cb79b43..e689d56 100644 (file)
@@ -46,7 +46,7 @@ public:
     bool multiThreaded() const override { return false; }
 
     // Note: CompletionTask should not hold a reference to the Plan otherwise there will be a reference cycle.
-    OMGPlan(Context*, Ref<Module>&&, uint32_t functionIndex, MemoryMode, CompletionTask&&);
+    OMGPlan(Context*, Ref<Module>&&, uint32_t functionIndex, CodeBlock&, CompletionTask&&);
 
 private:
     // For some reason friendship doesn't extend to parent classes...
index 1ade8a8..79c640b 100644 (file)
@@ -82,7 +82,7 @@ static void triggerOMGReplacementCompile(TierUpCount& tierUp, OMGCallee* replace
     if (compile) {
         dataLogLnIf(Options::verboseOSR(), "triggerOMGReplacement for ", functionIndex);
         // We need to compile the code.
-        Ref<Plan> plan = adoptRef(*new OMGPlan(instance->context(), Ref<Wasm::Module>(instance->module()), functionIndex, codeBlock.mode(), Plan::dontFinalize()));
+        Ref<Plan> plan = adoptRef(*new OMGPlan(instance->context(), Ref<Wasm::Module>(instance->module()), functionIndex, codeBlock, Plan::dontFinalize()));
         ensureWorklist().enqueue(plan.copyRef());
         if (UNLIKELY(!Options::useConcurrentJIT()))
             plan->waitForCompletion();
@@ -197,9 +197,14 @@ static void doOSREntry(Instance* instance, Probe::Context& context, BBQCallee& c
 #else
 #error Unsupported architecture.
 #endif
-    // 4. Configure argument registers to jump to OSR entry from the caller of this runtime function.
+
+    // 4. Update the callee slot in the call frame to the OSR entry callee.
+    Register* stackPointer = bitwise_cast<Register*>(context.sp());
+    *(stackPointer + CallFrameSlot::callee - 1) = bitwise_cast<Register>(CalleeBits::boxWasm(&osrEntryCallee));
+
+    // 5. Configure argument registers to jump to OSR entry from the caller of this runtime function.
     context.gpr(GPRInfo::argumentGPR0) = bitwise_cast<UCPURegister>(buffer);
-    context.gpr(GPRInfo::argumentGPR1) = bitwise_cast<UCPURegister>(osrEntryCallee.entrypoint().executableAddress<>());
+    context.gpr(GPRInfo::argumentGPR1) = bitwise_cast<UCPURegister>(osrEntryCallee.code().executableAddress<>());
 }
 
 void JIT_OPERATION triggerOSREntryNow(Probe::Context& context)
@@ -365,7 +370,7 @@ void JIT_OPERATION triggerOSREntryNow(Probe::Context& context)
 
     if (startOSREntryCompilation) {
         dataLogLnIf(Options::verboseOSR(), "triggerOMGOSR for ", functionIndex);
-        Ref<Plan> plan = adoptRef(*new OMGForOSREntryPlan(instance->context(), Ref<Wasm::Module>(instance->module()), Ref<Wasm::BBQCallee>(callee), functionIndex, loopIndex, codeBlock.mode(), Plan::dontFinalize()));
+        Ref<Plan> plan = adoptRef(*new OMGForOSREntryPlan(instance->context(), Ref<Wasm::Module>(instance->module()), Ref<Wasm::BBQCallee>(callee), functionIndex, loopIndex, codeBlock, Plan::dontFinalize()));
         ensureWorklist().enqueue(plan.copyRef());
         if (UNLIKELY(!Options::useConcurrentJIT()))
             plan->waitForCompletion();
index 7352202..f13f645 100644 (file)
 
 namespace JSC { namespace Wasm {
 
-std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext& compilationContext, const Signature& signature, Vector<UnlinkedWasmToWasmCall>* unlinkedWasmToWasmCalls, const ModuleInformation& info, MemoryMode mode, unsigned functionIndex)
+void createJSToWasmWrapper(CompilationContext& compilationContext, const Signature& signature, EmbedderEntrypointCallee* self, const ModuleInformation& info, MemoryMode mode)
 {
     CCallHelpers& jit = *compilationContext.embedderEntrypointJIT;
 
-    auto result = makeUnique<InternalFunction>();
     jit.emitFunctionPrologue();
 
     // FIXME Stop using 0 as codeBlocks. https://bugs.webkit.org/show_bug.cgi?id=165321
     jit.store64(CCallHelpers::TrustedImm64(0), CCallHelpers::Address(GPRInfo::callFrameRegister, CallFrameSlot::codeBlock * static_cast<int>(sizeof(Register))));
-    MacroAssembler::DataLabelPtr calleeMoveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), GPRInfo::nonPreservedNonReturnGPR);
-    jit.storePtr(GPRInfo::nonPreservedNonReturnGPR, CCallHelpers::Address(GPRInfo::callFrameRegister, CallFrameSlot::callee * static_cast<int>(sizeof(Register))));
-    CodeLocationDataLabelPtr<WasmEntryPtrTag>* linkedCalleeMove = &result->calleeMoveLocation;
-    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
-        *linkedCalleeMove = linkBuffer.locationOf<WasmEntryPtrTag>(calleeMoveLocation);
-    });
+    jit.storePtr(CCallHelpers::TrustedImmPtr(CalleeBits::boxWasm(self)), CCallHelpers::Address(GPRInfo::callFrameRegister, CallFrameSlot::callee * static_cast<int>(sizeof(Register))));
 
     const PinnedRegisterInfo& pinnedRegs = PinnedRegisterInfo::get();
     RegisterSet toSave = pinnedRegs.toSave(mode);
@@ -69,7 +63,7 @@ std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext& comp
 #endif
 
     RegisterAtOffsetList registersToSpill(toSave, RegisterAtOffsetList::OffsetBaseType::FramePointerBased);
-    result->entrypoint.calleeSaveRegisters = registersToSpill;
+    self->entrypoint().calleeSaveRegisters = registersToSpill;
 
     unsigned totalFrameSize = registersToSpill.size() * sizeof(void*);
     totalFrameSize += WasmCallingConvention::headerSizeInBytes();
@@ -125,9 +119,10 @@ std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext& comp
         }
 
         emitThrowWasmToJSException(jit, GPRInfo::argumentGPR2, argumentsIncludeI64 ? ExceptionType::I64ArgumentType : ExceptionType::I64ReturnType);
-        return result;
+        return;
     }
 
+    compilationContext.embedderMoveAndCall = UnlinkedMoveAndCall { };
     GPRReg wasmContextInstanceGPR = pinnedRegs.wasmContextInstancePointer;
 
     {
@@ -198,6 +193,10 @@ std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext& comp
 
             jsOffset += sizeof(EncodedJSValue);
         }
+
+        GPRReg calleeGPR = wasmCallingConventionAir().prologueScratch(0);
+        compilationContext.embedderMoveAndCall->moveLocation = jit.moveWithPatch(MacroAssembler::TrustedImmPtr(nullptr), calleeGPR);
+        jit.storePtr(calleeGPR, calleeFrame.withOffset(CallFrameSlot::callee * sizeof(Register)));
     }
 
     if (!!info.memory) {
@@ -221,12 +220,7 @@ std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext& comp
         jit.cageConditionally(Gigacage::Primitive, baseMemory, scratchOrSize, scratchOrSize);
     }
 
-    CCallHelpers::Call call = jit.threadSafePatchableNearCall();
-    unsigned functionIndexSpace = functionIndex + info.importFunctionCount();
-    ASSERT(functionIndexSpace < info.functionIndexSpaceSize());
-    jit.addLinkTask([unlinkedWasmToWasmCalls, call, functionIndexSpace] (LinkBuffer& linkBuffer) {
-        unlinkedWasmToWasmCalls->append({ linkBuffer.locationOfNearCall<WasmEntryPtrTag>(call), functionIndexSpace });
-    });
+    compilationContext.embedderMoveAndCall->callLocation = jit.threadSafePatchableNearCall();
 
     for (const RegisterAtOffset& regAtOffset : registersToSpill) {
         GPRReg reg = regAtOffset.reg().gpr();
@@ -266,8 +260,6 @@ std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext& comp
 
     jit.emitFunctionEpilogue();
     jit.ret();
-
-    return result;
 }
 
 } } // namespace JSC::Wasm
index b4b98d1..93aa24d 100644 (file)
@@ -41,7 +41,7 @@ namespace JSC {
 
 namespace Wasm {
 
-std::unique_ptr<InternalFunction> createJSToWasmWrapper(CompilationContext&, const Signature&, Vector<UnlinkedWasmToWasmCall>*, const ModuleInformation&, MemoryMode, uint32_t functionIndex);
+void createJSToWasmWrapper(CompilationContext&, const Signature&, EmbedderEntrypointCallee*, const ModuleInformation&, MemoryMode);
 
 } } // namespace JSC::Wasm
 
index 057aa70..fd3ae18 100644 (file)
@@ -465,6 +465,8 @@ MacroAssemblerCodePtr<JSEntryPtrTag> WebAssemblyFunction::jsCallEntrypointSlow()
         // FIXME: Currently we just do an indirect jump. But we should teach the Module
         // how to repatch us:
         // https://bugs.webkit.org/show_bug.cgi?id=196570
+        jit.loadPtr(boxedCalleeLoadLocation(), scratchGPR);
+        jit.storePtr(scratchGPR, CCallHelpers::Address(CCallHelpers::stackPointerRegister, CallFrameSlot::callee * sizeof(Register) - sizeof(CallerFrameAndPC)));
         jit.loadPtr(entrypointLoadLocation(), scratchGPR);
         jit.call(scratchGPR, WasmEntryPtrTag);
     }
@@ -536,10 +538,10 @@ MacroAssemblerCodePtr<JSEntryPtrTag> WebAssemblyFunction::jsCallEntrypointSlow()
     return m_jsCallEntrypoint.code();
 }
 
-WebAssemblyFunction* WebAssemblyFunction::create(VM& vm, JSGlobalObject* globalObject, Structure* structure, unsigned length, const String& name, JSWebAssemblyInstance* instance, Wasm::Callee& jsEntrypoint, Wasm::WasmToWasmImportableFunction::LoadLocation wasmToWasmEntrypointLoadLocation, Wasm::SignatureIndex signatureIndex)
+WebAssemblyFunction* WebAssemblyFunction::create(VM& vm, JSGlobalObject* globalObject, Structure* structure, unsigned length, const String& name, JSWebAssemblyInstance* instance, Wasm::Callee& jsEntrypoint, Wasm::WasmToWasmImportableFunction::LoadLocation wasmToWasmEntrypointLoadLocation, void** boxedCalleeLoadLocation, Wasm::SignatureIndex signatureIndex)
 {
     NativeExecutable* executable = vm.getHostFunction(callWebAssemblyFunction, NoIntrinsic, callHostFunctionAsConstructor, nullptr, name);
-    WebAssemblyFunction* function = new (NotNull, allocateCell<WebAssemblyFunction>(vm.heap)) WebAssemblyFunction(vm, globalObject, structure, jsEntrypoint, wasmToWasmEntrypointLoadLocation, signatureIndex);
+    WebAssemblyFunction* function = new (NotNull, allocateCell<WebAssemblyFunction>(vm.heap)) WebAssemblyFunction(vm, globalObject, structure, jsEntrypoint, wasmToWasmEntrypointLoadLocation, boxedCalleeLoadLocation, signatureIndex);
     function->finishCreation(vm, executable, length, name, instance);
     ASSERT_WITH_MESSAGE(!function->isLargeAllocation(), "WebAssemblyFunction should be allocated not in large allocation since it is JSCallee.");
     return function;
@@ -551,10 +553,10 @@ Structure* WebAssemblyFunction::createStructure(VM& vm, JSGlobalObject* globalOb
     return Structure::create(vm, globalObject, prototype, TypeInfo(JSFunctionType, StructureFlags), info());
 }
 
-WebAssemblyFunction::WebAssemblyFunction(VM& vm, JSGlobalObject* globalObject, Structure* structure, Wasm::Callee& jsEntrypoint, Wasm::WasmToWasmImportableFunction::LoadLocation wasmToWasmEntrypointLoadLocation, Wasm::SignatureIndex signatureIndex)
+WebAssemblyFunction::WebAssemblyFunction(VM& vm, JSGlobalObject* globalObject, Structure* structure, Wasm::Callee& jsEntrypoint, Wasm::WasmToWasmImportableFunction::LoadLocation wasmToWasmEntrypointLoadLocation, void** boxedCalleeLoadLocation, Wasm::SignatureIndex signatureIndex)
     : Base { vm, globalObject, structure }
-    , m_jsEntrypoint { jsEntrypoint.entrypoint() }
-    , m_importableFunction { signatureIndex, wasmToWasmEntrypointLoadLocation }
+    , m_jsEntrypoint { jsEntrypoint.code() }
+    , m_importableFunction { signatureIndex, wasmToWasmEntrypointLoadLocation, boxedCalleeLoadLocation }
 { }
 
 void WebAssemblyFunction::visitChildren(JSCell* cell, SlotVisitor& visitor)
index 98c3464..7ddf39d 100644 (file)
@@ -62,11 +62,12 @@ public:
 
     DECLARE_EXPORT_INFO;
 
-    JS_EXPORT_PRIVATE static WebAssemblyFunction* create(VM&, JSGlobalObject*, Structure*, unsigned, const String&, JSWebAssemblyInstance*, Wasm::Callee& jsEntrypoint, WasmToWasmImportableFunction::LoadLocation, Wasm::SignatureIndex);
+    JS_EXPORT_PRIVATE static WebAssemblyFunction* create(VM&, JSGlobalObject*, Structure*, unsigned, const String&, JSWebAssemblyInstance*, Wasm::Callee& jsEntrypoint, WasmToWasmImportableFunction::LoadLocation wasmEntrypointLoadLocation, void** boxedCalleeLoadLocation, Wasm::SignatureIndex);
     static Structure* createStructure(VM&, JSGlobalObject*, JSValue);
 
     Wasm::SignatureIndex signatureIndex() const { return m_importableFunction.signatureIndex; }
     WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation() const { return m_importableFunction.entrypointLoadLocation; }
+    void** boxedCalleeLoadLocation() const { return m_importableFunction.boxedCalleeLoadLocation; }
     WasmToWasmImportableFunction importableFunction() const { return m_importableFunction; }
 
     MacroAssemblerCodePtr<WasmEntryPtrTag> jsEntrypoint(ArityCheckMode arity)
@@ -78,6 +79,7 @@ public:
     }
 
     static ptrdiff_t offsetOfEntrypointLoadLocation() { return OBJECT_OFFSETOF(WebAssemblyFunction, m_importableFunction) + WasmToWasmImportableFunction::offsetOfEntrypointLoadLocation(); }
+    static ptrdiff_t offsetOfBoxedCalleeLoadLocation() { return OBJECT_OFFSETOF(WebAssemblyFunction, m_importableFunction) + WasmToWasmImportableFunction::offsetOfBoxedCalleeLoadLocation(); }
 
     MacroAssemblerCodePtr<JSEntryPtrTag> jsCallEntrypoint()
     {
@@ -91,7 +93,7 @@ public:
 
 private:
     static void visitChildren(JSCell*, SlotVisitor&);
-    WebAssemblyFunction(VM&, JSGlobalObject*, Structure*, Wasm::Callee& jsEntrypoint, WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation, Wasm::SignatureIndex);
+    WebAssemblyFunction(VM&, JSGlobalObject*, Structure*, Wasm::Callee& jsEntrypoint, WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation, void** boxedCalleeLoadLocation, Wasm::SignatureIndex);
 
     MacroAssemblerCodePtr<JSEntryPtrTag> jsCallEntrypointSlow();
     ptrdiff_t previousInstanceOffset() const;
index 92ddcf7..7af1a04 100644 (file)
@@ -193,6 +193,7 @@ void WebAssemblyModuleRecord::link(ExecState* exec, JSValue, JSObject* importObj
 
             Wasm::Instance* calleeInstance = nullptr;
             WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation = nullptr;
+            void** boxedCalleeLoadLocation = nullptr;
             JSObject* function = jsCast<JSObject*>(value);
 
             // ii. If v is an Exported Function Exotic Object:
@@ -205,6 +206,7 @@ void WebAssemblyModuleRecord::link(ExecState* exec, JSValue, JSObject* importObj
                     importedSignatureIndex = wasmFunction->signatureIndex();
                     calleeInstance = &wasmFunction->instance()->instance();
                     entrypointLoadLocation = wasmFunction->entrypointLoadLocation();
+                    boxedCalleeLoadLocation = wasmFunction->boxedCalleeLoadLocation();
                 } else {
                     importedSignatureIndex = wasmWrapperFunction->signatureIndex();
                     // b. Let closure be v.[[Closure]].
@@ -223,6 +225,7 @@ void WebAssemblyModuleRecord::link(ExecState* exec, JSValue, JSObject* importObj
             auto* info = m_instance->instance().importFunctionInfo(import.kindIndex);
             info->targetInstance = calleeInstance;
             info->wasmEntrypointLoadLocation = entrypointLoadLocation;
+            info->boxedCalleeLoadLocation = boxedCalleeLoadLocation;
             m_instance->instance().importFunction<WriteBarrier<JSObject>>(import.kindIndex)->set(vm, m_instance.get(), function);
             break;
         }
@@ -347,9 +350,10 @@ void WebAssemblyModuleRecord::link(ExecState* exec, JSValue, JSObject* importObj
             //     c. Return func.
             Wasm::Callee& embedderEntrypointCallee = codeBlock->embedderEntrypointCalleeFromFunctionIndexSpace(index);
             Wasm::WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation = codeBlock->entrypointLoadLocationFromFunctionIndexSpace(index);
+            void** boxedCalleeLoadLocation = codeBlock->boxedCalleeLoadLocationFromFunctionIndexSpace(index);
             Wasm::SignatureIndex signatureIndex = module->signatureIndexFromFunctionIndexSpace(index);
             const Wasm::Signature& signature = Wasm::SignatureInformation::get(signatureIndex);
-            WebAssemblyFunction* function = WebAssemblyFunction::create(vm, globalObject, globalObject->webAssemblyFunctionStructure(), signature.argumentCount(), field, m_instance.get(), embedderEntrypointCallee, entrypointLoadLocation, signatureIndex);
+            WebAssemblyFunction* function = WebAssemblyFunction::create(vm, globalObject, globalObject->webAssemblyFunctionStructure(), signature.argumentCount(), field, m_instance.get(), embedderEntrypointCallee, entrypointLoadLocation, boxedCalleeLoadLocation, signatureIndex);
             wrapper = function;
         }
 
@@ -460,7 +464,8 @@ void WebAssemblyModuleRecord::link(ExecState* exec, JSValue, JSObject* importObj
         } else {
             Wasm::Callee& embedderEntrypointCallee = codeBlock->embedderEntrypointCalleeFromFunctionIndexSpace(startFunctionIndexSpace);
             Wasm::WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation = codeBlock->entrypointLoadLocationFromFunctionIndexSpace(startFunctionIndexSpace);
-            WebAssemblyFunction* function = WebAssemblyFunction::create(vm, globalObject, globalObject->webAssemblyFunctionStructure(), signature.argumentCount(), "start", m_instance.get(), embedderEntrypointCallee, entrypointLoadLocation, signatureIndex);
+            void** boxedCalleeLoadLocation = codeBlock->boxedCalleeLoadLocationFromFunctionIndexSpace(startFunctionIndexSpace);
+            WebAssemblyFunction* function = WebAssemblyFunction::create(vm, globalObject, globalObject->webAssemblyFunctionStructure(), signature.argumentCount(), "start", m_instance.get(), embedderEntrypointCallee, entrypointLoadLocation, boxedCalleeLoadLocation, signatureIndex);
             m_startFunction.set(vm, this, function);
         }
     }
@@ -575,13 +580,14 @@ JSValue WebAssemblyModuleRecord::evaluate(ExecState* exec)
 
             Wasm::Callee& embedderEntrypointCallee = codeBlock->embedderEntrypointCalleeFromFunctionIndexSpace(functionIndex);
             Wasm::WasmToWasmImportableFunction::LoadLocation entrypointLoadLocation = codeBlock->entrypointLoadLocationFromFunctionIndexSpace(functionIndex);
+            void** boxedCalleeLoadLocation = codeBlock->boxedCalleeLoadLocationFromFunctionIndexSpace(functionIndex);
             const Wasm::Signature& signature = Wasm::SignatureInformation::get(signatureIndex);
             // FIXME: Say we export local function "foo" at function index 0.
             // What if we also set it to the table an Element w/ index 0.
             // Does (new Instance(...)).exports.foo === table.get(0)?
             // https://bugs.webkit.org/show_bug.cgi?id=165825
             WebAssemblyFunction* function = WebAssemblyFunction::create(
-                vm, globalObject, globalObject->webAssemblyFunctionStructure(), signature.argumentCount(), String(), m_instance.get(), embedderEntrypointCallee, entrypointLoadLocation, signatureIndex);
+                vm, globalObject, globalObject->webAssemblyFunctionStructure(), signature.argumentCount(), String(), m_instance.get(), embedderEntrypointCallee, entrypointLoadLocation, boxedCalleeLoadLocation, signatureIndex);
 
             m_instance->table(tableIndex)->set(elementIndex, function);
             ++elementIndex;
index cde4f3c..62862b4 100644 (file)
@@ -60,7 +60,9 @@ WebAssemblyWrapperFunction* WebAssemblyWrapperFunction::create(VM& vm, JSGlobalO
     ASSERT_WITH_MESSAGE(!function->inherits<WebAssemblyWrapperFunction>(vm), "We should never double wrap a wrapper function.");
     String name = "";
     NativeExecutable* executable = vm.getHostFunction(callWebAssemblyWrapperFunction, NoIntrinsic, callHostFunctionAsConstructor, nullptr, name);
-    WebAssemblyWrapperFunction* result = new (NotNull, allocateCell<WebAssemblyWrapperFunction>(vm.heap)) WebAssemblyWrapperFunction(vm, globalObject, structure, Wasm::WasmToWasmImportableFunction { signatureIndex, &instance->instance().importFunctionInfo(importIndex)->wasmToEmbedderStub } );
+    auto importFunctionInfo = instance->instance().importFunctionInfo(importIndex);
+    static void* nullBoxedCallee = nullptr;
+    WebAssemblyWrapperFunction* result = new (NotNull, allocateCell<WebAssemblyWrapperFunction>(vm.heap)) WebAssemblyWrapperFunction(vm, globalObject, structure, Wasm::WasmToWasmImportableFunction { signatureIndex, &importFunctionInfo->wasmToEmbedderStub, &nullBoxedCallee });
     const Wasm::Signature& signature = Wasm::SignatureInformation::get(signatureIndex);
     result->finishCreation(vm, executable, signature.argumentCount(), name, function, instance);
     return result;