GC should have a Baker barrier for concurrent copying
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 12 Oct 2015 22:41:01 +0000 (22:41 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 12 Oct 2015 22:41:01 +0000 (22:41 +0000)
https://bugs.webkit.org/show_bug.cgi?id=149852

Reviewed by Geoffrey Garen.

This adds a Baker-style read barrier [1] to copied space accesses. This barrier incurs some
overhead (0%-2% depending on benchmark suite), but what it buys is the ability to make the GC copy
phase concurrent.

The barrier relies on copied space pointers having two "space bits" in the low pointer bits. The
space bits indicate whether the backing store is being copied right now or not, and if it is being
copied, what stage of copying it's in. Two barrier variants are supported:

Read only barrier: if you load a backing store and immediately load from it without doing anything
else, you can just mask off the bits. In the worst case, you'll get the old backing store while
some copying thread is already allocating and populating the new version of the backing store. But
in that case, forwarding to the new backing store will not enable you to load a more up-to-date
value from the backing store. So, just masking the bits is enough. The read-only barrier is only
used in ICs where we know that we are only reading, and opportunistically within the DFG and FTL
thanks to the CopyBarrierOptimizationPhase. We never explicitly emit a read-only barrier in those
compilers; instead the phase will turn a GetButterfly into GetButterflyReadOnly if it proves that a
bunch of requirements are met.

Normal barrier: if the space bits are non-zero, call a slow path. The slow path will either do
nothing (if the copy phase hasn't started yet), or it will copy the backing store and update the
pointer (if the copy phase hasn't gotten around to copying this particular backing store), or it
will wait for the copying thread to finish (if some thread is copying this backing store right
now), or it will do nothing (if by the time we called into the slow path the backing store was
already copied). This is just like Baker's CAR/CDR barrier, but with a lock thrown in to handle
concurrent execution.

This is a 1% slow-down on SunSpider, a 1.5% slow-down on Octane, a 1.5% slow-down on Kraken, and a
0% slow-down on AsmBench. Note that the Octane slow-down is excluding the SplayLatency benchmark.
That benchmark will eventually speed up a lot once we finish doing all of this stuff. Probably, the
JetStream splay-latency will see an even larger speed-up, since our version of the latency tests do
a better job of punishing bad worst-case behavior.

[1] http://dspace.mit.edu/bitstream/handle/1721.1/41976/AI_WP_139.pdf, look for the CAR and CDR
procedures on page 9.

* CMakeLists.txt:
* JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
* JavaScriptCore.xcodeproj/project.pbxproj:
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generate):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGCopyBarrierOptimizationPhase.cpp: Added.
(JSC::DFG::performCopyBarrierOptimization):
* dfg/DFGCopyBarrierOptimizationPhase.h: Added.
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGLICMPhase.cpp:
(JSC::DFG::LICMPhase::run):
* dfg/DFGNodeType.h:
* dfg/DFGOperations.cpp:
* dfg/DFGOperations.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
(JSC::DFG::PredictionPropagationPhase::propagate):
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetIndexedPropertyStorage):
(JSC::DFG::SpeculativeJIT::compileGetTypedArrayByteOffset):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
(JSC::DFG::SpeculativeJIT::temporaryRegisterForPutByVal):
* dfg/DFGSpeculativeJIT.h:
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToLLVM.cpp:
(JSC::FTL::DFG::LowerDFGToLLVM::compileNode):
(JSC::FTL::DFG::LowerDFGToLLVM::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToLLVM::compileGetButterflyReadOnly):
(JSC::FTL::DFG::LowerDFGToLLVM::compileConstantStoragePointer):
(JSC::FTL::DFG::LowerDFGToLLVM::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToLLVM::compileCheckArray):
(JSC::FTL::DFG::LowerDFGToLLVM::compileGetTypedArrayByteOffset):
(JSC::FTL::DFG::LowerDFGToLLVM::compileMultiGetByOffset):
(JSC::FTL::DFG::LowerDFGToLLVM::compileMultiPutByOffset):
(JSC::FTL::DFG::LowerDFGToLLVM::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToLLVM::storageForTransition):
(JSC::FTL::DFG::LowerDFGToLLVM::getById):
(JSC::FTL::DFG::LowerDFGToLLVM::loadButterflyWithBarrier):
(JSC::FTL::DFG::LowerDFGToLLVM::loadVectorWithBarrier):
(JSC::FTL::DFG::LowerDFGToLLVM::copyBarrier):
(JSC::FTL::DFG::LowerDFGToLLVM::loadButterflyReadOnly):
(JSC::FTL::DFG::LowerDFGToLLVM::loadVectorReadOnly):
(JSC::FTL::DFG::LowerDFGToLLVM::removeSpaceBits):
(JSC::FTL::DFG::LowerDFGToLLVM::baseIndex):
* ftl/FTLOperations.cpp:
(JSC::FTL::operationNewObjectWithButterfly):
(JSC::FTL::operationPopulateObjectInOSR):
* ftl/FTLOutput.h:
(JSC::FTL::Output::testNonZero32):
(JSC::FTL::Output::testIsZero64):
(JSC::FTL::Output::testNonZero64):
(JSC::FTL::Output::testIsZeroPtr):
(JSC::FTL::Output::testNonZeroPtr):
(JSC::FTL::Output::select):
(JSC::FTL::Output::extractValue):
* heap/CopyBarrier.h: Copied from Source/JavaScriptCore/heap/CopyWriteBarrier.h.
(JSC::CopyBarrierBase::CopyBarrierBase):
(JSC::CopyBarrierBase::operator!):
(JSC::CopyBarrierBase::operator bool):
(JSC::CopyBarrierBase::getWithoutBarrier):
(JSC::CopyBarrierBase::get):
(JSC::CopyBarrierBase::copyState):
(JSC::CopyBarrierBase::setCopyState):
(JSC::CopyBarrierBase::clear):
(JSC::CopyBarrierBase::set):
(JSC::CopyBarrierBase::setWithoutBarrier):
(JSC::CopyBarrierBase::weakCASWithoutBarrier):
(JSC::CopyBarrier::CopyBarrier):
(JSC::CopyBarrier::getWithoutBarrier):
(JSC::CopyBarrier::get):
(JSC::CopyBarrier::set):
(JSC::CopyBarrier::setWithoutBarrier):
(JSC::CopyBarrier::weakCASWithoutBarrier):
(JSC::CopyWriteBarrier::CopyWriteBarrier): Deleted.
(JSC::CopyWriteBarrier::operator!): Deleted.
(JSC::CopyWriteBarrier::operator bool): Deleted.
(JSC::CopyWriteBarrier::get): Deleted.
(JSC::CopyWriteBarrier::operator*): Deleted.
(JSC::CopyWriteBarrier::operator->): Deleted.
(JSC::CopyWriteBarrier::set): Deleted.
(JSC::CopyWriteBarrier::setWithoutWriteBarrier): Deleted.
(JSC::CopyWriteBarrier::clear): Deleted.
* heap/CopyVisitorInlines.h:
(JSC::CopyVisitor::checkIfShouldCopy):
* heap/CopyWriteBarrier.h: Removed.
* heap/Heap.cpp:
(JSC::Heap::addToRememberedSet):
(JSC::Heap::copyBarrier):
(JSC::Heap::collectAndSweep):
* heap/Heap.h:
(JSC::Heap::writeBarrierBuffer):
* heap/HeapInlines.h:
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::branchStructure):
(JSC::AssemblyHelpers::branchIfNotToSpace):
(JSC::AssemblyHelpers::removeSpaceBits):
(JSC::AssemblyHelpers::addressForByteOffset):
* jit/JIT.cpp:
(JSC::JIT::privateCompileMainPass):
(JSC::JIT::privateCompileSlowCases):
* jit/JITOpcodes.cpp:
(JSC::JIT::emitSlow_op_has_indexed_property):
(JSC::JIT::emit_op_get_direct_pname):
(JSC::JIT::emitSlow_op_get_direct_pname):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_get_direct_pname):
(JSC::JIT::emitSlow_op_get_direct_pname):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitSlow_op_get_by_val):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emitSlow_op_put_by_val):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emitSlow_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitSlow_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter64.asm:
* runtime/DirectArguments.cpp:
(JSC::DirectArguments::visitChildren):
(JSC::DirectArguments::copyBackingStore):
(JSC::DirectArguments::overrideThings):
(JSC::DirectArguments::overrideThingsIfNecessary):
(JSC::DirectArguments::overrideArgument):
(JSC::DirectArguments::copyToArguments):
* runtime/DirectArguments.h:
(JSC::DirectArguments::canAccessIndexQuickly):
(JSC::DirectArguments::canAccessArgumentIndexQuicklyInDFG):
* runtime/JSArray.cpp:
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::fastConcatWith):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithAnyIndexingType):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::JSArrayBufferView):
(JSC::JSArrayBufferView::finishCreation):
(JSC::JSArrayBufferView::finalize):
* runtime/JSArrayBufferView.h:
(JSC::JSArrayBufferView::vector):
(JSC::JSArrayBufferView::length):
* runtime/JSArrayBufferViewInlines.h:
(JSC::JSArrayBufferView::neuter):
(JSC::JSArrayBufferView::byteOffset):
* runtime/JSGenericTypedArrayView.h:
(JSC::JSGenericTypedArrayView::typedVector):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::visitChildren):
(JSC::JSGenericTypedArrayView<Adaptor>::copyBackingStore):
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSMap.h:
(JSC::JSMap::JSMap):
* runtime/JSObject.cpp:
(JSC::JSObject::copyButterfly):
(JSC::JSObject::visitChildren):
(JSC::JSObject::copyBackingStore):
(JSC::JSObject::getOwnPropertySlotByIndex):
(JSC::JSObject::putByIndex):
(JSC::JSObject::enterDictionaryIndexingMode):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToDouble):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToContiguous):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToContiguous):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::setIndexQuicklyToUndecided):
(JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
(JSC::JSObject::deletePropertyByIndex):
(JSC::JSObject::getOwnPropertyNames):
(JSC::JSObject::putIndexedDescriptor):
(JSC::JSObject::defineOwnIndexedProperty):
(JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::reallocateAndShrinkButterfly):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::getOwnPropertyDescriptor):
(JSC::JSObject::getEnumerableLength):
* runtime/JSObject.h:
(JSC::JSObject::getArrayLength):
(JSC::JSObject::getVectorLength):
(JSC::JSObject::canGetIndexQuickly):
(JSC::JSObject::getIndexQuickly):
(JSC::JSObject::tryGetIndexQuickly):
(JSC::JSObject::canSetIndexQuickly):
(JSC::JSObject::canSetIndexQuicklyForPutDirect):
(JSC::JSObject::setIndexQuickly):
(JSC::JSObject::initializeIndex):
(JSC::JSObject::hasSparseMap):
(JSC::JSObject::inSparseIndexingMode):
(JSC::JSObject::inlineStorage):
(JSC::JSObject::butterfly):
(JSC::JSObject::outOfLineStorage):
(JSC::JSObject::locationForOffset):
(JSC::JSObject::ensureInt32):
(JSC::JSObject::ensureDouble):
(JSC::JSObject::ensureContiguous):
(JSC::JSObject::ensureArrayStorage):
(JSC::JSObject::arrayStorage):
(JSC::JSObject::arrayStorageOrNull):
(JSC::JSObject::ensureLength):
(JSC::JSObject::putDirectWithoutTransition):
* runtime/JSSet.h:
(JSC::JSSet::JSSet):
* runtime/MapData.h:
(JSC::JSIterator>::MapDataImpl):
(JSC::JSIterator>::IteratorData::next):
(JSC::JSIterator>::IteratorData::refreshCursor):
* runtime/MapDataInlines.h:
(JSC::JSIterator>::clear):
(JSC::JSIterator>::find):
(JSC::JSIterator>::add):
(JSC::JSIterator>::remove):
(JSC::JSIterator>::replaceAndPackBackingStore):
(JSC::JSIterator>::replaceBackingStore):
(JSC::JSIterator>::ensureSpaceForAppend):
(JSC::JSIterator>::visitChildren):
(JSC::JSIterator>::copyBackingStore):
* runtime/Options.h:

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@190896 268f45cc-cd09-0410-ab3c-d52691b4dbfc

58 files changed:
PerformanceTests/JetStream/Octane2/crypto.js
Source/JavaScriptCore/CMakeLists.txt
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp
Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h
Source/JavaScriptCore/dfg/DFGArgumentsEliminationPhase.cpp
Source/JavaScriptCore/dfg/DFGClobberize.h
Source/JavaScriptCore/dfg/DFGCopyBarrierOptimizationPhase.cpp [new file with mode: 0644]
Source/JavaScriptCore/dfg/DFGCopyBarrierOptimizationPhase.h [moved from Source/JavaScriptCore/heap/CopyWriteBarrier.h with 53% similarity]
Source/JavaScriptCore/dfg/DFGDoesGC.cpp
Source/JavaScriptCore/dfg/DFGFixupPhase.cpp
Source/JavaScriptCore/dfg/DFGHeapLocation.cpp
Source/JavaScriptCore/dfg/DFGHeapLocation.h
Source/JavaScriptCore/dfg/DFGLICMPhase.cpp
Source/JavaScriptCore/dfg/DFGNodeType.h
Source/JavaScriptCore/dfg/DFGOperations.cpp
Source/JavaScriptCore/dfg/DFGOperations.h
Source/JavaScriptCore/dfg/DFGPlan.cpp
Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp
Source/JavaScriptCore/dfg/DFGSafeToExecute.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp
Source/JavaScriptCore/dfg/DFGTypeCheckHoistingPhase.cpp
Source/JavaScriptCore/ftl/FTLCapabilities.cpp
Source/JavaScriptCore/ftl/FTLLowerDFGToLLVM.cpp
Source/JavaScriptCore/ftl/FTLOperations.cpp
Source/JavaScriptCore/ftl/FTLOutput.h
Source/JavaScriptCore/heap/CopyBarrier.h [new file with mode: 0644]
Source/JavaScriptCore/heap/CopyVisitorInlines.h
Source/JavaScriptCore/heap/Heap.cpp
Source/JavaScriptCore/heap/Heap.h
Source/JavaScriptCore/heap/HeapInlines.h
Source/JavaScriptCore/jit/AssemblyHelpers.h
Source/JavaScriptCore/jit/JIT.cpp
Source/JavaScriptCore/jit/JITOpcodes.cpp
Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
Source/JavaScriptCore/jit/JITPropertyAccess.cpp
Source/JavaScriptCore/llint/LowLevelInterpreter.asm
Source/JavaScriptCore/llint/LowLevelInterpreter64.asm
Source/JavaScriptCore/runtime/DirectArguments.cpp
Source/JavaScriptCore/runtime/DirectArguments.h
Source/JavaScriptCore/runtime/JSArray.cpp
Source/JavaScriptCore/runtime/JSArrayBufferView.cpp
Source/JavaScriptCore/runtime/JSArrayBufferView.h
Source/JavaScriptCore/runtime/JSArrayBufferViewInlines.h
Source/JavaScriptCore/runtime/JSGenericTypedArrayView.h
Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h
Source/JavaScriptCore/runtime/JSMap.h
Source/JavaScriptCore/runtime/JSObject.cpp
Source/JavaScriptCore/runtime/JSObject.h
Source/JavaScriptCore/runtime/JSSet.h
Source/JavaScriptCore/runtime/MapData.h
Source/JavaScriptCore/runtime/MapDataInlines.h
Source/JavaScriptCore/runtime/Options.h

index 9932ff7..75adbbd 100644 (file)
@@ -580,11 +580,18 @@ function montRevert(x) {
 }
 
 // x = x/R mod m (HAC 14.32)
+//var outerCounter = 0;
+//var innerCounter = 0;
+var globalThingy = 0;
 function montReduce(x) {
   var x_array = x.array;
   while(x.t <= this.mt2)       // pad x so am has enough room later
     x_array[x.t++] = 0;
+  //outerCounter++;
+  globalThingy = 665;
   for(var i = 0; i < this.m.t; ++i) {
+    //innerCounter++;
+    globalThingy = 666;
     // faster way of calculating u0 = x[i]*mp mod DV
     var j = x_array[i]&0x7fff;
     var u0 = (j*this.mpl+(((j*this.mph+(x_array[i]>>15)*this.mpl)&this.um)<<15))&BI_DM;
index 0b91f71..b804922 100644 (file)
@@ -168,6 +168,7 @@ set(JavaScriptCore_SOURCES
     dfg/DFGCompilationMode.cpp
     dfg/DFGConstantFoldingPhase.cpp
     dfg/DFGConstantHoistingPhase.cpp
+    dfg/DFGCopyBarrierOptimizationPhase.cpp
     dfg/DFGCriticalEdgeBreakingPhase.cpp
     dfg/DFGDCEPhase.cpp
     dfg/DFGDesiredIdentifiers.cpp
index 18ef573..907f8d1 100644 (file)
@@ -1,3 +1,309 @@
+2015-10-08  Filip Pizlo  <fpizlo@apple.com>
+
+        GC should have a Baker barrier for concurrent copying
+        https://bugs.webkit.org/show_bug.cgi?id=149852
+
+        Reviewed by Geoffrey Garen.
+
+        This adds a Baker-style read barrier [1] to copied space accesses. This barrier incurs some
+        overhead (0%-2% depending on benchmark suite), but what it buys is the ability to make the GC copy
+        phase concurrent.
+
+        The barrier relies on copied space pointers having two "space bits" in the low pointer bits. The
+        space bits indicate whether the backing store is being copied right now or not, and if it is being
+        copied, what stage of copying it's in. Two barrier variants are supported:
+
+        Read only barrier: if you load a backing store and immediately load from it without doing anything
+        else, you can just mask off the bits. In the worst case, you'll get the old backing store while
+        some copying thread is already allocating and populating the new version of the backing store. But
+        in that case, forwarding to the new backing store will not enable you to load a more up-to-date
+        value from the backing store. So, just masking the bits is enough. The read-only barrier is only
+        used in ICs where we know that we are only reading, and opportunistically within the DFG and FTL
+        thanks to the CopyBarrierOptimizationPhase. We never explicitly emit a read-only barrier in those
+        compilers; instead the phase will turn a GetButterfly into GetButterflyReadOnly if it proves that a
+        bunch of requirements are met.
+
+        Normal barrier: if the space bits are non-zero, call a slow path. The slow path will either do
+        nothing (if the copy phase hasn't started yet), or it will copy the backing store and update the
+        pointer (if the copy phase hasn't gotten around to copying this particular backing store), or it
+        will wait for the copying thread to finish (if some thread is copying this backing store right
+        now), or it will do nothing (if by the time we called into the slow path the backing store was
+        already copied). This is just like Baker's CAR/CDR barrier, but with a lock thrown in to handle
+        concurrent execution.
+
+        This is a 1% slow-down on SunSpider, a 1.5% slow-down on Octane, a 1.5% slow-down on Kraken, and a
+        0% slow-down on AsmBench. Note that the Octane slow-down is excluding the SplayLatency benchmark.
+        That benchmark will eventually speed up a lot once we finish doing all of this stuff. Probably, the
+        JetStream splay-latency will see an even larger speed-up, since our version of the latency tests do
+        a better job of punishing bad worst-case behavior.
+
+        [1] http://dspace.mit.edu/bitstream/handle/1721.1/41976/AI_WP_139.pdf, look for the CAR and CDR
+        procedures on page 9.
+
+        * CMakeLists.txt:
+        * JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * bytecode/PolymorphicAccess.cpp:
+        (JSC::AccessCase::generate):
+        * dfg/DFGAbstractInterpreterInlines.h:
+        (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
+        * dfg/DFGArgumentsEliminationPhase.cpp:
+        * dfg/DFGClobberize.h:
+        (JSC::DFG::clobberize):
+        * dfg/DFGCopyBarrierOptimizationPhase.cpp: Added.
+        (JSC::DFG::performCopyBarrierOptimization):
+        * dfg/DFGCopyBarrierOptimizationPhase.h: Added.
+        * dfg/DFGDoesGC.cpp:
+        (JSC::DFG::doesGC):
+        * dfg/DFGFixupPhase.cpp:
+        (JSC::DFG::FixupPhase::fixupNode):
+        * dfg/DFGHeapLocation.cpp:
+        (WTF::printInternal):
+        * dfg/DFGHeapLocation.h:
+        * dfg/DFGLICMPhase.cpp:
+        (JSC::DFG::LICMPhase::run):
+        * dfg/DFGNodeType.h:
+        * dfg/DFGOperations.cpp:
+        * dfg/DFGOperations.h:
+        * dfg/DFGPlan.cpp:
+        (JSC::DFG::Plan::compileInThreadImpl):
+        * dfg/DFGPredictionPropagationPhase.cpp:
+        (JSC::DFG::PredictionPropagationPhase::propagate):
+        * dfg/DFGSafeToExecute.h:
+        (JSC::DFG::safeToExecute):
+        * dfg/DFGSpeculativeJIT.cpp:
+        (JSC::DFG::SpeculativeJIT::compileGetIndexedPropertyStorage):
+        (JSC::DFG::SpeculativeJIT::compileGetTypedArrayByteOffset):
+        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
+        (JSC::DFG::SpeculativeJIT::compileGetButterfly):
+        (JSC::DFG::SpeculativeJIT::temporaryRegisterForPutByVal):
+        * dfg/DFGSpeculativeJIT.h:
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGTypeCheckHoistingPhase.cpp:
+        (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
+        (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
+        * ftl/FTLCapabilities.cpp:
+        (JSC::FTL::canCompile):
+        * ftl/FTLLowerDFGToLLVM.cpp:
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileNode):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileGetButterfly):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileGetButterflyReadOnly):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileConstantStoragePointer):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileGetIndexedPropertyStorage):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileCheckArray):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileGetTypedArrayByteOffset):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileMultiGetByOffset):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileMultiPutByOffset):
+        (JSC::FTL::DFG::LowerDFGToLLVM::compileGetDirectPname):
+        (JSC::FTL::DFG::LowerDFGToLLVM::storageForTransition):
+        (JSC::FTL::DFG::LowerDFGToLLVM::getById):
+        (JSC::FTL::DFG::LowerDFGToLLVM::loadButterflyWithBarrier):
+        (JSC::FTL::DFG::LowerDFGToLLVM::loadVectorWithBarrier):
+        (JSC::FTL::DFG::LowerDFGToLLVM::copyBarrier):
+        (JSC::FTL::DFG::LowerDFGToLLVM::loadButterflyReadOnly):
+        (JSC::FTL::DFG::LowerDFGToLLVM::loadVectorReadOnly):
+        (JSC::FTL::DFG::LowerDFGToLLVM::removeSpaceBits):
+        (JSC::FTL::DFG::LowerDFGToLLVM::baseIndex):
+        * ftl/FTLOperations.cpp:
+        (JSC::FTL::operationNewObjectWithButterfly):
+        (JSC::FTL::operationPopulateObjectInOSR):
+        * ftl/FTLOutput.h:
+        (JSC::FTL::Output::testNonZero32):
+        (JSC::FTL::Output::testIsZero64):
+        (JSC::FTL::Output::testNonZero64):
+        (JSC::FTL::Output::testIsZeroPtr):
+        (JSC::FTL::Output::testNonZeroPtr):
+        (JSC::FTL::Output::select):
+        (JSC::FTL::Output::extractValue):
+        * heap/CopyBarrier.h: Copied from Source/JavaScriptCore/heap/CopyWriteBarrier.h.
+        (JSC::CopyBarrierBase::CopyBarrierBase):
+        (JSC::CopyBarrierBase::operator!):
+        (JSC::CopyBarrierBase::operator bool):
+        (JSC::CopyBarrierBase::getWithoutBarrier):
+        (JSC::CopyBarrierBase::get):
+        (JSC::CopyBarrierBase::copyState):
+        (JSC::CopyBarrierBase::setCopyState):
+        (JSC::CopyBarrierBase::clear):
+        (JSC::CopyBarrierBase::set):
+        (JSC::CopyBarrierBase::setWithoutBarrier):
+        (JSC::CopyBarrierBase::weakCASWithoutBarrier):
+        (JSC::CopyBarrier::CopyBarrier):
+        (JSC::CopyBarrier::getWithoutBarrier):
+        (JSC::CopyBarrier::get):
+        (JSC::CopyBarrier::set):
+        (JSC::CopyBarrier::setWithoutBarrier):
+        (JSC::CopyBarrier::weakCASWithoutBarrier):
+        (JSC::CopyWriteBarrier::CopyWriteBarrier): Deleted.
+        (JSC::CopyWriteBarrier::operator!): Deleted.
+        (JSC::CopyWriteBarrier::operator bool): Deleted.
+        (JSC::CopyWriteBarrier::get): Deleted.
+        (JSC::CopyWriteBarrier::operator*): Deleted.
+        (JSC::CopyWriteBarrier::operator->): Deleted.
+        (JSC::CopyWriteBarrier::set): Deleted.
+        (JSC::CopyWriteBarrier::setWithoutWriteBarrier): Deleted.
+        (JSC::CopyWriteBarrier::clear): Deleted.
+        * heap/CopyVisitorInlines.h:
+        (JSC::CopyVisitor::checkIfShouldCopy):
+        * heap/CopyWriteBarrier.h: Removed.
+        * heap/Heap.cpp:
+        (JSC::Heap::addToRememberedSet):
+        (JSC::Heap::copyBarrier):
+        (JSC::Heap::collectAndSweep):
+        * heap/Heap.h:
+        (JSC::Heap::writeBarrierBuffer):
+        * heap/HeapInlines.h:
+        * jit/AssemblyHelpers.h:
+        (JSC::AssemblyHelpers::branchStructure):
+        (JSC::AssemblyHelpers::branchIfNotToSpace):
+        (JSC::AssemblyHelpers::removeSpaceBits):
+        (JSC::AssemblyHelpers::addressForByteOffset):
+        * jit/JIT.cpp:
+        (JSC::JIT::privateCompileMainPass):
+        (JSC::JIT::privateCompileSlowCases):
+        * jit/JITOpcodes.cpp:
+        (JSC::JIT::emitSlow_op_has_indexed_property):
+        (JSC::JIT::emit_op_get_direct_pname):
+        (JSC::JIT::emitSlow_op_get_direct_pname):
+        * jit/JITOpcodes32_64.cpp:
+        (JSC::JIT::emit_op_get_direct_pname):
+        (JSC::JIT::emitSlow_op_get_direct_pname):
+        * jit/JITPropertyAccess.cpp:
+        (JSC::JIT::emitDoubleLoad):
+        (JSC::JIT::emitContiguousLoad):
+        (JSC::JIT::emitArrayStorageLoad):
+        (JSC::JIT::emitSlow_op_get_by_val):
+        (JSC::JIT::emitGenericContiguousPutByVal):
+        (JSC::JIT::emitArrayStoragePutByVal):
+        (JSC::JIT::emitSlow_op_put_by_val):
+        (JSC::JIT::emit_op_get_from_scope):
+        (JSC::JIT::emitSlow_op_get_from_scope):
+        (JSC::JIT::emit_op_put_to_scope):
+        (JSC::JIT::emitSlow_op_put_to_scope):
+        (JSC::JIT::emitIntTypedArrayGetByVal):
+        (JSC::JIT::emitFloatTypedArrayGetByVal):
+        (JSC::JIT::emitIntTypedArrayPutByVal):
+        (JSC::JIT::emitFloatTypedArrayPutByVal):
+        * llint/LowLevelInterpreter.asm:
+        * llint/LowLevelInterpreter64.asm:
+        * runtime/DirectArguments.cpp:
+        (JSC::DirectArguments::visitChildren):
+        (JSC::DirectArguments::copyBackingStore):
+        (JSC::DirectArguments::overrideThings):
+        (JSC::DirectArguments::overrideThingsIfNecessary):
+        (JSC::DirectArguments::overrideArgument):
+        (JSC::DirectArguments::copyToArguments):
+        * runtime/DirectArguments.h:
+        (JSC::DirectArguments::canAccessIndexQuickly):
+        (JSC::DirectArguments::canAccessArgumentIndexQuicklyInDFG):
+        * runtime/JSArray.cpp:
+        (JSC::JSArray::setLength):
+        (JSC::JSArray::pop):
+        (JSC::JSArray::push):
+        (JSC::JSArray::fastSlice):
+        (JSC::JSArray::fastConcatWith):
+        (JSC::JSArray::shiftCountWithArrayStorage):
+        (JSC::JSArray::shiftCountWithAnyIndexingType):
+        (JSC::JSArray::unshiftCountWithAnyIndexingType):
+        (JSC::JSArray::fillArgList):
+        (JSC::JSArray::copyToArguments):
+        * runtime/JSArrayBufferView.cpp:
+        (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
+        (JSC::JSArrayBufferView::JSArrayBufferView):
+        (JSC::JSArrayBufferView::finishCreation):
+        (JSC::JSArrayBufferView::finalize):
+        * runtime/JSArrayBufferView.h:
+        (JSC::JSArrayBufferView::vector):
+        (JSC::JSArrayBufferView::length):
+        * runtime/JSArrayBufferViewInlines.h:
+        (JSC::JSArrayBufferView::neuter):
+        (JSC::JSArrayBufferView::byteOffset):
+        * runtime/JSGenericTypedArrayView.h:
+        (JSC::JSGenericTypedArrayView::typedVector):
+        * runtime/JSGenericTypedArrayViewInlines.h:
+        (JSC::JSGenericTypedArrayView<Adaptor>::visitChildren):
+        (JSC::JSGenericTypedArrayView<Adaptor>::copyBackingStore):
+        (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
+        * runtime/JSMap.h:
+        (JSC::JSMap::JSMap):
+        * runtime/JSObject.cpp:
+        (JSC::JSObject::copyButterfly):
+        (JSC::JSObject::visitChildren):
+        (JSC::JSObject::copyBackingStore):
+        (JSC::JSObject::getOwnPropertySlotByIndex):
+        (JSC::JSObject::putByIndex):
+        (JSC::JSObject::enterDictionaryIndexingMode):
+        (JSC::JSObject::createInitialIndexedStorage):
+        (JSC::JSObject::createArrayStorage):
+        (JSC::JSObject::convertUndecidedToInt32):
+        (JSC::JSObject::convertUndecidedToDouble):
+        (JSC::JSObject::convertUndecidedToContiguous):
+        (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
+        (JSC::JSObject::convertUndecidedToArrayStorage):
+        (JSC::JSObject::convertInt32ToDouble):
+        (JSC::JSObject::convertInt32ToContiguous):
+        (JSC::JSObject::convertInt32ToArrayStorage):
+        (JSC::JSObject::convertDoubleToContiguous):
+        (JSC::JSObject::convertDoubleToArrayStorage):
+        (JSC::JSObject::convertContiguousToArrayStorage):
+        (JSC::JSObject::setIndexQuicklyToUndecided):
+        (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
+        (JSC::JSObject::deletePropertyByIndex):
+        (JSC::JSObject::getOwnPropertyNames):
+        (JSC::JSObject::putIndexedDescriptor):
+        (JSC::JSObject::defineOwnIndexedProperty):
+        (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
+        (JSC::JSObject::putDirectIndexBeyondVectorLength):
+        (JSC::JSObject::getNewVectorLength):
+        (JSC::JSObject::ensureLengthSlow):
+        (JSC::JSObject::reallocateAndShrinkButterfly):
+        (JSC::JSObject::growOutOfLineStorage):
+        (JSC::JSObject::getOwnPropertyDescriptor):
+        (JSC::JSObject::getEnumerableLength):
+        * runtime/JSObject.h:
+        (JSC::JSObject::getArrayLength):
+        (JSC::JSObject::getVectorLength):
+        (JSC::JSObject::canGetIndexQuickly):
+        (JSC::JSObject::getIndexQuickly):
+        (JSC::JSObject::tryGetIndexQuickly):
+        (JSC::JSObject::canSetIndexQuickly):
+        (JSC::JSObject::canSetIndexQuicklyForPutDirect):
+        (JSC::JSObject::setIndexQuickly):
+        (JSC::JSObject::initializeIndex):
+        (JSC::JSObject::hasSparseMap):
+        (JSC::JSObject::inSparseIndexingMode):
+        (JSC::JSObject::inlineStorage):
+        (JSC::JSObject::butterfly):
+        (JSC::JSObject::outOfLineStorage):
+        (JSC::JSObject::locationForOffset):
+        (JSC::JSObject::ensureInt32):
+        (JSC::JSObject::ensureDouble):
+        (JSC::JSObject::ensureContiguous):
+        (JSC::JSObject::ensureArrayStorage):
+        (JSC::JSObject::arrayStorage):
+        (JSC::JSObject::arrayStorageOrNull):
+        (JSC::JSObject::ensureLength):
+        (JSC::JSObject::putDirectWithoutTransition):
+        * runtime/JSSet.h:
+        (JSC::JSSet::JSSet):
+        * runtime/MapData.h:
+        (JSC::JSIterator>::MapDataImpl):
+        (JSC::JSIterator>::IteratorData::next):
+        (JSC::JSIterator>::IteratorData::refreshCursor):
+        * runtime/MapDataInlines.h:
+        (JSC::JSIterator>::clear):
+        (JSC::JSIterator>::find):
+        (JSC::JSIterator>::add):
+        (JSC::JSIterator>::remove):
+        (JSC::JSIterator>::replaceAndPackBackingStore):
+        (JSC::JSIterator>::replaceBackingStore):
+        (JSC::JSIterator>::ensureSpaceForAppend):
+        (JSC::JSIterator>::visitChildren):
+        (JSC::JSIterator>::copyBackingStore):
+        * runtime/Options.h:
+
 2015-10-12  Saam barati  <sbarati@apple.com>
 
         Update JSC features.json
index 0ba180a..6bf28f7 100644 (file)
     <ClCompile Include="..\dfg\DFGCompilationMode.cpp" />
     <ClCompile Include="..\dfg\DFGConstantFoldingPhase.cpp" />
     <ClCompile Include="..\dfg\DFGConstantHoistingPhase.cpp" />
+    <ClCompile Include="..\dfg\DFGCopyBarrierOptimizationPhase.cpp" />
     <ClCompile Include="..\dfg\DFGCPSRethreadingPhase.cpp" />
     <ClCompile Include="..\dfg\DFGCriticalEdgeBreakingPhase.cpp" />
     <ClCompile Include="..\dfg\DFGCSEPhase.cpp" />
     <ClInclude Include="..\dfg\DFGCompilationMode.h" />
     <ClInclude Include="..\dfg\DFGConstantFoldingPhase.h" />
     <ClInclude Include="..\dfg\DFGConstantHoistingPhase.h" />
+    <ClInclude Include="..\dfg\DFGCopyBarrierOptimizationPhase.h" />
     <ClInclude Include="..\dfg\DFGCPSRethreadingPhase.h" />
     <ClInclude Include="..\dfg\DFGCriticalEdgeBreakingPhase.h" />
     <ClInclude Include="..\dfg\DFGCSEPhase.h" />
     <ClInclude Include="..\heap\CopiedBlockInlines.h" />
     <ClInclude Include="..\heap\CopiedSpace.h" />
     <ClInclude Include="..\heap\CopiedSpaceInlines.h" />
+    <ClInclude Include="..\heap\CopyBarrier.h" />
     <ClInclude Include="..\heap\CopyToken.h" />
     <ClInclude Include="..\heap\CopyVisitor.h" />
     <ClInclude Include="..\heap\CopyVisitorInlines.h" />
     <ClInclude Include="..\heap\CopyWorkList.h" />
-    <ClInclude Include="..\heap\CopyWriteBarrier.h" />
     <ClInclude Include="..\heap\DeferGC.h" />
     <ClInclude Include="..\heap\EdenGCActivityCallback.h" />
     <ClInclude Include="..\heap\FullGCActivityCallback.h" />
index ee685c4..8416528 100644 (file)
@@ -86,6 +86,8 @@
                0F04396E1B03DC0B009598B7 /* DFGCombinedLiveness.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */; };
                0F05C3B41683CF9200BAF45B /* DFGArrayifySlowPathGenerator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */; };
                0F0776BF14FF002B00102332 /* JITCompilationEffort.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0776BD14FF002800102332 /* JITCompilationEffort.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F0981F71BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F0981F51BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.cpp */; };
+               0F0981F81BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0981F61BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.h */; };
                0F0A75221B94BFA900110660 /* InferredType.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F0A75201B94BFA900110660 /* InferredType.cpp */; };
                0F0A75231B94BFA900110660 /* InferredType.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0A75211B94BFA900110660 /* InferredType.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F0B839C14BCF46300885B4F /* LLIntThunks.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F0B839714BCF45A00885B4F /* LLIntThunks.cpp */; };
                0F79085619A290B200F6310C /* DFGStructureRegistrationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F79085419A290B200F6310C /* DFGStructureRegistrationPhase.h */; };
                0F7B294B14C3CD2F007C3DB1 /* DFGCapabilities.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FD82E1F14172C2F00179C94 /* DFGCapabilities.h */; };
                0F7B294D14C3CD4C007C3DB1 /* DFGCommon.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FC0977E1469EBC400CF2442 /* DFGCommon.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F7C11AD1BC3862C00C74CDB /* CopyBarrier.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F7C11AC1BC3862C00C74CDB /* CopyBarrier.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F8023EA1613832B00A0BA45 /* ByValInfo.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8023E91613832300A0BA45 /* ByValInfo.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F8335B71639C1E6001443B5 /* ArrayAllocationProfile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F8335B41639C1E3001443B5 /* ArrayAllocationProfile.cpp */; };
                0F8335B81639C1EA001443B5 /* ArrayAllocationProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
                2A4BB7F318A41179008A0FCD /* JSManagedValueInternal.h in Headers */ = {isa = PBXBuildFile; fileRef = 2A4BB7F218A41179008A0FCD /* JSManagedValueInternal.h */; };
                2A4EC90B1860D6C20094F782 /* WriteBarrierBuffer.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2A4EC9091860D6C20094F782 /* WriteBarrierBuffer.cpp */; };
                2A4EC90C1860D6C20094F782 /* WriteBarrierBuffer.h in Headers */ = {isa = PBXBuildFile; fileRef = 2A4EC90A1860D6C20094F782 /* WriteBarrierBuffer.h */; settings = {ATTRIBUTES = (Private, ); }; };
-               2A68295B1875F80500B6C3E2 /* CopyWriteBarrier.h in Headers */ = {isa = PBXBuildFile; fileRef = 2A68295A1875F80500B6C3E2 /* CopyWriteBarrier.h */; settings = {ATTRIBUTES = (Private, ); }; };
                2A6F462617E959CE00C45C98 /* HeapOperation.h in Headers */ = {isa = PBXBuildFile; fileRef = 2A6F462517E959CE00C45C98 /* HeapOperation.h */; settings = {ATTRIBUTES = (Private, ); }; };
                2A7A58EF1808A4C40020BDF7 /* DeferGC.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2A7A58EE1808A4C40020BDF7 /* DeferGC.cpp */; };
                2A83638518D7D0EE0000EBCC /* EdenGCActivityCallback.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 2A83638318D7D0EE0000EBCC /* EdenGCActivityCallback.cpp */; };
                0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCombinedLiveness.h; path = dfg/DFGCombinedLiveness.h; sourceTree = "<group>"; };
                0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGArrayifySlowPathGenerator.h; path = dfg/DFGArrayifySlowPathGenerator.h; sourceTree = "<group>"; };
                0F0776BD14FF002800102332 /* JITCompilationEffort.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITCompilationEffort.h; sourceTree = "<group>"; };
+               0F0981F51BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGCopyBarrierOptimizationPhase.cpp; path = dfg/DFGCopyBarrierOptimizationPhase.cpp; sourceTree = "<group>"; };
+               0F0981F61BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCopyBarrierOptimizationPhase.h; path = dfg/DFGCopyBarrierOptimizationPhase.h; sourceTree = "<group>"; };
                0F0A75201B94BFA900110660 /* InferredType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InferredType.cpp; sourceTree = "<group>"; };
                0F0A75211B94BFA900110660 /* InferredType.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InferredType.h; sourceTree = "<group>"; };
                0F0B839714BCF45A00885B4F /* LLIntThunks.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = LLIntThunks.cpp; path = llint/LLIntThunks.cpp; sourceTree = "<group>"; };
                0F7700911402FF280078EB39 /* SamplingCounter.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SamplingCounter.cpp; sourceTree = "<group>"; };
                0F79085319A290B200F6310C /* DFGStructureRegistrationPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGStructureRegistrationPhase.cpp; path = dfg/DFGStructureRegistrationPhase.cpp; sourceTree = "<group>"; };
                0F79085419A290B200F6310C /* DFGStructureRegistrationPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStructureRegistrationPhase.h; path = dfg/DFGStructureRegistrationPhase.h; sourceTree = "<group>"; };
+               0F7C11AC1BC3862C00C74CDB /* CopyBarrier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CopyBarrier.h; sourceTree = "<group>"; };
                0F8023E91613832300A0BA45 /* ByValInfo.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ByValInfo.h; sourceTree = "<group>"; };
                0F8335B41639C1E3001443B5 /* ArrayAllocationProfile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ArrayAllocationProfile.cpp; sourceTree = "<group>"; };
                0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ArrayAllocationProfile.h; sourceTree = "<group>"; };
                2A4BB7F218A41179008A0FCD /* JSManagedValueInternal.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSManagedValueInternal.h; sourceTree = "<group>"; };
                2A4EC9091860D6C20094F782 /* WriteBarrierBuffer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WriteBarrierBuffer.cpp; sourceTree = "<group>"; };
                2A4EC90A1860D6C20094F782 /* WriteBarrierBuffer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WriteBarrierBuffer.h; sourceTree = "<group>"; };
-               2A68295A1875F80500B6C3E2 /* CopyWriteBarrier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CopyWriteBarrier.h; sourceTree = "<group>"; };
                2A6F462517E959CE00C45C98 /* HeapOperation.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapOperation.h; sourceTree = "<group>"; };
                2A7A58EE1808A4C40020BDF7 /* DeferGC.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = DeferGC.cpp; sourceTree = "<group>"; };
                2A83638318D7D0EE0000EBCC /* EdenGCActivityCallback.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = EdenGCActivityCallback.cpp; sourceTree = "<group>"; };
                                C240305314B404C90079EB64 /* CopiedSpace.cpp */,
                                C2EAA3F8149A830800FCE112 /* CopiedSpace.h */,
                                C2C8D02B14A3C6B200578E65 /* CopiedSpaceInlines.h */,
+                               0F7C11AC1BC3862C00C74CDB /* CopyBarrier.h */,
                                0F5A52CF17ADD717008ECB2D /* CopyToken.h */,
                                C2239D1216262BDD005AC5FD /* CopyVisitor.cpp */,
                                C2239D1316262BDD005AC5FD /* CopyVisitor.h */,
                                C2239D1416262BDD005AC5FD /* CopyVisitorInlines.h */,
                                C218D13F1655CFD50062BB81 /* CopyWorkList.h */,
-                               2A68295A1875F80500B6C3E2 /* CopyWriteBarrier.h */,
                                2A7A58EE1808A4C40020BDF7 /* DeferGC.cpp */,
                                0F136D4B174AD69B0075B354 /* DeferGC.h */,
                                2A83638318D7D0EE0000EBCC /* EdenGCActivityCallback.cpp */,
                                0F3B3A18153E68EF003ED0FF /* DFGConstantFoldingPhase.h */,
                                0FED67B71B26256D0066CE15 /* DFGConstantHoistingPhase.cpp */,
                                0FED67B81B26256D0066CE15 /* DFGConstantHoistingPhase.h */,
+                               0F0981F51BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.cpp */,
+                               0F0981F61BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.h */,
                                0FBE0F6B16C1DB010082C5E8 /* DFGCPSRethreadingPhase.cpp */,
                                0FBE0F6C16C1DB010082C5E8 /* DFGCPSRethreadingPhase.h */,
                                A7D89CE617A0B8CC00773AD8 /* DFGCriticalEdgeBreakingPhase.cpp */,
                                C2239D1816262BDD005AC5FD /* CopyVisitor.h in Headers */,
                                C2239D1916262BDD005AC5FD /* CopyVisitorInlines.h in Headers */,
                                C218D1401655CFD50062BB81 /* CopyWorkList.h in Headers */,
-                               2A68295B1875F80500B6C3E2 /* CopyWriteBarrier.h in Headers */,
                                C4F4B6F41A05C944005CAB76 /* cpp_generator.py in Headers */,
                                C4F4B6F31A05C944005CAB76 /* cpp_generator_templates.py in Headers */,
                                5DE6E5B30E1728EC00180407 /* create_hash_table in Headers */,
                                A7A4AE1017973B4D005612B1 /* JITStubsX86Common.h in Headers */,
                                0F5EF91F16878F7D003E5C25 /* JITThunks.h in Headers */,
                                0FC712E317CD8793008CC93C /* JITToDFGDeferredCompilationCallback.h in Headers */,
+                               0F0981F81BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.h in Headers */,
                                A76F54A313B28AAB00EF2BCE /* JITWriteBarrier.h in Headers */,
                                840480131021A1D9008E7F01 /* JSAPIValueWrapper.h in Headers */,
                                C2CF39C216E15A8100DD69BE /* JSAPIWrapperObject.h in Headers */,
                                868916B0155F286300CB2B9A /* PrivateName.h in Headers */,
                                BC18C4500E16F5CD00B34460 /* Profile.h in Headers */,
                                95CD45770E1C4FDD0085358E /* ProfileGenerator.h in Headers */,
+                               0F7C11AD1BC3862C00C74CDB /* CopyBarrier.h in Headers */,
                                BC18C4510E16F5CD00B34460 /* ProfileNode.h in Headers */,
                                0FF729A5166AD351000F5BA3 /* ProfilerBytecode.h in Headers */,
                                0FF729B9166AD360000F5BA3 /* ProfilerBytecodes.h in Headers */,
                                709FB86B1AE335C60039D069 /* WeakSetPrototype.cpp in Sources */,
                                2A4EC90B1860D6C20094F782 /* WriteBarrierBuffer.cpp in Sources */,
                                0FC8150B14043C0E00CFA603 /* WriteBarrierSupport.cpp in Sources */,
+                               0F0981F71BC5E565004814F8 /* DFGCopyBarrierOptimizationPhase.cpp in Sources */,
                                A7E5AB3A1799E4B200D2833D /* X86Disassembler.cpp in Sources */,
                                863C6D9C1521111A00585E4E /* YarrCanonicalizeUCS2.cpp in Sources */,
                                86704B8412DBA33700A9FE7B /* YarrInterpreter.cpp in Sources */,
index e394c16..23460e8 100644 (file)
@@ -607,6 +607,7 @@ void AccessCase::generate(AccessGenerationState& state)
                 jit.loadPtr(
                     CCallHelpers::Address(baseForAccessGPR, JSObject::butterflyOffset()),
                     loadedValueGPR);
+                jit.removeSpaceBits(loadedValueGPR);
                 storageGPR = loadedValueGPR;
             }
 
@@ -893,6 +894,7 @@ void AccessCase::generate(AccessGenerationState& state)
                     offsetInInlineStorage(m_offset) * sizeof(JSValue)));
         } else {
             jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR);
+            state.failAndIgnore.append(jit.branchIfNotToSpace(scratchGPR));
             jit.storeValue(
                 valueRegs,
                 CCallHelpers::Address(
@@ -956,6 +958,7 @@ void AccessCase::generate(AccessGenerationState& state)
                 ASSERT(newSize > oldSize);
             
                 jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR3);
+                slowPath.append(jit.branchIfNotToSpace(scratchGPR3));
                 jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
                 slowPath.append(
                     jit.branchSubPtr(
@@ -999,8 +1002,10 @@ void AccessCase::generate(AccessGenerationState& state)
                     JSObject::offsetOfInlineStorage() +
                     offsetInInlineStorage(m_offset) * sizeof(JSValue)));
         } else {
-            if (!scratchGPRHasStorage)
+            if (!scratchGPRHasStorage) {
                 jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR);
+                state.failAndIgnore.append(jit.branchIfNotToSpace(scratchGPR));
+            }
             jit.storeValue(
                 valueRegs,
                 CCallHelpers::Address(scratchGPR, offsetInButterfly(m_offset) * sizeof(JSValue)));
@@ -1087,6 +1092,7 @@ void AccessCase::generate(AccessGenerationState& state)
 
     case ArrayLength: {
         jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR);
+        jit.removeSpaceBits(scratchGPR);
         jit.load32(CCallHelpers::Address(scratchGPR, ArrayStorage::lengthOffset()), scratchGPR);
         state.failAndIgnore.append(
             jit.branch32(CCallHelpers::LessThan, scratchGPR, CCallHelpers::TrustedImm32(0)));
index 69183f7..7856753 100644 (file)
@@ -1968,6 +1968,7 @@ bool AbstractInterpreter<AbstractStateType>::executeEffects(unsigned clobberLimi
         }
         break;
     case GetButterfly:
+    case GetButterflyReadOnly:
     case AllocatePropertyStorage:
     case ReallocatePropertyStorage:
         forNode(node).clear(); // The result is not a JS value.
index 99e8542..17343ec 100644 (file)
@@ -192,6 +192,7 @@ private:
                     break;
                     
                 case GetButterfly:
+                case GetButterflyReadOnly:
                     // This barely works. The danger is that the GetButterfly is used by something that
                     // does something escaping to a candidate. Fortunately, the only butterfly-using ops
                     // that we exempt here also use the candidate directly. If there ever was a
index c8040ce..16006e3 100644 (file)
@@ -731,6 +731,13 @@ void clobberize(Graph& graph, Node* node, const ReadFunctor& read, const WriteFu
         read(JSObject_butterfly);
         def(HeapLocation(ButterflyLoc, JSObject_butterfly, node->child1()), LazyNode(node));
         return;
+
+    case GetButterflyReadOnly:
+        // This rule is separate to prevent CSE of GetButterfly with GetButterflyReadOnly. But in reality,
+        // this works because we don't introduce GetButterflyReadOnly until the bitter end of compilation.
+        read(JSObject_butterfly);
+        def(HeapLocation(ButterflyReadOnlyLoc, JSObject_butterfly, node->child1()), LazyNode(node));
+        return;
         
     case Arrayify:
     case ArrayifyToStructure:
diff --git a/Source/JavaScriptCore/dfg/DFGCopyBarrierOptimizationPhase.cpp b/Source/JavaScriptCore/dfg/DFGCopyBarrierOptimizationPhase.cpp
new file mode 100644 (file)
index 0000000..68e08c9
--- /dev/null
@@ -0,0 +1,133 @@
+/*
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "DFGCopyBarrierOptimizationPhase.h"
+
+#if ENABLE(DFG_JIT)
+
+#include "DFGClobberize.h"
+#include "DFGDoesGC.h"
+#include "DFGGraph.h"
+#include "DFGPhase.h"
+#include "JSCInlines.h"
+
+namespace JSC { namespace DFG {
+
+namespace {
+
+bool verbose = false;
+
+class CopyBarrierOptimizationPhase : public Phase {
+public:
+    CopyBarrierOptimizationPhase(Graph& graph)
+        : Phase(graph, "copy barrier optimization")
+    {
+    }
+    
+    bool run()
+    {
+        if (verbose) {
+            dataLog("Starting copy barrier optimization:\n");
+            m_graph.dump();
+        }
+
+        // First convert all GetButterfly nodes into GetButterflyReadOnly.
+        for (BasicBlock* block : m_graph.blocksInNaturalOrder()) {
+            for (Node* node : *block) {
+                if (node->op() == GetButterfly)
+                    node->setOp(GetButterflyReadOnly);
+            }
+        }
+
+        // Anytime we use a GetButterflyReadOnly in a setting that may write to the heap, or if we're in a
+        // new epoch, convert it to a GetButterfly. The epoch gets incremented at basic block boundaries,
+        // anytime we GC, and anytime a barrier on the butterfly may be executed. We traverse the program
+        // in pre-order so that we always see uses after defs. Note that this is a fixpoint because if we
+        // turn a GetButterflyReadOnly into a GetButterfly, then we've introduced a butterfly reallocation.
+        bool changed = true;
+        Epoch currentEpoch = Epoch::first();
+        m_graph.clearEpochs();
+        while (changed) {
+            changed = false;
+            for (BasicBlock* block : m_graph.blocksInPreOrder()) {
+                currentEpoch.bump();
+                for (Node* node : *block) {
+                    bool writesToHeap = writesOverlap(m_graph, node, Heap);
+
+                    bool reallocatesButterfly = false;
+                    if (doesGC(m_graph, node) || writesOverlap(m_graph, node, JSObject_butterfly))
+                        reallocatesButterfly = true;
+                    else {
+                        // This is not an exhaustive list of things that will execute copy barriers. Most
+                        // things that execute copy barriers also do GC or have writes that overlap the
+                        // butterfly heap, and we catch that above.
+                        switch (node->op()) {
+                        case GetButterfly:
+                        case MultiPutByOffset:
+                            reallocatesButterfly = true;
+                            break;
+                        default:
+                            break;
+                        }
+                    }
+                
+                    m_graph.doToChildren(
+                        node,
+                        [&] (Edge edge) {
+                            if (edge->op() != GetButterflyReadOnly)
+                                return;
+                        
+                            if (writesToHeap || currentEpoch != edge->epoch()) {
+                                changed = true;
+                                edge->setOp(GetButterfly);
+                            }
+                        });
+
+                    if (reallocatesButterfly)
+                        currentEpoch.bump();
+
+                    node->setEpoch(currentEpoch);
+                }
+            }
+        }
+        
+        // This phase always thinks that it changes the graph. That's OK, because it's a late phase.
+        return true;
+    }
+};
+
+} // anonymous namespace
+
+bool performCopyBarrierOptimization(Graph& graph)
+{
+    SamplingRegion samplingRegion("DFG Copy Barrier Optimization Phase");
+    return runPhase<CopyBarrierOptimizationPhase>(graph);
+}
+
+} } // namespace JSC::DFG
+
+#endif // ENABLE(DFG_JIT)
+
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
  * THE POSSIBILITY OF SUCH DAMAGE.
  */
 
-#ifndef CopyWriteBarrier_h
-#define CopyWriteBarrier_h
+#ifndef DFGCopyBarrierOptimizationPhase_h
+#define DFGCopyBarrierOptimizationPhase_h
 
-#include "Heap.h"
+#if ENABLE(DFG_JIT)
 
-namespace JSC {
+namespace JSC { namespace DFG {
 
-template <typename T>
-class CopyWriteBarrier {
-public:
-    CopyWriteBarrier()
-        : m_value(0)
-    {
-    }
-    
-    CopyWriteBarrier(VM& vm, const JSCell* owner, T& value)
-    {
-        this->set(vm, owner, &value);
-    }
-    
-    CopyWriteBarrier(VM& vm, const JSCell* owner, T* value)
-    {
-        this->set(vm, owner, value);
-    }
-    
-    bool operator!() const { return !m_value; }
-    
-    explicit operator bool() const { return m_value; }
-    
-    T* get() const
-    {
-        return m_value;
-    }
-    
-    T* operator*() const
-    {
-        return get();
-    }
-    
-    T* operator->() const
-    {
-        return get();
-    }
-    
-    void set(VM& vm, const JSCell* owner, T* value)
-    {
-        this->m_value = value;
-        vm.heap.writeBarrier(owner);
-    }
-    
-    void setWithoutWriteBarrier(T* value)
-    {
-        this->m_value = value;
-    }
-    
-    void clear() { m_value = 0; }
+class Graph;
 
-private:
-    T* m_value;
-};
+// Converts GetButterfly nodes into GetButterflyReadOnly nodes whenever the butterfly is only used for
+// read-only operations.
+bool performCopyBarrierOptimization(Graph&);
 
-} // namespace JSC
+} } // namespace JSC::DFG
+
+#endif // ENABLE(DFG_JIT)
+
+#endif // DFGCopyBarrierOptimizationPhase_h
 
-#endif // CopyWriteBarrier_h
index 7e4bf13..5c57e02 100644 (file)
@@ -98,6 +98,7 @@ bool doesGC(Graph& graph, Node* node)
     case CheckStructure:
     case GetExecutable:
     case GetButterfly:
+    case GetButterflyReadOnly:
     case CheckArray:
     case GetScope:
     case LoadArrowFunctionThis:
index 790ca75..0fe818c 100644 (file)
@@ -1065,7 +1065,8 @@ private:
         case CheckCell:
         case CheckHasInstance:
         case CreateThis:
-        case GetButterfly: {
+        case GetButterfly:
+        case GetButterflyReadOnly: {
             fixEdge<CellUse>(node->child1());
             break;
         }
index 2ca344a..6923656 100644 (file)
@@ -92,6 +92,10 @@ void printInternal(PrintStream& out, LocationKind kind)
         out.print("ButterflyLoc");
         return;
         
+    case ButterflyReadOnlyLoc:
+        out.print("ButterflyReadOnlyLoc");
+        return;
+        
     case CheckHasInstanceLoc:
         out.print("CheckHasInstanceLoc");
         return;
index 3d3a94c..495d300 100644 (file)
@@ -39,6 +39,7 @@ enum LocationKind {
     
     ArrayLengthLoc,
     ButterflyLoc,
+    ButterflyReadOnlyLoc,
     CheckHasInstanceLoc,
     ClosureVariableLoc,
     DirectArgumentsLoc,
index ceff560..c45dca6 100644 (file)
@@ -73,6 +73,11 @@ public:
         
         m_graph.m_dominators.computeIfNecessary(m_graph);
         m_graph.m_naturalLoops.computeIfNecessary(m_graph);
+
+        if (verbose) {
+            dataLog("Graph before LICM:\n");
+            m_graph.dump();
+        }
         
         m_data.resize(m_graph.m_naturalLoops.numLoops());
         
index 87c7074..1349a5d 100644 (file)
@@ -187,6 +187,7 @@ namespace JSC { namespace DFG {
     macro(AllocatePropertyStorage, NodeMustGenerate | NodeResultStorage) \
     macro(ReallocatePropertyStorage, NodeMustGenerate | NodeResultStorage) \
     macro(GetButterfly, NodeResultStorage) \
+    macro(GetButterflyReadOnly, NodeResultStorage) /* A node used to replace GetButterfly at the bitter end of compilation. */\
     macro(CheckArray, NodeMustGenerate) \
     macro(Arrayify, NodeMustGenerate) \
     macro(ArrayifyToStructure, NodeMustGenerate) \
index d6c4740..cf6911b 100644 (file)
@@ -643,7 +643,9 @@ char* JIT_OPERATION operationNewArrayWithSize(ExecState* exec, Structure* arrayS
     if (UNLIKELY(size < 0))
         return bitwise_cast<char*>(exec->vm().throwException(exec, createRangeError(exec, ASCIILiteral("Array size is not a small enough positive integer."))));
 
-    return bitwise_cast<char*>(JSArray::create(*vm, arrayStructure, size));
+    JSArray* result = JSArray::create(*vm, arrayStructure, size);
+    result->butterfly(); // Ensure that the backing store is in to-space.
+    return bitwise_cast<char*>(result);
 }
 
 char* JIT_OPERATION operationNewArrayBuffer(ExecState* exec, Structure* arrayStructure, size_t start, size_t size)
@@ -1176,6 +1178,24 @@ int32_t JIT_OPERATION operationSwitchStringAndGetBranchOffset(ExecState* exec, s
     return exec->codeBlock()->stringSwitchJumpTable(tableIndex).offsetForValue(string->value(exec).impl(), std::numeric_limits<int32_t>::min());
 }
 
+char* JIT_OPERATION operationGetButterfly(ExecState* exec, JSCell* cell)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+
+    dataLog("Ran the barrier.\n");
+
+    return bitwise_cast<char*>(jsCast<JSObject*>(cell)->butterfly());
+}
+
+char* JIT_OPERATION operationGetArrayBufferVector(ExecState* exec, JSCell* cell)
+{
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+
+    return bitwise_cast<char*>(jsCast<JSArrayBufferView*>(cell)->vector());
+}
+
 void JIT_OPERATION operationNotifyWrite(ExecState* exec, WatchpointSet* set)
 {
     VM& vm = exec->vm();
index 7aa12d8..d1bc778 100644 (file)
@@ -130,6 +130,8 @@ JSCell* JIT_OPERATION operationStrCat3(ExecState*, EncodedJSValue, EncodedJSValu
 char* JIT_OPERATION operationFindSwitchImmTargetForDouble(ExecState*, EncodedJSValue, size_t tableIndex);
 char* JIT_OPERATION operationSwitchString(ExecState*, size_t tableIndex, JSString*);
 int32_t JIT_OPERATION operationSwitchStringAndGetBranchOffset(ExecState*, size_t tableIndex, JSString*);
+char* JIT_OPERATION operationGetButterfly(ExecState*, JSCell*);
+char* JIT_OPERATION operationGetArrayBufferVector(ExecState*, JSCell*);
 void JIT_OPERATION operationNotifyWrite(ExecState*, WatchpointSet*);
 void JIT_OPERATION operationThrowStackOverflowForVarargs(ExecState*) WTF_INTERNAL;
 int32_t JIT_OPERATION operationSizeOfVarargs(ExecState*, EncodedJSValue arguments, int32_t firstVarArgOffset);
index 793276b..6b4150b 100644 (file)
@@ -38,6 +38,7 @@
 #include "DFGCleanUpPhase.h"
 #include "DFGConstantFoldingPhase.h"
 #include "DFGConstantHoistingPhase.h"
+#include "DFGCopyBarrierOptimizationPhase.h"
 #include "DFGCriticalEdgeBreakingPhase.h"
 #include "DFGDCEPhase.h"
 #include "DFGFailedFinalizer.h"
@@ -358,6 +359,8 @@ Plan::CompilationPath Plan::compileInThreadImpl(LongLivedState& longLivedState)
         performCleanUp(dfg);
         performCPSRethreading(dfg);
         performDCE(dfg);
+        if (Options::enableCopyBarrierOptimization())
+            performCopyBarrierOptimization(dfg);
         performPhantomInsertion(dfg);
         performStackLayout(dfg);
         performVirtualRegisterAllocation(dfg);
@@ -437,6 +440,8 @@ Plan::CompilationPath Plan::compileInThreadImpl(LongLivedState& longLivedState)
             performMovHintRemoval(dfg);
         performCleanUp(dfg);
         performDCE(dfg); // We rely on this to kill dead code that won't be recognized as dead by LLVM.
+        if (Options::enableCopyBarrierOptimization())
+            performCopyBarrierOptimization(dfg);
         performStackLayout(dfg);
         performLivenessAnalysis(dfg);
         performOSRAvailabilityAnalysis(dfg);
index a39a059..98961a2 100644 (file)
@@ -442,7 +442,8 @@ private:
             break;
         }
             
-        case GetButterfly: 
+        case GetButterfly:
+        case GetButterflyReadOnly:
         case GetIndexedPropertyStorage:
         case AllocatePropertyStorage:
         case ReallocatePropertyStorage: {
index ac288b2..9e3580e 100644 (file)
@@ -191,6 +191,7 @@ bool safeToExecute(AbstractStateType& state, Graph& graph, Node* node)
     case CheckStructure:
     case GetExecutable:
     case GetButterfly:
+    case GetButterflyReadOnly:
     case CheckArray:
     case Arrayify:
     case ArrayifyToStructure:
index 5ad0462..84378b7 100755 (executable)
@@ -4388,6 +4388,11 @@ void SpeculativeJIT::compileGetIndexedPropertyStorage(Node* node)
         m_jit.loadPtr(
             MacroAssembler::Address(baseReg, JSArrayBufferView::offsetOfVector()),
             storageReg);
+
+        addSlowPathGenerator(
+            slowPathCall(
+                m_jit.branchIfNotToSpace(storageReg),
+                this, operationGetArrayBufferVector, storageReg, baseReg));
         break;
     }
     
@@ -4408,9 +4413,11 @@ void SpeculativeJIT::compileGetTypedArrayByteOffset(Node* node)
         MacroAssembler::NotEqual,
         MacroAssembler::Address(baseGPR, JSArrayBufferView::offsetOfMode()),
         TrustedImm32(WastefulTypedArray));
-    
+
     m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSObject::butterflyOffset()), dataGPR);
+    m_jit.removeSpaceBits(dataGPR);
     m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSArrayBufferView::offsetOfVector()), vectorGPR);
+    m_jit.removeSpaceBits(vectorGPR);
     m_jit.loadPtr(MacroAssembler::Address(dataGPR, Butterfly::offsetOfArrayBuffer()), dataGPR);
     m_jit.loadPtr(MacroAssembler::Address(dataGPR, ArrayBuffer::offsetOfData()), dataGPR);
     m_jit.subPtr(dataGPR, vectorGPR);
@@ -4421,7 +4428,7 @@ void SpeculativeJIT::compileGetTypedArrayByteOffset(Node* node)
     m_jit.move(TrustedImmPtr(0), vectorGPR);
     
     done.link(&m_jit);
-    
+
     int32Result(vectorGPR, node);
 }
 
@@ -5489,6 +5496,36 @@ void SpeculativeJIT::compileReallocatePropertyStorage(Node* node)
     storageResult(scratchGPR1, node);
 }
 
+void SpeculativeJIT::compileGetButterfly(Node* node)
+{
+    SpeculateCellOperand base(this, node->child1());
+    GPRTemporary result(this, Reuse, base);
+    
+    GPRReg baseGPR = base.gpr();
+    GPRReg resultGPR = result.gpr();
+    
+    m_jit.loadPtr(JITCompiler::Address(baseGPR, JSObject::butterflyOffset()), resultGPR);
+
+    switch (node->op()) {
+    case GetButterfly:
+        addSlowPathGenerator(
+            slowPathCall(
+                m_jit.branchIfNotToSpace(resultGPR),
+                this, operationGetButterfly, resultGPR, baseGPR));
+        break;
+
+    case GetButterflyReadOnly:
+        m_jit.removeSpaceBits(resultGPR);
+        break;
+
+    default:
+        DFG_CRASH(m_jit.graph(), node, "Bad node type");
+        break;
+    }
+    
+    storageResult(resultGPR, node);
+}
+
 GPRReg SpeculativeJIT::temporaryRegisterForPutByVal(GPRTemporary& temporary, ArrayMode arrayMode)
 {
     if (!putByValWillNeedExtraRegister(arrayMode))
index b420eee..ce3c596 100755 (executable)
@@ -2147,6 +2147,7 @@ public:
     
     void compileAllocatePropertyStorage(Node*);
     void compileReallocatePropertyStorage(Node*);
+    void compileGetButterfly(Node*);
     
 #if USE(JSVALUE32_64)
     template<typename BaseOperandType, typename PropertyOperandType, typename ValueOperandType, typename TagType>
index 8b2a098..a44157b 100644 (file)
@@ -4023,18 +4023,10 @@ void SpeculativeJIT::compile(Node* node)
         compileReallocatePropertyStorage(node);
         break;
         
-    case GetButterfly: {
-        SpeculateCellOperand base(this, node->child1());
-        GPRTemporary result(this, Reuse, base);
-        
-        GPRReg baseGPR = base.gpr();
-        GPRReg resultGPR = result.gpr();
-        
-        m_jit.loadPtr(JITCompiler::Address(baseGPR, JSObject::butterflyOffset()), resultGPR);
-        
-        storageResult(resultGPR, node);
+    case GetButterfly:
+    case GetButterflyReadOnly:
+        compileGetButterfly(node);
         break;
-    }
 
     case GetIndexedPropertyStorage: {
         compileGetIndexedPropertyStorage(node);
@@ -4623,10 +4615,16 @@ void SpeculativeJIT::compile(Node* node)
         GPRReg indexGPR = index.gpr();
         GPRReg enumeratorGPR = enumerator.gpr();
 
+        MacroAssembler::JumpList slowPath;
+
         // Check the structure
         m_jit.load32(MacroAssembler::Address(baseGPR, JSCell::structureIDOffset()), scratchGPR);
-        MacroAssembler::Jump wrongStructure = m_jit.branch32(MacroAssembler::NotEqual, 
-            scratchGPR, MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset()));
+        slowPath.append(
+            m_jit.branch32(
+                MacroAssembler::NotEqual, 
+                scratchGPR,
+                MacroAssembler::Address(
+                    enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset())));
         
         // Compute the offset
         // If index is less than the enumerator's cached inline storage, then it's an inline access
@@ -4648,14 +4646,15 @@ void SpeculativeJIT::compile(Node* node)
         m_jit.signExtend32ToPtr(scratchGPR, scratchGPR);
         // We use resultPayloadGPR as a temporary here. We have to make sure clobber it after getting the 
         // value out of indexGPR and enumeratorGPR because resultPayloadGPR could reuse either of those registers.
-        m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSObject::butterflyOffset()), resultPayloadGPR); 
+        m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSObject::butterflyOffset()), resultPayloadGPR);
+        slowPath.append(m_jit.branchIfNotToSpace(resultPayloadGPR));
         int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
         m_jit.load32(MacroAssembler::BaseIndex(resultPayloadGPR, scratchGPR, MacroAssembler::TimesEight, offsetOfFirstProperty + OBJECT_OFFSETOF(JSValue, u.asBits.tag)), resultTagGPR);
         m_jit.load32(MacroAssembler::BaseIndex(resultPayloadGPR, scratchGPR, MacroAssembler::TimesEight, offsetOfFirstProperty + OBJECT_OFFSETOF(JSValue, u.asBits.payload)), resultPayloadGPR);
 
         done.link(&m_jit);
 
-        addSlowPathGenerator(slowPathCall(wrongStructure, this, operationGetByValCell, resultTagGPR, resultPayloadGPR, baseGPR, propertyGPR));
+        addSlowPathGenerator(slowPathCall(slowPath, this, operationGetByValCell, resultTagGPR, resultPayloadGPR, baseGPR, propertyGPR));
 #endif
 
         jsValueResult(resultTagGPR, resultPayloadGPR, node);
index 845bd8e..cb0806c 100644 (file)
@@ -4037,18 +4037,10 @@ void SpeculativeJIT::compile(Node* node)
         compileReallocatePropertyStorage(node);
         break;
         
-    case GetButterfly: {
-        SpeculateCellOperand base(this, node->child1());
-        GPRTemporary result(this, Reuse, base);
-        
-        GPRReg baseGPR = base.gpr();
-        GPRReg resultGPR = result.gpr();
-        
-        m_jit.loadPtr(JITCompiler::Address(baseGPR, JSObject::butterflyOffset()), resultGPR);
-        
-        storageResult(resultGPR, node);
+    case GetButterfly:
+    case GetButterflyReadOnly:
+        compileGetButterfly(node);
         break;
-    }
 
     case GetIndexedPropertyStorage: {
         compileGetIndexedPropertyStorage(node);
@@ -4644,10 +4636,16 @@ void SpeculativeJIT::compile(Node* node)
         GPRReg scratch1GPR = scratch1.gpr();
         GPRReg scratch2GPR = scratch2.gpr();
 
+        MacroAssembler::JumpList slowPath;
+
         // Check the structure
         m_jit.load32(MacroAssembler::Address(baseGPR, JSCell::structureIDOffset()), scratch1GPR);
-        MacroAssembler::Jump wrongStructure = m_jit.branch32(MacroAssembler::NotEqual, 
-            scratch1GPR, MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset()));
+        slowPath.append(
+            m_jit.branch32(
+                MacroAssembler::NotEqual, 
+                scratch1GPR,
+                MacroAssembler::Address(
+                    enumeratorGPR, JSPropertyNameEnumerator::cachedStructureIDOffset())));
         
         // Compute the offset
         // If index is less than the enumerator's cached inline storage, then it's an inline access
@@ -4661,6 +4659,7 @@ void SpeculativeJIT::compile(Node* node)
         // Otherwise it's out of line
         outOfLineAccess.link(&m_jit);
         m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSObject::butterflyOffset()), scratch2GPR);
+        slowPath.append(m_jit.branchIfNotToSpace(scratch2GPR));
         m_jit.move(indexGPR, scratch1GPR);
         m_jit.sub32(MacroAssembler::Address(enumeratorGPR, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), scratch1GPR);
         m_jit.neg32(scratch1GPR);
@@ -4670,7 +4669,7 @@ void SpeculativeJIT::compile(Node* node)
 
         done.link(&m_jit);
 
-        addSlowPathGenerator(slowPathCall(wrongStructure, this, operationGetByVal, resultGPR, baseGPR, propertyGPR));
+        addSlowPathGenerator(slowPathCall(slowPath, this, operationGetByVal, resultGPR, baseGPR, propertyGPR));
 
         jsValueResult(resultGPR, node);
         break;
index d1e4519..052f4b9 100644 (file)
@@ -247,6 +247,7 @@ private:
                 case AllocatePropertyStorage:
                 case ReallocatePropertyStorage:
                 case GetButterfly:
+                case GetButterflyReadOnly:
                 case GetByVal:
                 case PutByValDirect:
                 case PutByVal:
@@ -325,6 +326,7 @@ private:
                 case PutStructure:
                 case ReallocatePropertyStorage:
                 case GetButterfly:
+                case GetButterflyReadOnly:
                 case GetByVal:
                 case PutByValDirect:
                 case PutByVal:
index 1e96e2b..e2f8ad5 100644 (file)
@@ -68,6 +68,7 @@ inline CapabilityLevel canCompile(Node* node)
     case ArrayifyToStructure:
     case PutStructure:
     case GetButterfly:
+    case GetButterflyReadOnly:
     case NewObject:
     case NewArray:
     case NewArrayBuffer:
index 8deaa49..faf8766 100644 (file)
@@ -566,6 +566,9 @@ private:
         case GetButterfly:
             compileGetButterfly();
             break;
+        case GetButterflyReadOnly:
+            compileGetButterflyReadOnly();
+            break;
         case ConstantStoragePointer:
             compileConstantStoragePointer();
             break;
@@ -2282,7 +2285,12 @@ private:
     
     void compileGetButterfly()
     {
-        setStorage(m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly));
+        setStorage(loadButterflyWithBarrier(lowCell(m_node->child1())));
+    }
+
+    void compileGetButterflyReadOnly()
+    {
+        setStorage(loadButterflyReadOnly(lowCell(m_node->child1())));
     }
     
     void compileConstantStoragePointer()
@@ -2317,7 +2325,7 @@ private:
             return;
         }
         
-        setStorage(m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector));
+        setStorage(loadVectorWithBarrier(cell));
     }
     
     void compileCheckArray()
@@ -2337,27 +2345,25 @@ private:
     {
         LValue basePtr = lowCell(m_node->child1());    
 
-        LBasicBlock simpleCase = FTL_NEW_BLOCK(m_out, ("wasteless typed array"));
-        LBasicBlock wastefulCase = FTL_NEW_BLOCK(m_out, ("wasteful typed array"));
-        LBasicBlock continuation = FTL_NEW_BLOCK(m_out, ("continuation branch"));
+        LBasicBlock simpleCase = FTL_NEW_BLOCK(m_out, ("GetTypedArrayByteOffset wasteless typed array"));
+        LBasicBlock wastefulCase = FTL_NEW_BLOCK(m_out, ("GetTypedArrayByteOffset wasteful typed array"));
+        LBasicBlock continuation = FTL_NEW_BLOCK(m_out, ("GetTypedArrayByteOffset continuation"));
         
         LValue mode = m_out.load32(basePtr, m_heaps.JSArrayBufferView_mode);
         m_out.branch(
             m_out.notEqual(mode, m_out.constInt32(WastefulTypedArray)),
             unsure(simpleCase), unsure(wastefulCase));
 
-        // begin simple case        
         LBasicBlock lastNext = m_out.appendTo(simpleCase, wastefulCase);
 
         ValueFromBlock simpleOut = m_out.anchor(m_out.constIntPtr(0));
 
         m_out.jump(continuation);
 
-        // begin wasteful case
         m_out.appendTo(wastefulCase, continuation);
 
-        LValue vectorPtr = m_out.loadPtr(basePtr, m_heaps.JSArrayBufferView_vector);
-        LValue butterflyPtr = m_out.loadPtr(basePtr, m_heaps.JSObject_butterfly);
+        LValue vectorPtr = loadVectorReadOnly(basePtr);
+        LValue butterflyPtr = loadButterflyReadOnly(basePtr);
         LValue arrayBufferPtr = m_out.loadPtr(butterflyPtr, m_heaps.Butterfly_arrayBuffer);
         LValue dataPtr = m_out.loadPtr(arrayBufferPtr, m_heaps.ArrayBuffer_data);
 
@@ -2366,7 +2372,6 @@ private:
         m_out.jump(continuation);
         m_out.appendTo(continuation, lastNext);
 
-        // output
         setInt32(m_out.castToInt32(m_out.phi(m_out.intPtr, simpleOut, wastefulOut)));
     }
     
@@ -4050,7 +4055,7 @@ private:
                 else
                     propertyBase = weakPointer(method.prototype()->value().asCell());
                 if (!isInlineOffset(method.offset()))
-                    propertyBase = m_out.loadPtr(propertyBase, m_heaps.JSObject_butterfly);
+                    propertyBase = loadButterflyReadOnly(propertyBase);
                 result = loadProperty(
                     propertyBase, data.identifierNumber, method.offset());
                 break;
@@ -4118,7 +4123,7 @@ private:
                 if (isInlineOffset(variant.offset()))
                     storage = base;
                 else
-                    storage = m_out.loadPtr(base, m_heaps.JSObject_butterfly);
+                    storage = loadButterflyWithBarrier(base);
             } else {
                 m_graph.m_plan.transitions.addLazily(
                     codeBlock(), m_node->origin.semantic.codeOriginOwner(),
@@ -5352,7 +5357,7 @@ private:
         m_out.jump(continuation);
 
         m_out.appendTo(outOfLineLoad, slowCase);
-        LValue storage = m_out.loadPtr(base, m_heaps.JSObject_butterfly);
+        LValue storage = loadButterflyReadOnly(base);
         LValue realIndex = m_out.signExt(
             m_out.neg(m_out.sub(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedInlineCapacity))), 
             m_out.int64);
@@ -5999,14 +6004,14 @@ private:
             return object;
         
         if (previousStructure->outOfLineCapacity() == nextStructure->outOfLineCapacity())
-            return m_out.loadPtr(object, m_heaps.JSObject_butterfly);
+            return loadButterflyWithBarrier(object);
         
         LValue result;
         if (!previousStructure->outOfLineCapacity())
             result = allocatePropertyStorage(object, previousStructure);
         else {
             result = reallocatePropertyStorage(
-                object, m_out.loadPtr(object, m_heaps.JSObject_butterfly),
+                object, loadButterflyWithBarrier(object),
                 previousStructure, nextStructure);
         }
         
@@ -6120,6 +6125,59 @@ private:
         
         return call;
     }
+
+    LValue loadButterflyWithBarrier(LValue object)
+    {
+        return copyBarrier(
+            object, m_out.loadPtr(object, m_heaps.JSObject_butterfly), operationGetButterfly);
+    }
+    
+    LValue loadVectorWithBarrier(LValue object)
+    {
+        return copyBarrier(
+            object, m_out.loadPtr(object, m_heaps.JSArrayBufferView_vector),
+            operationGetArrayBufferVector);
+    }
+    
+    LValue copyBarrier(LValue object, LValue pointer, P_JITOperation_EC slowPathFunction)
+    {
+        LBasicBlock slowPath = FTL_NEW_BLOCK(m_out, ("loadButterflyWithBarrier slow path"));
+        LBasicBlock continuation = FTL_NEW_BLOCK(m_out, ("loadButterflyWithBarrier continuation"));
+
+        ValueFromBlock fastResult = m_out.anchor(pointer);
+        m_out.branch(
+            m_out.testIsZeroPtr(pointer, m_out.constIntPtr(CopyBarrierBase::spaceBits)),
+            usually(continuation), rarely(slowPath));
+
+        LBasicBlock lastNext = m_out.appendTo(slowPath, continuation);
+
+        LValue call = lazySlowPath(
+            [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
+                return createLazyCallGenerator(
+                    slowPathFunction, locations[0].directGPR(), locations[1].directGPR());
+            }, object);
+        ValueFromBlock slowResult = m_out.anchor(call);
+        m_out.jump(continuation);
+
+        m_out.appendTo(continuation, lastNext);
+        return m_out.phi(m_out.intPtr, fastResult, slowResult);
+    }
+
+    LValue loadButterflyReadOnly(LValue object)
+    {
+        return removeSpaceBits(m_out.loadPtr(object, m_heaps.JSObject_butterfly));
+    }
+
+    LValue loadVectorReadOnly(LValue object)
+    {
+        return removeSpaceBits(m_out.loadPtr(object, m_heaps.JSArrayBufferView_vector));
+    }
+
+    LValue removeSpaceBits(LValue storage)
+    {
+        return m_out.bitAnd(
+            storage, m_out.constIntPtr(~static_cast<intptr_t>(CopyBarrierBase::spaceBits)));
+    }
     
     TypedPointer baseIndex(IndexedAbstractHeap& heap, LValue storage, LValue index, Edge edge, ptrdiff_t offset = 0)
     {
index 80eab6f..c3672e8 100644 (file)
@@ -48,7 +48,9 @@ extern "C" JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState* exec
     Butterfly* butterfly = Butterfly::create(
         vm, nullptr, 0, structure->outOfLineCapacity(), false, IndexingHeader(), 0);
     
-    return JSFinalObject::create(exec, structure, butterfly);
+    JSObject* result = JSFinalObject::create(exec, structure, butterfly);
+    result->butterfly(); // Ensure that the butterfly is in to-space.
+    return result;
 }
 
 extern "C" void JIT_OPERATION operationPopulateObjectInOSR(
index e2d973d..3c23f49 100644 (file)
@@ -362,6 +362,8 @@ public:
     LValue testNonZero32(LValue value, LValue mask) { return notZero32(bitAnd(value, mask)); }
     LValue testIsZero64(LValue value, LValue mask) { return isZero64(bitAnd(value, mask)); }
     LValue testNonZero64(LValue value, LValue mask) { return notZero64(bitAnd(value, mask)); }
+    LValue testIsZeroPtr(LValue value, LValue mask) { return isNull(bitAnd(value, mask)); }
+    LValue testNonZeroPtr(LValue value, LValue mask) { return notNull(bitAnd(value, mask)); }
     
     LValue select(LValue value, LValue taken, LValue notTaken) { return buildSelect(m_builder, value, taken, notTaken); }
     LValue extractValue(LValue aggVal, unsigned index) { return buildExtractValue(m_builder, aggVal, index); }
diff --git a/Source/JavaScriptCore/heap/CopyBarrier.h b/Source/JavaScriptCore/heap/CopyBarrier.h
new file mode 100644 (file)
index 0000000..ba957be
--- /dev/null
@@ -0,0 +1,181 @@
+/*
+ * Copyright (C) 2014 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef CopyBarrier_h
+#define CopyBarrier_h
+
+#include "Heap.h"
+
+namespace JSC {
+
+enum class CopyState {
+    // The backing store is not planned to get copied in this epoch. If you keep a pointer to the backing
+    // store on the stack, it will not get copied. If you don't keep it on the stack, it may get copied
+    // starting at the next handshake (that is, it may transition from ToSpace to CopyPlanned, but
+    // CopyPlanned means ToSpace prior to the handshake that starts the copy phase).
+    ToSpace,
+
+    // The marking phase has selected this backing store to be copied. If we are not yet in the copying
+    // phase, this backing store is still in to-space. All that is needed in such a case is to mask off the
+    // low bits. If we are in the copying phase, this means that the object points to from-space. The
+    // barrier should first copy the object - or wait for copying to finish - before using the object.
+    CopyPlanned,
+
+    // The object is being copied right now. Anyone wanting to use the object must wait for the object to
+    // finish being copied. Notifications about copying use the ParkingLot combined with these bits. If the
+    // state is CopyingAndWaiting, then when the copying finishes, whatever thread was doing it will
+    // unparkAll() on the address of the CopyBarrierBase. So, to wait for copying to finish, CAS this to
+    // CopyingAndWaiting and then parkConditionally on the barrier address.
+    Copying,
+
+    // The object is being copied right now, and there are one or more threads parked. Those threads want
+    // to be unparked when copying is done. So, whichever thread does the copying needs to call unparkAll()
+    // on the barrier address after copying is done.
+    CopyingAndWaiting
+};
+
+class CopyBarrierBase {
+public:
+    static const unsigned spaceBits = 3;
+
+    CopyBarrierBase()
+        : m_value(nullptr)
+    {
+    }
+    
+    bool operator!() const { return !m_value; }
+    
+    explicit operator bool() const { return m_value; }
+
+    void* getWithoutBarrier() const
+    {
+        return m_value;
+    }
+
+    // When we are in the concurrent copying phase, this method may lock the barrier object (i.e. the field
+    // pointing to copied space) and call directly into the owning object's copyBackingStore() method.
+    void* get(const JSCell* owner) const
+    {
+        void* result = m_value;
+        if (UNLIKELY(bitwise_cast<uintptr_t>(result) & spaceBits))
+            return Heap::copyBarrier(owner, m_value);
+        return result;
+    }
+
+    CopyState copyState() const
+    {
+        return static_cast<CopyState>(bitwise_cast<uintptr_t>(m_value) & spaceBits);
+    }
+
+    // This only works when you know that there is nobody else concurrently messing with this CopyBarrier.
+    // That's hard to guarantee, though there are a few unusual places where this ends up being safe.
+    // Usually you want to use CopyBarrier::weakCAS().
+    void setCopyState(CopyState copyState)
+    {
+        WTF::storeStoreFence();
+        uintptr_t value = bitwise_cast<uintptr_t>(m_value);
+        value &= ~static_cast<uintptr_t>(spaceBits);
+        value |= static_cast<uintptr_t>(copyState);
+        m_value = bitwise_cast<void*>(value);
+    }
+
+    void clear() { m_value = nullptr; }
+
+protected:
+    CopyBarrierBase(VM& vm, const JSCell* owner, void* value)
+    {
+        this->set(vm, owner, value);
+    }
+    
+    void set(VM& vm, const JSCell* owner, void* value)
+    {
+        this->m_value = value;
+        vm.heap.writeBarrier(owner);
+    }
+    
+    void setWithoutBarrier(void* value)
+    {
+        this->m_value = value;
+    }
+
+    bool weakCASWithoutBarrier(
+        void* oldPointer, CopyState oldCopyState, void* newPointer, CopyState newCopyState)
+    {
+        uintptr_t oldValue = bitwise_cast<uintptr_t>(oldPointer) | static_cast<uintptr_t>(oldCopyState);
+        uintptr_t newValue = bitwise_cast<uintptr_t>(newPointer) | static_cast<uintptr_t>(newCopyState);
+        return WTF::weakCompareAndSwap(
+            &m_value, bitwise_cast<void*>(oldValue), bitwise_cast<void*>(newValue));
+    }
+
+private:
+    mutable void* m_value;
+};
+
+template <typename T>
+class CopyBarrier : public CopyBarrierBase {
+public:
+    CopyBarrier()
+    {
+    }
+    
+    CopyBarrier(VM& vm, const JSCell* owner, T& value)
+        : CopyBarrierBase(vm, owner, &value)
+    {
+    }
+    
+    CopyBarrier(VM& vm, const JSCell* owner, T* value)
+        : CopyBarrierBase(vm, owner, value)
+    {
+    }
+
+    T* getWithoutBarrier() const
+    {
+        return bitwise_cast<T*>(CopyBarrierBase::getWithoutBarrier());
+    }
+    
+    T* get(const JSCell* owner) const
+    {
+        return bitwise_cast<T*>(CopyBarrierBase::get(owner));
+    }
+    
+    void set(VM& vm, const JSCell* owner, T* value)
+    {
+        CopyBarrierBase::set(vm, owner, value);
+    }
+    
+    void setWithoutBarrier(T* value)
+    {
+        CopyBarrierBase::setWithoutBarrier(value);
+    }
+
+    bool weakCASWithoutBarrier(T* oldPointer, CopyState oldCopyState, T* newPointer, CopyState newCopyState)
+    {
+        return CopyBarrierBase::weakCASWithoutBarrier(oldPointer, oldCopyState, newPointer, newCopyState);
+    }
+};
+
+} // namespace JSC
+
+#endif // CopyBarrier_h
index 2cf1156..70cc672 100644 (file)
@@ -33,6 +33,8 @@ namespace JSC {
 
 inline bool CopyVisitor::checkIfShouldCopy(void* oldPtr)
 {
+    if (!oldPtr)
+        return false;
     CopiedBlock* block = CopiedSpace::blockFor(oldPtr);
     if (block->isOversize() || block->isPinned())
         return false;
index 6ba1576..777249a 100644 (file)
@@ -1000,6 +1000,12 @@ void Heap::addToRememberedSet(const JSCell* cell)
     m_slotVisitor.appendToMarkStack(const_cast<JSCell*>(cell));
 }
 
+void* Heap::copyBarrier(const JSCell*, void*& pointer)
+{
+    // Do nothing for now.
+    return pointer;
+}
+
 void Heap::collectAndSweep(HeapOperation collectionType)
 {
     if (!m_isSafeToCollect)
index 971faa5..ebb7a69 100644 (file)
@@ -108,6 +108,8 @@ public:
     void writeBarrier(const JSCell*, JSValue);
     void writeBarrier(const JSCell*, JSCell*);
 
+    JS_EXPORT_PRIVATE static void* copyBarrier(const JSCell* owner, void*& copiedSpacePointer);
+
     WriteBarrierBuffer& writeBarrierBuffer() { return m_writeBarrierBuffer; }
     void flushWriteBarrierBuffer(JSCell*);
 
index df3a199..ed55514 100644 (file)
@@ -26,6 +26,7 @@
 #ifndef HeapInlines_h
 #define HeapInlines_h
 
+#include "CopyBarrier.h"
 #include "Heap.h"
 #include "JSCell.h"
 #include "Structure.h"
index 9875c85..0e10b22 100644 (file)
@@ -29,6 +29,7 @@
 #if ENABLE(JIT)
 
 #include "CodeBlock.h"
+#include "CopyBarrier.h"
 #include "FPRInfo.h"
 #include "GPRInfo.h"
 #include "InlineCallFrame.h"
@@ -754,6 +755,16 @@ public:
         return branchPtr(condition, leftHandSide, TrustedImmPtr(structure));
 #endif
     }
+
+    Jump branchIfNotToSpace(GPRReg storageGPR)
+    {
+        return branchTest32(NonZero, storageGPR, TrustedImm32(CopyBarrierBase::spaceBits));
+    }
+
+    void removeSpaceBits(GPRReg storageGPR)
+    {
+        andPtr(TrustedImmPtr(~static_cast<uintptr_t>(CopyBarrierBase::spaceBits)), storageGPR);
+    }
     
     static Address addressForByteOffset(ptrdiff_t byteOffset)
     {
index e8fcd64..b8f2162 100644 (file)
@@ -180,6 +180,8 @@ void JIT::privateCompileMainPass()
         
         if (Options::eagerlyUpdateTopCallFrame())
             updateTopCallFrame();
+
+        unsigned bytecodeOffset = m_bytecodeOffset;
         
         switch (opcodeID) {
         DEFINE_SLOW_OP(del_by_val)
@@ -314,6 +316,9 @@ void JIT::privateCompileMainPass()
         default:
             RELEASE_ASSERT_NOT_REACHED();
         }
+
+        if (false)
+            dataLog("At ", bytecodeOffset, ": ", m_slowCases.size(), "\n");
     }
 
     RELEASE_ASSERT(m_callLinkInfoIndex == m_callCompilationInfo.size());
@@ -433,6 +438,9 @@ void JIT::privateCompileSlowCases()
             RELEASE_ASSERT_NOT_REACHED();
         }
 
+        if (false)
+            dataLog("At ", firstTo, " slow: ", iter - m_slowCases.begin(), "\n");
+
         RELEASE_ASSERT_WITH_MESSAGE(iter == m_slowCases.end() || firstTo != iter->to, "Not enough jumps linked in slow case codegen.");
         RELEASE_ASSERT_WITH_MESSAGE(firstTo == (iter - 1)->to, "Too many jumps linked in slow case codegen.");
         
index 840e262..66ebd51 100755 (executable)
@@ -1153,6 +1153,7 @@ void JIT::emitSlow_op_has_indexed_property(Instruction* currentInstruction, Vect
     
     linkSlowCaseIfNotJSCell(iter, base); // base cell check
     linkSlowCase(iter); // base array check
+    linkSlowCase(iter); // read barrier
     linkSlowCase(iter); // vector length check
     linkSlowCase(iter); // empty value
     
@@ -1196,6 +1197,7 @@ void JIT::emit_op_get_direct_pname(Instruction* currentInstruction)
     // Otherwise it's out of line
     outOfLineAccess.link(this);
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT0);
+    addSlowCase(branchIfNotToSpace(regT0));
     sub32(Address(regT2, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), regT1);
     neg32(regT1);
     signExtend32ToPtr(regT1, regT1);
@@ -1212,6 +1214,7 @@ void JIT::emitSlow_op_get_direct_pname(Instruction* currentInstruction, Vector<S
     int base = currentInstruction[2].u.operand;
     linkSlowCaseIfNotJSCell(iter, base);
     linkSlowCase(iter);
+    linkSlowCase(iter);
 
     JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_get_direct_pname);
     slowPathCall.call();
index 6ce41fd..7aea8da 100644 (file)
@@ -1174,6 +1174,7 @@ void JIT::emit_op_get_direct_pname(Instruction* currentInstruction)
     // Otherwise it's out of line
     outOfLineAccess.link(this);
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT0);
+    addSlowCase(branchIfNotToSpace(regT0));
     sub32(Address(regT1, JSPropertyNameEnumerator::cachedInlineCapacityOffset()), regT2);
     neg32(regT2);
     int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
@@ -1190,6 +1191,7 @@ void JIT::emitSlow_op_get_direct_pname(Instruction* currentInstruction, Vector<S
     int base = currentInstruction[2].u.operand;
     linkSlowCaseIfNotJSCell(iter, base);
     linkSlowCase(iter);
+    linkSlowCase(iter);
 
     JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_get_direct_pname);
     slowPathCall.call();
index e3c4998..2892c6d 100644 (file)
@@ -165,6 +165,7 @@ JIT::JumpList JIT::emitDoubleLoad(Instruction*, PatchableJump& badType)
     
     badType = patchableBranch32(NotEqual, regT2, TrustedImm32(DoubleShape));
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT2);
+    slowCases.append(branchIfNotToSpace(regT2));
     slowCases.append(branch32(AboveOrEqual, regT1, Address(regT2, Butterfly::offsetOfPublicLength())));
     loadDouble(BaseIndex(regT2, regT1, TimesEight), fpRegT0);
     slowCases.append(branchDouble(DoubleNotEqualOrUnordered, fpRegT0, fpRegT0));
@@ -178,6 +179,7 @@ JIT::JumpList JIT::emitContiguousLoad(Instruction*, PatchableJump& badType, Inde
     
     badType = patchableBranch32(NotEqual, regT2, TrustedImm32(expectedShape));
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT2);
+    slowCases.append(branchIfNotToSpace(regT2));
     slowCases.append(branch32(AboveOrEqual, regT1, Address(regT2, Butterfly::offsetOfPublicLength())));
     load64(BaseIndex(regT2, regT1, TimesEight), regT0);
     slowCases.append(branchTest64(Zero, regT0));
@@ -193,6 +195,7 @@ JIT::JumpList JIT::emitArrayStorageLoad(Instruction*, PatchableJump& badType)
     badType = patchableBranch32(Above, regT3, TrustedImm32(SlowPutArrayStorageShape - ArrayStorageShape));
 
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT2);
+    slowCases.append(branchIfNotToSpace(regT2));
     slowCases.append(branch32(AboveOrEqual, regT1, Address(regT2, ArrayStorage::vectorLengthOffset())));
 
     load64(BaseIndex(regT2, regT1, TimesEight, ArrayStorage::vectorOffset()), regT0);
@@ -251,6 +254,7 @@ void JIT::emitSlow_op_get_by_val(Instruction* currentInstruction, Vector<SlowCas
     notString.link(this);
     nonCell.link(this);
     
+    linkSlowCase(iter); // read barrier
     linkSlowCase(iter); // vector length check
     linkSlowCase(iter); // empty value
     
@@ -323,6 +327,7 @@ JIT::JumpList JIT::emitGenericContiguousPutByVal(Instruction* currentInstruction
     badType = patchableBranch32(NotEqual, regT2, TrustedImm32(indexingShape));
     
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT2);
+    slowCases.append(branchIfNotToSpace(regT2));
     Jump outOfBounds = branch32(AboveOrEqual, regT1, Address(regT2, Butterfly::offsetOfPublicLength()));
 
     Label storeResult = label();
@@ -378,6 +383,7 @@ JIT::JumpList JIT::emitArrayStoragePutByVal(Instruction* currentInstruction, Pat
     
     badType = patchableBranch32(NotEqual, regT2, TrustedImm32(ArrayStorageShape));
     loadPtr(Address(regT0, JSObject::butterflyOffset()), regT2);
+    slowCases.append(branchIfNotToSpace(regT2));
     slowCases.append(branch32(AboveOrEqual, regT1, Address(regT2, ArrayStorage::vectorLengthOffset())));
 
     Jump empty = branchTest64(Zero, BaseIndex(regT2, regT1, TimesEight, OBJECT_OFFSETOF(ArrayStorage, m_vector[0])));
@@ -448,8 +454,9 @@ void JIT::emitSlow_op_put_by_val(Instruction* currentInstruction, Vector<SlowCas
     linkSlowCase(iter); // property int32 check
     linkSlowCase(iter); // base not array check
     
+    linkSlowCase(iter); // read barrier
     linkSlowCase(iter); // out of bounds
-
+    
     JITArrayMode mode = chooseArrayMode(profile);
     switch (mode) {
     case JITInt32:
@@ -780,6 +787,7 @@ void JIT::emit_op_get_from_scope(Instruction* currentInstruction)
                 isOutOfLine.link(this);
             }
             loadPtr(Address(base, JSObject::butterflyOffset()), scratch);
+            addSlowCase(branchIfNotToSpace(scratch));
             neg32(offset);
             signExtend32ToPtr(offset, offset);
             load64(BaseIndex(scratch, offset, TimesEight, (firstOutOfLineOffset - 2) * sizeof(EncodedJSValue)), result);
@@ -859,8 +867,10 @@ void JIT::emitSlow_op_get_from_scope(Instruction* currentInstruction, Vector<Slo
     if (resolveType == GlobalVar || resolveType == ClosureVar)
         return;
 
-    if (resolveType == GlobalProperty || resolveType == GlobalPropertyWithVarInjectionChecks)
-        linkSlowCase(iter);
+    if (resolveType == GlobalProperty || resolveType == GlobalPropertyWithVarInjectionChecks) {
+        linkSlowCase(iter); // bad structure
+        linkSlowCase(iter); // read barrier
+    }
 
     if (resolveType == GlobalLexicalVarWithVarInjectionChecks) // Var injections check.
         linkSlowCase(iter);
@@ -869,6 +879,7 @@ void JIT::emitSlow_op_get_from_scope(Instruction* currentInstruction, Vector<Slo
         // GlobalProperty/GlobalPropertyWithVarInjectionChecks
         linkSlowCase(iter); // emitLoadWithStructureCheck
         linkSlowCase(iter); // emitLoadWithStructureCheck
+        linkSlowCase(iter); // read barrier
         // GlobalLexicalVar
         linkSlowCase(iter); // TDZ check.
         // GlobalLexicalVarWithVarInjectionChecks.
@@ -922,6 +933,7 @@ void JIT::emit_op_put_to_scope(Instruction* currentInstruction)
             emitGetVirtualRegister(value, regT2);
             
             loadPtr(Address(regT0, JSObject::butterflyOffset()), regT0);
+            addSlowCase(branchIfNotToSpace(regT0));
             loadPtr(operandSlot, regT1);
             negPtr(regT1);
             storePtr(regT2, BaseIndex(regT0, regT1, TimesEight, (firstOutOfLineOffset - 2) * sizeof(EncodedJSValue)));
@@ -1015,14 +1027,17 @@ void JIT::emitSlow_op_put_to_scope(Instruction* currentInstruction, Vector<SlowC
          || resolveType == LocalClosureVar)
         && currentInstruction[5].u.watchpointSet->state() != IsInvalidated)
         linkCount++;
-    if (resolveType == GlobalProperty || resolveType == GlobalPropertyWithVarInjectionChecks)
-        linkCount++;
+    if (resolveType == GlobalProperty || resolveType == GlobalPropertyWithVarInjectionChecks) {
+        linkCount++; // bad structure
+        linkCount++; // read barrier
+    }
     if (getPutInfo.initializationMode() != Initialization && (resolveType == GlobalLexicalVar || resolveType == GlobalLexicalVarWithVarInjectionChecks)) // TDZ check.
         linkCount++;
     if (resolveType == UnresolvedProperty || resolveType == UnresolvedPropertyWithVarInjectionChecks) {
         // GlobalProperty/GlobalPropertyWithVarInjectionsCheck
         linkCount++; // emitLoadWithStructureCheck
         linkCount++; // emitLoadWithStructureCheck
+        linkCount++; // read barrier
 
         // GlobalLexicalVar
         bool needsTDZCheck = getPutInfo.initializationMode() != Initialization;
@@ -1440,23 +1455,24 @@ JIT::JumpList JIT::emitIntTypedArrayGetByVal(Instruction*, PatchableJump& badTyp
     load8(Address(base, JSCell::typeInfoTypeOffset()), scratch);
     badType = patchableBranch32(NotEqual, scratch, TrustedImm32(typeForTypedArrayType(type)));
     slowCases.append(branch32(AboveOrEqual, property, Address(base, JSArrayBufferView::offsetOfLength())));
-    loadPtr(Address(base, JSArrayBufferView::offsetOfVector()), base);
+    loadPtr(Address(base, JSArrayBufferView::offsetOfVector()), scratch);
+    slowCases.append(branchIfNotToSpace(scratch));
     
     switch (elementSize(type)) {
     case 1:
         if (isSigned(type))
-            load8SignedExtendTo32(BaseIndex(base, property, TimesOne), resultPayload);
+            load8SignedExtendTo32(BaseIndex(scratch, property, TimesOne), resultPayload);
         else
-            load8(BaseIndex(base, property, TimesOne), resultPayload);
+            load8(BaseIndex(scratch, property, TimesOne), resultPayload);
         break;
     case 2:
         if (isSigned(type))
-            load16SignedExtendTo32(BaseIndex(base, property, TimesTwo), resultPayload);
+            load16SignedExtendTo32(BaseIndex(scratch, property, TimesTwo), resultPayload);
         else
-            load16(BaseIndex(base, property, TimesTwo), resultPayload);
+            load16(BaseIndex(scratch, property, TimesTwo), resultPayload);
         break;
     case 4:
-        load32(BaseIndex(base, property, TimesFour), resultPayload);
+        load32(BaseIndex(scratch, property, TimesFour), resultPayload);
         break;
     default:
         CRASH();
@@ -1511,15 +1527,16 @@ JIT::JumpList JIT::emitFloatTypedArrayGetByVal(Instruction*, PatchableJump& badT
     load8(Address(base, JSCell::typeInfoTypeOffset()), scratch);
     badType = patchableBranch32(NotEqual, scratch, TrustedImm32(typeForTypedArrayType(type)));
     slowCases.append(branch32(AboveOrEqual, property, Address(base, JSArrayBufferView::offsetOfLength())));
-    loadPtr(Address(base, JSArrayBufferView::offsetOfVector()), base);
+    loadPtr(Address(base, JSArrayBufferView::offsetOfVector()), scratch);
+    slowCases.append(branchIfNotToSpace(scratch));
     
     switch (elementSize(type)) {
     case 4:
-        loadFloat(BaseIndex(base, property, TimesFour), fpRegT0);
+        loadFloat(BaseIndex(scratch, property, TimesFour), fpRegT0);
         convertFloatToDouble(fpRegT0, fpRegT0);
         break;
     case 8: {
-        loadDouble(BaseIndex(base, property, TimesEight), fpRegT0);
+        loadDouble(BaseIndex(scratch, property, TimesEight), fpRegT0);
         break;
     }
     default:
@@ -1579,6 +1596,7 @@ JIT::JumpList JIT::emitIntTypedArrayPutByVal(Instruction* currentInstruction, Pa
     // We would be loading this into base as in get_by_val, except that the slow
     // path expects the base to be unclobbered.
     loadPtr(Address(base, JSArrayBufferView::offsetOfVector()), lateScratch);
+    slowCases.append(branchIfNotToSpace(lateScratch));
     
     if (isClamped(type)) {
         ASSERT(elementSize(type) == 1);
@@ -1664,6 +1682,7 @@ JIT::JumpList JIT::emitFloatTypedArrayPutByVal(Instruction* currentInstruction,
     // We would be loading this into base as in get_by_val, except that the slow
     // path expects the base to be unclobbered.
     loadPtr(Address(base, JSArrayBufferView::offsetOfVector()), lateScratch);
+    slowCases.append(branchIfNotToSpace(lateScratch));
     
     switch (elementSize(type)) {
     case 4:
index 5dae706..23b4b0d 100644 (file)
@@ -220,6 +220,8 @@ const PutByIdSecondaryTypeObject = 0x30
 const PutByIdSecondaryTypeObjectOrOther = 0x38
 const PutByIdSecondaryTypeTop = 0x40
 
+const CopyBarrierSpaceBits = 3
+
 const CallOpCodeSize = 9
 
 if X86_64 or ARM64 or C_LOOP
@@ -661,6 +663,10 @@ macro preserveReturnAddressAfterCall(destinationRegister)
     end
 end
 
+macro copyBarrier(value, slow)
+    btpnz value, CopyBarrierSpaceBits, slow
+end
+
 macro functionPrologue()
     if X86 or X86_WIN or X86_64 or X86_64_WIN
         push cfr
index d55cad9..a7c5549 100644 (file)
@@ -1188,9 +1188,10 @@ _llint_op_is_object:
     dispatch(3)
 
 
-macro loadPropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value)
+macro loadPropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value, slow)
     bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline
     loadp JSObject::m_butterfly[objectAndStorage], objectAndStorage
+    copyBarrier(objectAndStorage, slow)
     negi propertyOffsetAsInt
     sxi2q propertyOffsetAsInt, propertyOffsetAsInt
     jmp .ready
@@ -1201,9 +1202,10 @@ macro loadPropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value)
 end
 
 
-macro storePropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value)
+macro storePropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value, slow)
     bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline
     loadp JSObject::m_butterfly[objectAndStorage], objectAndStorage
+    copyBarrier(objectAndStorage, slow)
     negi propertyOffsetAsInt
     sxi2q propertyOffsetAsInt, propertyOffsetAsInt
     jmp .ready
@@ -1222,7 +1224,7 @@ _llint_op_get_by_id:
     bineq t2, t1, .opGetByIdSlow
     loadisFromInstruction(5, t1)
     loadisFromInstruction(1, t2)
-    loadPropertyAtVariableOffset(t1, t3, t0)
+    loadPropertyAtVariableOffset(t1, t3, t0, .opGetByIdSlow)
     storeq t0, [cfr, t2, 8]
     valueProfile(t0, 8, t1)
     dispatch(9)
@@ -1243,6 +1245,7 @@ _llint_op_get_array_length:
     btiz t2, IndexingShapeMask, .opGetArrayLengthSlow
     loadisFromInstruction(1, t1)
     loadp JSObject::m_butterfly[t3], t0
+    copyBarrier(t0, .opGetArrayLengthSlow)
     loadi -sizeof IndexingHeader + IndexingHeader::u.lengths.publicLength[t0], t0
     bilt t0, 0, .opGetArrayLengthSlow
     orq tagTypeNumber, t0
@@ -1382,7 +1385,7 @@ _llint_op_put_by_id:
     loadisFromInstruction(3, t1)
     loadConstantOrVariable(t1, t2)
     loadisFromInstruction(5, t1)
-    storePropertyAtVariableOffset(t1, t0, t2)
+    storePropertyAtVariableOffset(t1, t0, t2, .opPutByIdSlow)
     dispatch(9)
 
 .opPutByIdSlow:
@@ -1401,6 +1404,7 @@ _llint_op_get_by_val:
     loadConstantOrVariableInt32(t3, t1, .opGetByValSlow)
     sxi2q t1, t1
     loadp JSObject::m_butterfly[t0], t3
+    copyBarrier(t3, .opGetByValSlow)
     andi IndexingShapeMask, t2
     bieq t2, Int32Shape, .opGetByValIsContiguous
     bineq t2, ContiguousShape, .opGetByValNotContiguous
@@ -1471,6 +1475,7 @@ macro putByVal(slowPath)
     loadConstantOrVariableInt32(t0, t3, .opPutByValSlow)
     sxi2q t3, t3
     loadp JSObject::m_butterfly[t1], t0
+    copyBarrier(t0, .opPutByValSlow)
     andi IndexingShapeMask, t2
     bineq t2, Int32Shape, .opPutByValNotInt32
     contiguousPutByVal(
@@ -1991,9 +1996,9 @@ macro loadWithStructureCheck(operand, slowPath)
     bpneq t2, t1, slowPath
 end
 
-macro getProperty()
+macro getProperty(slow)
     loadisFromInstruction(6, t1)
-    loadPropertyAtVariableOffset(t1, t0, t2)
+    loadPropertyAtVariableOffset(t1, t0, t2, slow)
     valueProfile(t2, 7, t0)
     loadisFromInstruction(1, t0)
     storeq t2, [cfr, t0, 8]
@@ -2024,7 +2029,7 @@ _llint_op_get_from_scope:
 #gGlobalProperty:
     bineq t0, GlobalProperty, .gGlobalVar
     loadWithStructureCheck(2, .gDynamic)
-    getProperty()
+    getProperty(.gDynamic)
     dispatch(8)
 
 .gGlobalVar:
@@ -2049,7 +2054,7 @@ _llint_op_get_from_scope:
 .gGlobalPropertyWithVarInjectionChecks:
     bineq t0, GlobalPropertyWithVarInjectionChecks, .gGlobalVarWithVarInjectionChecks
     loadWithStructureCheck(2, .gDynamic)
-    getProperty()
+    getProperty(.gDynamic)
     dispatch(8)
 
 .gGlobalVarWithVarInjectionChecks:
@@ -2079,11 +2084,11 @@ _llint_op_get_from_scope:
     dispatch(8)
 
 
-macro putProperty()
+macro putProperty(slow)
     loadisFromInstruction(3, t1)
     loadConstantOrVariable(t1, t2)
     loadisFromInstruction(6, t1)
-    storePropertyAtVariableOffset(t1, t0, t2)
+    storePropertyAtVariableOffset(t1, t0, t2, slow)
 end
 
 macro putGlobalVariable()
@@ -2141,7 +2146,7 @@ _llint_op_put_to_scope:
     bineq t0, GlobalProperty, .pGlobalVar
     writeBarrierOnOperands(1, 3)
     loadWithStructureCheck(1, .pDynamic)
-    putProperty()
+    putProperty(.pDynamic)
     dispatch(7)
 
 .pGlobalVar:
@@ -2168,7 +2173,7 @@ _llint_op_put_to_scope:
     bineq t0, GlobalPropertyWithVarInjectionChecks, .pGlobalVarWithVarInjectionChecks
     writeBarrierOnOperands(1, 3)
     loadWithStructureCheck(1, .pDynamic)
-    putProperty()
+    putProperty(.pDynamic)
     dispatch(7)
 
 .pGlobalVarWithVarInjectionChecks:
index 2cf688c..4f93545 100644 (file)
@@ -97,7 +97,7 @@ void DirectArguments::visitChildren(JSCell* thisCell, SlotVisitor& visitor)
     if (thisObject->m_overrides) {
         visitor.copyLater(
             thisObject, DirectArgumentsOverridesCopyToken,
-            thisObject->m_overrides.get(), thisObject->overridesSize());
+            thisObject->m_overrides.getWithoutBarrier(), thisObject->overridesSize());
     }
 }
 
@@ -108,14 +108,11 @@ void DirectArguments::copyBackingStore(JSCell* thisCell, CopyVisitor& visitor, C
     
     RELEASE_ASSERT(token == DirectArgumentsOverridesCopyToken);
     
-    bool* oldOverrides = thisObject->m_overrides.get();
-    if (!oldOverrides)
-        return;
-    
+    void* oldOverrides = thisObject->m_overrides.getWithoutBarrier();
     if (visitor.checkIfShouldCopy(oldOverrides)) {
         bool* newOverrides = static_cast<bool*>(visitor.allocateNewSpace(thisObject->overridesSize()));
         memcpy(newOverrides, oldOverrides, thisObject->m_length);
-        thisObject->m_overrides.setWithoutWriteBarrier(newOverrides);
+        thisObject->m_overrides.setWithoutBarrier(newOverrides);
         visitor.didCopy(oldOverrides, thisObject->overridesSize());
     }
 }
@@ -135,9 +132,10 @@ void DirectArguments::overrideThings(VM& vm)
     
     void* backingStore;
     RELEASE_ASSERT(vm.heap.tryAllocateStorage(this, overridesSize(), &backingStore));
-    m_overrides.set(vm, this, static_cast<bool*>(backingStore));
+    bool* overrides = static_cast<bool*>(backingStore);
+    m_overrides.set(vm, this, overrides);
     for (unsigned i = m_length; i--;)
-        m_overrides.get()[i] = false;
+        overrides[i] = false;
 }
 
 void DirectArguments::overrideThingsIfNecessary(VM& vm)
@@ -149,7 +147,7 @@ void DirectArguments::overrideThingsIfNecessary(VM& vm)
 void DirectArguments::overrideArgument(VM& vm, unsigned index)
 {
     overrideThingsIfNecessary(vm);
-    m_overrides.get()[index] = true;
+    m_overrides.get(this)[index] = true;
 }
 
 void DirectArguments::copyToArguments(ExecState* exec, VirtualRegister firstElementDest, unsigned offset, unsigned length)
index 14bb8bb..a19f8e3 100644 (file)
@@ -73,7 +73,7 @@ public:
     
     bool canAccessIndexQuickly(uint32_t i) const
     {
-        return i < m_length && (!m_overrides || !m_overrides.get()[i]);
+        return i < m_length && (!m_overrides || !m_overrides.get(this)[i]);
     }
 
     bool canAccessArgumentIndexQuicklyInDFG(uint32_t i) const
@@ -148,7 +148,7 @@ private:
     WriteBarrier<JSFunction> m_callee;
     uint32_t m_length; // Always the actual length of captured arguments and never what was stored into the length property.
     uint32_t m_minCapacity; // The max of this and length determines the capacity of this object. It may be the actual capacity, or maybe something smaller. We arrange it this way to be kind to the JITs.
-    CopyWriteBarrier<bool> m_overrides; // If non-null, it means that length, callee, and caller are fully materialized properties.
+    CopyBarrier<bool> m_overrides; // If non-null, it means that length, callee, and caller are fully materialized properties.
 };
 
 } // namespace JSC
index daa87ba..afdbe0d 100644 (file)
@@ -392,6 +392,7 @@ bool JSArray::setLengthWithArrayStorage(ExecState* exec, unsigned newLength, boo
 
 bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException)
 {
+    Butterfly* butterfly = m_butterfly.get(this);
     switch (indexingType()) {
     case ArrayClass:
         if (!newLength)
@@ -408,7 +409,7 @@ bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException
     case ArrayWithInt32:
     case ArrayWithDouble:
     case ArrayWithContiguous: {
-        if (newLength == m_butterfly->publicLength())
+        if (newLength == butterfly->publicLength())
             return true;
         if (newLength >= MAX_ARRAY_INDEX // This case ensures that we can do fast push.
             || (newLength >= MIN_SPARSE_ARRAY_INDEX
@@ -417,12 +418,12 @@ bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException
                 exec, newLength, throwException,
                 ensureArrayStorage(exec->vm()));
         }
-        if (newLength > m_butterfly->publicLength()) {
+        if (newLength > butterfly->publicLength()) {
             ensureLength(exec->vm(), newLength);
             return true;
         }
 
-        unsigned lengthToClear = m_butterfly->publicLength() - newLength;
+        unsigned lengthToClear = butterfly->publicLength() - newLength;
         unsigned costToAllocateNewButterfly = 64; // a heuristic.
         if (lengthToClear > newLength && lengthToClear > costToAllocateNewButterfly) {
             reallocateAndShrinkButterfly(exec->vm(), newLength);
@@ -430,13 +431,13 @@ bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException
         }
 
         if (indexingType() == ArrayWithDouble) {
-            for (unsigned i = m_butterfly->publicLength(); i-- > newLength;)
-                m_butterfly->contiguousDouble()[i] = PNaN;
+            for (unsigned i = butterfly->publicLength(); i-- > newLength;)
+                butterfly->contiguousDouble()[i] = PNaN;
         } else {
-            for (unsigned i = m_butterfly->publicLength(); i-- > newLength;)
-                m_butterfly->contiguous()[i].clear();
+            for (unsigned i = butterfly->publicLength(); i-- > newLength;)
+                butterfly->contiguous()[i].clear();
         }
-        m_butterfly->setPublicLength(newLength);
+        butterfly->setPublicLength(newLength);
         return true;
     }
         
@@ -452,51 +453,53 @@ bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException
 
 JSValue JSArray::pop(ExecState* exec)
 {
+    Butterfly* butterfly = m_butterfly.get(this);
+    
     switch (indexingType()) {
     case ArrayClass:
         return jsUndefined();
         
     case ArrayWithUndecided:
-        if (!m_butterfly->publicLength())
+        if (!butterfly->publicLength())
             return jsUndefined();
         // We have nothing but holes. So, drop down to the slow version.
         break;
         
     case ArrayWithInt32:
     case ArrayWithContiguous: {
-        unsigned length = m_butterfly->publicLength();
+        unsigned length = butterfly->publicLength();
         
         if (!length--)
             return jsUndefined();
         
-        RELEASE_ASSERT(length < m_butterfly->vectorLength());
-        JSValue value = m_butterfly->contiguous()[length].get();
+        RELEASE_ASSERT(length < butterfly->vectorLength());
+        JSValue value = butterfly->contiguous()[length].get();
         if (value) {
-            m_butterfly->contiguous()[length].clear();
-            m_butterfly->setPublicLength(length);
+            butterfly->contiguous()[length].clear();
+            butterfly->setPublicLength(length);
             return value;
         }
         break;
     }
         
     case ArrayWithDouble: {
-        unsigned length = m_butterfly->publicLength();
+        unsigned length = butterfly->publicLength();
         
         if (!length--)
             return jsUndefined();
         
-        RELEASE_ASSERT(length < m_butterfly->vectorLength());
-        double value = m_butterfly->contiguousDouble()[length];
+        RELEASE_ASSERT(length < butterfly->vectorLength());
+        double value = butterfly->contiguousDouble()[length];
         if (value == value) {
-            m_butterfly->contiguousDouble()[length] = PNaN;
-            m_butterfly->setPublicLength(length);
+            butterfly->contiguousDouble()[length] = PNaN;
+            butterfly->setPublicLength(length);
             return JSValue(JSValue::EncodeAsDouble, value);
         }
         break;
     }
         
     case ARRAY_WITH_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = m_butterfly->arrayStorage();
+        ArrayStorage* storage = butterfly->arrayStorage();
     
         unsigned length = storage->length();
         if (!length) {
@@ -547,6 +550,8 @@ JSValue JSArray::pop(ExecState* exec)
 //  - pushing to an array of length 2^32-1 stores the property, but throws a range error.
 void JSArray::push(ExecState* exec, JSValue value)
 {
+    Butterfly* butterfly = m_butterfly.get(this);
+    
     switch (indexingType()) {
     case ArrayClass: {
         createInitialUndecided(exec->vm(), 0);
@@ -566,11 +571,11 @@ void JSArray::push(ExecState* exec, JSValue value)
             return;
         }
 
-        unsigned length = m_butterfly->publicLength();
-        ASSERT(length <= m_butterfly->vectorLength());
-        if (length < m_butterfly->vectorLength()) {
-            m_butterfly->contiguousInt32()[length].setWithoutWriteBarrier(value);
-            m_butterfly->setPublicLength(length + 1);
+        unsigned length = butterfly->publicLength();
+        ASSERT(length <= butterfly->vectorLength());
+        if (length < butterfly->vectorLength()) {
+            butterfly->contiguousInt32()[length].setWithoutWriteBarrier(value);
+            butterfly->setPublicLength(length + 1);
             return;
         }
         
@@ -586,11 +591,11 @@ void JSArray::push(ExecState* exec, JSValue value)
     }
 
     case ArrayWithContiguous: {
-        unsigned length = m_butterfly->publicLength();
-        ASSERT(length <= m_butterfly->vectorLength());
-        if (length < m_butterfly->vectorLength()) {
-            m_butterfly->contiguous()[length].set(exec->vm(), this, value);
-            m_butterfly->setPublicLength(length + 1);
+        unsigned length = butterfly->publicLength();
+        ASSERT(length <= butterfly->vectorLength());
+        if (length < butterfly->vectorLength()) {
+            butterfly->contiguous()[length].set(exec->vm(), this, value);
+            butterfly->setPublicLength(length + 1);
             return;
         }
         
@@ -618,11 +623,11 @@ void JSArray::push(ExecState* exec, JSValue value)
             return;
         }
 
-        unsigned length = m_butterfly->publicLength();
-        ASSERT(length <= m_butterfly->vectorLength());
-        if (length < m_butterfly->vectorLength()) {
-            m_butterfly->contiguousDouble()[length] = valueAsDouble;
-            m_butterfly->setPublicLength(length + 1);
+        unsigned length = butterfly->publicLength();
+        ASSERT(length <= butterfly->vectorLength());
+        if (length < butterfly->vectorLength()) {
+            butterfly->contiguousDouble()[length] = valueAsDouble;
+            butterfly->setPublicLength(length + 1);
             return;
         }
         
@@ -648,7 +653,7 @@ void JSArray::push(ExecState* exec, JSValue value)
     }
         
     case ArrayWithArrayStorage: {
-        ArrayStorage* storage = m_butterfly->arrayStorage();
+        ArrayStorage* storage = butterfly->arrayStorage();
 
         // Fast case - push within vector, always update m_length & m_numValuesInVector.
         unsigned length = storage->length();
@@ -696,9 +701,9 @@ JSArray* JSArray::fastSlice(ExecState& exec, unsigned startIndex, unsigned count
 
         auto& resultButterfly = *resultArray->butterfly();
         if (arrayType == ArrayWithDouble)
-            memcpy(resultButterfly.contiguousDouble().data(), m_butterfly->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);
+            memcpy(resultButterfly.contiguousDouble().data(), m_butterfly.get(this)->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);
         else
-            memcpy(resultButterfly.contiguous().data(), m_butterfly->contiguous().data() + startIndex, sizeof(JSValue) * count);
+            memcpy(resultButterfly.contiguous().data(), m_butterfly.get(this)->contiguous().data() + startIndex, sizeof(JSValue) * count);
         resultButterfly.setPublicLength(count);
 
         return resultArray;
@@ -715,8 +720,8 @@ EncodedJSValue JSArray::fastConcatWith(ExecState& exec, JSArray& otherArray)
     VM& vm = exec.vm();
     ASSERT(newArrayType == fastConcatType(vm, *this, otherArray));
 
-    unsigned thisArraySize = m_butterfly->publicLength();
-    unsigned otherArraySize = otherArray.m_butterfly->publicLength();
+    unsigned thisArraySize = m_butterfly.get(this)->publicLength();
+    unsigned otherArraySize = otherArray.m_butterfly.get(this)->publicLength();
     ASSERT(thisArraySize + otherArraySize < MIN_SPARSE_ARRAY_INDEX);
 
     Structure* resultStructure = exec.lexicalGlobalObject()->arrayStructureForIndexingTypeDuringAllocation(newArrayType);
@@ -728,11 +733,11 @@ EncodedJSValue JSArray::fastConcatWith(ExecState& exec, JSArray& otherArray)
     auto& otherButterfly = *otherArray.butterfly();
     if (newArrayType == ArrayWithDouble) {
         auto buffer = resultButterfly.contiguousDouble().data();
-        memcpy(buffer, m_butterfly->contiguousDouble().data(), sizeof(JSValue) * thisArraySize);
+        memcpy(buffer, m_butterfly.get(this)->contiguousDouble().data(), sizeof(JSValue) * thisArraySize);
         memcpy(buffer + thisArraySize, otherButterfly.contiguousDouble().data(), sizeof(JSValue) * otherArraySize);
     } else {
         auto buffer = resultButterfly.contiguous().data();
-        memcpy(buffer, m_butterfly->contiguous().data(), sizeof(JSValue) * thisArraySize);
+        memcpy(buffer, m_butterfly.get(this)->contiguous().data(), sizeof(JSValue) * thisArraySize);
         memcpy(buffer + thisArraySize, otherButterfly.contiguous().data(), sizeof(JSValue) * otherArraySize);
     }
 
@@ -805,8 +810,9 @@ bool JSArray::shiftCountWithArrayStorage(VM& vm, unsigned startIndex, unsigned c
         // Adjust the Butterfly and the index bias. We only need to do this here because we're changing
         // the start of the Butterfly, which needs to point at the first indexed property in the used
         // portion of the vector.
-        m_butterfly.setWithoutWriteBarrier(m_butterfly->shift(structure(), count));
-        storage = m_butterfly->arrayStorage();
+        Butterfly* butterfly = m_butterfly.get(this)->shift(structure(), count);
+        m_butterfly.setWithoutBarrier(butterfly);
+        storage = butterfly->arrayStorage();
         storage->m_indexBias += count;
 
         // Since we're consuming part of the vector by moving its beginning to the left,
@@ -848,6 +854,8 @@ bool JSArray::shiftCountWithAnyIndexingType(ExecState* exec, unsigned& startInde
 {
     VM& vm = exec->vm();
     RELEASE_ASSERT(count > 0);
+
+    Butterfly* butterfly = m_butterfly.get(this);
     
     switch (indexingType()) {
     case ArrayClass:
@@ -859,7 +867,7 @@ bool JSArray::shiftCountWithAnyIndexingType(ExecState* exec, unsigned& startInde
         
     case ArrayWithInt32:
     case ArrayWithContiguous: {
-        unsigned oldLength = m_butterfly->publicLength();
+        unsigned oldLength = butterfly->publicLength();
         RELEASE_ASSERT(count <= oldLength);
         
         // We may have to walk the entire array to do the shift. We're willing to do
@@ -874,28 +882,28 @@ bool JSArray::shiftCountWithAnyIndexingType(ExecState* exec, unsigned& startInde
         unsigned end = oldLength - count;
         if (this->structure(vm)->holesMustForwardToPrototype(vm)) {
             for (unsigned i = startIndex; i < end; ++i) {
-                JSValue v = m_butterfly->contiguous()[i + count].get();
+                JSValue v = butterfly->contiguous()[i + count].get();
                 if (UNLIKELY(!v)) {
                     startIndex = i;
                     return shiftCountWithArrayStorage(vm, startIndex, count, ensureArrayStorage(vm));
                 }
-                m_butterfly->contiguous()[i].setWithoutWriteBarrier(v);
+                butterfly->contiguous()[i].setWithoutWriteBarrier(v);
             }
         } else {
-            memmove(m_butterfly->contiguous().data() + startIndex, 
-                m_butterfly->contiguous().data() + startIndex + count, 
+            memmove(butterfly->contiguous().data() + startIndex, 
+                butterfly->contiguous().data() + startIndex + count, 
                 sizeof(JSValue) * (end - startIndex));
         }
 
         for (unsigned i = end; i < oldLength; ++i)
-            m_butterfly->contiguous()[i].clear();
+            butterfly->contiguous()[i].clear();
         
-        m_butterfly->setPublicLength(oldLength - count);
+        butterfly->setPublicLength(oldLength - count);
         return true;
     }
         
     case ArrayWithDouble: {
-        unsigned oldLength = m_butterfly->publicLength();
+        unsigned oldLength = butterfly->publicLength();
         RELEASE_ASSERT(count <= oldLength);
         
         // We may have to walk the entire array to do the shift. We're willing to do
@@ -910,22 +918,22 @@ bool JSArray::shiftCountWithAnyIndexingType(ExecState* exec, unsigned& startInde
         unsigned end = oldLength - count;
         if (this->structure(vm)->holesMustForwardToPrototype(vm)) {
             for (unsigned i = startIndex; i < end; ++i) {
-                double v = m_butterfly->contiguousDouble()[i + count];
+                double v = butterfly->contiguousDouble()[i + count];
                 if (UNLIKELY(v != v)) {
                     startIndex = i;
                     return shiftCountWithArrayStorage(vm, startIndex, count, ensureArrayStorage(vm));
                 }
-                m_butterfly->contiguousDouble()[i] = v;
+                butterfly->contiguousDouble()[i] = v;
             }
         } else {
-            memmove(m_butterfly->contiguousDouble().data() + startIndex,
-                m_butterfly->contiguousDouble().data() + startIndex + count,
+            memmove(butterfly->contiguousDouble().data() + startIndex,
+                butterfly->contiguousDouble().data() + startIndex + count,
                 sizeof(JSValue) * (end - startIndex));
         }
         for (unsigned i = end; i < oldLength; ++i)
-            m_butterfly->contiguousDouble()[i] = PNaN;
+            butterfly->contiguousDouble()[i] = PNaN;
         
-        m_butterfly->setPublicLength(oldLength - count);
+        butterfly->setPublicLength(oldLength - count);
         return true;
     }
         
@@ -986,6 +994,8 @@ bool JSArray::unshiftCountWithArrayStorage(ExecState* exec, unsigned startIndex,
 
 bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startIndex, unsigned count)
 {
+    Butterfly* butterfly = m_butterfly.get(this);
+    
     switch (indexingType()) {
     case ArrayClass:
     case ArrayWithUndecided:
@@ -994,7 +1004,7 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
 
     case ArrayWithInt32:
     case ArrayWithContiguous: {
-        unsigned oldLength = m_butterfly->publicLength();
+        unsigned oldLength = butterfly->publicLength();
         
         // We may have to walk the entire array to do the unshift. We're willing to do so
         // only if it's not horribly slow.
@@ -1002,19 +1012,20 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
             return unshiftCountWithArrayStorage(exec, startIndex, count, ensureArrayStorage(exec->vm()));
         
         ensureLength(exec->vm(), oldLength + count);
+        butterfly = m_butterfly.get(this);
 
         // We have to check for holes before we start moving things around so that we don't get halfway 
         // through shifting and then realize we should have been in ArrayStorage mode.
         for (unsigned i = oldLength; i-- > startIndex;) {
-            JSValue v = m_butterfly->contiguous()[i].get();
+            JSValue v = butterfly->contiguous()[i].get();
             if (UNLIKELY(!v))
                 return unshiftCountWithArrayStorage(exec, startIndex, count, ensureArrayStorage(exec->vm()));
         }
 
         for (unsigned i = oldLength; i-- > startIndex;) {
-            JSValue v = m_butterfly->contiguous()[i].get();
+            JSValue v = butterfly->contiguous()[i].get();
             ASSERT(v);
-            m_butterfly->contiguous()[i + count].setWithoutWriteBarrier(v);
+            butterfly->contiguous()[i + count].setWithoutWriteBarrier(v);
         }
         
         // NOTE: we're leaving being garbage in the part of the array that we shifted out
@@ -1026,7 +1037,7 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
     }
         
     case ArrayWithDouble: {
-        unsigned oldLength = m_butterfly->publicLength();
+        unsigned oldLength = butterfly->publicLength();
         
         // We may have to walk the entire array to do the unshift. We're willing to do so
         // only if it's not horribly slow.
@@ -1034,19 +1045,20 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
             return unshiftCountWithArrayStorage(exec, startIndex, count, ensureArrayStorage(exec->vm()));
         
         ensureLength(exec->vm(), oldLength + count);
+        butterfly = m_butterfly.get(this);
         
         // We have to check for holes before we start moving things around so that we don't get halfway 
         // through shifting and then realize we should have been in ArrayStorage mode.
         for (unsigned i = oldLength; i-- > startIndex;) {
-            double v = m_butterfly->contiguousDouble()[i];
+            double v = butterfly->contiguousDouble()[i];
             if (UNLIKELY(v != v))
                 return unshiftCountWithArrayStorage(exec, startIndex, count, ensureArrayStorage(exec->vm()));
         }
 
         for (unsigned i = oldLength; i-- > startIndex;) {
-            double v = m_butterfly->contiguousDouble()[i];
+            double v = butterfly->contiguousDouble()[i];
             ASSERT(v == v);
-            m_butterfly->contiguousDouble()[i + count] = v;
+            butterfly->contiguousDouble()[i + count] = v;
         }
         
         // NOTE: we're leaving being garbage in the part of the array that we shifted out
@@ -1072,6 +1084,8 @@ void JSArray::fillArgList(ExecState* exec, MarkedArgumentBuffer& args)
     unsigned i = 0;
     unsigned vectorEnd;
     WriteBarrier<Unknown>* vector;
+
+    Butterfly* butterfly = m_butterfly.get(this);
     
     switch (indexingType()) {
     case ArrayClass:
@@ -1085,16 +1099,16 @@ void JSArray::fillArgList(ExecState* exec, MarkedArgumentBuffer& args)
         
     case ArrayWithInt32:
     case ArrayWithContiguous: {
-        vectorEnd = m_butterfly->publicLength();
-        vector = m_butterfly->contiguous().data();
+        vectorEnd = butterfly->publicLength();
+        vector = butterfly->contiguous().data();
         break;
     }
         
     case ArrayWithDouble: {
         vector = 0;
         vectorEnd = 0;
-        for (; i < m_butterfly->publicLength(); ++i) {
-            double v = butterfly()->contiguousDouble()[i];
+        for (; i < butterfly->publicLength(); ++i) {
+            double v = butterfly->contiguousDouble()[i];
             if (v != v)
                 break;
             args.append(JSValue(JSValue::EncodeAsDouble, v));
@@ -1103,7 +1117,7 @@ void JSArray::fillArgList(ExecState* exec, MarkedArgumentBuffer& args)
     }
     
     case ARRAY_WITH_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = m_butterfly->arrayStorage();
+        ArrayStorage* storage = butterfly->arrayStorage();
         
         vector = storage->m_vector;
         vectorEnd = min(storage->length(), storage->vectorLength());
@@ -1141,6 +1155,8 @@ void JSArray::copyToArguments(ExecState* exec, VirtualRegister firstElementDest,
     // FIXME: What prevents this from being called with a RuntimeArray? The length function will always return 0 in that case.
     ASSERT(length == this->length());
 
+    Butterfly* butterfly = m_butterfly.get(this);
+    
     switch (indexingType()) {
     case ArrayClass:
         return;
@@ -1153,17 +1169,17 @@ void JSArray::copyToArguments(ExecState* exec, VirtualRegister firstElementDest,
         
     case ArrayWithInt32:
     case ArrayWithContiguous: {
-        vector = m_butterfly->contiguous().data();
-        vectorEnd = m_butterfly->publicLength();
+        vector = butterfly->contiguous().data();
+        vectorEnd = butterfly->publicLength();
         break;
     }
         
     case ArrayWithDouble: {
         vector = 0;
         vectorEnd = 0;
-        for (; i < m_butterfly->publicLength(); ++i) {
-            ASSERT(i < butterfly()->vectorLength());
-            double v = m_butterfly->contiguousDouble()[i];
+        for (; i < butterfly->publicLength(); ++i) {
+            ASSERT(i < butterfly->vectorLength());
+            double v = butterfly->contiguousDouble()[i];
             if (v != v)
                 break;
             exec->r(firstElementDest + i - offset) = JSValue(JSValue::EncodeAsDouble, v);
@@ -1172,7 +1188,7 @@ void JSArray::copyToArguments(ExecState* exec, VirtualRegister firstElementDest,
     }
         
     case ARRAY_WITH_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = m_butterfly->arrayStorage();
+        ArrayStorage* storage = butterfly->arrayStorage();
         vector = storage->m_vector;
         vectorEnd = min(length, storage->vectorLength());
         break;
index a3398db..a7281aa 100644 (file)
@@ -88,10 +88,10 @@ JSArrayBufferView::ConstructionContext::ConstructionContext(
     VM& vm, Structure* structure, PassRefPtr<ArrayBuffer> arrayBuffer,
     unsigned byteOffset, unsigned length)
     : m_structure(structure)
-    , m_vector(static_cast<uint8_t*>(arrayBuffer->data()) + byteOffset)
     , m_length(length)
     , m_mode(WastefulTypedArray)
 {
+    m_vector = static_cast<uint8_t*>(arrayBuffer->data()) + byteOffset;
     IndexingHeader indexingHeader;
     indexingHeader.setArrayBuffer(arrayBuffer.get());
     m_butterfly = Butterfly::create(vm, 0, 0, 0, true, indexingHeader, 0);
@@ -101,19 +101,19 @@ JSArrayBufferView::ConstructionContext::ConstructionContext(
     Structure* structure, PassRefPtr<ArrayBuffer> arrayBuffer,
     unsigned byteOffset, unsigned length, DataViewTag)
     : m_structure(structure)
-    , m_vector(static_cast<uint8_t*>(arrayBuffer->data()) + byteOffset)
     , m_length(length)
     , m_mode(DataViewMode)
     , m_butterfly(0)
 {
+    m_vector = static_cast<uint8_t*>(arrayBuffer->data()) + byteOffset;
 }
 
 JSArrayBufferView::JSArrayBufferView(VM& vm, ConstructionContext& context)
     : Base(vm, context.structure(), context.butterfly())
-    , m_vector(context.vector())
     , m_length(context.length())
     , m_mode(context.mode())
 {
+    m_vector.setWithoutBarrier(static_cast<char*>(context.vector()));
 }
 
 void JSArrayBufferView::finishCreation(VM& vm)
@@ -215,7 +215,7 @@ void JSArrayBufferView::finalize(JSCell* cell)
     JSArrayBufferView* thisObject = static_cast<JSArrayBufferView*>(cell);
     ASSERT(thisObject->m_mode == OversizeTypedArray || thisObject->m_mode == WastefulTypedArray);
     if (thisObject->m_mode == OversizeTypedArray)
-        fastFree(thisObject->m_vector);
+        fastFree(thisObject->m_vector.getWithoutBarrier());
 }
 
 } // namespace JSC
index 0514452..2c77b62 100644 (file)
@@ -161,7 +161,7 @@ public:
     PassRefPtr<ArrayBufferView> impl();
     void neuter();
     
-    void* vector() { return m_vector; }
+    void* vector() { return m_vector.get(this); }
     unsigned byteOffset();
     unsigned length() const { return m_length; }
 
@@ -177,7 +177,7 @@ private:
 protected:
     ArrayBuffer* existingBufferInButterfly();
 
-    void* m_vector;
+    CopyBarrier<char> m_vector; // this is really a void*, but void would not work here.
     uint32_t m_length;
     TypedArrayMode m_mode;
 };
index de4e858..cfbe009 100644 (file)
@@ -59,7 +59,7 @@ inline void JSArrayBufferView::neuter()
 {
     ASSERT(hasArrayBuffer());
     m_length = 0;
-    m_vector = 0;
+    m_vector.clear();
 }
 
 inline unsigned JSArrayBufferView::byteOffset()
@@ -68,7 +68,7 @@ inline unsigned JSArrayBufferView::byteOffset()
         return 0;
     
     ptrdiff_t delta =
-        static_cast<uint8_t*>(m_vector) - static_cast<uint8_t*>(buffer()->data());
+        bitwise_cast<uint8_t*>(m_vector.get(this)) - static_cast<uint8_t*>(buffer()->data());
     
     unsigned result = static_cast<unsigned>(delta);
     ASSERT(static_cast<ptrdiff_t>(result) == delta);
index 6e55a31..9f985d4 100644 (file)
@@ -108,11 +108,11 @@ public:
     
     const typename Adaptor::Type* typedVector() const
     {
-        return static_cast<const typename Adaptor::Type*>(m_vector);
+        return bitwise_cast<const typename Adaptor::Type*>(m_vector.get(this));
     }
     typename Adaptor::Type* typedVector()
     {
-        return static_cast<typename Adaptor::Type*>(m_vector);
+        return bitwise_cast<typename Adaptor::Type*>(m_vector.get(this));
     }
 
     // These methods are meant to match indexed access methods that JSObject
index 957d4af..eb2c712 100644 (file)
@@ -441,7 +441,7 @@ void JSGenericTypedArrayView<Adaptor>::visitChildren(JSCell* cell, SlotVisitor&
     switch (thisObject->m_mode) {
     case FastTypedArray: {
         if (thisObject->m_vector)
-            visitor.copyLater(thisObject, TypedArrayVectorCopyToken, thisObject->m_vector, thisObject->byteSize());
+            visitor.copyLater(thisObject, TypedArrayVectorCopyToken, thisObject->m_vector.getWithoutBarrier(), thisObject->byteSize());
         break;
     }
         
@@ -468,12 +468,12 @@ void JSGenericTypedArrayView<Adaptor>::copyBackingStore(
     JSGenericTypedArrayView* thisObject = jsCast<JSGenericTypedArrayView*>(cell);
     
     if (token == TypedArrayVectorCopyToken
-        && visitor.checkIfShouldCopy(thisObject->m_vector)) {
+        && visitor.checkIfShouldCopy(thisObject->m_vector.getWithoutBarrier())) {
         ASSERT(thisObject->m_vector);
-        void* oldVector = thisObject->m_vector;
+        void* oldVector = thisObject->m_vector.get(thisObject);
         void* newVector = visitor.allocateNewSpace(thisObject->byteSize());
         memcpy(newVector, oldVector, thisObject->byteSize());
-        thisObject->m_vector = newVector;
+        thisObject->m_vector.setWithoutBarrier(static_cast<char*>(newVector));
         visitor.didCopy(oldVector, thisObject->byteSize());
     }
     
@@ -508,8 +508,8 @@ ArrayBuffer* JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory(JSArrayBuf
         && !thisObject->butterfly() && size >= sizeof(IndexingHeader)) {
         ASSERT(thisObject->m_vector);
         // Reuse already allocated memory if at all possible.
-        thisObject->m_butterfly.setWithoutWriteBarrier(
-            static_cast<IndexingHeader*>(thisObject->m_vector)->butterfly());
+        thisObject->m_butterfly.setWithoutBarrier(
+            bitwise_cast<IndexingHeader*>(thisObject->m_vector.get(thisObject))->butterfly());
     } else {
         VM& vm = *heap->vm();
         thisObject->m_butterfly.set(vm, thisObject, Butterfly::createOrGrowArrayRight(
@@ -521,14 +521,14 @@ ArrayBuffer* JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory(JSArrayBuf
     
     switch (thisObject->m_mode) {
     case FastTypedArray:
-        buffer = ArrayBuffer::create(thisObject->m_vector, thisObject->byteLength());
+        buffer = ArrayBuffer::create(thisObject->m_vector.get(thisObject), thisObject->byteLength());
         break;
         
     case OversizeTypedArray:
         // FIXME: consider doing something like "subtracting" from extra memory
         // cost, since right now this case will cause the GC to think that we reallocated
         // the whole buffer.
-        buffer = ArrayBuffer::createAdopted(thisObject->m_vector, thisObject->byteLength());
+        buffer = ArrayBuffer::createAdopted(thisObject->m_vector.get(thisObject), thisObject->byteLength());
         break;
         
     default:
@@ -537,7 +537,7 @@ ArrayBuffer* JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory(JSArrayBuf
     }
 
     thisObject->butterfly()->indexingHeader()->setArrayBuffer(buffer.get());
-    thisObject->m_vector = buffer->data();
+    thisObject->m_vector.setWithoutBarrier(static_cast<char*>(buffer->data()));
     thisObject->m_mode = WastefulTypedArray;
     heap->addReference(thisObject, buffer.get());
     
index 837f7e3..213047c 100644 (file)
@@ -117,7 +117,7 @@ public:
 private:
     JSMap(VM& vm, Structure* structure)
         : Base(vm, structure)
-        , m_mapData(vm)
+        , m_mapData(vm, this)
     {
     }
 
index 0289016..de5df9e 100644 (file)
@@ -150,7 +150,7 @@ ALWAYS_INLINE void JSObject::copyButterfly(CopyVisitor& visitor, Butterfly* butt
             memcpy(currentTarget, currentSource, count * sizeof(EncodedJSValue));
         }
         
-        m_butterfly.setWithoutWriteBarrier(newButterfly);
+        m_butterfly.setWithoutBarrier(newButterfly);
         visitor.didCopy(butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
     } 
 }
@@ -206,7 +206,7 @@ void JSObject::visitChildren(JSCell* cell, SlotVisitor& visitor)
     
     JSCell::visitChildren(thisObject, visitor);
 
-    Butterfly* butterfly = thisObject->butterfly();
+    Butterfly* butterfly = thisObject->m_butterfly.getWithoutBarrier();
     if (butterfly)
         thisObject->visitButterfly(visitor, butterfly, thisObject->structure(visitor.vm())->outOfLineSize());
 
@@ -223,7 +223,7 @@ void JSObject::copyBackingStore(JSCell* cell, CopyVisitor& visitor, CopyToken to
     if (token != ButterflyCopyToken)
         return;
     
-    Butterfly* butterfly = thisObject->butterfly();
+    Butterfly* butterfly = thisObject->m_butterfly.getWithoutBarrier();
     if (butterfly)
         thisObject->copyButterfly(visitor, butterfly, thisObject->structure()->outOfLineSize());
 }
@@ -341,7 +341,7 @@ bool JSObject::getOwnPropertySlotByIndex(JSObject* thisObject, ExecState* exec,
     }
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly.get(thisObject)->arrayStorage();
         if (i >= storage->length())
             return false;
         
@@ -514,7 +514,7 @@ void JSObject::putByIndex(JSCell* cell, ExecState* exec, unsigned propertyName,
         
     case NonArrayWithArrayStorage:
     case ArrayWithArrayStorage: {
-        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly.get(thisObject)->arrayStorage();
         
         if (propertyName >= storage->vectorLength())
             break;
@@ -536,7 +536,7 @@ void JSObject::putByIndex(JSCell* cell, ExecState* exec, unsigned propertyName,
         
     case NonArrayWithSlowPutArrayStorage:
     case ArrayWithSlowPutArrayStorage: {
-        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly.get(thisObject)->arrayStorage();
         
         if (propertyName >= storage->vectorLength())
             break;
@@ -613,7 +613,7 @@ void JSObject::enterDictionaryIndexingMode(VM& vm)
         enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, ensureArrayStorageSlow(vm));
         break;
     case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-        enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage());
+        enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly.get(this)->arrayStorage());
         break;
         
     default:
@@ -643,7 +643,7 @@ Butterfly* JSObject::createInitialIndexedStorage(VM& vm, unsigned length, size_t
     ASSERT(!indexingShouldBeSparse());
     unsigned vectorLength = std::max(length, BASE_VECTOR_LEN);
     Butterfly* newButterfly = Butterfly::createOrGrowArrayRight(
-        m_butterfly.get(), vm, this, structure(), structure()->outOfLineCapacity(), false, 0,
+        m_butterfly.get(this), vm, this, structure(), structure()->outOfLineCapacity(), false, 0,
         elementSize * vectorLength);
     newButterfly->setPublicLength(length);
     newButterfly->setVectorLength(vectorLength);
@@ -695,7 +695,7 @@ ArrayStorage* JSObject::createArrayStorage(VM& vm, unsigned length, unsigned vec
     IndexingType oldType = indexingType();
     ASSERT_UNUSED(oldType, !hasIndexedProperties(oldType));
     Butterfly* newButterfly = Butterfly::createOrGrowArrayRight(
-        m_butterfly.get(), vm, this, structure, structure->outOfLineCapacity(), false, 0,
+        m_butterfly.get(this), vm, this, structure, structure->outOfLineCapacity(), false, 0,
         ArrayStorage::sizeFor(vectorLength));
     RELEASE_ASSERT(newButterfly);
 
@@ -719,31 +719,32 @@ ContiguousJSValues JSObject::convertUndecidedToInt32(VM& vm)
 {
     ASSERT(hasUndecided(indexingType()));
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), AllocateInt32));
-    return m_butterfly->contiguousInt32();
+    return m_butterfly.get(this)->contiguousInt32();
 }
 
 ContiguousDoubles JSObject::convertUndecidedToDouble(VM& vm)
 {
     ASSERT(hasUndecided(indexingType()));
-    
-    for (unsigned i = m_butterfly->vectorLength(); i--;)
-        m_butterfly->contiguousDouble()[i] = PNaN;
+
+    Butterfly* butterfly = m_butterfly.get(this);
+    for (unsigned i = butterfly->vectorLength(); i--;)
+        butterfly->contiguousDouble()[i] = PNaN;
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), AllocateDouble));
-    return m_butterfly->contiguousDouble();
+    return m_butterfly.get(this)->contiguousDouble();
 }
 
 ContiguousJSValues JSObject::convertUndecidedToContiguous(VM& vm)
 {
     ASSERT(hasUndecided(indexingType()));
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), AllocateContiguous));
-    return m_butterfly->contiguous();
+    return m_butterfly.get(this)->contiguous();
 }
 
 ArrayStorage* JSObject::constructConvertedArrayStorageWithoutCopyingElements(VM& vm, unsigned neededLength)
 {
     Structure* structure = this->structure(vm);
-    unsigned publicLength = m_butterfly->publicLength();
+    unsigned publicLength = m_butterfly.get(this)->publicLength();
     unsigned propertyCapacity = structure->outOfLineCapacity();
     unsigned propertySize = structure->outOfLineSize();
     
@@ -752,7 +753,7 @@ ArrayStorage* JSObject::constructConvertedArrayStorageWithoutCopyingElements(VM&
     
     memcpy(
         newButterfly->propertyStorage() - propertySize,
-        m_butterfly->propertyStorage() - propertySize,
+        m_butterfly.get(this)->propertyStorage() - propertySize,
         propertySize * sizeof(EncodedJSValue));
     
     ArrayStorage* newStorage = newButterfly->arrayStorage();
@@ -770,7 +771,7 @@ ArrayStorage* JSObject::convertUndecidedToArrayStorage(VM& vm, NonPropertyTransi
     DeferGC deferGC(vm.heap);
     ASSERT(hasUndecided(indexingType()));
 
-    unsigned vectorLength = m_butterfly->vectorLength();
+    unsigned vectorLength = m_butterfly.get(this)->vectorLength();
     ArrayStorage* storage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
     // No need to copy elements.
     
@@ -787,9 +788,10 @@ ArrayStorage* JSObject::convertUndecidedToArrayStorage(VM& vm)
 ContiguousDoubles JSObject::convertInt32ToDouble(VM& vm)
 {
     ASSERT(hasInt32(indexingType()));
-    
-    for (unsigned i = m_butterfly->vectorLength(); i--;) {
-        WriteBarrier<Unknown>* current = &m_butterfly->contiguousInt32()[i];
+
+    Butterfly* butterfly = m_butterfly.get(this);
+    for (unsigned i = butterfly->vectorLength(); i--;) {
+        WriteBarrier<Unknown>* current = &butterfly->contiguousInt32()[i];
         double* currentAsDouble = bitwise_cast<double*>(current);
         JSValue v = current->get();
         if (!v) {
@@ -801,7 +803,7 @@ ContiguousDoubles JSObject::convertInt32ToDouble(VM& vm)
     }
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), AllocateDouble));
-    return m_butterfly->contiguousDouble();
+    return m_butterfly.get(this)->contiguousDouble();
 }
 
 ContiguousJSValues JSObject::convertInt32ToContiguous(VM& vm)
@@ -809,7 +811,7 @@ ContiguousJSValues JSObject::convertInt32ToContiguous(VM& vm)
     ASSERT(hasInt32(indexingType()));
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), AllocateContiguous));
-    return m_butterfly->contiguous();
+    return m_butterfly.get(this)->contiguous();
 }
 
 ArrayStorage* JSObject::convertInt32ToArrayStorage(VM& vm, NonPropertyTransition transition)
@@ -817,10 +819,11 @@ ArrayStorage* JSObject::convertInt32ToArrayStorage(VM& vm, NonPropertyTransition
     DeferGC deferGC(vm.heap);
     ASSERT(hasInt32(indexingType()));
 
-    unsigned vectorLength = m_butterfly->vectorLength();
+    unsigned vectorLength = m_butterfly.get(this)->vectorLength();
     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
-    for (unsigned i = 0; i < m_butterfly->publicLength(); i++) {
-        JSValue v = m_butterfly->contiguous()[i].get();
+    Butterfly* butterfly = m_butterfly.get(this);
+    for (unsigned i = 0; i < butterfly->publicLength(); i++) {
+        JSValue v = butterfly->contiguous()[i].get();
         if (v) {
             newStorage->m_vector[i].setWithoutWriteBarrier(v);
             newStorage->m_numValuesInVector++;
@@ -841,9 +844,10 @@ ArrayStorage* JSObject::convertInt32ToArrayStorage(VM& vm)
 ContiguousJSValues JSObject::convertDoubleToContiguous(VM& vm)
 {
     ASSERT(hasDouble(indexingType()));
-    
-    for (unsigned i = m_butterfly->vectorLength(); i--;) {
-        double* current = &m_butterfly->contiguousDouble()[i];
+
+    Butterfly* butterfly = m_butterfly.get(this);
+    for (unsigned i = butterfly->vectorLength(); i--;) {
+        double* current = &butterfly->contiguousDouble()[i];
         WriteBarrier<Unknown>* currentAsValue = bitwise_cast<WriteBarrier<Unknown>*>(current);
         double value = *current;
         if (value != value) {
@@ -855,7 +859,7 @@ ContiguousJSValues JSObject::convertDoubleToContiguous(VM& vm)
     }
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), AllocateContiguous));
-    return m_butterfly->contiguous();
+    return m_butterfly.get(this)->contiguous();
 }
 
 ArrayStorage* JSObject::convertDoubleToArrayStorage(VM& vm, NonPropertyTransition transition)
@@ -863,10 +867,11 @@ ArrayStorage* JSObject::convertDoubleToArrayStorage(VM& vm, NonPropertyTransitio
     DeferGC deferGC(vm.heap);
     ASSERT(hasDouble(indexingType()));
 
-    unsigned vectorLength = m_butterfly->vectorLength();
+    unsigned vectorLength = m_butterfly.get(this)->vectorLength();
     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
-    for (unsigned i = 0; i < m_butterfly->publicLength(); i++) {
-        double value = m_butterfly->contiguousDouble()[i];
+    Butterfly* butterfly = m_butterfly.get(this);
+    for (unsigned i = 0; i < butterfly->publicLength(); i++) {
+        double value = butterfly->contiguousDouble()[i];
         if (value == value) {
             newStorage->m_vector[i].setWithoutWriteBarrier(JSValue(JSValue::EncodeAsDouble, value));
             newStorage->m_numValuesInVector++;
@@ -889,10 +894,11 @@ ArrayStorage* JSObject::convertContiguousToArrayStorage(VM& vm, NonPropertyTrans
     DeferGC deferGC(vm.heap);
     ASSERT(hasContiguous(indexingType()));
 
-    unsigned vectorLength = m_butterfly->vectorLength();
+    unsigned vectorLength = m_butterfly.get(this)->vectorLength();
     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
-    for (unsigned i = 0; i < m_butterfly->publicLength(); i++) {
-        JSValue v = m_butterfly->contiguous()[i].get();
+    Butterfly* butterfly = m_butterfly.get(this);
+    for (unsigned i = 0; i < butterfly->publicLength(); i++) {
+        JSValue v = butterfly->contiguous()[i].get();
         if (v) {
             newStorage->m_vector[i].setWithoutWriteBarrier(v);
             newStorage->m_numValuesInVector++;
@@ -957,8 +963,8 @@ void JSObject::convertInt32ForValue(VM& vm, JSValue value)
 
 void JSObject::setIndexQuicklyToUndecided(VM& vm, unsigned index, JSValue value)
 {
-    ASSERT(index < m_butterfly->publicLength());
-    ASSERT(index < m_butterfly->vectorLength());
+    ASSERT(index < m_butterfly.get(this)->publicLength());
+    ASSERT(index < m_butterfly.get(this)->vectorLength());
     convertUndecidedForValue(vm, value);
     setIndexQuickly(vm, index, value);
 }
@@ -1114,7 +1120,7 @@ ArrayStorage* JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode(V
         return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, convertContiguousToArrayStorage(vm));
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-        return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage());
+        return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly.get(this)->arrayStorage());
         
     default:
         CRASH();
@@ -1356,7 +1362,7 @@ bool JSObject::deletePropertyByIndex(JSCell* cell, ExecState* exec, unsigned i)
     }
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly.get(thisObject)->arrayStorage();
         
         if (i < storage->vectorLength()) {
             WriteBarrier<Unknown>& valueSlot = storage->m_vector[i];
@@ -1546,7 +1552,7 @@ void JSObject::getOwnPropertyNames(JSObject* object, ExecState* exec, PropertyNa
         }
             
         case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-            ArrayStorage* storage = object->m_butterfly->arrayStorage();
+            ArrayStorage* storage = object->m_butterfly.get(object)->arrayStorage();
             
             unsigned usedVectorLength = std::min(storage->length(), storage->vectorLength());
             for (unsigned i = 0; i < usedVectorLength; ++i) {
@@ -1699,7 +1705,7 @@ NEVER_INLINE void JSObject::fillGetterPropertySlot(PropertySlot& slot, JSValue g
 void JSObject::putIndexedDescriptor(ExecState* exec, SparseArrayEntry* entryInMap, const PropertyDescriptor& descriptor, PropertyDescriptor& oldDescriptor)
 {
     VM& vm = exec->vm();
-    auto map = m_butterfly->arrayStorage()->m_sparseMap.get();
+    auto map = m_butterfly.get(this)->arrayStorage()->m_sparseMap.get();
 
     if (descriptor.isDataDescriptor()) {
         if (descriptor.value())
@@ -1758,7 +1764,7 @@ bool JSObject::defineOwnIndexedProperty(ExecState* exec, unsigned index, const P
     if (descriptor.attributes() & (ReadOnly | Accessor))
         notifyPresenceOfIndexedAccessors(exec->vm());
 
-    SparseArrayValueMap* map = m_butterfly->arrayStorage()->m_sparseMap.get();
+    SparseArrayValueMap* map = m_butterfly.get(this)->arrayStorage()->m_sparseMap.get();
     RELEASE_ASSERT(map);
 
     // 1. Let current be the result of calling the [[GetOwnProperty]] internal method of O with property name P.
@@ -1790,8 +1796,9 @@ bool JSObject::defineOwnIndexedProperty(ExecState* exec, unsigned index, const P
         entryInMap->get(defaults);
 
         putIndexedDescriptor(exec, entryInMap, descriptor, defaults);
-        if (index >= m_butterfly->arrayStorage()->length())
-            m_butterfly->arrayStorage()->setLength(index + 1);
+        Butterfly* butterfly = m_butterfly.get(this);
+        if (index >= butterfly->arrayStorage()->length())
+            butterfly->arrayStorage()->setLength(index + 1);
         return true;
     }
 
@@ -1911,17 +1918,19 @@ void JSObject::putByIndexBeyondVectorLengthWithoutAttributes(ExecState* exec, un
 {
     ASSERT((indexingType() & IndexingShapeMask) == indexingShape);
     ASSERT(!indexingShouldBeSparse());
+
+    Butterfly* butterfly = m_butterfly.get(this);
     
     // For us to get here, the index is either greater than the public length, or greater than
     // or equal to the vector length.
-    ASSERT(i >= m_butterfly->vectorLength());
+    ASSERT(i >= butterfly->vectorLength());
     
     VM& vm = exec->vm();
     
     if (i >= MAX_ARRAY_INDEX - 1
         || (i >= MIN_SPARSE_ARRAY_INDEX
-            && !isDenseEnoughForVector(i, countElements<indexingShape>(butterfly())))
-        || indexIsSufficientlyBeyondLengthForSparseMap(i, m_butterfly->vectorLength())) {
+            && !isDenseEnoughForVector(i, countElements<indexingShape>(butterfly)))
+        || indexIsSufficientlyBeyondLengthForSparseMap(i, butterfly->vectorLength())) {
         ASSERT(i <= MAX_ARRAY_INDEX);
         ensureArrayStorageSlow(vm);
         SparseArrayValueMap* map = allocateSparseIndexMap(vm);
@@ -1932,24 +1941,25 @@ void JSObject::putByIndexBeyondVectorLengthWithoutAttributes(ExecState* exec, un
     }
 
     ensureLength(vm, i + 1);
+    butterfly = m_butterfly.get(this);
 
-    RELEASE_ASSERT(i < m_butterfly->vectorLength());
+    RELEASE_ASSERT(i < butterfly->vectorLength());
     switch (indexingShape) {
     case Int32Shape:
         ASSERT(value.isInt32());
-        m_butterfly->contiguousInt32()[i].setWithoutWriteBarrier(value);
+        butterfly->contiguousInt32()[i].setWithoutWriteBarrier(value);
         break;
         
     case DoubleShape: {
         ASSERT(value.isNumber());
         double valueAsDouble = value.asNumber();
         ASSERT(valueAsDouble == valueAsDouble);
-        m_butterfly->contiguousDouble()[i] = valueAsDouble;
+        butterfly->contiguousDouble()[i] = valueAsDouble;
         break;
     }
         
     case ContiguousShape:
-        m_butterfly->contiguous()[i].set(vm, this, value);
+        butterfly->contiguous()[i].set(vm, this, value);
         break;
         
     default:
@@ -2223,7 +2233,7 @@ bool JSObject::putDirectIndexBeyondVectorLength(ExecState* exec, unsigned i, JSV
         
     case ALL_INT32_INDEXING_TYPES: {
         if (attributes) {
-            if (i < m_butterfly->vectorLength())
+            if (i < m_butterfly.get(this)->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
             return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertInt32ToArrayStorage(vm));
         }
@@ -2237,7 +2247,7 @@ bool JSObject::putDirectIndexBeyondVectorLength(ExecState* exec, unsigned i, JSV
         
     case ALL_DOUBLE_INDEXING_TYPES: {
         if (attributes) {
-            if (i < m_butterfly->vectorLength())
+            if (i < m_butterfly.get(this)->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
             return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertDoubleToArrayStorage(vm));
         }
@@ -2256,7 +2266,7 @@ bool JSObject::putDirectIndexBeyondVectorLength(ExecState* exec, unsigned i, JSV
         
     case ALL_CONTIGUOUS_INDEXING_TYPES: {
         if (attributes) {
-            if (i < m_butterfly->vectorLength())
+            if (i < m_butterfly.get(this)->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
             return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertContiguousToArrayStorage(vm));
         }
@@ -2266,7 +2276,7 @@ bool JSObject::putDirectIndexBeyondVectorLength(ExecState* exec, unsigned i, JSV
 
     case ALL_ARRAY_STORAGE_INDEXING_TYPES:
         if (attributes) {
-            if (i < m_butterfly->vectorLength())
+            if (i < m_butterfly.get(this)->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
         }
         return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, arrayStorage());
@@ -2347,8 +2357,8 @@ ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned desiredLength)
     unsigned length;
     
     if (hasIndexedProperties(indexingType())) {
-        vectorLength = m_butterfly->vectorLength();
-        length = m_butterfly->publicLength();
+        vectorLength = m_butterfly.get(this)->vectorLength();
+        length = m_butterfly.get(this)->publicLength();
     } else {
         vectorLength = 0;
         length = 0;
@@ -2454,25 +2464,28 @@ bool JSObject::increaseVectorLength(VM& vm, unsigned newLength)
 
 void JSObject::ensureLengthSlow(VM& vm, unsigned length)
 {
+    Butterfly* butterfly = m_butterfly.get(this);
+    
     ASSERT(length < MAX_ARRAY_INDEX);
     ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
-    ASSERT(length > m_butterfly->vectorLength());
+    ASSERT(length > butterfly->vectorLength());
     
     unsigned newVectorLength = std::min(
         length << 1,
         MAX_STORAGE_VECTOR_LENGTH);
-    unsigned oldVectorLength = m_butterfly->vectorLength();
+    unsigned oldVectorLength = butterfly->vectorLength();
     DeferGC deferGC(vm.heap);
-    m_butterfly.set(vm, this, m_butterfly->growArrayRight(
+    butterfly = butterfly->growArrayRight(
         vm, this, structure(), structure()->outOfLineCapacity(), true,
         oldVectorLength * sizeof(EncodedJSValue),
-        newVectorLength * sizeof(EncodedJSValue)));
+        newVectorLength * sizeof(EncodedJSValue));
+    m_butterfly.set(vm, this, butterfly);
 
-    m_butterfly->setVectorLength(newVectorLength);
+    butterfly->setVectorLength(newVectorLength);
 
     if (hasDouble(indexingType())) {
         for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
-            m_butterfly->contiguousDouble().data()[i] = PNaN;
+            butterfly->contiguousDouble().data()[i] = PNaN;
     }
 }
 
@@ -2485,10 +2498,10 @@ void JSObject::reallocateAndShrinkButterfly(VM& vm, unsigned length)
     ASSERT(!m_butterfly->indexingHeader()->preCapacity(structure()));
 
     DeferGC deferGC(vm.heap);
-    Butterfly* newButterfly = m_butterfly->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
+    Butterfly* newButterfly = m_butterfly.get(this)->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
     m_butterfly.set(vm, this, newButterfly);
-    m_butterfly->setVectorLength(length);
-    m_butterfly->setPublicLength(length);
+    newButterfly->setVectorLength(length);
+    newButterfly->setPublicLength(length);
 }
 
 Butterfly* JSObject::growOutOfLineStorage(VM& vm, size_t oldSize, size_t newSize)
@@ -2498,7 +2511,7 @@ Butterfly* JSObject::growOutOfLineStorage(VM& vm, size_t oldSize, size_t newSize
     // It's important that this function not rely on structure(), for the property
     // capacity, since we might have already mutated the structure in-place.
     
-    return Butterfly::createOrGrowPropertyStorage(m_butterfly.get(), vm, this, structure(vm), oldSize, newSize);
+    return Butterfly::createOrGrowPropertyStorage(m_butterfly.get(this), vm, this, structure(vm), oldSize, newSize);
 }
 
 bool JSObject::getOwnPropertyDescriptor(ExecState* exec, PropertyName propertyName, PropertyDescriptor& descriptor)
@@ -2785,7 +2798,7 @@ uint32_t JSObject::getEnumerableLength(ExecState* exec, JSObject* object)
     }
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = object->m_butterfly->arrayStorage();
+        ArrayStorage* storage = object->m_butterfly.get(object)->arrayStorage();
         if (storage->m_sparseMap.get())
             return 0;
         
index 93fcb69..c406ba1 100644 (file)
@@ -30,7 +30,7 @@
 #include "CallFrame.h"
 #include "ClassInfo.h"
 #include "CommonIdentifiers.h"
-#include "CopyWriteBarrier.h"
+#include "CopyBarrier.h"
 #include "CustomGetterSetter.h"
 #include "DeferGC.h"
 #include "Heap.h"
@@ -131,14 +131,14 @@ public:
     {
         if (!hasIndexedProperties(indexingType()))
             return 0;
-        return m_butterfly->publicLength();
+        return m_butterfly.get(this)->publicLength();
     }
         
     unsigned getVectorLength()
     {
         if (!hasIndexedProperties(indexingType()))
             return 0;
-        return m_butterfly->vectorLength();
+        return m_butterfly.get(this)->vectorLength();
     }
         
     JS_EXPORT_PRIVATE static void put(JSCell*, ExecState*, PropertyName, JSValue, PutPropertySlot&);
@@ -181,23 +181,24 @@ public:
     
     bool canGetIndexQuickly(unsigned i)
     {
+        Butterfly* butterfly = m_butterfly.get(this);
         switch (indexingType()) {
         case ALL_BLANK_INDEXING_TYPES:
         case ALL_UNDECIDED_INDEXING_TYPES:
             return false;
         case ALL_INT32_INDEXING_TYPES:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
-            return i < m_butterfly->vectorLength() && m_butterfly->contiguous()[i];
+            return i < butterfly->vectorLength() && butterfly->contiguous()[i];
         case ALL_DOUBLE_INDEXING_TYPES: {
-            if (i >= m_butterfly->vectorLength())
+            if (i >= butterfly->vectorLength())
                 return false;
-            double value = m_butterfly->contiguousDouble()[i];
+            double value = butterfly->contiguousDouble()[i];
             if (value != value)
                 return false;
             return true;
         }
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return i < m_butterfly->arrayStorage()->vectorLength() && m_butterfly->arrayStorage()->m_vector[i];
+            return i < butterfly->arrayStorage()->vectorLength() && butterfly->arrayStorage()->m_vector[i];
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -206,15 +207,16 @@ public:
         
     JSValue getIndexQuickly(unsigned i)
     {
+        Butterfly* butterfly = m_butterfly.get(this);
         switch (indexingType()) {
         case ALL_INT32_INDEXING_TYPES:
-            return jsNumber(m_butterfly->contiguous()[i].get().asInt32());
+            return jsNumber(butterfly->contiguous()[i].get().asInt32());
         case ALL_CONTIGUOUS_INDEXING_TYPES:
-            return m_butterfly->contiguous()[i].get();
+            return butterfly->contiguous()[i].get();
         case ALL_DOUBLE_INDEXING_TYPES:
-            return JSValue(JSValue::EncodeAsDouble, m_butterfly->contiguousDouble()[i]);
+            return JSValue(JSValue::EncodeAsDouble, butterfly->contiguousDouble()[i]);
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return m_butterfly->arrayStorage()->m_vector[i].get();
+            return butterfly->arrayStorage()->m_vector[i].get();
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return JSValue();
@@ -223,32 +225,33 @@ public:
         
     JSValue tryGetIndexQuickly(unsigned i) const
     {
+        Butterfly* butterfly = m_butterfly.get(this);
         switch (indexingType()) {
         case ALL_BLANK_INDEXING_TYPES:
         case ALL_UNDECIDED_INDEXING_TYPES:
             break;
         case ALL_INT32_INDEXING_TYPES:
-            if (i < m_butterfly->publicLength()) {
-                JSValue result = m_butterfly->contiguous()[i].get();
+            if (i < butterfly->publicLength()) {
+                JSValue result = butterfly->contiguous()[i].get();
                 ASSERT(result.isInt32() || !result);
                 return result;
             }
             break;
         case ALL_CONTIGUOUS_INDEXING_TYPES:
-            if (i < m_butterfly->publicLength())
-                return m_butterfly->contiguous()[i].get();
+            if (i < butterfly->publicLength())
+                return butterfly->contiguous()[i].get();
             break;
         case ALL_DOUBLE_INDEXING_TYPES: {
-            if (i >= m_butterfly->publicLength())
+            if (i >= butterfly->publicLength())
                 break;
-            double result = m_butterfly->contiguousDouble()[i];
+            double result = butterfly->contiguousDouble()[i];
             if (result != result)
                 break;
             return JSValue(JSValue::EncodeAsDouble, result);
         }
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            if (i < m_butterfly->arrayStorage()->vectorLength())
-                return m_butterfly->arrayStorage()->m_vector[i].get();
+            if (i < butterfly->arrayStorage()->vectorLength())
+                return butterfly->arrayStorage()->m_vector[i].get();
             break;
         default:
             RELEASE_ASSERT_NOT_REACHED();
@@ -276,6 +279,7 @@ public:
         
     bool canSetIndexQuickly(unsigned i)
     {
+        Butterfly* butterfly = m_butterfly.get(this);
         switch (indexingType()) {
         case ALL_BLANK_INDEXING_TYPES:
         case ALL_UNDECIDED_INDEXING_TYPES:
@@ -285,11 +289,11 @@ public:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
         case NonArrayWithArrayStorage:
         case ArrayWithArrayStorage:
-            return i < m_butterfly->vectorLength();
+            return i < butterfly->vectorLength();
         case NonArrayWithSlowPutArrayStorage:
         case ArrayWithSlowPutArrayStorage:
-            return i < m_butterfly->arrayStorage()->vectorLength()
-                && !!m_butterfly->arrayStorage()->m_vector[i];
+            return i < butterfly->arrayStorage()->vectorLength()
+                && !!butterfly->arrayStorage()->m_vector[i];
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -306,7 +310,7 @@ public:
         case ALL_DOUBLE_INDEXING_TYPES:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return i < m_butterfly->vectorLength();
+            return i < m_butterfly.get(this)->vectorLength();
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -315,9 +319,10 @@ public:
         
     void setIndexQuickly(VM& vm, unsigned i, JSValue v)
     {
+        Butterfly* butterfly = m_butterfly.get(this);
         switch (indexingType()) {
         case ALL_INT32_INDEXING_TYPES: {
-            ASSERT(i < m_butterfly->vectorLength());
+            ASSERT(i < butterfly->vectorLength());
             if (!v.isInt32()) {
                 convertInt32ToDoubleOrContiguousWhilePerformingSetIndex(vm, i, v);
                 return;
@@ -325,14 +330,14 @@ public:
             FALLTHROUGH;
         }
         case ALL_CONTIGUOUS_INDEXING_TYPES: {
-            ASSERT(i < m_butterfly->vectorLength());
-            m_butterfly->contiguous()[i].set(vm, this, v);
-            if (i >= m_butterfly->publicLength())
-                m_butterfly->setPublicLength(i + 1);
+            ASSERT(i < butterfly->vectorLength());
+            butterfly->contiguous()[i].set(vm, this, v);
+            if (i >= butterfly->publicLength())
+                butterfly->setPublicLength(i + 1);
             break;
         }
         case ALL_DOUBLE_INDEXING_TYPES: {
-            ASSERT(i < m_butterfly->vectorLength());
+            ASSERT(i < butterfly->vectorLength());
             if (!v.isNumber()) {
                 convertDoubleToContiguousWhilePerformingSetIndex(vm, i, v);
                 return;
@@ -342,13 +347,13 @@ public:
                 convertDoubleToContiguousWhilePerformingSetIndex(vm, i, v);
                 return;
             }
-            m_butterfly->contiguousDouble()[i] = value;
-            if (i >= m_butterfly->publicLength())
-                m_butterfly->setPublicLength(i + 1);
+            butterfly->contiguousDouble()[i] = value;
+            if (i >= butterfly->publicLength())
+                butterfly->setPublicLength(i + 1);
             break;
         }
         case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-            ArrayStorage* storage = m_butterfly->arrayStorage();
+            ArrayStorage* storage = butterfly->arrayStorage();
             WriteBarrier<Unknown>& x = storage->m_vector[i];
             JSValue old = x.get();
             x.set(vm, this, v);
@@ -371,14 +376,15 @@ public:
 
     void initializeIndex(VM& vm, unsigned i, JSValue v, IndexingType indexingType)
     {
+        Butterfly* butterfly = m_butterfly.get(this);
         switch (indexingType) {
         case ALL_UNDECIDED_INDEXING_TYPES: {
             setIndexQuicklyToUndecided(vm, i, v);
             break;
         }
         case ALL_INT32_INDEXING_TYPES: {
-            ASSERT(i < m_butterfly->publicLength());
-            ASSERT(i < m_butterfly->vectorLength());
+            ASSERT(i < butterfly->publicLength());
+            ASSERT(i < butterfly->vectorLength());
             if (!v.isInt32()) {
                 convertInt32ToDoubleOrContiguousWhilePerformingSetIndex(vm, i, v);
                 break;
@@ -386,14 +392,14 @@ public:
             FALLTHROUGH;
         }
         case ALL_CONTIGUOUS_INDEXING_TYPES: {
-            ASSERT(i < m_butterfly->publicLength());
-            ASSERT(i < m_butterfly->vectorLength());
-            m_butterfly->contiguous()[i].set(vm, this, v);
+            ASSERT(i < butterfly->publicLength());
+            ASSERT(i < butterfly->vectorLength());
+            butterfly->contiguous()[i].set(vm, this, v);
             break;
         }
         case ALL_DOUBLE_INDEXING_TYPES: {
-            ASSERT(i < m_butterfly->publicLength());
-            ASSERT(i < m_butterfly->vectorLength());
+            ASSERT(i < butterfly->publicLength());
+            ASSERT(i < butterfly->vectorLength());
             if (!v.isNumber()) {
                 convertDoubleToContiguousWhilePerformingSetIndex(vm, i, v);
                 return;
@@ -403,11 +409,11 @@ public:
                 convertDoubleToContiguousWhilePerformingSetIndex(vm, i, v);
                 return;
             }
-            m_butterfly->contiguousDouble()[i] = value;
+            butterfly->contiguousDouble()[i] = value;
             break;
         }
         case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-            ArrayStorage* storage = m_butterfly->arrayStorage();
+            ArrayStorage* storage = butterfly->arrayStorage();
             ASSERT(i < storage->length());
             ASSERT(i < storage->m_numValuesInVector);
             storage->m_vector[i].set(vm, this, v);
@@ -428,7 +434,7 @@ public:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
             return false;
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return !!m_butterfly->arrayStorage()->m_sparseMap;
+            return !!m_butterfly.get(this)->arrayStorage()->m_sparseMap;
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -445,7 +451,7 @@ public:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
             return false;
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return m_butterfly->arrayStorage()->inSparseMode();
+            return m_butterfly.get(this)->arrayStorage()->inSparseMode();
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -550,11 +556,11 @@ public:
         return inlineStorageUnsafe();
     }
         
-    const Butterfly* butterfly() const { return m_butterfly.get(); }
-    Butterfly* butterfly() { return m_butterfly.get(); }
+    const Butterfly* butterfly() const { return m_butterfly.get(this); }
+    Butterfly* butterfly() { return m_butterfly.get(this); }
         
-    ConstPropertyStorage outOfLineStorage() const { return m_butterfly->propertyStorage(); }
-    PropertyStorage outOfLineStorage() { return m_butterfly->propertyStorage(); }
+    ConstPropertyStorage outOfLineStorage() const { return m_butterfly.get(this)->propertyStorage(); }
+    PropertyStorage outOfLineStorage() { return m_butterfly.get(this)->propertyStorage(); }
 
     const WriteBarrierBase<Unknown>* locationForOffset(PropertyOffset offset) const
     {
@@ -656,7 +662,7 @@ public:
     ContiguousJSValues ensureInt32(VM& vm)
     {
         if (LIKELY(hasInt32(indexingType())))
-            return m_butterfly->contiguousInt32();
+            return m_butterfly.get(this)->contiguousInt32();
             
         return ensureInt32Slow(vm);
     }
@@ -668,7 +674,7 @@ public:
     ContiguousDoubles ensureDouble(VM& vm)
     {
         if (LIKELY(hasDouble(indexingType())))
-            return m_butterfly->contiguousDouble();
+            return m_butterfly.get(this)->contiguousDouble();
             
         return ensureDoubleSlow(vm);
     }
@@ -678,7 +684,7 @@ public:
     ContiguousJSValues ensureContiguous(VM& vm)
     {
         if (LIKELY(hasContiguous(indexingType())))
-            return m_butterfly->contiguous();
+            return m_butterfly.get(this)->contiguous();
             
         return ensureContiguousSlow(vm);
     }
@@ -690,7 +696,7 @@ public:
     ArrayStorage* ensureArrayStorage(VM& vm)
     {
         if (LIKELY(hasAnyArrayStorage(indexingType())))
-            return m_butterfly->arrayStorage();
+            return m_butterfly.get(this)->arrayStorage();
 
         return ensureArrayStorageSlow(vm);
     }
@@ -736,7 +742,7 @@ protected:
     ArrayStorage* arrayStorage()
     {
         ASSERT(hasAnyArrayStorage(indexingType()));
-        return m_butterfly->arrayStorage();
+        return m_butterfly.get(this)->arrayStorage();
     }
         
     // Call this if you want to predicate some actions on whether or not the
@@ -745,7 +751,7 @@ protected:
     {
         switch (indexingType()) {
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return m_butterfly->arrayStorage();
+            return m_butterfly.get(this)->arrayStorage();
                 
         default:
             return 0;
@@ -809,12 +815,12 @@ protected:
     {
         ASSERT(length < MAX_ARRAY_INDEX);
         ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
-            
-        if (m_butterfly->vectorLength() < length)
+
+        if (m_butterfly.get(this)->vectorLength() < length)
             ensureLengthSlow(vm, length);
             
-        if (m_butterfly->publicLength() < length)
-            m_butterfly->setPublicLength(length);
+        if (m_butterfly.get(this)->publicLength() < length)
+            m_butterfly.get(this)->setPublicLength(length);
     }
         
     // Call this if you want to shrink the butterfly backing store, and you're
@@ -874,7 +880,7 @@ private:
     JS_EXPORT_PRIVATE ArrayStorage* ensureArrayStorageSlow(VM&);
 
 protected:
-    CopyWriteBarrier<Butterfly> m_butterfly;
+    CopyBarrier<Butterfly> m_butterfly;
 #if USE(JSVALUE32_64)
 private:
     uint32_t m_padding;
@@ -1348,7 +1354,7 @@ inline void JSObject::putDirectWithoutTransition(VM& vm, PropertyName propertyNa
     DeferGC deferGC(vm.heap);
     ASSERT(!value.isGetterSetter() && !(attributes & Accessor));
     ASSERT(!value.isCustomGetterSetter());
-    Butterfly* newButterfly = m_butterfly.get();
+    Butterfly* newButterfly = m_butterfly.get(this);
     if (structure()->putWillGrowOutOfLineStorage())
         newButterfly = growOutOfLineStorage(vm, structure()->outOfLineCapacity(), structure()->suggestedNewOutOfLineStorageCapacity());
     Structure* structure = this->structure();
index e55362e..61bd919 100644 (file)
@@ -112,7 +112,7 @@ public:
 private:
     JSSet(VM& vm, Structure* structure)
         : Base(vm, structure)
-        , m_setData(vm)
+        , m_setData(vm, this)
     {
     }
 
index 3b5863e..c308a03 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2013 Apple Inc. All rights reserved.
+ * Copyright (C) 2013, 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -26,6 +26,7 @@
 #ifndef MapData_h
 #define MapData_h
 
+#include "CopyBarrier.h"
 #include "JSCell.h"
 #include "WeakGCMapInlines.h"
 #include <wtf/HashFunctions.h>
@@ -94,7 +95,7 @@ public:
         JSValue value;
     };
 
-    MapDataImpl(VM&);
+    MapDataImpl(VM&, JSCell* owner);
 
     void set(ExecState*, JSCell* owner, KeyType, JSValue);
     JSValue get(ExecState*, KeyType);
@@ -136,16 +137,17 @@ private:
     int32_t m_capacity;
     int32_t m_size;
     int32_t m_deletedCount;
-    Entry* m_entries;
+    JSCell* m_owner;
+    CopyBarrier<Entry> m_entries;
     WeakGCMap<JSIterator*, JSIterator> m_iterators;
 };
 
 template<typename Entry, typename JSIterator>
-ALWAYS_INLINE MapDataImpl<Entry, JSIterator>::MapDataImpl(VM& vm)
+ALWAYS_INLINE MapDataImpl<Entry, JSIterator>::MapDataImpl(VM& vm, JSCell* owner)
     : m_capacity(0)
     , m_size(0)
     , m_deletedCount(0)
-    , m_entries(nullptr)
+    , m_owner(owner)
     , m_iterators(vm)
 {
 }
@@ -182,7 +184,7 @@ ALWAYS_INLINE bool MapDataImpl<Entry, JSIterator>::IteratorData::next(WTF::KeyVa
 {
     if (!ensureSlot())
         return false;
-    Entry* entry = &m_mapData->m_entries[m_index];
+    Entry* entry = &m_mapData->m_entries.get(m_mapData->m_owner)[m_index];
     pair = WTF::KeyValuePair<JSValue, JSValue>(entry->key().get(), entry->value().get());
     m_index += 1;
     return true;
@@ -204,7 +206,7 @@ ALWAYS_INLINE int32_t MapDataImpl<Entry, JSIterator>::IteratorData::refreshCurso
     if (isFinished())
         return m_index;
 
-    Entry* entries = m_mapData->m_entries;
+    Entry* entries = m_mapData->m_entries.get(m_mapData->m_owner);
     size_t end = m_mapData->m_size;
     while (static_cast<size_t>(m_index) < end && !entries[m_index].key())
         m_index++;
index 176acc1..5f006ce 100644 (file)
@@ -47,7 +47,7 @@ inline void MapDataImpl<Entry, JSIterator>::clear()
     m_capacity = 0;
     m_size = 0;
     m_deletedCount = 0;
-    m_entries = nullptr;
+    m_entries.clear();
     m_iterators.forEach([](JSIterator* iterator, JSIterator*) {
         iterator->iteratorData()->didRemoveAllEntries();
     });
@@ -60,25 +60,25 @@ inline Entry* MapDataImpl<Entry, JSIterator>::find(ExecState* exec, KeyType key)
         auto iter = m_stringKeyedTable.find(asString(key.value)->value(exec).impl());
         if (iter == m_stringKeyedTable.end())
             return 0;
-        return &m_entries[iter->value];
+        return &m_entries.get(m_owner)[iter->value];
     }
     if (key.value.isSymbol()) {
         auto iter = m_symbolKeyedTable.find(asSymbol(key.value)->privateName().uid());
         if (iter == m_symbolKeyedTable.end())
             return 0;
-        return &m_entries[iter->value];
+        return &m_entries.get(m_owner)[iter->value];
     }
     if (key.value.isCell()) {
         auto iter = m_cellKeyedTable.find(key.value.asCell());
         if (iter == m_cellKeyedTable.end())
             return 0;
-        return &m_entries[iter->value];
+        return &m_entries.get(m_owner)[iter->value];
     }
 
     auto iter = m_valueKeyedTable.find(JSValue::encode(key.value));
     if (iter == m_valueKeyedTable.end())
         return 0;
-    return &m_entries[iter->value];
+    return &m_entries.get(m_owner)[iter->value];
 }
 
 template<typename Entry, typename JSIterator>
@@ -93,14 +93,14 @@ inline Entry* MapDataImpl<Entry, JSIterator>::add(ExecState* exec, JSCell* owner
 {
     typename Map::iterator location = map.find(key);
     if (location != map.end())
-        return &m_entries[location->value];
+        return &m_entries.get(m_owner)[location->value];
 
     if (!ensureSpaceForAppend(exec, owner))
         return 0;
 
     auto result = map.add(key, m_size);
     RELEASE_ASSERT(result.isNewEntry);
-    Entry* entry = &m_entries[m_size++];
+    Entry* entry = &m_entries.get(m_owner)[m_size++];
     new (entry) Entry();
     entry->setKey(exec->vm(), owner, keyValue.value);
     return entry;
@@ -164,7 +164,7 @@ inline bool MapDataImpl<Entry, JSIterator>::remove(ExecState* exec, KeyType key)
         location = iter->value;
         m_valueKeyedTable.remove(iter);
     }
-    m_entries[location].clear();
+    m_entries.get(m_owner)[location].clear();
     m_deletedCount++;
     return true;
 }
@@ -176,7 +176,7 @@ inline void MapDataImpl<Entry, JSIterator>::replaceAndPackBackingStore(Entry* de
     int32_t newEnd = 0;
     RELEASE_ASSERT(newCapacity > 0);
     for (int32_t i = 0; i < m_size; i++) {
-        Entry& entry = m_entries[i];
+        Entry& entry = m_entries.getWithoutBarrier()[i];
         if (!entry.key()) {
             m_iterators.forEach([newEnd](JSIterator* iterator, JSIterator*) {
                 iterator->iteratorData()->didRemoveEntry(newEnd);
@@ -195,20 +195,20 @@ inline void MapDataImpl<Entry, JSIterator>::replaceAndPackBackingStore(Entry* de
 
     // Fixup for the hashmaps
     for (auto ptr = m_valueKeyedTable.begin(); ptr != m_valueKeyedTable.end(); ++ptr)
-        ptr->value = m_entries[ptr->value].key().get().asInt32();
+        ptr->value = m_entries.getWithoutBarrier()[ptr->value].key().get().asInt32();
     for (auto ptr = m_cellKeyedTable.begin(); ptr != m_cellKeyedTable.end(); ++ptr)
-        ptr->value = m_entries[ptr->value].key().get().asInt32();
+        ptr->value = m_entries.getWithoutBarrier()[ptr->value].key().get().asInt32();
     for (auto ptr = m_stringKeyedTable.begin(); ptr != m_stringKeyedTable.end(); ++ptr)
-        ptr->value = m_entries[ptr->value].key().get().asInt32();
+        ptr->value = m_entries.getWithoutBarrier()[ptr->value].key().get().asInt32();
     for (auto ptr = m_symbolKeyedTable.begin(); ptr != m_symbolKeyedTable.end(); ++ptr)
-        ptr->value = m_entries[ptr->value].key().get().asInt32();
+        ptr->value = m_entries.getWithoutBarrier()[ptr->value].key().get().asInt32();
 
     ASSERT((m_size - newEnd) == m_deletedCount);
     m_deletedCount = 0;
 
     m_capacity = newCapacity;
     m_size = newEnd;
-    m_entries = destination;
+    m_entries.setWithoutBarrier(destination);
 }
 
 template<typename Entry, typename JSIterator>
@@ -217,9 +217,9 @@ inline void MapDataImpl<Entry, JSIterator>::replaceBackingStore(Entry* destinati
     ASSERT(!shouldPack());
     RELEASE_ASSERT(newCapacity > 0);
     ASSERT(newCapacity >= m_capacity);
-    memcpy(destination, m_entries, sizeof(Entry) * m_size);
+    memcpy(destination, m_entries.getWithoutBarrier(), sizeof(Entry) * m_size);
     m_capacity = newCapacity;
-    m_entries = destination;
+    m_entries.setWithoutBarrier(destination);
 }
 
 template<typename Entry, typename JSIterator>
@@ -236,6 +236,8 @@ inline CheckedBoolean MapDataImpl<Entry, JSIterator>::ensureSpaceForAppend(ExecS
         return false;
     }
     Entry* newEntries = static_cast<Entry*>(newStorage);
+    // Do a read barrier to ensure that m_entries points to to-space for the remainder of this GC epoch.
+    m_entries.get(m_owner); 
     if (shouldPack())
         replaceAndPackBackingStore(newEntries, requiredSize);
     else
@@ -247,7 +249,7 @@ inline CheckedBoolean MapDataImpl<Entry, JSIterator>::ensureSpaceForAppend(ExecS
 template<typename Entry, typename JSIterator>
 inline void MapDataImpl<Entry, JSIterator>::visitChildren(JSCell* owner, SlotVisitor& visitor)
 {
-    Entry* entries = m_entries;
+    Entry* entries = m_entries.getWithoutBarrier();
     if (!entries)
         return;
     if (m_deletedCount) {
@@ -261,14 +263,14 @@ inline void MapDataImpl<Entry, JSIterator>::visitChildren(JSCell* owner, SlotVis
         visitor.appendValues(reinterpret_cast<WriteBarrier<Unknown>*>(&entries[0]), m_size * (sizeof(Entry) / sizeof(WriteBarrier<Unknown>)));
     }
 
-    visitor.copyLater(owner, MapBackingStoreCopyToken, entries, capacityInBytes());
+    visitor.copyLater(owner, MapBackingStoreCopyToken, m_entries.getWithoutBarrier(), capacityInBytes());
 }
 
 template<typename Entry, typename JSIterator>
 inline void MapDataImpl<Entry, JSIterator>::copyBackingStore(CopyVisitor& visitor, CopyToken token)
 {
-    if (token == MapBackingStoreCopyToken && visitor.checkIfShouldCopy(m_entries)) {
-        Entry* oldEntries = m_entries;
+    if (token == MapBackingStoreCopyToken && visitor.checkIfShouldCopy(m_entries.getWithoutBarrier())) {
+        Entry* oldEntries = m_entries.getWithoutBarrier();
         Entry* newEntries = static_cast<Entry*>(visitor.allocateNewSpace(capacityInBytes()));
         if (shouldPack())
             replaceAndPackBackingStore(newEntries, m_capacity);
index b17f492..be4f228 100644 (file)
@@ -203,6 +203,7 @@ typedef const char* optionString;
     v(bool, createPreHeaders, true, nullptr) \
     v(bool, enableMovHintRemoval, true, nullptr) \
     v(bool, enableObjectAllocationSinking, true, nullptr) \
+    v(bool, enableCopyBarrierOptimization, true, nullptr) \
     \
     v(bool, enableConcurrentJIT, true, "allows the DFG / FTL compilation in threads other than the executing JS thread") \
     v(unsigned, numberOfDFGCompilerThreads, computeNumberOfWorkerThreads(2, 2) - 1, nullptr) \