Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 7 Aug 2017 21:31:49 +0000 (21:31 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 7 Aug 2017 21:31:49 +0000 (21:31 +0000)
https://bugs.webkit.org/show_bug.cgi?id=174919

Reviewed by Keith Miller.
Source/bmalloc:

This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of
HeapKind, PrimitiveGigacage and JSValueGigacage.

The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and
NO_INLINE into BNO_INLINE.

* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::refillAllocator):
(bmalloc::Allocator::allocateLarge):
(bmalloc::Allocator::allocateLogSizeClass):
* bmalloc/AsyncTask.h:
* bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h.
* bmalloc/Cache.cpp:
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
* bmalloc/Deallocator.cpp:
* bmalloc/Gigacage.cpp:
(Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::ensureGigacage):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::Callbacks::Callbacks): Deleted.
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.
* bmalloc/Gigacage.h:
(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::forEachKind):
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::gigacageBasePtr):
* bmalloc/Heap.h:
* bmalloc/HeapKind.h:
(bmalloc::isGigacage):
(bmalloc::gigacageKind):
(bmalloc::heapKind):
* bmalloc/Inline.h: Removed.
* bmalloc/Map.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
(bmalloc::PerProcess<T>::get):
(bmalloc::PerProcess<T>::getSlowCase):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::getFastCase):
* bmalloc/Vector.h:
(bmalloc::Vector<T>::push):
(bmalloc::Vector<T>::shrinkCapacity):
(bmalloc::Vector<T>::growCapacity):

Source/JavaScriptCore:

This adapts JSC to there being two gigacages.

To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than
singletons. I don't think we were gaining anything by making them be singletons.

This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of
gigacages. We'll have one of those allocators per cage.

From there, this change teaches everyone who previously knew about cages that there are two cages.
This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is
easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's
not so obvious, so this change introduces some helpers to make it easy to define what cage you want
to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h

A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and
CagedPtr. This removes one layer of "get()" calls from a bunch of places.

* JavaScriptCore.xcodeproj/project.pbxproj:
* bytecode/AccessCase.cpp:
(JSC::AccessCase::generateImpl):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileNewTypedArray):
(JSC::DFG::SpeculativeJIT::emitAllocateButterfly):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/FastMallocAlignedMemoryAllocator.cpp:
(JSC::FastMallocAlignedMemoryAllocator::instance): Deleted.
* heap/FastMallocAlignedMemoryAllocator.h:
* heap/GigacageAlignedMemoryAllocator.cpp:
(JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator):
(JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::dump const):
(JSC::GigacageAlignedMemoryAllocator::instance): Deleted.
* heap/GigacageAlignedMemoryAllocator.h:
* jsc.cpp:
(primitiveGigacageDisabled):
(jscmain):
(gigacageDisabled): Deleted.
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
* runtime/AuxiliaryBarrier.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::tryCreate):
(JSC::Butterfly::growArrayRight):
* runtime/CagedBarrierPtr.h: Added.
(JSC::CagedBarrierPtr::CagedBarrierPtr):
(JSC::CagedBarrierPtr::clear):
(JSC::CagedBarrierPtr::set):
(JSC::CagedBarrierPtr::get const):
(JSC::CagedBarrierPtr::getMayBeNull const):
(JSC::CagedBarrierPtr::operator== const):
(JSC::CagedBarrierPtr::operator!= const):
(JSC::CagedBarrierPtr::operator bool const):
(JSC::CagedBarrierPtr::setWithoutBarrier):
(JSC::CagedBarrierPtr::operator* const):
(JSC::CagedBarrierPtr::operator-> const):
(JSC::CagedBarrierPtr::operator[] const):
* runtime/DirectArguments.cpp:
(JSC::DirectArguments::overrideThings):
(JSC::DirectArguments::unmapArgument):
* runtime/DirectArguments.h:
(JSC::DirectArguments::isMappedArgument const):
* runtime/GenericArguments.h:
* runtime/GenericArgumentsInlines.h:
(JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
(JSC::GenericArguments<Type>::setModifiedArgumentDescriptor):
(JSC::GenericArguments<Type>::isModifiedArgumentDescriptor):
* runtime/HashMapImpl.cpp:
(JSC::HashMapImpl<HashMapBucket>::visitChildren):
* runtime/HashMapImpl.h:
(JSC::HashMapBuffer::create):
(JSC::HashMapImpl::buffer const):
(JSC::HashMapImpl::rehash):
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitializedRestricted):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithAnyIndexingType):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::JSArray::tryCreate):
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.cpp:
(JSC::JSObject::heapSnapshot):
(JSC::JSObject::getOwnPropertySlotByIndex):
(JSC::JSObject::putByIndex):
(JSC::JSObject::enterDictionaryIndexingMode):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToDouble):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToContiguous):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToContiguous):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::setIndexQuicklyToUndecided):
(JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
(JSC::JSObject::deletePropertyByIndex):
(JSC::JSObject::getOwnPropertyNames):
(JSC::JSObject::putIndexedDescriptor):
(JSC::JSObject::defineOwnIndexedProperty):
(JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
(JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::reallocateAndShrinkButterfly):
(JSC::JSObject::allocateMoreOutOfLineStorage):
(JSC::JSObject::getEnumerableLength):
* runtime/JSObject.h:
(JSC::JSObject::getArrayLength const):
(JSC::JSObject::getVectorLength):
(JSC::JSObject::putDirectIndex):
(JSC::JSObject::canGetIndexQuickly):
(JSC::JSObject::getIndexQuickly):
(JSC::JSObject::tryGetIndexQuickly const):
(JSC::JSObject::canSetIndexQuickly):
(JSC::JSObject::setIndexQuickly):
(JSC::JSObject::initializeIndex):
(JSC::JSObject::initializeIndexWithoutBarrier):
(JSC::JSObject::hasSparseMap):
(JSC::JSObject::inSparseIndexingMode):
(JSC::JSObject::butterfly const):
(JSC::JSObject::butterfly):
(JSC::JSObject::outOfLineStorage const):
(JSC::JSObject::outOfLineStorage):
(JSC::JSObject::ensureInt32):
(JSC::JSObject::ensureDouble):
(JSC::JSObject::ensureContiguous):
(JSC::JSObject::ensureArrayStorage):
(JSC::JSObject::arrayStorage):
(JSC::JSObject::arrayStorageOrNull):
(JSC::JSObject::ensureLength):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::primitiveGigacageDisabledCallback):
(JSC::VM::primitiveGigacageDisabled):
(JSC::VM::gigacageDisabledCallback): Deleted.
(JSC::VM::gigacageDisabled): Deleted.
* runtime/VM.h:
(JSC::VM::gigacageAuxiliarySpace):
(JSC::VM::firePrimitiveGigacageEnabledIfNecessary):
(JSC::VM::primitiveGigacageEnabled):
(JSC::VM::fireGigacageEnabledIfNecessary): Deleted.
(JSC::VM::gigacageEnabled): Deleted.
* wasm/WasmMemory.cpp:
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::grow):

Source/WebCore:

No new tests because no change in behavior.

Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be
specific about which Gigacage we're using.

* bindings/js/WebCoreJSClientData.cpp:
(WebCore::JSVMClientData::JSVMClientData):
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):

Source/WebKit:

The disable callback is all about the primitive gigacage.

* WebProcess/WebProcess.cpp:
(WebKit::primitiveGigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
(WebKit::gigacageDisabled): Deleted.

Source/WTF:

This mirrors the changes from bmalloc/Gigacage.h.

Also it teaches CagedPtr how to reason about multiple gigacages.

* wtf/CagedPtr.h:
(WTF::CagedPtr::get const):
(WTF::CagedPtr::operator[] const):
* wtf/Gigacage.cpp:
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h:
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc

56 files changed:
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/bytecode/AccessCase.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.cpp
Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.h
Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.cpp
Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.h
Source/JavaScriptCore/jsc.cpp
Source/JavaScriptCore/llint/LowLevelInterpreter64.asm
Source/JavaScriptCore/runtime/ArrayBuffer.cpp
Source/JavaScriptCore/runtime/AuxiliaryBarrier.h
Source/JavaScriptCore/runtime/ButterflyInlines.h
Source/JavaScriptCore/runtime/CagedBarrierPtr.h [new file with mode: 0644]
Source/JavaScriptCore/runtime/DirectArguments.cpp
Source/JavaScriptCore/runtime/DirectArguments.h
Source/JavaScriptCore/runtime/GenericArguments.h
Source/JavaScriptCore/runtime/GenericArgumentsInlines.h
Source/JavaScriptCore/runtime/HashMapImpl.cpp
Source/JavaScriptCore/runtime/HashMapImpl.h
Source/JavaScriptCore/runtime/JSArray.cpp
Source/JavaScriptCore/runtime/JSArray.h
Source/JavaScriptCore/runtime/JSArrayBufferView.cpp
Source/JavaScriptCore/runtime/JSLock.cpp
Source/JavaScriptCore/runtime/JSObject.cpp
Source/JavaScriptCore/runtime/JSObject.h
Source/JavaScriptCore/runtime/RegExpMatchesArray.h
Source/JavaScriptCore/runtime/VM.cpp
Source/JavaScriptCore/runtime/VM.h
Source/JavaScriptCore/wasm/WasmMemory.cpp
Source/WTF/ChangeLog
Source/WTF/wtf/CagedPtr.h
Source/WTF/wtf/Gigacage.cpp
Source/WTF/wtf/Gigacage.h
Source/WebCore/ChangeLog
Source/WebCore/bindings/js/WebCoreJSClientData.cpp
Source/WebCore/platform/graphics/cocoa/GPUBufferMetal.mm
Source/WebKit/ChangeLog
Source/WebKit/WebProcess/WebProcess.cpp
Source/bmalloc/ChangeLog
Source/bmalloc/bmalloc.xcodeproj/project.pbxproj
Source/bmalloc/bmalloc/Allocator.cpp
Source/bmalloc/bmalloc/AsyncTask.h
Source/bmalloc/bmalloc/BInline.h [moved from Source/bmalloc/bmalloc/Inline.h with 89% similarity]
Source/bmalloc/bmalloc/Cache.cpp
Source/bmalloc/bmalloc/Deallocator.cpp
Source/bmalloc/bmalloc/Gigacage.cpp
Source/bmalloc/bmalloc/Gigacage.h
Source/bmalloc/bmalloc/Heap.cpp
Source/bmalloc/bmalloc/Heap.h
Source/bmalloc/bmalloc/HeapKind.h
Source/bmalloc/bmalloc/Map.h
Source/bmalloc/bmalloc/PerProcess.h
Source/bmalloc/bmalloc/PerThread.h
Source/bmalloc/bmalloc/Vector.h

index a9b2c2e9011f2134a2fae1ebd37b86b60902de20..1f01970b939642e22191d9aaed23949cb5b253f8 100644 (file)
@@ -1,3 +1,192 @@
+2017-08-06  Filip Pizlo  <fpizlo@apple.com>
+
+        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
+        https://bugs.webkit.org/show_bug.cgi?id=174919
+
+        Reviewed by Keith Miller.
+        
+        This adapts JSC to there being two gigacages.
+        
+        To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than
+        singletons. I don't think we were gaining anything by making them be singletons.
+        
+        This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of
+        gigacages. We'll have one of those allocators per cage.
+        
+        From there, this change teaches everyone who previously knew about cages that there are two cages.
+        This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is
+        easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's
+        not so obvious, so this change introduces some helpers to make it easy to define what cage you want
+        to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h
+        
+        A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and
+        CagedPtr. This removes one layer of "get()" calls from a bunch of places.
+
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * bytecode/AccessCase.cpp:
+        (JSC::AccessCase::generateImpl):
+        * dfg/DFGSpeculativeJIT.cpp:
+        (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
+        (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
+        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
+        (JSC::DFG::SpeculativeJIT::compileNewTypedArray):
+        (JSC::DFG::SpeculativeJIT::emitAllocateButterfly):
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
+        (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
+        (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
+        (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
+        (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
+        (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
+        (JSC::FTL::DFG::LowerDFGToB3::caged):
+        * heap/FastMallocAlignedMemoryAllocator.cpp:
+        (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted.
+        * heap/FastMallocAlignedMemoryAllocator.h:
+        * heap/GigacageAlignedMemoryAllocator.cpp:
+        (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator):
+        (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory):
+        (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory):
+        (JSC::GigacageAlignedMemoryAllocator::dump const):
+        (JSC::GigacageAlignedMemoryAllocator::instance): Deleted.
+        * heap/GigacageAlignedMemoryAllocator.h:
+        * jsc.cpp:
+        (primitiveGigacageDisabled):
+        (jscmain):
+        (gigacageDisabled): Deleted.
+        * llint/LowLevelInterpreter64.asm:
+        * runtime/ArrayBuffer.cpp:
+        (JSC::ArrayBufferContents::tryAllocate):
+        (JSC::ArrayBuffer::createAdopted):
+        (JSC::ArrayBuffer::createFromBytes):
+        * runtime/AuxiliaryBarrier.h:
+        * runtime/ButterflyInlines.h:
+        (JSC::Butterfly::createUninitialized):
+        (JSC::Butterfly::tryCreate):
+        (JSC::Butterfly::growArrayRight):
+        * runtime/CagedBarrierPtr.h: Added.
+        (JSC::CagedBarrierPtr::CagedBarrierPtr):
+        (JSC::CagedBarrierPtr::clear):
+        (JSC::CagedBarrierPtr::set):
+        (JSC::CagedBarrierPtr::get const):
+        (JSC::CagedBarrierPtr::getMayBeNull const):
+        (JSC::CagedBarrierPtr::operator== const):
+        (JSC::CagedBarrierPtr::operator!= const):
+        (JSC::CagedBarrierPtr::operator bool const):
+        (JSC::CagedBarrierPtr::setWithoutBarrier):
+        (JSC::CagedBarrierPtr::operator* const):
+        (JSC::CagedBarrierPtr::operator-> const):
+        (JSC::CagedBarrierPtr::operator[] const):
+        * runtime/DirectArguments.cpp:
+        (JSC::DirectArguments::overrideThings):
+        (JSC::DirectArguments::unmapArgument):
+        * runtime/DirectArguments.h:
+        (JSC::DirectArguments::isMappedArgument const):
+        * runtime/GenericArguments.h:
+        * runtime/GenericArgumentsInlines.h:
+        (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
+        (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor):
+        (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor):
+        * runtime/HashMapImpl.cpp:
+        (JSC::HashMapImpl<HashMapBucket>::visitChildren):
+        * runtime/HashMapImpl.h:
+        (JSC::HashMapBuffer::create):
+        (JSC::HashMapImpl::buffer const):
+        (JSC::HashMapImpl::rehash):
+        * runtime/JSArray.cpp:
+        (JSC::JSArray::tryCreateUninitializedRestricted):
+        (JSC::JSArray::unshiftCountSlowCase):
+        (JSC::JSArray::setLength):
+        (JSC::JSArray::pop):
+        (JSC::JSArray::push):
+        (JSC::JSArray::fastSlice):
+        (JSC::JSArray::shiftCountWithArrayStorage):
+        (JSC::JSArray::shiftCountWithAnyIndexingType):
+        (JSC::JSArray::unshiftCountWithAnyIndexingType):
+        (JSC::JSArray::fillArgList):
+        (JSC::JSArray::copyToArguments):
+        * runtime/JSArray.h:
+        (JSC::JSArray::tryCreate):
+        * runtime/JSArrayBufferView.cpp:
+        (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
+        (JSC::JSArrayBufferView::finalize):
+        * runtime/JSLock.cpp:
+        (JSC::JSLock::didAcquireLock):
+        * runtime/JSObject.cpp:
+        (JSC::JSObject::heapSnapshot):
+        (JSC::JSObject::getOwnPropertySlotByIndex):
+        (JSC::JSObject::putByIndex):
+        (JSC::JSObject::enterDictionaryIndexingMode):
+        (JSC::JSObject::createInitialIndexedStorage):
+        (JSC::JSObject::createArrayStorage):
+        (JSC::JSObject::convertUndecidedToInt32):
+        (JSC::JSObject::convertUndecidedToDouble):
+        (JSC::JSObject::convertUndecidedToContiguous):
+        (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
+        (JSC::JSObject::convertUndecidedToArrayStorage):
+        (JSC::JSObject::convertInt32ToDouble):
+        (JSC::JSObject::convertInt32ToContiguous):
+        (JSC::JSObject::convertInt32ToArrayStorage):
+        (JSC::JSObject::convertDoubleToContiguous):
+        (JSC::JSObject::convertDoubleToArrayStorage):
+        (JSC::JSObject::convertContiguousToArrayStorage):
+        (JSC::JSObject::setIndexQuicklyToUndecided):
+        (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
+        (JSC::JSObject::deletePropertyByIndex):
+        (JSC::JSObject::getOwnPropertyNames):
+        (JSC::JSObject::putIndexedDescriptor):
+        (JSC::JSObject::defineOwnIndexedProperty):
+        (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
+        (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength):
+        (JSC::JSObject::getNewVectorLength):
+        (JSC::JSObject::ensureLengthSlow):
+        (JSC::JSObject::reallocateAndShrinkButterfly):
+        (JSC::JSObject::allocateMoreOutOfLineStorage):
+        (JSC::JSObject::getEnumerableLength):
+        * runtime/JSObject.h:
+        (JSC::JSObject::getArrayLength const):
+        (JSC::JSObject::getVectorLength):
+        (JSC::JSObject::putDirectIndex):
+        (JSC::JSObject::canGetIndexQuickly):
+        (JSC::JSObject::getIndexQuickly):
+        (JSC::JSObject::tryGetIndexQuickly const):
+        (JSC::JSObject::canSetIndexQuickly):
+        (JSC::JSObject::setIndexQuickly):
+        (JSC::JSObject::initializeIndex):
+        (JSC::JSObject::initializeIndexWithoutBarrier):
+        (JSC::JSObject::hasSparseMap):
+        (JSC::JSObject::inSparseIndexingMode):
+        (JSC::JSObject::butterfly const):
+        (JSC::JSObject::butterfly):
+        (JSC::JSObject::outOfLineStorage const):
+        (JSC::JSObject::outOfLineStorage):
+        (JSC::JSObject::ensureInt32):
+        (JSC::JSObject::ensureDouble):
+        (JSC::JSObject::ensureContiguous):
+        (JSC::JSObject::ensureArrayStorage):
+        (JSC::JSObject::arrayStorage):
+        (JSC::JSObject::arrayStorageOrNull):
+        (JSC::JSObject::ensureLength):
+        * runtime/RegExpMatchesArray.h:
+        (JSC::tryCreateUninitializedRegExpMatchesArray):
+        * runtime/VM.cpp:
+        (JSC::VM::VM):
+        (JSC::VM::~VM):
+        (JSC::VM::primitiveGigacageDisabledCallback):
+        (JSC::VM::primitiveGigacageDisabled):
+        (JSC::VM::gigacageDisabledCallback): Deleted.
+        (JSC::VM::gigacageDisabled): Deleted.
+        * runtime/VM.h:
+        (JSC::VM::gigacageAuxiliarySpace):
+        (JSC::VM::firePrimitiveGigacageEnabledIfNecessary):
+        (JSC::VM::primitiveGigacageEnabled):
+        (JSC::VM::fireGigacageEnabledIfNecessary): Deleted.
+        (JSC::VM::gigacageEnabled): Deleted.
+        * wasm/WasmMemory.cpp:
+        (JSC::Wasm::Memory::create):
+        (JSC::Wasm::Memory::~Memory):
+        (JSC::Wasm::Memory::grow):
+
 2017-08-07  Commit Queue  <commit-queue@webkit.org>
 
         Unreviewed, rolling out r220144.
index 9b59b2309a10e0591d5ec9800d1b81729e4c2b30..f9847e792327385bd2dd80398a7697b7ec7fec3e 100644 (file)
                0FEC3C571F33A45300F59B6C /* FastMallocAlignedMemoryAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C551F33A45300F59B6C /* FastMallocAlignedMemoryAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FEC3C5A1F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC3C581F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp */; };
                0FEC3C5B1F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C591F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h */; };
+               0FEC3C601F379F5300F59B6C /* CagedBarrierPtr.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FEC84FE1BDACDAC0080FF74 /* B3ArgumentRegValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC84B41BDACDAC0080FF74 /* B3ArgumentRegValue.cpp */; };
                0FEC84FF1BDACDAC0080FF74 /* B3ArgumentRegValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FEC84B51BDACDAC0080FF74 /* B3ArgumentRegValue.h */; };
                0FEC85001BDACDAC0080FF74 /* B3BasicBlock.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FEC84B61BDACDAC0080FF74 /* B3BasicBlock.cpp */; };
                0FEC3C551F33A45300F59B6C /* FastMallocAlignedMemoryAllocator.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = FastMallocAlignedMemoryAllocator.h; sourceTree = "<group>"; };
                0FEC3C581F33A48900F59B6C /* GigacageAlignedMemoryAllocator.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = GigacageAlignedMemoryAllocator.cpp; sourceTree = "<group>"; };
                0FEC3C591F33A48900F59B6C /* GigacageAlignedMemoryAllocator.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = GigacageAlignedMemoryAllocator.h; sourceTree = "<group>"; };
+               0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = CagedBarrierPtr.h; sourceTree = "<group>"; };
                0FEC84B41BDACDAC0080FF74 /* B3ArgumentRegValue.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = B3ArgumentRegValue.cpp; path = b3/B3ArgumentRegValue.cpp; sourceTree = "<group>"; };
                0FEC84B51BDACDAC0080FF74 /* B3ArgumentRegValue.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3ArgumentRegValue.h; path = b3/B3ArgumentRegValue.h; sourceTree = "<group>"; };
                0FEC84B61BDACDAC0080FF74 /* B3BasicBlock.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = B3BasicBlock.cpp; path = b3/B3BasicBlock.cpp; sourceTree = "<group>"; };
                                9E729409190F0306001A91B5 /* BundlePath.mm */,
                                0FB7F38B15ED8E3800F167B2 /* Butterfly.h */,
                                0FB7F38C15ED8E3800F167B2 /* ButterflyInlines.h */,
+                               0FEC3C5F1F379F5300F59B6C /* CagedBarrierPtr.h */,
                                BCA62DFE0E2826230004F30D /* CallData.cpp */,
                                145C507F0D9DF63B0088F6B9 /* CallData.h */,
                                FE80C1981D775FB4008510C0 /* CatchScope.cpp */,
                                0FB14E2318130955009B6B4D /* DFGInlineCacheWrapperInlines.h in Headers */,
                                A704D90617A0BAA8006BA554 /* DFGInPlaceAbstractState.h in Headers */,
                                0F2BDC21151E803B00CD8910 /* DFGInsertionSet.h in Headers */,
+                               0FEC3C601F379F5300F59B6C /* CagedBarrierPtr.h in Headers */,
                                0F300B7C18AB1B1400A6D72E /* DFGIntegerCheckCombiningPhase.h in Headers */,
                                0F898F321B27689F0083A33C /* DFGIntegerRangeOptimizationPhase.h in Headers */,
                                0FC97F3E18202119002C9B26 /* DFGInvalidationPointInjectionPhase.h in Headers */,
index 89ac5be9bccff0c015e05499d395ecc7105afffd..675f09d8521f1cdf57b1ee1a6c123db6a2950bbe 100644 (file)
@@ -858,7 +858,7 @@ void AccessCase::generateImpl(AccessGenerationState& state)
             size_t newSize = newStructure()->outOfLineCapacity() * sizeof(JSValue);
 
             if (allocatingInline) {
-                MarkedAllocator* allocator = vm.auxiliarySpace.allocatorFor(newSize);
+                MarkedAllocator* allocator = vm.jsValueGigacageAuxiliarySpace.allocatorFor(newSize);
 
                 if (!allocator) {
                     // Yuck, this case would suck!
index 5c8cf92e242faeb171de53104ab9370dcadd879f..adef6aea81e9af730c143c0a5786254b45a0849e 100644 (file)
@@ -109,7 +109,7 @@ void SpeculativeJIT::emitAllocateRawObject(GPRReg resultGPR, RegisteredStructure
     m_jit.move(TrustedImmPtr(0), storageGPR);
     
     if (size) {
-        if (MarkedAllocator* allocator = m_jit.vm()->auxiliarySpace.allocatorFor(size)) {
+        if (MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(size)) {
             m_jit.move(TrustedImmPtr(allocator), scratchGPR);
             m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases);
             
@@ -7868,7 +7868,7 @@ void SpeculativeJIT::compileAllocatePropertyStorage(Node* node)
     
     size_t size = initialOutOfLineCapacity * sizeof(JSValue);
 
-    MarkedAllocator* allocator = m_jit.vm()->auxiliarySpace.allocatorFor(size);
+    MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(size);
 
     if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
         SpeculateCellOperand base(this, node->child1());
@@ -7913,7 +7913,7 @@ void SpeculativeJIT::compileReallocatePropertyStorage(Node* node)
     size_t newSize = oldSize * outOfLineGrowthFactor;
     ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue));
     
-    MarkedAllocator* allocator = m_jit.vm()->auxiliarySpace.allocatorFor(newSize);
+    MarkedAllocator* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorFor(newSize);
 
     if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
         SpeculateCellOperand base(this, node->child1());
@@ -8494,7 +8494,7 @@ void SpeculativeJIT::compileNewTypedArray(Node* node)
         m_jit.and32(TrustedImm32(~7), scratchGPR);
     }
     m_jit.emitAllocateVariableSized(
-        storageGPR, m_jit.vm()->auxiliarySpace, scratchGPR, scratchGPR,
+        storageGPR, m_jit.vm()->primitiveGigacageAuxiliarySpace, scratchGPR, scratchGPR,
         scratchGPR2, slowCases);
     
     MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, sizeGPR);
@@ -10112,7 +10112,7 @@ void SpeculativeJIT::emitAllocateButterfly(GPRReg storageResultGPR, GPRReg sizeG
     m_jit.lshift32(TrustedImm32(3), scratch1);
     m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratch1, scratch2);
     m_jit.emitAllocateVariableSized(
-        storageResultGPR, m_jit.vm()->auxiliarySpace, scratch2, scratch1, scratch3, slowCases);
+        storageResultGPR, m_jit.vm()->jsValueGigacageAuxiliarySpace, scratch2, scratch1, scratch3, slowCases);
     m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageResultGPR);
 
     m_jit.store32(sizeGPR, MacroAssembler::Address(storageResultGPR, Butterfly::offsetOfPublicLength()));
index a2dd4b86ca0024f4879bfaca3964c4cff6a042b5..142eb5b3f92e28bb3728a439f3e0d3bfff71932c 100644 (file)
@@ -3235,7 +3235,7 @@ private:
     {
         LValue butterfly = m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly);
         if (m_node->op() != GetButterflyWithoutCaging)
-            butterfly = caged(butterfly);
+            butterfly = caged(Gigacage::JSValue, butterfly);
         setStorage(butterfly);
     }
 
@@ -3272,7 +3272,7 @@ private:
         }
 
         DFG_ASSERT(m_graph, m_node, isTypedView(m_node->arrayMode().typedArrayType()));
-        setStorage(caged(m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
+        setStorage(caged(Gigacage::Primitive, m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
     }
     
     void compileCheckArray()
@@ -5040,7 +5040,7 @@ private:
                     m_out.constIntPtr(~static_cast<intptr_t>(7)));
             }
         
-            LValue allocator = allocatorForSize(vm().auxiliarySpace, byteSize, slowCase);
+            LValue allocator = allocatorForSize(vm().primitiveGigacageAuxiliarySpace, byteSize, slowCase);
             LValue storage = allocateHeapCell(allocator, slowCase);
             
             splatWords(
@@ -8869,7 +8869,7 @@ private:
             m_out.neg(m_out.sub(index, m_out.load32(enumerator, m_heaps.JSPropertyNameEnumerator_cachedInlineCapacity))));
         int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
         ValueFromBlock outOfLineResult = m_out.anchor(
-            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(storage), realIndex, ScaleEight, offsetOfFirstProperty)));
+            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(Gigacage::JSValue, storage), realIndex, ScaleEight, offsetOfFirstProperty)));
         m_out.jump(continuation);
 
         m_out.appendTo(slowCase, continuation);
@@ -9049,7 +9049,7 @@ private:
                 ValueFromBlock noButterfly = m_out.anchor(m_out.intPtrZero);
                 
                 LValue startOfStorage = allocateHeapCell(
-                    allocatorForSize(vm().auxiliarySpace, butterflySize, slowPath),
+                    allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, slowPath),
                     slowPath);
 
                 LValue fastButterflyValue = m_out.add(
@@ -10005,7 +10005,7 @@ private:
         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
 
         size_t sizeInBytes = sizeInValues * sizeof(JSValue);
-        MarkedAllocator* allocator = vm().auxiliarySpace.allocatorFor(sizeInBytes);
+        MarkedAllocator* allocator = vm().jsValueGigacageAuxiliarySpace.allocatorFor(sizeInBytes);
         LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
         ValueFromBlock fastButterfly = m_out.anchor(
             m_out.add(m_out.constIntPtr(sizeInBytes + sizeof(IndexingHeader)), startOfStorage));
@@ -11233,7 +11233,7 @@ private:
         LValue butterflySize = m_out.add(
             payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
             
-        LValue allocator = allocatorForSize(vm().auxiliarySpace, butterflySize, failCase);
+        LValue allocator = allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, failCase);
         LValue startOfStorage = allocateHeapCell(allocator, failCase);
             
         LValue butterfly = m_out.add(startOfStorage, m_out.constIntPtr(sizeof(IndexingHeader)));
@@ -11614,34 +11614,35 @@ private:
         }
     }
     
-    LValue caged(LValue ptr)
+    LValue caged(Gigacage::Kind kind, LValue ptr)
     {
-        if (vm().gigacageEnabled().isStillValid()) {
-            m_graph.watchpoints().addLazily(vm().gigacageEnabled());
-            
-            LValue basePtr = m_out.constIntPtr(g_gigacageBasePtr);
-            LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
-            
-            // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
-            // codegen for 2-operand andq on x86-64.
-            LValue masked = m_out.bitAnd(ptr, mask);
-            
-            // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
-            // compute here will get reassociated and folded with g_gigacageBasePtr. There's a world in which
-            // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
-            // it's much easier to just block B3's badness here. That's what we do for now.
-            PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
-            patchpoint->appendSomeRegister(basePtr);
-            patchpoint->appendSomeRegister(masked);
-            patchpoint->setGenerator(
-                [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
-                    jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
-                });
-            patchpoint->effects = Effects::none();
-            return patchpoint;
+        if (kind == Gigacage::Primitive) {
+            if (vm().primitiveGigacageEnabled().isStillValid())
+                m_graph.watchpoints().addLazily(vm().primitiveGigacageEnabled());
+            else
+                return ptr;
         }
         
-        return ptr;
+        LValue basePtr = m_out.constIntPtr(Gigacage::basePtr(kind));
+        LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
+        
+        // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
+        // codegen for 2-operand andq on x86-64.
+        LValue masked = m_out.bitAnd(ptr, mask);
+        
+        // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
+        // compute here will get reassociated and folded with Gigacage::basePtr. There's a world in which
+        // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
+        // it's much easier to just block B3's badness here. That's what we do for now.
+        PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
+        patchpoint->appendSomeRegister(basePtr);
+        patchpoint->appendSomeRegister(masked);
+        patchpoint->setGenerator(
+            [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
+                jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
+            });
+        patchpoint->effects = Effects::none();
+        return patchpoint;
     }
     
     void buildSwitch(SwitchData* data, LType type, LValue switchValue)
index 8d5d7b4af1be05d67434679c0963077235de029b..45f6bb751b129ec069509292b8d5e559b4574296 100644 (file)
 
 namespace JSC {
 
-FastMallocAlignedMemoryAllocator& FastMallocAlignedMemoryAllocator::instance()
-{
-    static FastMallocAlignedMemoryAllocator* result;
-    static std::once_flag onceFlag;
-    std::call_once(
-        onceFlag,
-        [] {
-            result = new FastMallocAlignedMemoryAllocator();
-        });
-    return *result;
-}
-
 FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator()
 {
 }
index fb31ea5fbdf2f5a7c4debcb05fb9f5508dcb8a08..bdd57b708343c7e86674eef164a152192a36dd4b 100644 (file)
@@ -31,17 +31,13 @@ namespace JSC {
 
 class FastMallocAlignedMemoryAllocator : public AlignedMemoryAllocator {
 public:
-    JS_EXPORT_PRIVATE static FastMallocAlignedMemoryAllocator& instance();
-
+    FastMallocAlignedMemoryAllocator();
     ~FastMallocAlignedMemoryAllocator();
     
     void* tryAllocateAlignedMemory(size_t alignment, size_t size) override;
     void freeAlignedMemory(void*) override;
     
     void dump(PrintStream&) const override;
-
-private:
-    FastMallocAlignedMemoryAllocator();
 };
 
 } // namespace JSC
index dae2219c8fb17e8c19ed15162fc08aa34c298b4e..d6796e61b3d54a61e73f416b589401f4ce1784a0 100644 (file)
 #include "config.h"
 #include "GigacageAlignedMemoryAllocator.h"
 
-#include <mutex>
-#include <wtf/Gigacage.h>
-
 namespace JSC {
 
-GigacageAlignedMemoryAllocator& GigacageAlignedMemoryAllocator::instance()
-{
-    static GigacageAlignedMemoryAllocator* result;
-    static std::once_flag onceFlag;
-    std::call_once(
-        onceFlag,
-        [] {
-            result = new GigacageAlignedMemoryAllocator();
-        });
-    return *result;
-}
-
-GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator()
+GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator(Gigacage::Kind kind)
+    : m_kind(kind)
 {
 }
 
@@ -53,17 +39,17 @@ GigacageAlignedMemoryAllocator::~GigacageAlignedMemoryAllocator()
 
 void* GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory(size_t alignment, size_t size)
 {
-    return Gigacage::tryAlignedMalloc(alignment, size);
+    return Gigacage::tryAlignedMalloc(m_kind, alignment, size);
 }
 
 void GigacageAlignedMemoryAllocator::freeAlignedMemory(void* basePtr)
 {
-    Gigacage::alignedFree(basePtr);
+    Gigacage::alignedFree(m_kind, basePtr);
 }
 
 void GigacageAlignedMemoryAllocator::dump(PrintStream& out) const
 {
-    out.print("Gigacage");
+    out.print(Gigacage::name(m_kind), "Gigacage");
 }
 
 } // namespace JSC
index 2b5fe04241fc7a39c16a5a2c83064f924f58686b..4d119a9ea27d565f6c19b34ecef83ca6d4728244 100644 (file)
 #pragma once
 
 #include "AlignedMemoryAllocator.h"
+#include <wtf/Gigacage.h>
 
 namespace JSC {
 
 class GigacageAlignedMemoryAllocator : public AlignedMemoryAllocator {
 public:
-    // FIXME: This shouldn't be a singleton. There should be different instances for primaries, JSValues,
-    // and other things.
-    // https://bugs.webkit.org/show_bug.cgi?id=174919
-    static GigacageAlignedMemoryAllocator& instance();
-
+    GigacageAlignedMemoryAllocator(Gigacage::Kind);
     ~GigacageAlignedMemoryAllocator();
     
     void* tryAllocateAlignedMemory(size_t alignment, size_t size) override;
@@ -44,7 +41,7 @@ public:
     void dump(PrintStream&) const override;
 
 private:
-    GigacageAlignedMemoryAllocator();
+    Gigacage::Kind m_kind;
 };
 
 } // namespace JSC
index 9e3db46b4bc26f1c26c4d826fabbf69598551c96..34f09c4f7b95aa81a8d8d1575347310713490c28 100644 (file)
@@ -3802,9 +3802,9 @@ int runJSC(CommandLine options, bool isWorker, const Func& func)
     return result;
 }
 
-static void gigacageDisabled(void*)
+static void primitiveGigacageDisabled(void*)
 {
-    dataLog("Gigacage disabled! Aborting.\n");
+    dataLog("Primitive gigacage disabled! Aborting.\n");
     UNREACHABLE_FOR_PLATFORM();
 }
 
@@ -3827,7 +3827,7 @@ int jscmain(int argc, char** argv)
     JSC::Wasm::enableFastMemory();
 #endif
     if (Gigacage::shouldBeEnabled())
-        Gigacage::addDisableCallback(gigacageDisabled, nullptr);
+        Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabled, nullptr);
 
     int result;
     result = runJSC(
index c7325c0b3ee83bc817603c37c8a8340b41ba6000..1d5e055a2fc2b8ebefdbf6f5252339d095758b2c 100644 (file)
@@ -376,10 +376,10 @@ macro checkSwitchToJITForLoop()
         end)
 end
 
-macro loadCaged(source, dest, scratch)
+macro loadCaged(basePtr, source, dest, scratch)
     loadp source, dest
     if GIGACAGE_ENABLED and not C_LOOP
-        loadp _g_gigacageBasePtr, scratch
+        loadp basePtr, scratch
         btpz scratch, .done
         andp constexpr GIGACAGE_MASK, dest
         addp scratch, dest
@@ -1209,7 +1209,7 @@ _llint_op_is_object:
 
 macro loadPropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value)
     bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline
-    loadCaged(JSObject::m_butterfly[objectAndStorage], objectAndStorage, value)
+    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[objectAndStorage], objectAndStorage, value)
     negi propertyOffsetAsInt
     sxi2q propertyOffsetAsInt, propertyOffsetAsInt
     jmp .ready
@@ -1222,7 +1222,7 @@ end
 
 macro storePropertyAtVariableOffset(propertyOffsetAsInt, objectAndStorage, value, scratch)
     bilt propertyOffsetAsInt, firstOutOfLineOffset, .isInline
-    loadCaged(JSObject::m_butterfly[objectAndStorage], objectAndStorage, scratch)
+    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[objectAndStorage], objectAndStorage, scratch)
     negi propertyOffsetAsInt
     sxi2q propertyOffsetAsInt, propertyOffsetAsInt
     jmp .ready
@@ -1298,7 +1298,7 @@ _llint_op_get_array_length:
     btiz t2, IsArray, .opGetArrayLengthSlow
     btiz t2, IndexingShapeMask, .opGetArrayLengthSlow
     loadisFromInstruction(1, t1)
-    loadCaged(JSObject::m_butterfly[t3], t0, t2)
+    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t3], t0, t2)
     loadi -sizeof IndexingHeader + IndexingHeader::u.lengths.publicLength[t0], t0
     bilt t0, 0, .opGetArrayLengthSlow
     orq tagTypeNumber, t0
@@ -1481,7 +1481,7 @@ _llint_op_get_by_val:
     loadisFromInstruction(3, t3)
     loadConstantOrVariableInt32(t3, t1, .opGetByValSlow)
     sxi2q t1, t1
-    loadCaged(JSObject::m_butterfly[t0], t3, t5)
+    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t0], t3, t5)
     andi IndexingShapeMask, t2
     bieq t2, Int32Shape, .opGetByValIsContiguous
     bineq t2, ContiguousShape, .opGetByValNotContiguous
@@ -1528,7 +1528,7 @@ _llint_op_get_by_val:
     bia t2, LastArrayType - FirstArrayType, .opGetByValSlow
     
     # Sweet, now we know that we have a typed array. Do some basic things now.
-    loadCaged(JSArrayBufferView::m_vector[t0], t3, t5)
+    loadCaged(_g_primitiveGigacageBasePtr, JSArrayBufferView::m_vector[t0], t3, t5)
     biaeq t1, JSArrayBufferView::m_length[t0], .opGetByValSlow
     
     # Now bisect through the various types. Note that we can treat Uint8ArrayType and
@@ -1619,7 +1619,7 @@ macro putByVal(slowPath)
     loadisFromInstruction(2, t0)
     loadConstantOrVariableInt32(t0, t3, .opPutByValSlow)
     sxi2q t3, t3
-    loadCaged(JSObject::m_butterfly[t1], t0, t5)
+    loadCaged(_g_jsValueGigacageBasePtr, JSObject::m_butterfly[t1], t0, t5)
     andi IndexingShapeMask, t2
     bineq t2, Int32Shape, .opPutByValNotInt32
     contiguousPutByVal(
index 6add4cec1d71fe093ceb95d6b485fde39b76a588..ccf988e2a06061bde4e6bf4adeaae16fdee37465 100644 (file)
@@ -106,7 +106,7 @@ void ArrayBufferContents::tryAllocate(unsigned numElements, unsigned elementByte
     size_t size = static_cast<size_t>(numElements) * static_cast<size_t>(elementByteSize);
     if (!size)
         size = 1; // Make sure malloc actually allocates something, but not too much. We use null to mean that the buffer is neutered.
-    m_data = Gigacage::tryMalloc(size);
+    m_data = Gigacage::tryMalloc(Gigacage::Primitive, size);
     if (!m_data) {
         reset();
         return;
@@ -116,7 +116,7 @@ void ArrayBufferContents::tryAllocate(unsigned numElements, unsigned elementByte
         memset(m_data, 0, size);
 
     m_sizeInBytes = numElements * elementByteSize;
-    m_destructor = [] (void* p) { Gigacage::free(p); };
+    m_destructor = [] (void* p) { Gigacage::free(Gigacage::Primitive, p); };
 }
 
 void ArrayBufferContents::makeShared()
@@ -187,7 +187,7 @@ Ref<ArrayBuffer> ArrayBuffer::create(ArrayBufferContents&& contents)
 //   from the cage.
 Ref<ArrayBuffer> ArrayBuffer::createAdopted(const void* data, unsigned byteLength)
 {
-    return createFromBytes(data, byteLength, [] (void* p) { Gigacage::free(p); });
+    return createFromBytes(data, byteLength, [] (void* p) { Gigacage::free(Gigacage::Primitive, p); });
 }
 
 // FIXME: We cannot use this except if the memory comes from the cage.
@@ -198,8 +198,8 @@ Ref<ArrayBuffer> ArrayBuffer::createAdopted(const void* data, unsigned byteLengt
 // - WebAssembly. Wasm should allocate from the cage.
 Ref<ArrayBuffer> ArrayBuffer::createFromBytes(const void* data, unsigned byteLength, ArrayBufferDestructorFunction&& destructor)
 {
-    if (data && byteLength && !Gigacage::isCaged(data))
-        Gigacage::disableGigacage();
+    if (data && byteLength && !Gigacage::isCaged(Gigacage::Primitive, data))
+        Gigacage::disablePrimitiveGigacage();
     
     ArrayBufferContents contents(const_cast<void*>(data), byteLength, WTFMove(destructor));
     return create(WTFMove(contents));
index 3b7322f13565eb11089fb6991676323b2432d7ee..6a6cb674308da3dfcc7380a5cffc7920b92c8069 100644 (file)
@@ -36,6 +36,8 @@ class VM;
 template<typename T>
 class AuxiliaryBarrier {
 public:
+    typedef T Type;
+    
     AuxiliaryBarrier(): m_value() { }
     
     template<typename U>
index 88c6f8a676e6cf64846e38e3a4b0de91d47c7aa5..2aa26c4b7ecab3cbf758391110acca40b2b5996e 100644 (file)
@@ -62,7 +62,7 @@ ALWAYS_INLINE unsigned Butterfly::optimalContiguousVectorLength(Structure* struc
 inline Butterfly* Butterfly::createUninitialized(VM& vm, JSCell*, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes)
 {
     size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
-    void* base = vm.auxiliarySpace.allocate(size);
+    void* base = vm.jsValueGigacageAuxiliarySpace.allocate(size);
     Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
     return result;
 }
@@ -70,7 +70,7 @@ inline Butterfly* Butterfly::createUninitialized(VM& vm, JSCell*, size_t preCapa
 inline Butterfly* Butterfly::tryCreate(VM& vm, JSCell*, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, const IndexingHeader& indexingHeader, size_t indexingPayloadSizeInBytes)
 {
     size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
-    void* base = vm.auxiliarySpace.tryAllocate(size);
+    void* base = vm.jsValueGigacageAuxiliarySpace.tryAllocate(size);
     if (!base)
         return nullptr;
     Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
@@ -148,7 +148,7 @@ inline Butterfly* Butterfly::growArrayRight(
     void* theBase = base(0, propertyCapacity);
     size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes);
     size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes);
-    void* newBase = vm.auxiliarySpace.tryAllocate(newSize);
+    void* newBase = vm.jsValueGigacageAuxiliarySpace.tryAllocate(newSize);
     if (!newBase)
         return nullptr;
     // FIXME: This probably shouldn't be a memcpy.
diff --git a/Source/JavaScriptCore/runtime/CagedBarrierPtr.h b/Source/JavaScriptCore/runtime/CagedBarrierPtr.h
new file mode 100644 (file)
index 0000000..73ba08f
--- /dev/null
@@ -0,0 +1,91 @@
+/*
+ * Copyright (C) 2017 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "AuxiliaryBarrier.h"
+#include <wtf/CagedPtr.h>
+
+namespace JSC {
+
+class JSCell;
+class VM;
+
+// This is a convenient combo of AuxiliaryBarrier and CagedPtr.
+
+template<Gigacage::Kind passedKind, typename T>
+class CagedBarrierPtr {
+public:
+    static constexpr Gigacage::Kind kind = passedKind;
+    typedef T Type;
+    
+    CagedBarrierPtr() { }
+    
+    template<typename U>
+    CagedBarrierPtr(VM& vm, JSCell* cell, U&& value)
+    {
+        m_barrier.set(vm, cell, std::forward<U>(value));
+    }
+    
+    void clear() { m_barrier.clear(); }
+    
+    template<typename U>
+    void set(VM& vm, JSCell* cell, U&& value)
+    {
+        m_barrier.set(vm, cell, std::forward<U>(value));
+    }
+    
+    T* get() const { return m_barrier.get().get(); }
+    T* getMayBeNull() const { return m_barrier.get().getMayBeNull(); }
+    
+    bool operator==(const CagedBarrierPtr& other) const
+    {
+        return getMayBeNull() == other.getMayBeNull();
+    }
+    
+    bool operator!=(const CagedBarrierPtr& other) const
+    {
+        return !(*this == other);
+    }
+    
+    explicit operator bool() const
+    {
+        return *this != CagedBarrierPtr();
+    }
+    
+    template<typename U>
+    void setWithoutBarrier(U&& value) { m_barrier.setWithoutBarrier(std::forward<U>(value)); }
+    
+    T& operator*() const { return *get(); }
+    T* operator->() const { return get(); }
+    
+    template<typename IndexType>
+    T& operator[](IndexType index) const { return get()[index]; }
+    
+private:
+    AuxiliaryBarrier<CagedPtr<kind, T>> m_barrier;
+};
+
+} // namespace JSC
index db632d0226c87a9f78d8261ba51d5a4773e8bb08..22239e0e260884f383c4fbfd6e048379f218516c 100644 (file)
@@ -118,7 +118,7 @@ void DirectArguments::overrideThings(VM& vm)
     putDirect(vm, vm.propertyNames->callee, m_callee.get(), DontEnum);
     putDirect(vm, vm.propertyNames->iteratorSymbol, globalObject()->arrayProtoValuesFunction(), DontEnum);
     
-    void* backingStore = vm.auxiliarySpace.tryAllocate(mappedArgumentsSize());
+    void* backingStore = vm.gigacageAuxiliarySpace(m_mappedArguments.kind).tryAllocate(mappedArgumentsSize());
     RELEASE_ASSERT(backingStore);
     bool* overrides = static_cast<bool*>(backingStore);
     m_mappedArguments.set(vm, this, overrides);
@@ -135,7 +135,7 @@ void DirectArguments::overrideThingsIfNecessary(VM& vm)
 void DirectArguments::unmapArgument(VM& vm, unsigned index)
 {
     overrideThingsIfNecessary(vm);
-    m_mappedArguments.get()[index] = true;
+    m_mappedArguments[index] = true;
 }
 
 void DirectArguments::copyToArguments(ExecState* exec, VirtualRegister firstElementDest, unsigned offset, unsigned length)
index d3ce619a8c954f12e3e4b920f3693461a5fa4164..05eb5ac9027abc9d128defa2a3408f48cb7c0b81 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
+ * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
 
 #pragma once
 
-#include "AuxiliaryBarrier.h"
+#include "CagedBarrierPtr.h"
 #include "DirectArgumentsOffset.h"
 #include "GenericArguments.h"
+#include <wtf/CagedPtr.h>
 
 namespace JSC {
 
@@ -73,7 +74,7 @@ public:
     
     bool isMappedArgument(uint32_t i) const
     {
-        return i < m_length && (!m_mappedArguments || !m_mappedArguments.get()[i]);
+        return i < m_length && (!m_mappedArguments || !m_mappedArguments[i]);
     }
 
     bool isMappedArgumentInDFG(uint32_t i) const
@@ -164,7 +165,7 @@ private:
     WriteBarrier<JSFunction> m_callee;
     uint32_t m_length; // Always the actual length of captured arguments and never what was stored into the length property.
     uint32_t m_minCapacity; // The max of this and length determines the capacity of this object. It may be the actual capacity, or maybe something smaller. We arrange it this way to be kind to the JITs.
-    AuxiliaryBarrier<bool*> m_mappedArguments; // If non-null, it means that length, callee, and caller are fully materialized properties.
+    CagedBarrierPtr<Gigacage::Primitive, bool> m_mappedArguments; // If non-null, it means that length, callee, and caller are fully materialized properties.
 };
 
 } // namespace JSC
index 5ba509f1caed4c07a0669f4dad34dca84ffab94b..6a6380e7f1b9d51e9d3a376f920e3e6d676b7166 100644 (file)
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2015 Apple Inc. All rights reserved.
+ * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -25,6 +25,7 @@
 
 #pragma once
 
+#include "CagedBarrierPtr.h"
 #include "JSObject.h"
 
 namespace JSC {
@@ -60,7 +61,7 @@ protected:
 
     void copyToArguments(ExecState*, VirtualRegister firstElementDest, unsigned offset, unsigned length);
     
-    AuxiliaryBarrier<bool*> m_modifiedArgumentsDescriptor;
+    CagedBarrierPtr<Gigacage::Primitive, bool> m_modifiedArgumentsDescriptor;
 };
 
 } // namespace JSC
index d9fb7a598697c8c2d0a8687cf10031071c1128c2..0b262f62a5f2a559fa4cd36c4356729010ce6d08 100644 (file)
@@ -261,7 +261,7 @@ void GenericArguments<Type>::initModifiedArgumentsDescriptor(VM& vm, unsigned ar
     RELEASE_ASSERT(!m_modifiedArgumentsDescriptor);
 
     if (argsLength) {
-        void* backingStore = vm.auxiliarySpace.tryAllocate(WTF::roundUpToMultipleOf<8>(argsLength));
+        void* backingStore = vm.gigacageAuxiliarySpace(m_modifiedArgumentsDescriptor.kind).tryAllocate(WTF::roundUpToMultipleOf<8>(argsLength));
         RELEASE_ASSERT(backingStore);
         bool* modifiedArguments = static_cast<bool*>(backingStore);
         m_modifiedArgumentsDescriptor.set(vm, this, modifiedArguments);
@@ -282,7 +282,7 @@ void GenericArguments<Type>::setModifiedArgumentDescriptor(VM& vm, unsigned inde
 {
     initModifiedArgumentsDescriptorIfNecessary(vm, length);
     if (index < length)
-        m_modifiedArgumentsDescriptor.get()[index] = true;
+        m_modifiedArgumentsDescriptor[index] = true;
 }
 
 template<typename Type>
@@ -291,7 +291,7 @@ bool GenericArguments<Type>::isModifiedArgumentDescriptor(unsigned index, unsign
     if (!m_modifiedArgumentsDescriptor)
         return false;
     if (index < length)
-        return m_modifiedArgumentsDescriptor.get()[index];
+        return m_modifiedArgumentsDescriptor[index];
     return false;
 }
 
index 5c757e55aa9e537c867df09d9e2bc9506964dc38..57ea346ca2731a40169521d2b76ba0689caee514 100644 (file)
@@ -62,7 +62,7 @@ void HashMapImpl<HashMapBucket>::visitChildren(JSCell* cell, SlotVisitor& visito
     visitor.append(thisObject->m_head);
     visitor.append(thisObject->m_tail);
     
-    if (HashMapBufferType* buffer = thisObject->m_buffer.get())
+    if (HashMapBufferType* buffer = thisObject->m_buffer.getMayBeNull())
         visitor.markAuxiliary(buffer);
 }
 
index 86e4cb554b81c41eca3c49ae23b132bfe8822763..d187176713353351b011930a9907d676240dcce4 100644 (file)
@@ -188,7 +188,7 @@ public:
     {
         auto scope = DECLARE_THROW_SCOPE(vm);
         size_t allocationSize = HashMapBuffer::allocationSize(capacity);
-        void* data = vm.auxiliarySpace.tryAllocate(allocationSize);
+        void* data = vm.jsValueGigacageAuxiliarySpace.tryAllocate(allocationSize);
         if (!data) {
             throwOutOfMemoryError(exec, scope);
             return nullptr;
@@ -302,7 +302,7 @@ public:
 
     ALWAYS_INLINE HashMapBucketType** buffer() const
     {
-        return m_buffer.get()->buffer();
+        return m_buffer->buffer();
     }
 
     void finishCreation(ExecState* exec, VM& vm)
@@ -591,7 +591,7 @@ private:
             makeAndSetNewBuffer(exec, vm);
             RETURN_IF_EXCEPTION(scope, void());
         } else {
-            m_buffer.get()->reset(m_capacity);
+            m_buffer->reset(m_capacity);
             assertBufferIsEmpty();
         }
 
@@ -655,7 +655,7 @@ private:
 
     WriteBarrier<HashMapBucketType> m_head;
     WriteBarrier<HashMapBucketType> m_tail;
-    AuxiliaryBarrier<HashMapBufferType*> m_buffer;
+    CagedBarrierPtr<Gigacage::JSValue, HashMapBufferType> m_buffer;
     uint32_t m_keyCount;
     uint32_t m_deleteCount;
     uint32_t m_capacity;
index eb3a3c454289d6b139639acd39ba66c65ab4dd17..8d8a6f3cddd9ad500dea200771234d732036692d 100644 (file)
@@ -81,7 +81,7 @@ JSArray* JSArray::tryCreateUninitializedRestricted(ObjectInitializationScope& sc
             || hasContiguous(indexingType));
 
         unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
-        void* temp = vm.auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
+        void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
         if (UNLIKELY(!temp))
             return nullptr;
         butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
@@ -98,7 +98,7 @@ JSArray* JSArray::tryCreateUninitializedRestricted(ObjectInitializationScope& sc
     } else {
         static const unsigned indexBias = 0;
         unsigned vectorLength = ArrayStorage::optimalVectorLength(indexBias, structure, initialLength);
-        void* temp = vm.auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));
+        void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));
         if (UNLIKELY(!temp))
             return nullptr;
         butterfly = Butterfly::fromBase(temp, indexBias, outOfLineStorage);
@@ -360,7 +360,7 @@ bool JSArray::unshiftCountSlowCase(const AbstractLocker&, VM& vm, DeferGC&, bool
         allocatedNewStorage = false;
     } else {
         size_t newSize = Butterfly::totalSize(0, propertyCapacity, true, ArrayStorage::sizeFor(desiredCapacity));
-        newAllocBase = vm.auxiliarySpace.tryAllocate(newSize);
+        newAllocBase = vm.jsValueGigacageAuxiliarySpace.tryAllocate(newSize);
         if (!newAllocBase)
             return false;
         newStorageCapacity = desiredCapacity;
@@ -550,7 +550,7 @@ bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException
     VM& vm = exec->vm();
     auto scope = DECLARE_THROW_SCOPE(vm);
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     switch (indexingType()) {
     case ArrayClass:
         if (!newLength)
@@ -620,7 +620,7 @@ JSValue JSArray::pop(ExecState* exec)
     VM& vm = exec->vm();
     auto scope = DECLARE_THROW_SCOPE(vm);
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     
     switch (indexingType()) {
     case ArrayClass:
@@ -722,7 +722,7 @@ void JSArray::push(ExecState* exec, JSValue value)
     VM& vm = exec->vm();
     auto scope = DECLARE_THROW_SCOPE(vm);
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     
     switch (indexingType()) {
     case ArrayClass: {
@@ -890,9 +890,9 @@ JSArray* JSArray::fastSlice(ExecState& exec, unsigned startIndex, unsigned count
 
         auto& resultButterfly = *resultArray->butterfly();
         if (arrayType == ArrayWithDouble)
-            memcpy(resultButterfly.contiguousDouble().data(), m_butterfly.get()->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);
+            memcpy(resultButterfly.contiguousDouble().data(), m_butterfly->contiguousDouble().data() + startIndex, sizeof(JSValue) * count);
         else
-            memcpy(resultButterfly.contiguous().data(), m_butterfly.get()->contiguous().data() + startIndex, sizeof(JSValue) * count);
+            memcpy(resultButterfly.contiguous().data(), m_butterfly->contiguous().data() + startIndex, sizeof(JSValue) * count);
         resultButterfly.setPublicLength(count);
 
         return resultArray;
@@ -970,7 +970,7 @@ bool JSArray::shiftCountWithArrayStorage(VM& vm, unsigned startIndex, unsigned c
         // Adjust the Butterfly and the index bias. We only need to do this here because we're changing
         // the start of the Butterfly, which needs to point at the first indexed property in the used
         // portion of the vector.
-        Butterfly* butterfly = m_butterfly.get()->shift(structure(), count);
+        Butterfly* butterfly = m_butterfly->shift(structure(), count);
         setButterfly(vm, butterfly);
         storage = butterfly->arrayStorage();
         storage->m_indexBias += count;
@@ -1015,7 +1015,7 @@ bool JSArray::shiftCountWithAnyIndexingType(ExecState* exec, unsigned& startInde
     VM& vm = exec->vm();
     RELEASE_ASSERT(count > 0);
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     
     switch (indexingType()) {
     case ArrayClass:
@@ -1171,7 +1171,7 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
     VM& vm = exec->vm();
     auto scope = DECLARE_THROW_SCOPE(vm);
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     
     switch (indexingType()) {
     case ArrayClass:
@@ -1194,7 +1194,7 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
             throwOutOfMemoryError(exec, scope);
             return false;
         }
-        butterfly = m_butterfly.get().getMayBeNull();
+        butterfly = m_butterfly.getMayBeNull();
 
         // We have to check for holes before we start moving things around so that we don't get halfway 
         // through shifting and then realize we should have been in ArrayStorage mode.
@@ -1238,7 +1238,7 @@ bool JSArray::unshiftCountWithAnyIndexingType(ExecState* exec, unsigned startInd
             throwOutOfMemoryError(exec, scope);
             return false;
         }
-        butterfly = m_butterfly.get().getMayBeNull();
+        butterfly = m_butterfly.getMayBeNull();
         
         // We have to check for holes before we start moving things around so that we don't get halfway 
         // through shifting and then realize we should have been in ArrayStorage mode.
@@ -1281,7 +1281,7 @@ void JSArray::fillArgList(ExecState* exec, MarkedArgumentBuffer& args)
     unsigned vectorEnd;
     WriteBarrier<Unknown>* vector;
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     
     switch (indexingType()) {
     case ArrayClass:
@@ -1354,7 +1354,7 @@ void JSArray::copyToArguments(ExecState* exec, VirtualRegister firstElementDest,
     // FIXME: What prevents this from being called with a RuntimeArray? The length function will always return 0 in that case.
     ASSERT(length == this->length());
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     switch (indexingType()) {
     case ArrayClass:
         return;
index ef0746179d78dcb9da62b0fd72188195eee35726..e180cd03950c74764de493e72c61fcb0677aa552 100644 (file)
@@ -231,7 +231,7 @@ inline JSArray* JSArray::tryCreate(VM& vm, Structure* structure, unsigned initia
             return nullptr;
 
         unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
-        void* temp = vm.auxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
+        void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
         if (!temp)
             return nullptr;
         butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
index 30183dd081c0363f6cc3969688d582b38e83a3c8..ea5201101b0f39a084d75a0ff94835943a8171b5 100644 (file)
@@ -66,7 +66,7 @@ JSArrayBufferView::ConstructionContext::ConstructionContext(
         void* temp;
         size_t size = sizeOf(length, elementSize);
         if (size) {
-            temp = vm.auxiliarySpace.tryAllocate(nullptr, size);
+            temp = vm.primitiveGigacageAuxiliarySpace.tryAllocate(nullptr, size);
             if (!temp)
                 return;
         } else
@@ -90,7 +90,7 @@ JSArrayBufferView::ConstructionContext::ConstructionContext(
         return;
     
     size_t size = static_cast<size_t>(length) * static_cast<size_t>(elementSize);
-    m_vector = Gigacage::tryMalloc(size);
+    m_vector = Gigacage::tryMalloc(Gigacage::Primitive, size);
     if (!m_vector)
         return;
     if (mode == ZeroFill)
@@ -192,7 +192,7 @@ void JSArrayBufferView::finalize(JSCell* cell)
     JSArrayBufferView* thisObject = static_cast<JSArrayBufferView*>(cell);
     ASSERT(thisObject->m_mode == OversizeTypedArray || thisObject->m_mode == WastefulTypedArray);
     if (thisObject->m_mode == OversizeTypedArray)
-        Gigacage::free(thisObject->m_vector.get());
+        Gigacage::free(Gigacage::Primitive, thisObject->m_vector.get());
 }
 
 JSArrayBuffer* JSArrayBufferView::unsharedJSBuffer(ExecState* exec)
index 1e56fb24101961ed488b2278a5f13789bf2f3560..a37e24662e184bc98c79560b501c299c96da060c 100644 (file)
@@ -157,7 +157,7 @@ void JSLock::didAcquireLock()
     // Note: everything below must come after addCurrentThread().
     m_vm->traps().notifyGrabAllLocks();
     
-    m_vm->fireGigacageEnabledIfNecessary();
+    m_vm->firePrimitiveGigacageEnabledIfNecessary();
 
 #if ENABLE(SAMPLING_PROFILER)
     if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler())
index 3788a0fd8d37805df44f8011cb31b9982e3b53f4..1a33e873ee17336650ab07695dd2e441e84b8657 100644 (file)
@@ -463,7 +463,7 @@ void JSObject::heapSnapshot(JSCell* cell, HeapSnapshotBuilder& builder)
             builder.appendPropertyNameEdge(thisObject, toValue.asCell(), entry.key);
     }
 
-    Butterfly* butterfly = thisObject->m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = thisObject->m_butterfly.getMayBeNull();
     if (butterfly) {
         WriteBarrier<Unknown>* data = nullptr;
         uint32_t count = 0;
@@ -613,7 +613,7 @@ bool JSObject::getOwnPropertySlotByIndex(JSObject* thisObject, ExecState* exec,
     }
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
         if (i >= storage->length())
             return false;
         
@@ -875,7 +875,7 @@ bool JSObject::putByIndex(JSCell* cell, ExecState* exec, unsigned propertyName,
         
     case NonArrayWithArrayStorage:
     case ArrayWithArrayStorage: {
-        ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
         
         if (propertyName >= storage->vectorLength())
             break;
@@ -897,7 +897,7 @@ bool JSObject::putByIndex(JSCell* cell, ExecState* exec, unsigned propertyName,
         
     case NonArrayWithSlowPutArrayStorage:
     case ArrayWithSlowPutArrayStorage: {
-        ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
         
         if (propertyName >= storage->vectorLength())
             break;
@@ -979,7 +979,7 @@ void JSObject::enterDictionaryIndexingMode(VM& vm)
             enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, storage);
         break;
     case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-        enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly.get()->arrayStorage());
+        enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage());
         break;
         
     default:
@@ -1011,7 +1011,7 @@ Butterfly* JSObject::createInitialIndexedStorage(VM& vm, unsigned length)
     unsigned propertyCapacity = structure->outOfLineCapacity();
     unsigned vectorLength = Butterfly::optimalContiguousVectorLength(propertyCapacity, length);
     Butterfly* newButterfly = Butterfly::createOrGrowArrayRight(
-        m_butterfly.get().getMayBeNull(), vm, this, structure, propertyCapacity, false, 0,
+        m_butterfly.getMayBeNull(), vm, this, structure, propertyCapacity, false, 0,
         sizeof(EncodedJSValue) * vectorLength);
     newButterfly->setPublicLength(length);
     newButterfly->setVectorLength(vectorLength);
@@ -1099,7 +1099,7 @@ ArrayStorage* JSObject::createArrayStorage(VM& vm, unsigned length, unsigned vec
     IndexingType oldType = indexingType();
     ASSERT_UNUSED(oldType, !hasIndexedProperties(oldType));
 
-    Butterfly* newButterfly = createArrayStorageButterfly(vm, this, oldStructure, length, vectorLength, m_butterfly.get().getMayBeNull());
+    Butterfly* newButterfly = createArrayStorageButterfly(vm, this, oldStructure, length, vectorLength, m_butterfly.getMayBeNull());
     ArrayStorage* result = newButterfly->arrayStorage();
     Structure* newStructure = Structure::nonPropertyTransition(vm, oldStructure, oldStructure->suggestedArrayStorageTransition());
     nukeStructureAndSetButterfly(vm, oldStructureID, newButterfly);
@@ -1117,43 +1117,43 @@ ContiguousJSValues JSObject::convertUndecidedToInt32(VM& vm)
 {
     ASSERT(hasUndecided(indexingType()));
 
-    Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+    Butterfly* butterfly = m_butterfly.getMayBeNull();
     for (unsigned i = butterfly->vectorLength(); i--;)
         butterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
 
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32));
-    return m_butterfly.get()->contiguousInt32();
+    return m_butterfly->contiguousInt32();
 }
 
 ContiguousDoubles JSObject::convertUndecidedToDouble(VM& vm)
 {
     ASSERT(hasUndecided(indexingType()));
 
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = butterfly->vectorLength(); i--;)
         butterfly->contiguousDouble()[i] = PNaN;
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble));
-    return m_butterfly.get()->contiguousDouble();
+    return m_butterfly->contiguousDouble();
 }
 
 ContiguousJSValues JSObject::convertUndecidedToContiguous(VM& vm)
 {
     ASSERT(hasUndecided(indexingType()));
 
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = butterfly->vectorLength(); i--;)
         butterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
 
     WTF::storeStoreFence();
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
-    return m_butterfly.get()->contiguous();
+    return m_butterfly->contiguous();
 }
 
 ArrayStorage* JSObject::constructConvertedArrayStorageWithoutCopyingElements(VM& vm, unsigned neededLength)
 {
     Structure* structure = this->structure(vm);
-    unsigned publicLength = m_butterfly.get()->publicLength();
+    unsigned publicLength = m_butterfly->publicLength();
     unsigned propertyCapacity = structure->outOfLineCapacity();
     unsigned propertySize = structure->outOfLineSize();
     
@@ -1162,7 +1162,7 @@ ArrayStorage* JSObject::constructConvertedArrayStorageWithoutCopyingElements(VM&
     
     memcpy(
         newButterfly->propertyStorage() - propertySize,
-        m_butterfly.get()->propertyStorage() - propertySize,
+        m_butterfly->propertyStorage() - propertySize,
         propertySize * sizeof(EncodedJSValue));
     
     ArrayStorage* newStorage = newButterfly->arrayStorage();
@@ -1180,7 +1180,7 @@ ArrayStorage* JSObject::convertUndecidedToArrayStorage(VM& vm, NonPropertyTransi
     DeferGC deferGC(vm.heap);
     ASSERT(hasUndecided(indexingType()));
 
-    unsigned vectorLength = m_butterfly.get()->vectorLength();
+    unsigned vectorLength = m_butterfly->vectorLength();
     ArrayStorage* storage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
     
     for (unsigned i = vectorLength; i--;)
@@ -1203,7 +1203,7 @@ ContiguousDoubles JSObject::convertInt32ToDouble(VM& vm)
 {
     ASSERT(hasInt32(indexingType()));
 
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = butterfly->vectorLength(); i--;) {
         WriteBarrier<Unknown>* current = &butterfly->contiguousInt32()[i];
         double* currentAsDouble = bitwise_cast<double*>(current);
@@ -1218,7 +1218,7 @@ ContiguousDoubles JSObject::convertInt32ToDouble(VM& vm)
     }
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble));
-    return m_butterfly.get()->contiguousDouble();
+    return m_butterfly->contiguousDouble();
 }
 
 ContiguousJSValues JSObject::convertInt32ToContiguous(VM& vm)
@@ -1226,7 +1226,7 @@ ContiguousJSValues JSObject::convertInt32ToContiguous(VM& vm)
     ASSERT(hasInt32(indexingType()));
     
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
-    return m_butterfly.get()->contiguous();
+    return m_butterfly->contiguous();
 }
 
 ArrayStorage* JSObject::convertInt32ToArrayStorage(VM& vm, NonPropertyTransition transition)
@@ -1234,9 +1234,9 @@ ArrayStorage* JSObject::convertInt32ToArrayStorage(VM& vm, NonPropertyTransition
     DeferGC deferGC(vm.heap);
     ASSERT(hasInt32(indexingType()));
 
-    unsigned vectorLength = m_butterfly.get()->vectorLength();
+    unsigned vectorLength = m_butterfly->vectorLength();
     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = 0; i < vectorLength; i++) {
         JSValue v = butterfly->contiguous()[i].get();
         newStorage->m_vector[i].setWithoutWriteBarrier(v);
@@ -1261,7 +1261,7 @@ ContiguousJSValues JSObject::convertDoubleToContiguous(VM& vm)
 {
     ASSERT(hasDouble(indexingType()));
 
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = butterfly->vectorLength(); i--;) {
         double* current = &butterfly->contiguousDouble()[i];
         WriteBarrier<Unknown>* currentAsValue = bitwise_cast<WriteBarrier<Unknown>*>(current);
@@ -1276,7 +1276,7 @@ ContiguousJSValues JSObject::convertDoubleToContiguous(VM& vm)
     
     WTF::storeStoreFence();
     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
-    return m_butterfly.get()->contiguous();
+    return m_butterfly->contiguous();
 }
 
 ArrayStorage* JSObject::convertDoubleToArrayStorage(VM& vm, NonPropertyTransition transition)
@@ -1284,9 +1284,9 @@ ArrayStorage* JSObject::convertDoubleToArrayStorage(VM& vm, NonPropertyTransitio
     DeferGC deferGC(vm.heap);
     ASSERT(hasDouble(indexingType()));
 
-    unsigned vectorLength = m_butterfly.get()->vectorLength();
+    unsigned vectorLength = m_butterfly->vectorLength();
     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = 0; i < vectorLength; i++) {
         double value = butterfly->contiguousDouble()[i];
         if (value != value) {
@@ -1315,9 +1315,9 @@ ArrayStorage* JSObject::convertContiguousToArrayStorage(VM& vm, NonPropertyTrans
     DeferGC deferGC(vm.heap);
     ASSERT(hasContiguous(indexingType()));
 
-    unsigned vectorLength = m_butterfly.get()->vectorLength();
+    unsigned vectorLength = m_butterfly->vectorLength();
     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     for (unsigned i = 0; i < vectorLength; i++) {
         JSValue v = butterfly->contiguous()[i].get();
         newStorage->m_vector[i].setWithoutWriteBarrier(v);
@@ -1408,8 +1408,8 @@ void JSObject::convertInt32ForValue(VM& vm, JSValue value)
 
 void JSObject::setIndexQuicklyToUndecided(VM& vm, unsigned index, JSValue value)
 {
-    ASSERT(index < m_butterfly.get()->publicLength());
-    ASSERT(index < m_butterfly.get()->vectorLength());
+    ASSERT(index < m_butterfly->publicLength());
+    ASSERT(index < m_butterfly->vectorLength());
     convertUndecidedForValue(vm, value);
     setIndexQuickly(vm, index, value);
 }
@@ -1577,7 +1577,7 @@ ArrayStorage* JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode(V
         return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, convertContiguousToArrayStorage(vm));
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-        return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly.get()->arrayStorage());
+        return enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(vm, m_butterfly->arrayStorage());
         
     default:
         CRASH();
@@ -1862,7 +1862,7 @@ bool JSObject::deletePropertyByIndex(JSCell* cell, ExecState* exec, unsigned i)
     }
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = thisObject->m_butterfly.get()->arrayStorage();
+        ArrayStorage* storage = thisObject->m_butterfly->arrayStorage();
         
         if (i < storage->vectorLength()) {
             WriteBarrier<Unknown>& valueSlot = storage->m_vector[i];
@@ -2168,7 +2168,7 @@ void JSObject::getOwnPropertyNames(JSObject* object, ExecState* exec, PropertyNa
         }
             
         case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-            ArrayStorage* storage = object->m_butterfly.get()->arrayStorage();
+            ArrayStorage* storage = object->m_butterfly->arrayStorage();
             
             unsigned usedVectorLength = std::min(storage->length(), storage->vectorLength());
             for (unsigned i = 0; i < usedVectorLength; ++i) {
@@ -2325,7 +2325,7 @@ NEVER_INLINE void JSObject::fillGetterPropertySlot(PropertySlot& slot, JSCell* g
 bool JSObject::putIndexedDescriptor(ExecState* exec, SparseArrayEntry* entryInMap, const PropertyDescriptor& descriptor, PropertyDescriptor& oldDescriptor)
 {
     VM& vm = exec->vm();
-    auto map = m_butterfly.get()->arrayStorage()->m_sparseMap.get();
+    auto map = m_butterfly->arrayStorage()->m_sparseMap.get();
 
     if (descriptor.isDataDescriptor()) {
         if (descriptor.value())
@@ -2395,7 +2395,7 @@ bool JSObject::defineOwnIndexedProperty(ExecState* exec, unsigned index, const P
     if (descriptor.attributes() & (ReadOnly | Accessor))
         notifyPresenceOfIndexedAccessors(vm);
 
-    SparseArrayValueMap* map = m_butterfly.get()->arrayStorage()->m_sparseMap.get();
+    SparseArrayValueMap* map = m_butterfly->arrayStorage()->m_sparseMap.get();
     RELEASE_ASSERT(map);
     
     // 1. Let current be the result of calling the [[GetOwnProperty]] internal method of O with property name P.
@@ -2427,7 +2427,7 @@ bool JSObject::defineOwnIndexedProperty(ExecState* exec, unsigned index, const P
         entryInMap->get(defaults);
 
         putIndexedDescriptor(exec, entryInMap, descriptor, defaults);
-        Butterfly* butterfly = m_butterfly.get().get();
+        Butterfly* butterfly = m_butterfly.get();
         if (index >= butterfly->arrayStorage()->length())
             butterfly->arrayStorage()->setLength(index + 1);
         return true;
@@ -2559,7 +2559,7 @@ bool JSObject::putByIndexBeyondVectorLengthWithoutAttributes(ExecState* exec, un
     ASSERT((indexingType() & IndexingShapeMask) == indexingShape);
     ASSERT(!indexingShouldBeSparse());
 
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     
     // For us to get here, the index is either greater than the public length, or greater than
     // or equal to the vector length.
@@ -2581,7 +2581,7 @@ bool JSObject::putByIndexBeyondVectorLengthWithoutAttributes(ExecState* exec, un
         throwOutOfMemoryError(exec, scope);
         return false;
     }
-    butterfly = m_butterfly.get().get();
+    butterfly = m_butterfly.get();
 
     RELEASE_ASSERT(i < butterfly->vectorLength());
     switch (indexingShape) {
@@ -2869,7 +2869,7 @@ bool JSObject::putDirectIndexSlowOrBeyondVectorLength(ExecState* exec, unsigned
         
     case ALL_INT32_INDEXING_TYPES: {
         if (attributes) {
-            if (i < m_butterfly.get()->vectorLength())
+            if (i < m_butterfly->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
             return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertInt32ToArrayStorage(vm));
         }
@@ -2883,7 +2883,7 @@ bool JSObject::putDirectIndexSlowOrBeyondVectorLength(ExecState* exec, unsigned
         
     case ALL_DOUBLE_INDEXING_TYPES: {
         if (attributes) {
-            if (i < m_butterfly.get()->vectorLength())
+            if (i < m_butterfly->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
             return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertDoubleToArrayStorage(vm));
         }
@@ -2902,7 +2902,7 @@ bool JSObject::putDirectIndexSlowOrBeyondVectorLength(ExecState* exec, unsigned
         
     case ALL_CONTIGUOUS_INDEXING_TYPES: {
         if (attributes) {
-            if (i < m_butterfly.get()->vectorLength())
+            if (i < m_butterfly->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
             return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, convertContiguousToArrayStorage(vm));
         }
@@ -2912,7 +2912,7 @@ bool JSObject::putDirectIndexSlowOrBeyondVectorLength(ExecState* exec, unsigned
 
     case ALL_ARRAY_STORAGE_INDEXING_TYPES:
         if (attributes) {
-            if (i < m_butterfly.get()->vectorLength())
+            if (i < m_butterfly->vectorLength())
                 return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, ensureArrayStorageExistsAndEnterDictionaryIndexingMode(vm));
         }
         return putDirectIndexBeyondVectorLengthWithArrayStorage(exec, i, value, attributes, mode, arrayStorage());
@@ -3015,8 +3015,8 @@ ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned desiredLength)
     if (hasIndexedProperties(indexingType())) {
         if (ArrayStorage* storage = arrayStorageOrNull())
             indexBias = storage->m_indexBias;
-        vectorLength = m_butterfly.get()->vectorLength();
-        length = m_butterfly.get()->publicLength();
+        vectorLength = m_butterfly->vectorLength();
+        length = m_butterfly->publicLength();
     }
 
     return getNewVectorLength(indexBias, vectorLength, length, desiredLength);
@@ -3132,7 +3132,7 @@ bool JSObject::increaseVectorLength(VM& vm, unsigned newLength)
 
 bool JSObject::ensureLengthSlow(VM& vm, unsigned length)
 {
-    Butterfly* butterfly = m_butterfly.get().get();
+    Butterfly* butterfly = m_butterfly.get();
     
     ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH);
     ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
@@ -3188,11 +3188,11 @@ void JSObject::reallocateAndShrinkButterfly(VM& vm, unsigned length)
 {
     ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH);
     ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
-    ASSERT(m_butterfly.get()->vectorLength() > length);
-    ASSERT(!m_butterfly.get()->indexingHeader()->preCapacity(structure()));
+    ASSERT(m_butterfly->vectorLength() > length);
+    ASSERT(!m_butterfly->indexingHeader()->preCapacity(structure()));
 
     DeferGC deferGC(vm.heap);
-    Butterfly* newButterfly = m_butterfly.get()->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
+    Butterfly* newButterfly = m_butterfly->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
     newButterfly->setVectorLength(length);
     newButterfly->setPublicLength(length);
     WTF::storeStoreFence();
@@ -3206,7 +3206,7 @@ Butterfly* JSObject::allocateMoreOutOfLineStorage(VM& vm, size_t oldSize, size_t
     // It's important that this function not rely on structure(), for the property
     // capacity, since we might have already mutated the structure in-place.
 
-    return Butterfly::createOrGrowPropertyStorage(m_butterfly.get().getMayBeNull(), vm, this, structure(vm), oldSize, newSize);
+    return Butterfly::createOrGrowPropertyStorage(m_butterfly.getMayBeNull(), vm, this, structure(vm), oldSize, newSize);
 }
 
 static JSCustomGetterSetterFunction* getCustomGetterSetterFunctionForGetterSetter(ExecState* exec, PropertyName propertyName, CustomGetterSetter* getterSetter, JSCustomGetterSetterFunction::Type type)
@@ -3552,7 +3552,7 @@ uint32_t JSObject::getEnumerableLength(ExecState* exec, JSObject* object)
     }
         
     case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
-        ArrayStorage* storage = object->m_butterfly.get()->arrayStorage();
+        ArrayStorage* storage = object->m_butterfly->arrayStorage();
         if (storage->m_sparseMap.get())
             return 0;
         
index 683d96638c39c239e246449ef88b3c13f00c234c..968c54a2bc5a01cbb0acf2e5d2bfc546f1f3a595 100644 (file)
@@ -24,9 +24,9 @@
 
 #include "ArrayConventions.h"
 #include "ArrayStorage.h"
-#include "AuxiliaryBarrier.h"
 #include "Butterfly.h"
 #include "CPU.h"
+#include "CagedBarrierPtr.h"
 #include "CallFrame.h"
 #include "ClassInfo.h"
 #include "CustomGetterSetter.h"
@@ -43,7 +43,6 @@
 #include "VM.h"
 #include "JSString.h"
 #include "SparseArrayValueMap.h"
-#include <wtf/CagedPtr.h>
 #include <wtf/StdLibExtras.h>
 
 namespace JSC {
@@ -178,14 +177,14 @@ public:
     {
         if (!hasIndexedProperties(indexingType()))
             return 0;
-        return m_butterfly.get()->publicLength();
+        return m_butterfly->publicLength();
     }
         
     unsigned getVectorLength()
     {
         if (!hasIndexedProperties(indexingType()))
             return 0;
-        return m_butterfly.get()->vectorLength();
+        return m_butterfly->vectorLength();
     }
     
     static bool putInlineForJSObject(JSCell*, ExecState*, PropertyName, JSValue, PutPropertySlot&);
@@ -221,7 +220,7 @@ public:
             case ALL_DOUBLE_INDEXING_TYPES:
             case ALL_CONTIGUOUS_INDEXING_TYPES:
             case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-                return propertyName < m_butterfly.get()->vectorLength();
+                return propertyName < m_butterfly->vectorLength();
             default:
                 RELEASE_ASSERT_NOT_REACHED();
                 return false;
@@ -255,7 +254,7 @@ public:
     
     bool canGetIndexQuickly(unsigned i)
     {
-        Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+        Butterfly* butterfly = m_butterfly.getMayBeNull();
         switch (indexingType()) {
         case ALL_BLANK_INDEXING_TYPES:
         case ALL_UNDECIDED_INDEXING_TYPES:
@@ -281,7 +280,7 @@ public:
         
     JSValue getIndexQuickly(unsigned i)
     {
-        Butterfly* butterfly = m_butterfly.get().get();
+        Butterfly* butterfly = m_butterfly.get();
         switch (indexingType()) {
         case ALL_INT32_INDEXING_TYPES:
             return jsNumber(butterfly->contiguous()[i].get().asInt32());
@@ -299,7 +298,7 @@ public:
         
     JSValue tryGetIndexQuickly(unsigned i) const
     {
-        Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+        Butterfly* butterfly = m_butterfly.getMayBeNull();
         switch (indexingType()) {
         case ALL_BLANK_INDEXING_TYPES:
         case ALL_UNDECIDED_INDEXING_TYPES:
@@ -353,7 +352,7 @@ public:
         
     bool canSetIndexQuickly(unsigned i)
     {
-        Butterfly* butterfly = m_butterfly.get().getMayBeNull();
+        Butterfly* butterfly = m_butterfly.getMayBeNull();
         switch (indexingType()) {
         case ALL_BLANK_INDEXING_TYPES:
         case ALL_UNDECIDED_INDEXING_TYPES:
@@ -376,7 +375,7 @@ public:
         
     void setIndexQuickly(VM& vm, unsigned i, JSValue v)
     {
-        Butterfly* butterfly = m_butterfly.get().get();
+        Butterfly* butterfly = m_butterfly.get();
         switch (indexingType()) {
         case ALL_INT32_INDEXING_TYPES: {
             ASSERT(i < butterfly->vectorLength());
@@ -436,7 +435,7 @@ public:
     ALWAYS_INLINE void initializeIndex(ObjectInitializationScope& scope, unsigned i, JSValue v, IndexingType indexingType)
     {
         VM& vm = scope.vm();
-        Butterfly* butterfly = m_butterfly.get().get();
+        Butterfly* butterfly = m_butterfly.get();
         switch (indexingType) {
         case ALL_UNDECIDED_INDEXING_TYPES: {
             setIndexQuicklyToUndecided(vm, i, v);
@@ -493,7 +492,7 @@ public:
     // barriers. This implies not having any data format conversions.
     ALWAYS_INLINE void initializeIndexWithoutBarrier(ObjectInitializationScope&, unsigned i, JSValue v, IndexingType indexingType)
     {
-        Butterfly* butterfly = m_butterfly.get().get();
+        Butterfly* butterfly = m_butterfly.get();
         switch (indexingType) {
         case ALL_UNDECIDED_INDEXING_TYPES: {
             RELEASE_ASSERT_NOT_REACHED();
@@ -542,7 +541,7 @@ public:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
             return false;
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return !!m_butterfly.get()->arrayStorage()->m_sparseMap;
+            return !!m_butterfly->arrayStorage()->m_sparseMap;
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -559,7 +558,7 @@ public:
         case ALL_CONTIGUOUS_INDEXING_TYPES:
             return false;
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return m_butterfly.get()->arrayStorage()->inSparseMode();
+            return m_butterfly->arrayStorage()->inSparseMode();
         default:
             RELEASE_ASSERT_NOT_REACHED();
             return false;
@@ -669,11 +668,11 @@ public:
         return inlineStorageUnsafe();
     }
         
-    const Butterfly* butterfly() const { return m_butterfly.get().getMayBeNull(); }
-    Butterfly* butterfly() { return m_butterfly.get().getMayBeNull(); }
+    const Butterfly* butterfly() const { return m_butterfly.getMayBeNull(); }
+    Butterfly* butterfly() { return m_butterfly.getMayBeNull(); }
     
-    ConstPropertyStorage outOfLineStorage() const { return m_butterfly.get()->propertyStorage(); }
-    PropertyStorage outOfLineStorage() { return m_butterfly.get()->propertyStorage(); }
+    ConstPropertyStorage outOfLineStorage() const { return m_butterfly->propertyStorage(); }
+    PropertyStorage outOfLineStorage() { return m_butterfly->propertyStorage(); }
 
     const WriteBarrierBase<Unknown>* locationForOffset(PropertyOffset offset) const
     {
@@ -807,7 +806,7 @@ public:
     ContiguousJSValues ensureInt32(VM& vm)
     {
         if (LIKELY(hasInt32(indexingType())))
-            return m_butterfly.get()->contiguousInt32();
+            return m_butterfly->contiguousInt32();
             
         return ensureInt32Slow(vm);
     }
@@ -819,7 +818,7 @@ public:
     ContiguousDoubles ensureDouble(VM& vm)
     {
         if (LIKELY(hasDouble(indexingType())))
-            return m_butterfly.get()->contiguousDouble();
+            return m_butterfly->contiguousDouble();
             
         return ensureDoubleSlow(vm);
     }
@@ -829,7 +828,7 @@ public:
     ContiguousJSValues ensureContiguous(VM& vm)
     {
         if (LIKELY(hasContiguous(indexingType())))
-            return m_butterfly.get()->contiguous();
+            return m_butterfly->contiguous();
             
         return ensureContiguousSlow(vm);
     }
@@ -841,7 +840,7 @@ public:
     ArrayStorage* ensureArrayStorage(VM& vm)
     {
         if (LIKELY(hasAnyArrayStorage(indexingType())))
-            return m_butterfly.get()->arrayStorage();
+            return m_butterfly->arrayStorage();
 
         return ensureArrayStorageSlow(vm);
     }
@@ -893,7 +892,7 @@ protected:
     ArrayStorage* arrayStorage()
     {
         ASSERT(hasAnyArrayStorage(indexingType()));
-        return m_butterfly.get()->arrayStorage();
+        return m_butterfly->arrayStorage();
     }
         
     // Call this if you want to predicate some actions on whether or not the
@@ -902,7 +901,7 @@ protected:
     {
         switch (indexingType()) {
         case ALL_ARRAY_STORAGE_INDEXING_TYPES:
-            return m_butterfly.get()->arrayStorage();
+            return m_butterfly->arrayStorage();
                 
         default:
             return 0;
@@ -968,13 +967,13 @@ protected:
         ASSERT(length <= MAX_STORAGE_VECTOR_LENGTH);
         ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
 
-        if (m_butterfly.get()->vectorLength() < length) {
+        if (m_butterfly->vectorLength() < length) {
             if (!ensureLengthSlow(vm, length))
                 return false;
         }
             
-        if (m_butterfly.get()->publicLength() < length)
-            m_butterfly.get()->setPublicLength(length);
+        if (m_butterfly->publicLength() < length)
+            m_butterfly->setPublicLength(length);
         return true;
     }
         
@@ -1046,7 +1045,7 @@ private:
     PropertyOffset prepareToPutDirectWithoutTransition(VM&, PropertyName, unsigned attributes, StructureID, Structure*);
 
 protected:
-    AuxiliaryBarrier<CagedPtr<Butterfly>> m_butterfly;
+    CagedBarrierPtr<Gigacage::JSValue, Butterfly> m_butterfly;
 #if USE(JSVALUE32_64)
 private:
     uint32_t m_padding;
index a4c22a87dce3642582dc03c92866cab9bdf4603d..7e1938eb4c15dc4be8fa527c00da135d0591fee3 100644 (file)
@@ -41,7 +41,7 @@ ALWAYS_INLINE JSArray* tryCreateUninitializedRegExpMatchesArray(ObjectInitializa
 
     JSGlobalObject* globalObject = structure->globalObject();
     bool createUninitialized = globalObject->isOriginalArrayStructure(structure);
-    void* temp = vm.auxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
+    void* temp = vm.jsValueGigacageAuxiliarySpace.tryAllocate(deferralContext, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
     if (UNLIKELY(!temp))
         return nullptr;
     Butterfly* butterfly = Butterfly::fromBase(temp, 0, structure->outOfLineCapacity());
index 998c4ab860300e871b6de5b962a29030d9aed461..f5f4d2fe77b2e153e0537de868afb16b83b65256 100644 (file)
@@ -165,15 +165,19 @@ VM::VM(VMType vmType, HeapType heapType)
     , m_runLoop(CFRunLoopGetCurrent())
 #endif // USE(CF)
     , heap(this, heapType)
-    , auxiliarySpace("Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), &GigacageAlignedMemoryAllocator::instance())
-    , cellSpace("JSCell", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::JSCell), &FastMallocAlignedMemoryAllocator::instance())
-    , destructibleCellSpace("Destructible JSCell", heap, AllocatorAttributes(NeedsDestruction, HeapCell::JSCell), &FastMallocAlignedMemoryAllocator::instance())
-    , stringSpace("JSString", heap, &FastMallocAlignedMemoryAllocator::instance())
-    , destructibleObjectSpace("JSDestructibleObject", heap, &FastMallocAlignedMemoryAllocator::instance())
-    , eagerlySweptDestructibleObjectSpace("Eagerly Swept JSDestructibleObject", heap, &FastMallocAlignedMemoryAllocator::instance())
-    , segmentedVariableObjectSpace("JSSegmentedVariableObjectSpace", heap, &FastMallocAlignedMemoryAllocator::instance())
+    , fastMallocAllocator(std::make_unique<FastMallocAlignedMemoryAllocator>())
+    , primitiveGigacageAllocator(std::make_unique<GigacageAlignedMemoryAllocator>(Gigacage::Primitive))
+    , jsValueGigacageAllocator(std::make_unique<GigacageAlignedMemoryAllocator>(Gigacage::JSValue))
+    , primitiveGigacageAuxiliarySpace("Primitive Gigacage Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), primitiveGigacageAllocator.get())
+    , jsValueGigacageAuxiliarySpace("JSValue Gigacage Auxiliary", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::Auxiliary), jsValueGigacageAllocator.get())
+    , cellSpace("JSCell", heap, AllocatorAttributes(DoesNotNeedDestruction, HeapCell::JSCell), fastMallocAllocator.get())
+    , destructibleCellSpace("Destructible JSCell", heap, AllocatorAttributes(NeedsDestruction, HeapCell::JSCell), fastMallocAllocator.get())
+    , stringSpace("JSString", heap, fastMallocAllocator.get())
+    , destructibleObjectSpace("JSDestructibleObject", heap, fastMallocAllocator.get())
+    , eagerlySweptDestructibleObjectSpace("Eagerly Swept JSDestructibleObject", heap, fastMallocAllocator.get())
+    , segmentedVariableObjectSpace("JSSegmentedVariableObjectSpace", heap, fastMallocAllocator.get())
 #if ENABLE(WEBASSEMBLY)
-    , webAssemblyCodeBlockSpace("JSWebAssemblyCodeBlockSpace", heap, &FastMallocAlignedMemoryAllocator::instance())
+    , webAssemblyCodeBlockSpace("JSWebAssemblyCodeBlockSpace", heap, fastMallocAllocator.get())
 #endif
     , vmType(vmType)
     , clientData(0)
@@ -211,7 +215,7 @@ VM::VM(VMType vmType, HeapType heapType)
     , m_codeCache(std::make_unique<CodeCache>())
     , m_builtinExecutables(std::make_unique<BuiltinExecutables>(*this))
     , m_typeProfilerEnabledCount(0)
-    , m_gigacageEnabled(IsWatched)
+    , m_primitiveGigacageEnabled(IsWatched)
     , m_controlFlowProfilerEnabledCount(0)
     , m_shadowChicken(std::make_unique<ShadowChicken>())
 {
@@ -290,7 +294,7 @@ VM::VM(VMType vmType, HeapType heapType)
     initializeHostCallReturnValue(); // This is needed to convince the linker not to drop host call return support.
 #endif
     
-    Gigacage::addDisableCallback(gigacageDisabledCallback, this);
+    Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabledCallback, this);
 
     heap.notifyIsSafeToCollect();
     
@@ -354,7 +358,7 @@ VM::~VM()
 {
     auto destructionLocker = holdLock(s_destructionLock.read());
     
-    Gigacage::removeDisableCallback(gigacageDisabledCallback, this);
+    Gigacage::removePrimitiveDisableCallback(primitiveGigacageDisabledCallback, this);
     promiseDeferredTimer->stopRunningTasks();
 #if ENABLE(WEBASSEMBLY)
     if (Wasm::existingWorklistOrNull())
@@ -423,21 +427,21 @@ VM::~VM()
 #endif
 }
 
-void VM::gigacageDisabledCallback(void* argument)
+void VM::primitiveGigacageDisabledCallback(void* argument)
 {
-    static_cast<VM*>(argument)->gigacageDisabled();
+    static_cast<VM*>(argument)->primitiveGigacageDisabled();
 }
 
-void VM::gigacageDisabled()
+void VM::primitiveGigacageDisabled()
 {
     if (m_apiLock->currentThreadIsHoldingLock()) {
-        m_gigacageEnabled.fireAll(*this, "Gigacage disabled");
+        m_primitiveGigacageEnabled.fireAll(*this, "Primitive gigacage disabled");
         return;
     }
  
     // This is totally racy, and that's OK. The point is, it's up to the user to ensure that they pass the
     // uncaged buffer in a nicely synchronized manner.
-    m_needToFireGigacageEnabled = true;
+    m_needToFirePrimitiveGigacageEnabled = true;
 }
 
 void VM::setLastStackTop(void* lastStackTop)
index 18a3b410140006144c99b8660d9504e0166830c6..06ba1bb3975c7a06fe7538d489eb5a8e0a8f0d64 100644 (file)
@@ -62,6 +62,7 @@
 #include <wtf/Deque.h>
 #include <wtf/DoublyLinkedList.h>
 #include <wtf/Forward.h>
+#include <wtf/Gigacage.h>
 #include <wtf/HashMap.h>
 #include <wtf/HashSet.h>
 #include <wtf/StackBounds.h>
@@ -95,6 +96,8 @@ class DOMAttributeGetterSetter;
 class ExecState;
 class Exception;
 class ExceptionScope;
+class FastMallocAlignedMemoryAllocator;
+class GigacageAlignedMemoryAllocator;
 class HandleStack;
 class TypeProfiler;
 class TypeProfilerLog;
@@ -285,7 +288,30 @@ private:
 public:
     Heap heap;
     
-    Subspace auxiliarySpace;
+    std::unique_ptr<FastMallocAlignedMemoryAllocator> fastMallocAllocator;
+    std::unique_ptr<GigacageAlignedMemoryAllocator> primitiveGigacageAllocator;
+    std::unique_ptr<GigacageAlignedMemoryAllocator> jsValueGigacageAllocator;
+    
+    Subspace primitiveGigacageAuxiliarySpace; // Typed arrays, strings, bitvectors, etc go here.
+    Subspace jsValueGigacageAuxiliarySpace; // Butterflies, arrays of JSValues, etc go here.
+
+    // We make cross-cutting assumptions about typed arrays being in the primitive Gigacage and butterflies
+    // being in the JSValue gigacage. For some types, it's super obvious where they should go, and so we
+    // can hardcode that fact. But sometimes it's not clear, so we abstract it by having a Gigacage::Kind
+    // constant somewhere.
+    // FIXME: Maybe it would be better if everyone abstracted this?
+    // https://bugs.webkit.org/show_bug.cgi?id=175248
+    ALWAYS_INLINE Subspace& gigacageAuxiliarySpace(Gigacage::Kind kind)
+    {
+        switch (kind) {
+        case Gigacage::Primitive:
+            return primitiveGigacageAuxiliarySpace;
+        case Gigacage::JSValue:
+            return jsValueGigacageAuxiliarySpace;
+        }
+        RELEASE_ASSERT_NOT_REACHED();
+        return primitiveGigacageAuxiliarySpace;
+    }
     
     // Whenever possible, use subspaceFor<CellType>(vm) to get one of these subspaces.
     Subspace cellSpace;
@@ -524,11 +550,11 @@ public:
     void* lastStackTop() { return m_lastStackTop; }
     void setLastStackTop(void*);
     
-    void fireGigacageEnabledIfNecessary()
+    void firePrimitiveGigacageEnabledIfNecessary()
     {
-        if (m_needToFireGigacageEnabled) {
-            m_needToFireGigacageEnabled = false;
-            m_gigacageEnabled.fireAll(*this, "Gigacage disabled asynchronously");
+        if (m_needToFirePrimitiveGigacageEnabled) {
+            m_needToFirePrimitiveGigacageEnabled = false;
+            m_primitiveGigacageEnabled.fireAll(*this, "Primitive gigacage disabled asynchronously");
         }
     }
 
@@ -633,7 +659,7 @@ public:
     // FIXME: Use AtomicString once it got merged with Identifier.
     JS_EXPORT_PRIVATE void addImpureProperty(const String&);
     
-    InlineWatchpointSet& gigacageEnabled() { return m_gigacageEnabled; }
+    InlineWatchpointSet& primitiveGigacageEnabled() { return m_primitiveGigacageEnabled; }
 
     BuiltinExecutables* builtinExecutables() { return m_builtinExecutables.get(); }
 
@@ -741,8 +767,8 @@ private:
     void verifyExceptionCheckNeedIsSatisfied(unsigned depth, ExceptionEventLocation&);
 #endif
     
-    static void gigacageDisabledCallback(void*);
-    void gigacageDisabled();
+    static void primitiveGigacageDisabledCallback(void*);
+    void primitiveGigacageDisabled();
 
 #if ENABLE(ASSEMBLER)
     bool m_canUseAssembler;
@@ -787,8 +813,8 @@ private:
     std::unique_ptr<TypeProfiler> m_typeProfiler;
     std::unique_ptr<TypeProfilerLog> m_typeProfilerLog;
     unsigned m_typeProfilerEnabledCount;
-    bool m_needToFireGigacageEnabled { false };
-    InlineWatchpointSet m_gigacageEnabled;
+    bool m_needToFirePrimitiveGigacageEnabled { false };
+    InlineWatchpointSet m_primitiveGigacageEnabled;
     FunctionHasExecutedCache m_functionHasExecutedCache;
     std::unique_ptr<ControlFlowProfiler> m_controlFlowProfiler;
     unsigned m_controlFlowProfilerEnabledCount;
index db7731f57fea8573a2601cfcf7a32d252ebe5df4..94d064c36d3e7c4a7adc838b74a11625d00eb440 100644 (file)
@@ -100,7 +100,7 @@ public:
             if (m_memories.size() >= m_maxCount)
                 return MemoryResult(nullptr, MemoryResult::SyncGCAndRetry);
             
-            void* result = Gigacage::tryAllocateVirtualPages(Memory::fastMappedBytes());
+            void* result = Gigacage::tryAllocateVirtualPages(Gigacage::Primitive, Memory::fastMappedBytes());
             if (!result)
                 return MemoryResult(nullptr, MemoryResult::SyncGCAndRetry);
             
@@ -121,7 +121,7 @@ public:
     {
         {
             auto holder = holdLock(m_lock);
-            Gigacage::freeVirtualPages(basePtr, Memory::fastMappedBytes());
+            Gigacage::freeVirtualPages(Gigacage::Primitive, basePtr, Memory::fastMappedBytes());
             m_memories.removeFirst(basePtr);
         }
         
@@ -312,7 +312,7 @@ RefPtr<Memory> Memory::create(VM& vm, PageCount initial, PageCount maximum)
     if (!initialBytes)
         return adoptRef(new Memory(initial, maximum));
     
-    void* slowMemory = Gigacage::tryAlignedMalloc(WTF::pageSize(), initialBytes);
+    void* slowMemory = Gigacage::tryAlignedMalloc(Gigacage::Primitive, WTF::pageSize(), initialBytes);
     if (!slowMemory) {
         memoryManager().freePhysicalBytes(initialBytes);
         return nullptr;
@@ -331,7 +331,7 @@ Memory::~Memory()
             memoryManager().freeVirtualPages(m_memory);
             break;
         case MemoryMode::BoundsChecking:
-            Gigacage::alignedFree(m_memory);
+            Gigacage::alignedFree(Gigacage::Primitive, m_memory);
             break;
         }
     }
@@ -378,13 +378,13 @@ bool Memory::grow(VM& vm, PageCount newSize)
     case MemoryMode::BoundsChecking: {
         RELEASE_ASSERT(maximum().bytes() != 0);
         
-        void* newMemory = Gigacage::tryAlignedMalloc(WTF::pageSize(), desiredSize);
+        void* newMemory = Gigacage::tryAlignedMalloc(Gigacage::Primitive, WTF::pageSize(), desiredSize);
         if (!newMemory)
             return false;
         memcpy(newMemory, m_memory, m_size);
         memset(static_cast<char*>(newMemory) + m_size, 0, desiredSize - m_size);
         if (m_memory)
-            Gigacage::alignedFree(m_memory);
+            Gigacage::alignedFree(Gigacage::Primitive, m_memory);
         m_memory = newMemory;
         m_mappedCapacity = desiredSize;
         m_size = desiredSize;
index 53b314e2e57ab91fe431da2b19a147de8720d344..cb8a43d70c7d7c98c573cfcd335d10719f879df7 100644 (file)
@@ -1,3 +1,39 @@
+2017-08-06  Filip Pizlo  <fpizlo@apple.com>
+
+        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
+        https://bugs.webkit.org/show_bug.cgi?id=174919
+
+        Reviewed by Keith Miller.
+        
+        This mirrors the changes from bmalloc/Gigacage.h.
+        
+        Also it teaches CagedPtr how to reason about multiple gigacages.
+
+        * wtf/CagedPtr.h:
+        (WTF::CagedPtr::get const):
+        (WTF::CagedPtr::operator[] const):
+        * wtf/Gigacage.cpp:
+        (Gigacage::tryMalloc):
+        (Gigacage::tryAllocateVirtualPages):
+        (Gigacage::freeVirtualPages):
+        (Gigacage::tryAlignedMalloc):
+        (Gigacage::alignedFree):
+        (Gigacage::free):
+        * wtf/Gigacage.h:
+        (Gigacage::disablePrimitiveGigacage):
+        (Gigacage::addPrimitiveDisableCallback):
+        (Gigacage::removePrimitiveDisableCallback):
+        (Gigacage::name):
+        (Gigacage::basePtr):
+        (Gigacage::caged):
+        (Gigacage::isCaged):
+        (Gigacage::tryAlignedMalloc):
+        (Gigacage::alignedFree):
+        (Gigacage::free):
+        (Gigacage::disableGigacage): Deleted.
+        (Gigacage::addDisableCallback): Deleted.
+        (Gigacage::removeDisableCallback): Deleted.
+
 2017-08-07  Brian Burg  <bburg@apple.com>
 
         Remove CANVAS_PATH compilation guard
index 3f77645df999c49c7087a65a753bd303a6cb31af..031a69413ce09b2b6cae58c771f7a3aad2ed5fb3 100644 (file)
 
 namespace WTF {
 
-template<typename T>
+template<Gigacage::Kind passedKind, typename T>
 class CagedPtr {
 public:
+    static constexpr Gigacage::Kind kind = passedKind;
+    
     CagedPtr(T* ptr = nullptr)
         : m_ptr(ptr)
     {
@@ -40,7 +42,7 @@ public:
     T* get() const
     {
         ASSERT(m_ptr);
-        return Gigacage::caged(m_ptr);
+        return Gigacage::caged(kind, m_ptr);
     }
     
     T* getMayBeNull() const
@@ -67,6 +69,9 @@ public:
     
     T& operator*() const { return *get(); }
     T* operator->() const { return get(); }
+
+    template<typename IndexType>
+    T& operator[](IndexType index) const { return get()[index]; }
     
 private:
     T* m_ptr;
index be3115c5757e8540b4d3f6ea4c726f1e92787954..b59c03eb8604f8ce49ca212264c59639a06e5952 100644 (file)
 #if defined(USE_SYSTEM_MALLOC) && USE_SYSTEM_MALLOC
 
 extern "C" {
-const void* g_gigacageBasePtr;
+void* const g_gigacageBasePtr;
 }
 
 namespace Gigacage {
 
-void* tryMalloc(size_t size)
+void* tryMalloc(Kind, size_t size)
 {
     auto result = tryFastMalloc(size);
     void* realResult;
@@ -47,12 +47,12 @@ void* tryMalloc(size_t size)
     return nullptr;
 }
 
-void* tryAllocateVirtualPages(size_t size)
+void* tryAllocateVirtualPages(Kind, size_t size)
 {
     return OSAllocator::reserveUncommitted(size);
 }
 
-void freeVirtualPages(void* basePtr, size_t size)
+void freeVirtualPages(Kind, void* basePtr, size_t size)
 {
     OSAllocator::releaseDecommitted(basePtr, size);
 }
@@ -67,42 +67,51 @@ namespace Gigacage {
 // and stay scrambled except just before use.
 // https://bugs.webkit.org/show_bug.cgi?id=175035
 
-void* tryAlignedMalloc(size_t alignment, size_t size)
+void* tryAlignedMalloc(Kind kind, size_t alignment, size_t size)
 {
-    void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc::HeapKind::Gigacage);
+    void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc::heapKind(kind));
     WTF::compilerFence();
     return result;
 }
 
-void alignedFree(void* p)
+void alignedFree(Kind kind, void* p)
 {
-    bmalloc::api::free(p, bmalloc::HeapKind::Gigacage);
+    if (!p)
+        return;
+    RELEASE_ASSERT(isCaged(kind, p));
+    bmalloc::api::free(p, bmalloc::heapKind(kind));
     WTF::compilerFence();
 }
 
-void* tryMalloc(size_t size)
+void* tryMalloc(Kind kind, size_t size)
 {
-    void* result = bmalloc::api::tryMalloc(size, bmalloc::HeapKind::Gigacage);
+    void* result = bmalloc::api::tryMalloc(size, bmalloc::heapKind(kind));
     WTF::compilerFence();
     return result;
 }
 
-void free(void* p)
+void free(Kind kind, void* p)
 {
-    bmalloc::api::free(p, bmalloc::HeapKind::Gigacage);
+    if (!p)
+        return;
+    RELEASE_ASSERT(isCaged(kind, p));
+    bmalloc::api::free(p, bmalloc::heapKind(kind));
     WTF::compilerFence();
 }
 
-void* tryAllocateVirtualPages(size_t size)
+void* tryAllocateVirtualPages(Kind kind, size_t size)
 {
-    void* result = bmalloc::api::tryLargeMemalignVirtual(WTF::pageSize(), size, bmalloc::HeapKind::Gigacage);
+    void* result = bmalloc::api::tryLargeMemalignVirtual(WTF::pageSize(), size, bmalloc::heapKind(kind));
     WTF::compilerFence();
     return result;
 }
 
-void freeVirtualPages(void* basePtr, size_t)
+void freeVirtualPages(Kind kind, void* basePtr, size_t)
 {
-    bmalloc::api::freeLargeVirtual(basePtr, bmalloc::HeapKind::Gigacage);
+    if (!basePtr)
+        return;
+    RELEASE_ASSERT(isCaged(kind, basePtr));
+    bmalloc::api::freeLargeVirtual(basePtr, bmalloc::heapKind(kind));
     WTF::compilerFence();
 }
 
index dd2df2b2959e1d74961964a27e75de35a7cc6505..689ca0bbfe7d8c11032f204625741751981508ce 100644 (file)
 #define GIGACAGE_ENABLED 0
 
 extern "C" {
-extern WTF_EXPORTDATA const void* g_gigacageBasePtr;
+extern WTF_EXPORTDATA void* const g_gigacageBasePtr;
 }
 
 namespace Gigacage {
 
+enum Kind {
+    Primitive,
+    JSValue
+};
+
 inline void ensureGigacage() { }
-inline void disableGigacage() { }
+inline void disablePrimitiveGigacage() { }
 inline bool shouldBeEnabled() { return false; }
 
-inline void addDisableCallback(void (*)(void*), void*) { }
-inline void removeDisableCallback(void (*)(void*), void*) { }
+inline void addPrimitiveDisableCallback(void (*)(void*), void*) { }
+inline void removePrimitiveDisableCallback(void (*)(void*), void*) { }
+
+ALWAYS_INLINE const char* name(Kind kind)
+{
+    switch (kind) {
+    case Primitive:
+        return "Primitive";
+    case JSValue:
+        return "JSValue";
+    }
+    RELEASE_ASSERT_NOT_REACHED();
+    return nullptr;
+}
+
+ALWAYS_INLINE void* basePtr(Kind)
+{
+    return g_gigacageBasePtr;
+}
 
 template<typename T>
-inline T* caged(T* ptr) { return ptr; }
+inline T* caged(Kind, T* ptr) { return ptr; }
 
-inline bool isCaged(const void*) { return false; }
+inline bool isCaged(Kind, const void*) { return false; }
 
-inline void* tryAlignedMalloc(size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); }
-inline void alignedFree(void* p) { fastAlignedFree(p); }
-WTF_EXPORT_PRIVATE void* tryMalloc(size_t size);
-inline void free(void* p) { fastFree(p); }
+inline void* tryAlignedMalloc(Kind, size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); }
+inline void alignedFree(Kind, void* p) { fastAlignedFree(p); }
+WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t size);
+inline void free(Kind, void* p) { fastFree(p); }
 
-WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(size_t size);
-WTF_EXPORT_PRIVATE void freeVirtualPages(void* basePtr, size_t size);
+WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(Kind, size_t size);
+WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size);
 
 } // namespace Gigacage
 #else
@@ -63,13 +85,13 @@ WTF_EXPORT_PRIVATE void freeVirtualPages(void* basePtr, size_t size);
 
 namespace Gigacage {
 
-WTF_EXPORT_PRIVATE void* tryAlignedMalloc(size_t alignment, size_t size);
-WTF_EXPORT_PRIVATE void alignedFree(void*);
-WTF_EXPORT_PRIVATE void* tryMalloc(size_t);
-WTF_EXPORT_PRIVATE void free(void*);
+WTF_EXPORT_PRIVATE void* tryAlignedMalloc(Kind, size_t alignment, size_t size);
+WTF_EXPORT_PRIVATE void alignedFree(Kind, void*);
+WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t);
+WTF_EXPORT_PRIVATE void free(Kind, void*);
 
-WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(size_t size);
-WTF_EXPORT_PRIVATE void freeVirtualPages(void* basePtr, size_t size);
+WTF_EXPORT_PRIVATE void* tryAllocateVirtualPages(Kind, size_t size);
+WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size);
 
 } // namespace Gigacage
 #endif
index bd713d3626f50171caf18ffce673655d91e00c89..6b474aa50bb601d625cd80712df3fe63009270fa 100644 (file)
@@ -1,3 +1,20 @@
+2017-08-06  Filip Pizlo  <fpizlo@apple.com>
+
+        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
+        https://bugs.webkit.org/show_bug.cgi?id=174919
+
+        Reviewed by Keith Miller.
+
+        No new tests because no change in behavior.
+        
+        Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be
+        specific about which Gigacage we're using.
+
+        * bindings/js/WebCoreJSClientData.cpp:
+        (WebCore::JSVMClientData::JSVMClientData):
+        * platform/graphics/cocoa/GPUBufferMetal.mm:
+        (WebCore::GPUBuffer::GPUBuffer):
+
 2017-08-07  Basuke Suzuki  <Basuke.Suzuki@sony.com>
 
         [Curl] Add abstraction layer of cookie jar implementation for Curl port
index 2974a5537cdf913efe40fabee21883c672ecc426..fe73fce77d4773643c37c9a2bdc7cdc67ce35653 100644 (file)
@@ -44,8 +44,8 @@ namespace WebCore {
 JSVMClientData::JSVMClientData(VM& vm)
     : m_builtinFunctions(vm)
     , m_builtinNames(&vm)
-    , m_outputConstraintSpace("WebCore Wrapper w/ Output Constraint", vm.heap, &FastMallocAlignedMemoryAllocator::instance())
-    , m_globalObjectOutputConstraintSpace("WebCore Global Object w/ Output Constraint", vm.heap, &FastMallocAlignedMemoryAllocator::instance())
+    , m_outputConstraintSpace("WebCore Wrapper w/ Output Constraint", vm.heap, vm.fastMallocAllocator.get())
+    , m_globalObjectOutputConstraintSpace("WebCore Global Object w/ Output Constraint", vm.heap, vm.fastMallocAllocator.get())
 {
 }
 
index c1456097b068d7ec86ca0d627ef4a0dbf1e7b1bb..ad46220b25a3fdd1e8566351f0f1ecf3ae11993e 100644 (file)
@@ -45,11 +45,11 @@ GPUBuffer::GPUBuffer(GPUDevice* device, ArrayBufferView* data)
     
     size_t pageSize = WTF::pageSize();
     size_t pageAlignedSize = roundUpToMultipleOf(pageSize, data->byteLength());
-    void* pageAlignedCopy = Gigacage::tryAlignedMalloc(pageSize, pageAlignedSize);
+    void* pageAlignedCopy = Gigacage::tryAlignedMalloc(Gigacage::Primitive, pageSize, pageAlignedSize);
     if (!pageAlignedCopy)
         return;
     memcpy(pageAlignedCopy, data->baseAddress(), data->byteLength());
-    m_contents = ArrayBuffer::createFromBytes(pageAlignedCopy, data->byteLength(), [] (void* ptr) { Gigacage::alignedFree(ptr); });
+    m_contents = ArrayBuffer::createFromBytes(pageAlignedCopy, data->byteLength(), [] (void* ptr) { Gigacage::alignedFree(Gigacage::Primitive, ptr); });
     m_contents->ref();
     ArrayBuffer* capturedContents = m_contents.get();
     m_buffer = adoptNS((MTLBuffer *)[device->platformDevice() newBufferWithBytesNoCopy:m_contents->data() length:pageAlignedSize options:MTLResourceOptionCPUCacheModeDefault deallocator:^(void*, NSUInteger) { capturedContents->deref(); }]);
index 23d0deba703c310c3379f69f1df698cd74510e9c..fd15088817d4b21b797f1c6cc673585530423225 100644 (file)
@@ -1,3 +1,17 @@
+2017-08-06  Filip Pizlo  <fpizlo@apple.com>
+
+        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
+        https://bugs.webkit.org/show_bug.cgi?id=174919
+
+        Reviewed by Keith Miller.
+        
+        The disable callback is all about the primitive gigacage.
+
+        * WebProcess/WebProcess.cpp:
+        (WebKit::primitiveGigacageDisabled):
+        (WebKit::m_webSQLiteDatabaseTracker):
+        (WebKit::gigacageDisabled): Deleted.
+
 2017-08-07  Brian Burg  <bburg@apple.com>
 
         Remove CANVAS_PATH compilation guard
index a3ee40172252dd864a72ccd0eff23959893eddf8..18031a1a388c83f2b9e56d82ec8a28d3fb2c8306 100644 (file)
@@ -146,7 +146,7 @@ static const Seconds nonVisibleProcessCleanupDelay { 10_s };
 
 namespace WebKit {
 
-static void gigacageDisabled(void*)
+static void primitiveGigacageDisabled(void*)
 {
     UNREACHABLE_FOR_PLATFORM();
 }
@@ -203,7 +203,7 @@ WebProcess::WebProcess()
     });
 
     if (Gigacage::shouldBeEnabled())
-        Gigacage::addDisableCallback(gigacageDisabled, nullptr);
+        Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabled, nullptr);
 }
 
 WebProcess::~WebProcess()
index 61ad2ece653e2e4dde1baa0eb5793da6e166d5c0..5feac56d303c0dbe359260fd06f036b011e82e0e 100644 (file)
@@ -1,3 +1,68 @@
+2017-08-06  Filip Pizlo  <fpizlo@apple.com>
+
+        Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
+        https://bugs.webkit.org/show_bug.cgi?id=174919
+
+        Reviewed by Keith Miller.
+        
+        This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of
+        HeapKind, PrimitiveGigacage and JSValueGigacage.
+        
+        The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and
+        NO_INLINE into BNO_INLINE.
+
+        * bmalloc.xcodeproj/project.pbxproj:
+        * bmalloc/Allocator.cpp:
+        (bmalloc::Allocator::refillAllocatorSlowCase):
+        (bmalloc::Allocator::refillAllocator):
+        (bmalloc::Allocator::allocateLarge):
+        (bmalloc::Allocator::allocateLogSizeClass):
+        * bmalloc/AsyncTask.h:
+        * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h.
+        * bmalloc/Cache.cpp:
+        (bmalloc::Cache::tryAllocateSlowCaseNullCache):
+        (bmalloc::Cache::allocateSlowCaseNullCache):
+        (bmalloc::Cache::deallocateSlowCaseNullCache):
+        (bmalloc::Cache::reallocateSlowCaseNullCache):
+        * bmalloc/Deallocator.cpp:
+        * bmalloc/Gigacage.cpp:
+        (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
+        (Gigacage::ensureGigacage):
+        (Gigacage::disablePrimitiveGigacage):
+        (Gigacage::addPrimitiveDisableCallback):
+        (Gigacage::removePrimitiveDisableCallback):
+        (Gigacage::Callbacks::Callbacks): Deleted.
+        (Gigacage::disableGigacage): Deleted.
+        (Gigacage::addDisableCallback): Deleted.
+        (Gigacage::removeDisableCallback): Deleted.
+        * bmalloc/Gigacage.h:
+        (Gigacage::name):
+        (Gigacage::basePtr):
+        (Gigacage::forEachKind):
+        (Gigacage::caged):
+        (Gigacage::isCaged):
+        * bmalloc/Heap.cpp:
+        (bmalloc::Heap::Heap):
+        (bmalloc::Heap::usingGigacage):
+        (bmalloc::Heap::gigacageBasePtr):
+        * bmalloc/Heap.h:
+        * bmalloc/HeapKind.h:
+        (bmalloc::isGigacage):
+        (bmalloc::gigacageKind):
+        (bmalloc::heapKind):
+        * bmalloc/Inline.h: Removed.
+        * bmalloc/Map.h:
+        * bmalloc/PerProcess.h:
+        (bmalloc::PerProcess<T>::getFastCase):
+        (bmalloc::PerProcess<T>::get):
+        (bmalloc::PerProcess<T>::getSlowCase):
+        * bmalloc/PerThread.h:
+        (bmalloc::PerThread<T>::getFastCase):
+        * bmalloc/Vector.h:
+        (bmalloc::Vector<T>::push):
+        (bmalloc::Vector<T>::shrinkCapacity):
+        (bmalloc::Vector<T>::growCapacity):
+
 2017-08-02  Filip Pizlo  <fpizlo@apple.com>
 
         If Gigacage is disabled, bmalloc should service large aligned memory allocation requests through vmAllocate
index 0b01f6990e74fd8dd674ca26b888fbe3425fd307..12ced240ad7986b3f1a666bade60944945842231 100644 (file)
@@ -51,7 +51,7 @@
                14DD78C618F48D7500950702 /* AsyncTask.h in Headers */ = {isa = PBXBuildFile; fileRef = 1417F65218BA88A00076FA3F /* AsyncTask.h */; settings = {ATTRIBUTES = (Private, ); }; };
                14DD78C718F48D7500950702 /* BAssert.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E468189EEDE400546D68 /* BAssert.h */; settings = {ATTRIBUTES = (Private, ); }; };
                14DD78C818F48D7500950702 /* FixedVector.h in Headers */ = {isa = PBXBuildFile; fileRef = 14D9DB4517F2447100EAAB79 /* FixedVector.h */; settings = {ATTRIBUTES = (Private, ); }; };
-               14DD78C918F48D7500950702 /* Inline.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E460189DCE1E00546D68 /* Inline.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               14DD78C918F48D7500950702 /* BInline.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E460189DCE1E00546D68 /* BInline.h */; settings = {ATTRIBUTES = (Private, ); }; };
                14DD78CA18F48D7500950702 /* Mutex.h in Headers */ = {isa = PBXBuildFile; fileRef = 144DCED617A649D90093B2F2 /* Mutex.h */; settings = {ATTRIBUTES = (Private, ); }; };
                14DD78CB18F48D7500950702 /* PerProcess.h in Headers */ = {isa = PBXBuildFile; fileRef = 14446A0717A61FA400F9EA1D /* PerProcess.h */; settings = {ATTRIBUTES = (Private, ); }; };
                14DD78CC18F48D7500950702 /* PerThread.h in Headers */ = {isa = PBXBuildFile; fileRef = 144469FD17A61F1F00F9EA1D /* PerThread.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -94,7 +94,7 @@
                140FA00219CE429C00FFD3C8 /* BumpRange.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BumpRange.h; path = bmalloc/BumpRange.h; sourceTree = "<group>"; };
                140FA00419CE4B6800FFD3C8 /* LineMetadata.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LineMetadata.h; path = bmalloc/LineMetadata.h; sourceTree = "<group>"; };
                14105E8318E14374003A106E /* ObjectType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ObjectType.cpp; path = bmalloc/ObjectType.cpp; sourceTree = "<group>"; };
-               1413E460189DCE1E00546D68 /* Inline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Inline.h; path = bmalloc/Inline.h; sourceTree = "<group>"; };
+               1413E460189DCE1E00546D68 /* BInline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BInline.h; path = bmalloc/BInline.h; sourceTree = "<group>"; };
                1413E462189DE1CD00546D68 /* BumpAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; lineEnding = 0; name = BumpAllocator.h; path = bmalloc/BumpAllocator.h; sourceTree = "<group>"; };
                1413E468189EEDE400546D68 /* BAssert.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BAssert.h; path = bmalloc/BAssert.h; sourceTree = "<group>"; };
                1417F64F18B7280C0076FA3F /* Syscall.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Syscall.h; path = bmalloc/Syscall.h; sourceTree = "<group>"; };
                                6599C5CB1EC3F15900A2F7BB /* AvailableMemory.h */,
                                1413E468189EEDE400546D68 /* BAssert.h */,
                                0F5BF1721F23C5710029D91D /* BExport.h */,
+                               1413E460189DCE1E00546D68 /* BInline.h */,
                                14C919C818FCC59F0028DB43 /* BPlatform.h */,
                                14D9DB4517F2447100EAAB79 /* FixedVector.h */,
                                0F5BF1461F22A8B10029D91D /* HeapKind.h */,
-                               1413E460189DCE1E00546D68 /* Inline.h */,
                                141D9AFF1C8E51C0000ABBA0 /* List.h */,
                                4426E27E1C838EE0008EB042 /* Logging.cpp */,
                                4426E27F1C838EE0008EB042 /* Logging.h */,
                                14DD78C818F48D7500950702 /* FixedVector.h in Headers */,
                                1400274918F89C1300115C97 /* Heap.h in Headers */,
                                0F5BF1491F22A8D80029D91D /* PerHeapKind.h in Headers */,
-                               14DD78C918F48D7500950702 /* Inline.h in Headers */,
+                               14DD78C918F48D7500950702 /* BInline.h in Headers */,
                                144C07F51C7B70260051BB6A /* LargeMap.h in Headers */,
                                14C8992D1CC578330027A057 /* LargeRange.h in Headers */,
                                140FA00519CE4B6800FFD3C8 /* LineMetadata.h in Headers */,
index 8215251b6412f155f4f4c8ef76928a3ebec4a3e6..b442def927134f2e1921f402dd40a02d9c166eb5 100644 (file)
@@ -149,7 +149,7 @@ void Allocator::scavenge()
     }
 }
 
-NO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size_t sizeClass)
+BNO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size_t sizeClass)
 {
     BumpRangeCache& bumpRangeCache = m_bumpRangeCaches[sizeClass];
 
@@ -158,7 +158,7 @@ NO_INLINE void Allocator::refillAllocatorSlowCase(BumpAllocator& allocator, size
     m_heap.allocateSmallBumpRanges(lock, sizeClass, allocator, bumpRangeCache, m_deallocator.lineCache(lock));
 }
 
-INLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass)
+BINLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass)
 {
     BumpRangeCache& bumpRangeCache = m_bumpRangeCaches[sizeClass];
     if (!bumpRangeCache.size())
@@ -166,13 +166,13 @@ INLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClas
     return allocator.refill(bumpRangeCache.pop());
 }
 
-NO_INLINE void* Allocator::allocateLarge(size_t size)
+BNO_INLINE void* Allocator::allocateLarge(size_t size)
 {
     std::lock_guard<StaticMutex> lock(Heap::mutex());
     return m_heap.allocateLarge(lock, alignment, size);
 }
 
-NO_INLINE void* Allocator::allocateLogSizeClass(size_t size)
+BNO_INLINE void* Allocator::allocateLogSizeClass(size_t size)
 {
     size_t sizeClass = bmalloc::sizeClass(size);
     BumpAllocator& allocator = m_bumpAllocators[sizeClass];
index f83f51b8512b9c52268a373f726e1bcd6c8461ee..c6c0649a120faa65d3238abfc4c512b1a7e4b19f 100644 (file)
@@ -27,7 +27,7 @@
 #define AsyncTask_h
 
 #include "BAssert.h"
-#include "Inline.h"
+#include "BInline.h"
 #include "Mutex.h"
 #include "Sizes.h"
 #include <atomic>
similarity index 89%
rename from Source/bmalloc/bmalloc/Inline.h
rename to Source/bmalloc/bmalloc/BInline.h
index c005417a6ed13d90a1fc1d69e88d34caef24bd44..a8806c50c059fcf07ef26215e7594952520c0d7e 100644 (file)
  * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
  */
 
-#ifndef Inline_h
-#define Inline_h
+#ifndef BInline_h
+#define BInline_h
 
-#define INLINE __attribute__((always_inline)) inline
+#define BINLINE __attribute__((always_inline)) inline
 
-#define NO_INLINE __attribute__((noinline))
+#define BNO_INLINE __attribute__((noinline))
 
-#endif // Inline_h
+#endif // BInline_h
index c3c5bf90411c16287f3b0d65f8c8e5d0b23f3694..26f4202ee5dd527a9328ea4d6d9f2cbb3fbf92d9 100644 (file)
@@ -23,9 +23,9 @@
  * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
  */
 
+#include "BInline.h"
 #include "Cache.h"
 #include "Heap.h"
-#include "Inline.h"
 #include "PerProcess.h"
 
 namespace bmalloc {
@@ -46,27 +46,27 @@ Cache::Cache(HeapKind heapKind)
 {
 }
 
-NO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size)
+BNO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size)
 {
     return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().tryAllocate(size);
 }
 
-NO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t size)
+BNO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t size)
 {
     return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().allocate(size);
 }
 
-NO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t alignment, size_t size)
+BNO_INLINE void* Cache::allocateSlowCaseNullCache(HeapKind heapKind, size_t alignment, size_t size)
 {
     return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().allocate(alignment, size);
 }
 
-NO_INLINE void Cache::deallocateSlowCaseNullCache(HeapKind heapKind, void* object)
+BNO_INLINE void Cache::deallocateSlowCaseNullCache(HeapKind heapKind, void* object)
 {
     PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).deallocator().deallocate(object);
 }
 
-NO_INLINE void* Cache::reallocateSlowCaseNullCache(HeapKind heapKind, void* object, size_t newSize)
+BNO_INLINE void* Cache::reallocateSlowCaseNullCache(HeapKind heapKind, void* object, size_t newSize)
 {
     return PerThread<PerHeapKind<Cache>>::getSlowCase()->at(heapKind).allocator().reallocate(object, newSize);
 }
index e8b7b29c79109b88526fc1d519720a3c623242a3..44c72635b86251fcb12adfb733f869e44dd4752f 100644 (file)
  */
 
 #include "BAssert.h"
+#include "BInline.h"
 #include "Chunk.h"
 #include "Deallocator.h"
 #include "DebugHeap.h"
 #include "Heap.h"
-#include "Inline.h"
 #include "Object.h"
 #include "PerProcess.h"
 #include <algorithm>
index 6c1b9eef7d2e6bf7a906cfb78acdfc3b90794cd9..0e48fe93a11ad89d6bd971658ec43d0075e2d016 100644 (file)
@@ -34,7 +34,8 @@
 
 // FIXME: Ask dyld to put this in its own page, and mprotect the page after we ensure the gigacage.
 // https://bugs.webkit.org/show_bug.cgi?id=174972
-void* g_gigacageBasePtr;
+void* g_primitiveGigacageBasePtr;
+void* g_jsValueGigacageBasePtr;
 
 using namespace bmalloc;
 
@@ -53,8 +54,8 @@ struct Callback {
     void* argument { nullptr };
 };
 
-struct Callbacks {
-    Callbacks(std::lock_guard<StaticMutex>&) { }
+struct PrimitiveDisableCallbacks {
+    PrimitiveDisableCallbacks(std::lock_guard<StaticMutex>&) { }
     
     Vector<Callback> callbacks;
 };
@@ -69,52 +70,57 @@ void ensureGigacage()
             if (!shouldBeEnabled())
                 return;
             
-            void* basePtr = tryVMAllocate(GIGACAGE_SIZE, GIGACAGE_SIZE + GIGACAGE_RUNWAY);
-            if (!basePtr)
-                return;
-            
-            vmDeallocatePhysicalPages(basePtr, GIGACAGE_SIZE + GIGACAGE_RUNWAY);
-            
-            g_gigacageBasePtr = basePtr;
+            forEachKind(
+                [&] (Kind kind) {
+                    // FIXME: Randomize where this goes.
+                    // https://bugs.webkit.org/show_bug.cgi?id=175245
+                    basePtr(kind) = tryVMAllocate(GIGACAGE_SIZE, GIGACAGE_SIZE + GIGACAGE_RUNWAY);
+                    if (!basePtr(kind)) {
+                        fprintf(stderr, "FATAL: Could not allocate %s gigacage.\n", name(kind));
+                        BCRASH();
+                    }
+                    
+                    vmDeallocatePhysicalPages(basePtr(kind), GIGACAGE_SIZE + GIGACAGE_RUNWAY);
+                });
         });
 #endif // GIGACAGE_ENABLED
 }
 
-void disableGigacage()
+void disablePrimitiveGigacage()
 {
     ensureGigacage();
-    if (!g_gigacageBasePtr) {
+    if (!g_primitiveGigacageBasePtr) {
         // It was never enabled. That means that we never even saved any callbacks. Or, we had already disabled
         // it before, and already called the callbacks.
         return;
     }
     
-    Callbacks& callbacks = *PerProcess<Callbacks>::get();
-    std::unique_lock<StaticMutex> lock(PerProcess<Callbacks>::mutex());
+    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
+    std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
     for (Callback& callback : callbacks.callbacks)
         callback.function(callback.argument);
     callbacks.callbacks.shrink(0);
-    g_gigacageBasePtr = nullptr;
+    g_primitiveGigacageBasePtr = nullptr;
 }
 
-void addDisableCallback(void (*function)(void*), void* argument)
+void addPrimitiveDisableCallback(void (*function)(void*), void* argument)
 {
     ensureGigacage();
-    if (!g_gigacageBasePtr) {
+    if (!g_primitiveGigacageBasePtr) {
         // It was already disabled or we were never able to enable it.
         function(argument);
         return;
     }
     
-    Callbacks& callbacks = *PerProcess<Callbacks>::get();
-    std::unique_lock<StaticMutex> lock(PerProcess<Callbacks>::mutex());
+    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
+    std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
     callbacks.callbacks.push(Callback(function, argument));
 }
 
-void removeDisableCallback(void (*function)(void*), void* argument)
+void removePrimitiveDisableCallback(void (*function)(void*), void* argument)
 {
-    Callbacks& callbacks = *PerProcess<Callbacks>::get();
-    std::unique_lock<StaticMutex> lock(PerProcess<Callbacks>::mutex());
+    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
+    std::unique_lock<StaticMutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
     for (size_t i = 0; i < callbacks.callbacks.size(); ++i) {
         if (callbacks.callbacks[i].function == function
             && callbacks.callbacks[i].argument == argument) {
index 0bd0974791513269f9103957d5a4b18199097aec..b1cbf5a43f952f06547aa05c1f3842ee40f574f0 100644 (file)
@@ -27,6 +27,7 @@
 
 #include "BAssert.h"
 #include "BExport.h"
+#include "BInline.h"
 #include "BPlatform.h"
 #include <inttypes.h>
 
 #define GIGACAGE_ENABLED 0
 #endif
 
-extern "C" BEXPORT void* g_gigacageBasePtr;
+extern "C" BEXPORT void* g_primitiveGigacageBasePtr;
+extern "C" BEXPORT void* g_jsValueGigacageBasePtr;
 
 namespace Gigacage {
 
+enum Kind {
+    Primitive,
+    JSValue
+};
+
 BEXPORT void ensureGigacage();
 
-BEXPORT void disableGigacage();
+BEXPORT void disablePrimitiveGigacage();
+
+// This will call the disable callback immediately if the Primitive Gigacage is currently disabled.
+BEXPORT void addPrimitiveDisableCallback(void (*)(void*), void*);
+BEXPORT void removePrimitiveDisableCallback(void (*)(void*), void*);
+
+BINLINE const char* name(Kind kind)
+{
+    switch (kind) {
+    case Primitive:
+        return "Primitive";
+    case JSValue:
+        return "JSValue";
+    }
+    BCRASH();
+    return nullptr;
+}
+
+BINLINE void*& basePtr(Kind kind)
+{
+    switch (kind) {
+    case Primitive:
+        return g_primitiveGigacageBasePtr;
+    case JSValue:
+        return g_jsValueGigacageBasePtr;
+    }
+    BCRASH();
+    return g_primitiveGigacageBasePtr;
+}
 
-// This will call the disable callback immediately if the Gigacage is currently disabled.
-BEXPORT void addDisableCallback(void (*)(void*), void*);
-BEXPORT void removeDisableCallback(void (*)(void*), void*);
+template<typename Func>
+void forEachKind(const Func& func)
+{
+    func(Primitive);
+    func(JSValue);
+}
 
 template<typename T>
-T* caged(T* ptr)
+BINLINE T* caged(Kind kind, T* ptr)
 {
     BASSERT(ptr);
-    void* gigacageBasePtr = g_gigacageBasePtr;
+    void* gigacageBasePtr = basePtr(kind);
     if (!gigacageBasePtr)
         return ptr;
     return reinterpret_cast<T*>(
@@ -68,9 +106,9 @@ T* caged(T* ptr)
             reinterpret_cast<uintptr_t>(ptr) & static_cast<uintptr_t>(GIGACAGE_MASK)));
 }
 
-inline bool isCaged(const void* ptr)
+BINLINE bool isCaged(Kind kind, const void* ptr)
 {
-    return caged(ptr) == ptr;
+    return caged(kind, ptr) == ptr;
 }
 
 BEXPORT bool shouldBeEnabled();
index 8f4161b199602e8d6753def1ba9f961114b28aae..dbca2f4293a179a3131c13318b7a4c73af140884 100644 (file)
@@ -59,8 +59,8 @@ Heap::Heap(HeapKind kind, std::lock_guard<StaticMutex>&)
         Gigacage::ensureGigacage();
 #if GIGACAGE_ENABLED
         if (usingGigacage()) {
-            RELEASE_BASSERT(g_gigacageBasePtr);
-            m_largeFree.add(LargeRange(g_gigacageBasePtr, GIGACAGE_SIZE, 0));
+            RELEASE_BASSERT(gigacageBasePtr());
+            m_largeFree.add(LargeRange(gigacageBasePtr(), GIGACAGE_SIZE, 0));
         }
 #endif
     }
@@ -70,7 +70,12 @@ Heap::Heap(HeapKind kind, std::lock_guard<StaticMutex>&)
 
 bool Heap::usingGigacage()
 {
-    return m_kind == HeapKind::Gigacage && g_gigacageBasePtr;
+    return isGigacage(m_kind) && gigacageBasePtr();
+}
+
+void* Heap::gigacageBasePtr()
+{
+    return Gigacage::basePtr(gigacageKind(m_kind));
 }
 
 void Heap::initializeLineMetadata()
index aae9c31b0bd6a5bb9ae36050537a8abeb1e46278..0167f32675bbd40a0d0644b1902754b7aca7d821 100644 (file)
@@ -89,6 +89,7 @@ private:
     ~Heap() = delete;
     
     bool usingGigacage();
+    void* gigacageBasePtr(); // May crash if !usingGigacage().
     
     void initializeLineMetadata();
     void initializePageMetadata();
index 4b3f32644d871e91fbbf0d2875a4593e664ebaf2..94d4a2035afb631a8cb04ad207a77ba2fab7f2f9 100644 (file)
 
 #pragma once
 
+#include "BAssert.h"
+#include "BInline.h"
+#include "Gigacage.h"
+
 namespace bmalloc {
 
 enum class HeapKind {
     Primary,
-    Gigacage
+    PrimitiveGigacage,
+    JSValueGigacage
 };
 
-static constexpr unsigned numHeaps = 2;
+static constexpr unsigned numHeaps = 3;
+
+BINLINE bool isGigacage(HeapKind heapKind)
+{
+    switch (heapKind) {
+    case HeapKind::Primary:
+        return false;
+    case HeapKind::PrimitiveGigacage:
+    case HeapKind::JSValueGigacage:
+        return true;
+    }
+    BCRASH();
+    return false;
+}
+
+BINLINE Gigacage::Kind gigacageKind(HeapKind kind)
+{
+    switch (kind) {
+    case HeapKind::Primary:
+        BCRASH();
+        return Gigacage::Primitive;
+    case HeapKind::PrimitiveGigacage:
+        return Gigacage::Primitive;
+    case HeapKind::JSValueGigacage:
+        return Gigacage::JSValue;
+    }
+    BCRASH();
+    return Gigacage::Primitive;
+}
+
+BINLINE HeapKind heapKind(Gigacage::Kind kind)
+{
+    switch (kind) {
+    case Gigacage::Primitive:
+        return HeapKind::PrimitiveGigacage;
+    case Gigacage::JSValue:
+        return HeapKind::JSValueGigacage;
+    }
+    BCRASH();
+    return HeapKind::Primary;
+}
 
 } // namespace bmalloc
 
index 74ed8ca049f8938f130c23009fc5dd9f5dc92d7c..f3581783af880aa572d6fe9caa42f8bd8598c37f 100644 (file)
@@ -26,7 +26,7 @@
 #ifndef Map_h
 #define Map_h
 
-#include "Inline.h"
+#include "BInline.h"
 #include "Sizes.h"
 #include "Vector.h"
 
index b1fb6b4a94eaf2be8c66ef0211c3201408d82738..cfa30158166568204d5e2907cb1be18da632b298 100644 (file)
@@ -26,7 +26,7 @@
 #ifndef PerProcess_h
 #define PerProcess_h
 
-#include "Inline.h"
+#include "BInline.h"
 #include "Sizes.h"
 #include "StaticMutex.h"
 #include <mutex>
@@ -71,13 +71,13 @@ private:
 };
 
 template<typename T>
-INLINE T* PerProcess<T>::getFastCase()
+BINLINE T* PerProcess<T>::getFastCase()
 {
     return s_object.load(std::memory_order_consume);
 }
 
 template<typename T>
-INLINE T* PerProcess<T>::get()
+BINLINE T* PerProcess<T>::get()
 {
     T* object = getFastCase();
     if (!object)
@@ -86,7 +86,7 @@ INLINE T* PerProcess<T>::get()
 }
 
 template<typename T>
-NO_INLINE T* PerProcess<T>::getSlowCase()
+BNO_INLINE T* PerProcess<T>::getSlowCase()
 {
     std::lock_guard<StaticMutex> lock(s_mutex);
     if (!s_object.load(std::memory_order_consume)) {
index fd0aca029383538fe052f90319bcc10e1d2f89ef..b848f828e95490ec6d5d9903216d20b9c67de15d 100644 (file)
@@ -26,8 +26,8 @@
 #ifndef PerThread_h
 #define PerThread_h
 
+#include "BInline.h"
 #include "BPlatform.h"
-#include "Inline.h"
 #include "PerHeapKind.h"
 #include "VMAllocate.h"
 #include <mutex>
@@ -115,7 +115,7 @@ template<typename T> std::once_flag PerThreadStorage<T>::s_onceFlag;
 #endif
 
 template<typename T>
-INLINE T* PerThread<T>::getFastCase()
+BINLINE T* PerThread<T>::getFastCase()
 {
     return static_cast<T*>(PerThreadStorage<T>::get());
 }
index 28bbe1332933e128cd32995a6594359a724be913..9d876559e3c2fef47fb3e2bc6900d6c9a6b99942 100644 (file)
@@ -26,7 +26,7 @@
 #ifndef Vector_h
 #define Vector_h
 
-#include "Inline.h"
+#include "BInline.h"
 #include "VMAllocate.h"
 #include <cstddef>
 #include <cstring>
@@ -119,7 +119,7 @@ inline T& Vector<T>::operator[](size_t i)
 }
 
 template<typename T>
-INLINE void Vector<T>::push(const T& value)
+BINLINE void Vector<T>::push(const T& value)
 {
     if (m_size == m_capacity)
         growCapacity();
@@ -212,14 +212,14 @@ void Vector<T>::reallocateBuffer(size_t newCapacity)
 }
 
 template<typename T>
-NO_INLINE void Vector<T>::shrinkCapacity()
+BNO_INLINE void Vector<T>::shrinkCapacity()
 {
     size_t newCapacity = max(initialCapacity(), m_capacity / shrinkFactor);
     reallocateBuffer(newCapacity);
 }
 
 template<typename T>
-NO_INLINE void Vector<T>::growCapacity()
+BNO_INLINE void Vector<T>::growCapacity()
 {
     size_t newCapacity = max(initialCapacity(), m_size * growFactor);
     reallocateBuffer(newCapacity);