Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 7 Aug 2017 21:31:49 +0000 (21:31 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Mon, 7 Aug 2017 21:31:49 +0000 (21:31 +0000)
commitf368abfa53ea2c886c379c2abcb2ef2795badf3b
tree9270ddc359fa1d9430cde7dbef2ad3b476d815ab
parent4fc9d5849483ee8fd6b40f25188ab976f7db3812
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
https://bugs.webkit.org/show_bug.cgi?id=174919

Reviewed by Keith Miller.
Source/bmalloc:

This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of
HeapKind, PrimitiveGigacage and JSValueGigacage.

The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and
NO_INLINE into BNO_INLINE.

* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::refillAllocator):
(bmalloc::Allocator::allocateLarge):
(bmalloc::Allocator::allocateLogSizeClass):
* bmalloc/AsyncTask.h:
* bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h.
* bmalloc/Cache.cpp:
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
* bmalloc/Deallocator.cpp:
* bmalloc/Gigacage.cpp:
(Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::ensureGigacage):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::Callbacks::Callbacks): Deleted.
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.
* bmalloc/Gigacage.h:
(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::forEachKind):
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::gigacageBasePtr):
* bmalloc/Heap.h:
* bmalloc/HeapKind.h:
(bmalloc::isGigacage):
(bmalloc::gigacageKind):
(bmalloc::heapKind):
* bmalloc/Inline.h: Removed.
* bmalloc/Map.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
(bmalloc::PerProcess<T>::get):
(bmalloc::PerProcess<T>::getSlowCase):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::getFastCase):
* bmalloc/Vector.h:
(bmalloc::Vector<T>::push):
(bmalloc::Vector<T>::shrinkCapacity):
(bmalloc::Vector<T>::growCapacity):

Source/JavaScriptCore:

This adapts JSC to there being two gigacages.

To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than
singletons. I don't think we were gaining anything by making them be singletons.

This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of
gigacages. We'll have one of those allocators per cage.

From there, this change teaches everyone who previously knew about cages that there are two cages.
This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is
easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's
not so obvious, so this change introduces some helpers to make it easy to define what cage you want
to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h

A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and
CagedPtr. This removes one layer of "get()" calls from a bunch of places.

* JavaScriptCore.xcodeproj/project.pbxproj:
* bytecode/AccessCase.cpp:
(JSC::AccessCase::generateImpl):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileNewTypedArray):
(JSC::DFG::SpeculativeJIT::emitAllocateButterfly):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/FastMallocAlignedMemoryAllocator.cpp:
(JSC::FastMallocAlignedMemoryAllocator::instance): Deleted.
* heap/FastMallocAlignedMemoryAllocator.h:
* heap/GigacageAlignedMemoryAllocator.cpp:
(JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator):
(JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::dump const):
(JSC::GigacageAlignedMemoryAllocator::instance): Deleted.
* heap/GigacageAlignedMemoryAllocator.h:
* jsc.cpp:
(primitiveGigacageDisabled):
(jscmain):
(gigacageDisabled): Deleted.
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
* runtime/AuxiliaryBarrier.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::tryCreate):
(JSC::Butterfly::growArrayRight):
* runtime/CagedBarrierPtr.h: Added.
(JSC::CagedBarrierPtr::CagedBarrierPtr):
(JSC::CagedBarrierPtr::clear):
(JSC::CagedBarrierPtr::set):
(JSC::CagedBarrierPtr::get const):
(JSC::CagedBarrierPtr::getMayBeNull const):
(JSC::CagedBarrierPtr::operator== const):
(JSC::CagedBarrierPtr::operator!= const):
(JSC::CagedBarrierPtr::operator bool const):
(JSC::CagedBarrierPtr::setWithoutBarrier):
(JSC::CagedBarrierPtr::operator* const):
(JSC::CagedBarrierPtr::operator-> const):
(JSC::CagedBarrierPtr::operator[] const):
* runtime/DirectArguments.cpp:
(JSC::DirectArguments::overrideThings):
(JSC::DirectArguments::unmapArgument):
* runtime/DirectArguments.h:
(JSC::DirectArguments::isMappedArgument const):
* runtime/GenericArguments.h:
* runtime/GenericArgumentsInlines.h:
(JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
(JSC::GenericArguments<Type>::setModifiedArgumentDescriptor):
(JSC::GenericArguments<Type>::isModifiedArgumentDescriptor):
* runtime/HashMapImpl.cpp:
(JSC::HashMapImpl<HashMapBucket>::visitChildren):
* runtime/HashMapImpl.h:
(JSC::HashMapBuffer::create):
(JSC::HashMapImpl::buffer const):
(JSC::HashMapImpl::rehash):
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitializedRestricted):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithAnyIndexingType):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::JSArray::tryCreate):
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.cpp:
(JSC::JSObject::heapSnapshot):
(JSC::JSObject::getOwnPropertySlotByIndex):
(JSC::JSObject::putByIndex):
(JSC::JSObject::enterDictionaryIndexingMode):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToDouble):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToContiguous):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToContiguous):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::setIndexQuicklyToUndecided):
(JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
(JSC::JSObject::deletePropertyByIndex):
(JSC::JSObject::getOwnPropertyNames):
(JSC::JSObject::putIndexedDescriptor):
(JSC::JSObject::defineOwnIndexedProperty):
(JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
(JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::reallocateAndShrinkButterfly):
(JSC::JSObject::allocateMoreOutOfLineStorage):
(JSC::JSObject::getEnumerableLength):
* runtime/JSObject.h:
(JSC::JSObject::getArrayLength const):
(JSC::JSObject::getVectorLength):
(JSC::JSObject::putDirectIndex):
(JSC::JSObject::canGetIndexQuickly):
(JSC::JSObject::getIndexQuickly):
(JSC::JSObject::tryGetIndexQuickly const):
(JSC::JSObject::canSetIndexQuickly):
(JSC::JSObject::setIndexQuickly):
(JSC::JSObject::initializeIndex):
(JSC::JSObject::initializeIndexWithoutBarrier):
(JSC::JSObject::hasSparseMap):
(JSC::JSObject::inSparseIndexingMode):
(JSC::JSObject::butterfly const):
(JSC::JSObject::butterfly):
(JSC::JSObject::outOfLineStorage const):
(JSC::JSObject::outOfLineStorage):
(JSC::JSObject::ensureInt32):
(JSC::JSObject::ensureDouble):
(JSC::JSObject::ensureContiguous):
(JSC::JSObject::ensureArrayStorage):
(JSC::JSObject::arrayStorage):
(JSC::JSObject::arrayStorageOrNull):
(JSC::JSObject::ensureLength):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::primitiveGigacageDisabledCallback):
(JSC::VM::primitiveGigacageDisabled):
(JSC::VM::gigacageDisabledCallback): Deleted.
(JSC::VM::gigacageDisabled): Deleted.
* runtime/VM.h:
(JSC::VM::gigacageAuxiliarySpace):
(JSC::VM::firePrimitiveGigacageEnabledIfNecessary):
(JSC::VM::primitiveGigacageEnabled):
(JSC::VM::fireGigacageEnabledIfNecessary): Deleted.
(JSC::VM::gigacageEnabled): Deleted.
* wasm/WasmMemory.cpp:
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::grow):

Source/WebCore:

No new tests because no change in behavior.

Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be
specific about which Gigacage we're using.

* bindings/js/WebCoreJSClientData.cpp:
(WebCore::JSVMClientData::JSVMClientData):
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):

Source/WebKit:

The disable callback is all about the primitive gigacage.

* WebProcess/WebProcess.cpp:
(WebKit::primitiveGigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
(WebKit::gigacageDisabled): Deleted.

Source/WTF:

This mirrors the changes from bmalloc/Gigacage.h.

Also it teaches CagedPtr how to reason about multiple gigacages.

* wtf/CagedPtr.h:
(WTF::CagedPtr::get const):
(WTF::CagedPtr::operator[] const):
* wtf/Gigacage.cpp:
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h:
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
56 files changed:
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/bytecode/AccessCase.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.cpp
Source/JavaScriptCore/heap/FastMallocAlignedMemoryAllocator.h
Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.cpp
Source/JavaScriptCore/heap/GigacageAlignedMemoryAllocator.h
Source/JavaScriptCore/jsc.cpp
Source/JavaScriptCore/llint/LowLevelInterpreter64.asm
Source/JavaScriptCore/runtime/ArrayBuffer.cpp
Source/JavaScriptCore/runtime/AuxiliaryBarrier.h
Source/JavaScriptCore/runtime/ButterflyInlines.h
Source/JavaScriptCore/runtime/CagedBarrierPtr.h [new file with mode: 0644]
Source/JavaScriptCore/runtime/DirectArguments.cpp
Source/JavaScriptCore/runtime/DirectArguments.h
Source/JavaScriptCore/runtime/GenericArguments.h
Source/JavaScriptCore/runtime/GenericArgumentsInlines.h
Source/JavaScriptCore/runtime/HashMapImpl.cpp
Source/JavaScriptCore/runtime/HashMapImpl.h
Source/JavaScriptCore/runtime/JSArray.cpp
Source/JavaScriptCore/runtime/JSArray.h
Source/JavaScriptCore/runtime/JSArrayBufferView.cpp
Source/JavaScriptCore/runtime/JSLock.cpp
Source/JavaScriptCore/runtime/JSObject.cpp
Source/JavaScriptCore/runtime/JSObject.h
Source/JavaScriptCore/runtime/RegExpMatchesArray.h
Source/JavaScriptCore/runtime/VM.cpp
Source/JavaScriptCore/runtime/VM.h
Source/JavaScriptCore/wasm/WasmMemory.cpp
Source/WTF/ChangeLog
Source/WTF/wtf/CagedPtr.h
Source/WTF/wtf/Gigacage.cpp
Source/WTF/wtf/Gigacage.h
Source/WebCore/ChangeLog
Source/WebCore/bindings/js/WebCoreJSClientData.cpp
Source/WebCore/platform/graphics/cocoa/GPUBufferMetal.mm
Source/WebKit/ChangeLog
Source/WebKit/WebProcess/WebProcess.cpp
Source/bmalloc/ChangeLog
Source/bmalloc/bmalloc.xcodeproj/project.pbxproj
Source/bmalloc/bmalloc/Allocator.cpp
Source/bmalloc/bmalloc/AsyncTask.h
Source/bmalloc/bmalloc/BInline.h [moved from Source/bmalloc/bmalloc/Inline.h with 89% similarity]
Source/bmalloc/bmalloc/Cache.cpp
Source/bmalloc/bmalloc/Deallocator.cpp
Source/bmalloc/bmalloc/Gigacage.cpp
Source/bmalloc/bmalloc/Gigacage.h
Source/bmalloc/bmalloc/Heap.cpp
Source/bmalloc/bmalloc/Heap.h
Source/bmalloc/bmalloc/HeapKind.h
Source/bmalloc/bmalloc/Map.h
Source/bmalloc/bmalloc/PerProcess.h
Source/bmalloc/bmalloc/PerThread.h
Source/bmalloc/bmalloc/Vector.h