Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and...
authorfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 6 Sep 2016 01:02:22 +0000 (01:02 +0000)
committerfpizlo@apple.com <fpizlo@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Tue, 6 Sep 2016 01:02:22 +0000 (01:02 +0000)
https://bugs.webkit.org/show_bug.cgi?id=160125

Reviewed by Geoffrey Garen and Keith Miller.
JSTests:

Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.

* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.

Source/bmalloc:

I needed to tryMemalign, so I added such a thing.

* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):

Source/JavaScriptCore:

In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.

Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.

At a high level the mechanical changes are:

- We use AuxiliaryBarrier instead of CopyBarrier.

- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
  CheckedBoolean stuff, since it's so much more trouble than it's worth.

- The JITs have to emit inlined marked space allocations instead of inline copy space
  allocations.

- Everyone has to get used to zeroing their butterflies after allocation instead of relying
  on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
  space doesn't.

That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:

- MarkedSpace always causes internal fragmentation for array allocations because the vector
  length we choose when we resize usually leads to a cell size that doesn't correspond to any
  size class. I got around this by making array allocations usually round up vectorLength to
  the maximum allowed by the size class that we would have allocated in. Also,
  ensureLengthSlow() and friends first make sure that the requested length can't just be
  fulfilled with the current allocation size. This safeguard means that not every array
  allocation has to do size class queries. For example, the fast path of new Array(length)
  never does any size class queries, under the assumption that (1) the speed gained from
  avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
  size class query, is too small to offset the speed lost by doing the query on every
  allocation and (2) new Array(length) is a pretty good hint that resizing is not very
  likely.

- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
  changes MarkedSpace size classes to use a linear progression for very small sizes followed
  by a geometric progression that naturally transitions to a hyperbolic progression. We want
  hyperbolic sizes when we get close to blockSize: for example the largest size we want is
  payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
  next size down should be payloadSize / 3 rounded down, and so on. After the last precise
  size (80 bytes), we proceed using a geometric progression, but round up each size to
  minimize slop at the end of the block. This naturally causes the geometric progression to
  turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
  it can be controlled with runtime options. I found that a base of 1.4 works pretty well.

- Large allocations caused massive internal fragmentation, since the smallest large
  allocation had to use exactly blockSize, and the largest small allocation used
  blockSize / 2. The next size up - the first large allocation size to require two blocks -
  also had 50% internal fragmentation. This is because we required large allocations to be
  blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
  that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
  owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
  a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
  Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
  out great since (1) typed arrays won't use large allocations anyway since they have their
  own malloc fallback and (2) large array butterflies already have a 8 byte header, which
  means that the 8 byte base misalignment aligns the large array payload on a 16 byte
  boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
  rare as possible; for example, ExecState::vm() skips the check because we know that callees
  must be small allocations. It's also possible to use template tricks to do one check for
  cell container kind, and then invoke a function specialized for MarkedBlock or a function
  specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
  that get used from functions that are template-specialized like this. That's mostly to
  speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
  directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
  allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
  HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
  cutoff for large allocations is runtime-configurable, so long as you don't choose something
  so small that callees end up large. I found that 400 bytes is roughly optimal. This means
  that the MarkedBlock size classes end up being:

  16, 32, 48, 64, 80, 112, 160, 224, 320

  The next size class would have been 432, but that's above the 400 byte cutoff. All of this
  is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
  size classes you end up with by doing --dumpSizeClasses=true.

- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
  of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
  lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
  MarkedBlocks because the list that contains the interesting meta-data is allocated on the
  side, for better locality during a sequential walk. It's no longer necessary to scan
  MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
  maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
  bits because we now use versioned mark bits: to clear then, just increment the 64-bit
  heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
  because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
  the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
  together, these optimizations allowed me to reduce block size to 16KB without losing much
  performance. There is some small perf loss on JetStream/splay, but not enough to hurt
  JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
  progression on membuster. That's not possible yet, since there is still enough per-block
  overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
  this further: https://bugs.webkit.org/show_bug.cgi?id=161581.

- Even after all of that, copying butterflies was still faster because it allowed us to skip
  sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
  so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
  larger than O(live), especially in an eden collection. Copying satisfies this premise while
  mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
  MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
  that we walk the block to build a free list. This means walking dead space. The new
  allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
  The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
  pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
  we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
  quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
  is completely empty, which is the common case because of the generational hypothesis: the
  number of objects that survive an eden collection is a tiny fraction of the number of
  objects that had been allocated, and this fraction is so small that there are typically
  fewer than one survivors per MarkedBlock. This change was enough to make this change a net
  win over tip-of-tree.

- FTL now shares the same allocation fast paths as everything else, which is great, because
  bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
  control flow, since it won't be able to improve the machine code we write ourselves. GC
  fast paths are best written in assembly. So, I've empowered B3 to have even better support
  for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
  So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
  AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
  size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
  ability to constant-fold some allocator logic (in case we first assume that we're doing a
  variable-length allocation but then realize that the length is known). I think it makes
  sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
  since this makes lowering easier (you can constant fold during lowering more easily) and it
  reduces the amount of malloc traffic. In the future, we could teach B3 how to better
  constant-fold this code. That would require allowing loads to be constant-folded, which is
  doable but hella tricky.

- It used to be that if a logical object allocation required two physical allocations (first
  the butterfly and then the cell), then the JIT would emit the code in such a way that a
  failure in the second fast path would cause us to forget the successful first physical
  allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
  register to storing either the butterfly or null, because the butterfly register is anyway
  going to be free inside the first allocation. The only overhead here is zeroing the
  butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
  path, which can then either allocate a butterfly or not. So now we never waste a successful
  allocation. This patch implements such a solution both in DFG (where it's easy to do this
  since we control registers already) and in FTL (where it's annoying, because mutable
  "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
  duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
  one version of this patch, this wastage would resonate with some Kraken benchmark: the
  benchmark would always allocate N small things followed by one bigger thing. The problem
  was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
  the JSObject size class, which both the small and big thing shared for their cell, could
  hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
  it allocated the big thing. So, it would end up having to allocate the big thing's large
  butterfly twice, every single time! Ouch!

- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
  amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
  of the allocator to initialize the object to whatever state they need. It used to be that
  we would just use a dumb loop. I initially changed this so that we would end up in memset
  for large allocations, but this didn't actually help performance that much. I got a much
  better result by playing with different memsets written in assembly. First I wrote one
  using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
  "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
  allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
  memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
  double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
  of stores (if the size is known), but those paths in the JIT won't trigger for
  NewArrayWithSize() if the size is large, since that takes us to the
  operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
  optimizations here are all in JSArray::create() - that's the hot place for large arrays
  that need to be filled with holes.

All of this put together gives us neutral perf on JetStream,  membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.

We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.

This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.

* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):

Source/WebCore:

No new tests because no new WebCore behavior.

Just rewiring #includes.

* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:

Source/WebKit2:

Just rewiring some #includes.

* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:

Source/WTF:

I needed tryFastAlignedMalloc() so I added it.

* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.

Tools:

* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc

315 files changed:
JSTests/ChangeLog
JSTests/stress/array-storage-array-unshift.js [new file with mode: 0644]
JSTests/stress/contiguous-array-unshift.js [new file with mode: 0644]
JSTests/stress/double-array-unshift.js [new file with mode: 0644]
JSTests/stress/int32-array-unshift.js [new file with mode: 0644]
JSTests/stress/unshift-array-storage.js [new file with mode: 0644]
Source/JavaScriptCore/API/JSManagedValue.mm
Source/JavaScriptCore/API/JSTypedArray.cpp
Source/JavaScriptCore/API/ObjCCallbackFunction.mm
Source/JavaScriptCore/API/tests/testapi.mm
Source/JavaScriptCore/CMakeLists.txt
Source/JavaScriptCore/ChangeLog
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_implementation.py
Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_implementation.py
Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_implementation.py
Source/JavaScriptCore/assembler/AbstractMacroAssembler.h
Source/JavaScriptCore/assembler/MacroAssembler.h
Source/JavaScriptCore/assembler/MacroAssemblerARM64.h
Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.cpp [new file with mode: 0644]
Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h
Source/JavaScriptCore/b3/B3BasicBlock.cpp
Source/JavaScriptCore/b3/B3BasicBlock.h
Source/JavaScriptCore/b3/B3DuplicateTails.cpp
Source/JavaScriptCore/b3/B3StackmapGenerationParams.h
Source/JavaScriptCore/b3/testb3.cpp
Source/JavaScriptCore/bindings/ScriptValue.cpp
Source/JavaScriptCore/bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp
Source/JavaScriptCore/bytecode/BytecodeLivenessAnalysis.cpp
Source/JavaScriptCore/bytecode/BytecodeRewriter.cpp
Source/JavaScriptCore/bytecode/BytecodeUseDef.h
Source/JavaScriptCore/bytecode/CallLinkInfo.cpp
Source/JavaScriptCore/bytecode/CallLinkInfo.h
Source/JavaScriptCore/bytecode/CallLinkStatus.cpp
Source/JavaScriptCore/bytecode/CodeBlock.cpp
Source/JavaScriptCore/bytecode/CodeBlock.h
Source/JavaScriptCore/bytecode/ExecutionCounter.h
Source/JavaScriptCore/bytecode/Instruction.h
Source/JavaScriptCore/bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp
Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h
Source/JavaScriptCore/bytecode/Opcode.h
Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp
Source/JavaScriptCore/bytecode/PolymorphicAccess.h
Source/JavaScriptCore/bytecode/StructureStubInfo.cpp
Source/JavaScriptCore/bytecode/StructureStubInfo.h
Source/JavaScriptCore/bytecode/SuperSampler.cpp
Source/JavaScriptCore/bytecode/SuperSampler.h
Source/JavaScriptCore/bytecode/UnlinkedCodeBlock.cpp
Source/JavaScriptCore/bytecode/UnlinkedCodeBlock.h
Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.cpp
Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.h
Source/JavaScriptCore/dfg/DFGCallArrayAllocatorSlowPathGenerator.h
Source/JavaScriptCore/dfg/DFGOperations.cpp
Source/JavaScriptCore/dfg/DFGOperations.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp
Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp
Source/JavaScriptCore/dfg/DFGStrengthReductionPhase.cpp
Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h
Source/JavaScriptCore/ftl/FTLCompile.cpp
Source/JavaScriptCore/ftl/FTLJITFinalizer.cpp
Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
Source/JavaScriptCore/ftl/FTLOutput.cpp
Source/JavaScriptCore/ftl/FTLOutput.h
Source/JavaScriptCore/ftl/FTLValueFromBlock.h
Source/JavaScriptCore/ftl/FTLWeightedTarget.h
Source/JavaScriptCore/heap/CellContainer.h [new file with mode: 0644]
Source/JavaScriptCore/heap/CellContainerInlines.h [new file with mode: 0644]
Source/JavaScriptCore/heap/ConservativeRoots.cpp
Source/JavaScriptCore/heap/ConservativeRoots.h
Source/JavaScriptCore/heap/CopyToken.h
Source/JavaScriptCore/heap/FreeList.cpp [new file with mode: 0644]
Source/JavaScriptCore/heap/FreeList.h [new file with mode: 0644]
Source/JavaScriptCore/heap/GCTypeMap.h [new file with mode: 0644]
Source/JavaScriptCore/heap/Heap.cpp
Source/JavaScriptCore/heap/Heap.h
Source/JavaScriptCore/heap/HeapCell.h
Source/JavaScriptCore/heap/HeapCellInlines.h [new file with mode: 0644]
Source/JavaScriptCore/heap/HeapInlines.h
Source/JavaScriptCore/heap/HeapOperation.cpp [new file with mode: 0644]
Source/JavaScriptCore/heap/HeapOperation.h
Source/JavaScriptCore/heap/HeapUtil.h [new file with mode: 0644]
Source/JavaScriptCore/heap/IncrementalSweeper.cpp
Source/JavaScriptCore/heap/IncrementalSweeper.h
Source/JavaScriptCore/heap/LargeAllocation.cpp [new file with mode: 0644]
Source/JavaScriptCore/heap/LargeAllocation.h [new file with mode: 0644]
Source/JavaScriptCore/heap/MarkedAllocator.cpp
Source/JavaScriptCore/heap/MarkedAllocator.h
Source/JavaScriptCore/heap/MarkedBlock.cpp
Source/JavaScriptCore/heap/MarkedBlock.h
Source/JavaScriptCore/heap/MarkedSpace.cpp
Source/JavaScriptCore/heap/MarkedSpace.h
Source/JavaScriptCore/heap/SlotVisitor.cpp
Source/JavaScriptCore/heap/SlotVisitor.h
Source/JavaScriptCore/heap/WeakBlock.cpp
Source/JavaScriptCore/heap/WeakBlock.h
Source/JavaScriptCore/heap/WeakSet.cpp
Source/JavaScriptCore/heap/WeakSet.h
Source/JavaScriptCore/heap/WeakSetInlines.h
Source/JavaScriptCore/inspector/InjectedScriptManager.cpp
Source/JavaScriptCore/inspector/JSGlobalObjectInspectorController.cpp
Source/JavaScriptCore/inspector/JSJavaScriptCallFrame.cpp
Source/JavaScriptCore/inspector/ScriptDebugServer.cpp
Source/JavaScriptCore/inspector/agents/InspectorDebuggerAgent.cpp
Source/JavaScriptCore/interpreter/CachedCall.h
Source/JavaScriptCore/interpreter/Interpreter.cpp
Source/JavaScriptCore/interpreter/Interpreter.h
Source/JavaScriptCore/jit/AssemblyHelpers.h
Source/JavaScriptCore/jit/CCallHelpers.h
Source/JavaScriptCore/jit/GCAwareJITStubRoutine.cpp
Source/JavaScriptCore/jit/JIT.cpp
Source/JavaScriptCore/jit/JIT.h
Source/JavaScriptCore/jit/JITExceptions.cpp
Source/JavaScriptCore/jit/JITExceptions.h
Source/JavaScriptCore/jit/JITOpcodes.cpp
Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
Source/JavaScriptCore/jit/JITOperations.cpp
Source/JavaScriptCore/jit/JITOperations.h
Source/JavaScriptCore/jit/JITPropertyAccess.cpp
Source/JavaScriptCore/jit/JITThunks.cpp
Source/JavaScriptCore/jit/JITThunks.h
Source/JavaScriptCore/jsc.cpp
Source/JavaScriptCore/llint/LLIntData.cpp
Source/JavaScriptCore/llint/LLIntExceptions.cpp
Source/JavaScriptCore/llint/LLIntThunks.cpp
Source/JavaScriptCore/llint/LLIntThunks.h
Source/JavaScriptCore/llint/LowLevelInterpreter.asm
Source/JavaScriptCore/llint/LowLevelInterpreter.cpp
Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm
Source/JavaScriptCore/llint/LowLevelInterpreter64.asm
Source/JavaScriptCore/parser/ModuleAnalyzer.cpp
Source/JavaScriptCore/parser/NodeConstructors.h
Source/JavaScriptCore/parser/Nodes.h
Source/JavaScriptCore/profiler/ProfilerBytecode.cpp
Source/JavaScriptCore/profiler/ProfilerBytecode.h
Source/JavaScriptCore/profiler/ProfilerBytecodeSequence.cpp
Source/JavaScriptCore/runtime/ArrayConventions.cpp [new file with mode: 0644]
Source/JavaScriptCore/runtime/ArrayConventions.h
Source/JavaScriptCore/runtime/ArrayPrototype.cpp
Source/JavaScriptCore/runtime/ArrayStorage.h
Source/JavaScriptCore/runtime/AuxiliaryBarrier.h [new file with mode: 0644]
Source/JavaScriptCore/runtime/AuxiliaryBarrierInlines.h [new file with mode: 0644]
Source/JavaScriptCore/runtime/Butterfly.h
Source/JavaScriptCore/runtime/ButterflyInlines.h
Source/JavaScriptCore/runtime/ClonedArguments.cpp
Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.cpp
Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.h
Source/JavaScriptCore/runtime/DataView.cpp
Source/JavaScriptCore/runtime/DirectArguments.h
Source/JavaScriptCore/runtime/ECMAScriptSpecInternalFunctions.cpp
Source/JavaScriptCore/runtime/Error.cpp
Source/JavaScriptCore/runtime/Error.h
Source/JavaScriptCore/runtime/ErrorInstance.cpp
Source/JavaScriptCore/runtime/ErrorInstance.h
Source/JavaScriptCore/runtime/Exception.cpp
Source/JavaScriptCore/runtime/Exception.h
Source/JavaScriptCore/runtime/GeneratorPrototype.cpp
Source/JavaScriptCore/runtime/InternalFunction.cpp
Source/JavaScriptCore/runtime/IntlCollator.cpp
Source/JavaScriptCore/runtime/IntlCollatorConstructor.cpp
Source/JavaScriptCore/runtime/IntlCollatorPrototype.cpp
Source/JavaScriptCore/runtime/IntlDateTimeFormat.cpp
Source/JavaScriptCore/runtime/IntlDateTimeFormatConstructor.cpp
Source/JavaScriptCore/runtime/IntlDateTimeFormatPrototype.cpp
Source/JavaScriptCore/runtime/IntlNumberFormat.cpp
Source/JavaScriptCore/runtime/IntlNumberFormatConstructor.cpp
Source/JavaScriptCore/runtime/IntlNumberFormatPrototype.cpp
Source/JavaScriptCore/runtime/IntlObject.cpp
Source/JavaScriptCore/runtime/IteratorPrototype.cpp
Source/JavaScriptCore/runtime/JSArray.cpp
Source/JavaScriptCore/runtime/JSArray.h
Source/JavaScriptCore/runtime/JSArrayBufferView.h
Source/JavaScriptCore/runtime/JSCInlines.h
Source/JavaScriptCore/runtime/JSCJSValue.cpp
Source/JavaScriptCore/runtime/JSCallee.cpp
Source/JavaScriptCore/runtime/JSCell.cpp
Source/JavaScriptCore/runtime/JSCell.h
Source/JavaScriptCore/runtime/JSCellInlines.h
Source/JavaScriptCore/runtime/JSFunction.cpp
Source/JavaScriptCore/runtime/JSFunction.h
Source/JavaScriptCore/runtime/JSFunctionInlines.h
Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h
Source/JavaScriptCore/runtime/JSInternalPromise.cpp
Source/JavaScriptCore/runtime/JSInternalPromiseConstructor.cpp
Source/JavaScriptCore/runtime/JSInternalPromiseDeferred.cpp
Source/JavaScriptCore/runtime/JSInternalPromisePrototype.cpp
Source/JavaScriptCore/runtime/JSJob.cpp
Source/JavaScriptCore/runtime/JSMapIterator.cpp
Source/JavaScriptCore/runtime/JSModuleNamespaceObject.cpp
Source/JavaScriptCore/runtime/JSModuleRecord.cpp
Source/JavaScriptCore/runtime/JSObject.cpp
Source/JavaScriptCore/runtime/JSObject.h
Source/JavaScriptCore/runtime/JSObjectInlines.h
Source/JavaScriptCore/runtime/JSPromise.cpp
Source/JavaScriptCore/runtime/JSPromiseConstructor.cpp
Source/JavaScriptCore/runtime/JSPromiseDeferred.cpp
Source/JavaScriptCore/runtime/JSPromisePrototype.cpp
Source/JavaScriptCore/runtime/JSPropertyNameIterator.cpp
Source/JavaScriptCore/runtime/JSScope.cpp
Source/JavaScriptCore/runtime/JSScope.h
Source/JavaScriptCore/runtime/JSSetIterator.cpp
Source/JavaScriptCore/runtime/JSStringIterator.cpp
Source/JavaScriptCore/runtime/JSTemplateRegistryKey.cpp
Source/JavaScriptCore/runtime/JSTypedArrayViewConstructor.cpp
Source/JavaScriptCore/runtime/JSTypedArrayViewPrototype.cpp
Source/JavaScriptCore/runtime/JSWeakMap.cpp
Source/JavaScriptCore/runtime/JSWeakSet.cpp
Source/JavaScriptCore/runtime/MapConstructor.cpp
Source/JavaScriptCore/runtime/MapIteratorPrototype.cpp
Source/JavaScriptCore/runtime/MapPrototype.cpp
Source/JavaScriptCore/runtime/NativeErrorConstructor.cpp
Source/JavaScriptCore/runtime/NativeStdFunctionCell.cpp
Source/JavaScriptCore/runtime/Operations.h
Source/JavaScriptCore/runtime/Options.cpp
Source/JavaScriptCore/runtime/Options.h
Source/JavaScriptCore/runtime/PropertyTable.cpp
Source/JavaScriptCore/runtime/ProxyConstructor.cpp
Source/JavaScriptCore/runtime/ProxyObject.cpp
Source/JavaScriptCore/runtime/ProxyRevoke.cpp
Source/JavaScriptCore/runtime/RegExp.cpp
Source/JavaScriptCore/runtime/RegExp.h
Source/JavaScriptCore/runtime/RegExpConstructor.h
Source/JavaScriptCore/runtime/RegExpInlines.h
Source/JavaScriptCore/runtime/RegExpMatchesArray.h
Source/JavaScriptCore/runtime/RegExpPrototype.cpp
Source/JavaScriptCore/runtime/RuntimeType.cpp
Source/JavaScriptCore/runtime/SamplingProfiler.cpp
Source/JavaScriptCore/runtime/SetConstructor.cpp
Source/JavaScriptCore/runtime/SetIteratorPrototype.cpp
Source/JavaScriptCore/runtime/SetPrototype.cpp
Source/JavaScriptCore/runtime/StackFrame.cpp [new file with mode: 0644]
Source/JavaScriptCore/runtime/StackFrame.h [new file with mode: 0644]
Source/JavaScriptCore/runtime/StringConstructor.cpp
Source/JavaScriptCore/runtime/StringIteratorPrototype.cpp
Source/JavaScriptCore/runtime/StructureInlines.h
Source/JavaScriptCore/runtime/TemplateRegistry.cpp
Source/JavaScriptCore/runtime/TestRunnerUtils.cpp
Source/JavaScriptCore/runtime/TestRunnerUtils.h
Source/JavaScriptCore/runtime/ThrowScope.cpp
Source/JavaScriptCore/runtime/TypeProfilerLog.cpp
Source/JavaScriptCore/runtime/TypeSet.cpp
Source/JavaScriptCore/runtime/VM.cpp
Source/JavaScriptCore/runtime/VM.h
Source/JavaScriptCore/runtime/VMEntryScope.h
Source/JavaScriptCore/runtime/VMInlines.h
Source/JavaScriptCore/runtime/WeakMapConstructor.cpp
Source/JavaScriptCore/runtime/WeakMapData.cpp
Source/JavaScriptCore/runtime/WeakMapPrototype.cpp
Source/JavaScriptCore/runtime/WeakSetConstructor.cpp
Source/JavaScriptCore/runtime/WeakSetPrototype.cpp
Source/JavaScriptCore/testRegExp.cpp
Source/JavaScriptCore/tools/JSDollarVM.cpp
Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp
Source/WTF/ChangeLog
Source/WTF/wtf/FastMalloc.cpp
Source/WTF/wtf/FastMalloc.h
Source/WTF/wtf/ParkingLot.cpp
Source/WTF/wtf/ParkingLot.h
Source/WTF/wtf/ScopedLambda.h
Source/WTF/wtf/SentinelLinkedList.h
Source/WTF/wtf/SimpleStats.h
Source/WebCore/ChangeLog
Source/WebCore/ForwardingHeaders/heap/HeapInlines.h [new file with mode: 0644]
Source/WebCore/ForwardingHeaders/interpreter/Interpreter.h [deleted file]
Source/WebCore/ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h [new file with mode: 0644]
Source/WebCore/Modules/indexeddb/IDBCursorWithValue.cpp
Source/WebCore/Modules/indexeddb/client/TransactionOperation.cpp
Source/WebCore/Modules/indexeddb/server/SQLiteIDBBackingStore.cpp
Source/WebCore/Modules/indexeddb/server/UniqueIDBDatabase.cpp
Source/WebCore/bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp
Source/WebCore/bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp
Source/WebCore/bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp
Source/WebCore/bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp
Source/WebCore/bindings/js/JSClientRectCustom.cpp
Source/WebCore/bindings/js/JSDOMBinding.cpp
Source/WebCore/bindings/js/JSDOMBinding.h
Source/WebCore/bindings/js/JSDeviceMotionEventCustom.cpp
Source/WebCore/bindings/js/JSDeviceOrientationEventCustom.cpp
Source/WebCore/bindings/js/JSErrorEventCustom.cpp
Source/WebCore/bindings/js/JSIDBCursorWithValueCustom.cpp
Source/WebCore/bindings/js/JSIDBIndexCustom.cpp
Source/WebCore/bindings/js/JSPerformanceTimingCustom.cpp
Source/WebCore/bindings/js/JSPopStateEventCustom.cpp
Source/WebCore/bindings/js/JSWebGL2RenderingContextCustom.cpp
Source/WebCore/bindings/js/JSWorkerGlobalScopeCustom.cpp
Source/WebCore/bindings/js/WorkerScriptController.cpp
Source/WebCore/contentextensions/ContentExtensionParser.cpp
Source/WebCore/dom/ErrorEvent.cpp
Source/WebCore/html/HTMLCanvasElement.cpp
Source/WebCore/html/MediaDocument.cpp
Source/WebCore/inspector/CommandLineAPIModule.cpp
Source/WebCore/loader/EmptyClients.cpp
Source/WebCore/page/CaptionUserPreferences.cpp
Source/WebCore/page/Frame.cpp
Source/WebCore/page/PageGroup.cpp
Source/WebCore/page/UserContentController.cpp
Source/WebCore/platform/mock/mediasource/MockBox.cpp
Source/WebCore/testing/GCObservation.cpp
Source/WebKit2/ChangeLog
Source/WebKit2/UIProcess/ViewGestureController.cpp
Source/WebKit2/UIProcess/WebPageProxy.cpp
Source/WebKit2/UIProcess/WebProcessPool.cpp
Source/WebKit2/UIProcess/WebProcessProxy.cpp
Source/WebKit2/WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp
Source/WebKit2/WebProcess/Plugins/Netscape/JSNPObject.cpp
Source/bmalloc/ChangeLog
Source/bmalloc/bmalloc/Allocator.cpp
Source/bmalloc/bmalloc/Allocator.h
Source/bmalloc/bmalloc/Cache.h
Source/bmalloc/bmalloc/bmalloc.h
Tools/ChangeLog
Tools/DumpRenderTree/TestRunner.cpp
Tools/DumpRenderTree/mac/DumpRenderTree.mm
Tools/Scripts/run-jsc-stress-tests

index 111e75c..2703ead 100644 (file)
@@ -1,3 +1,18 @@
+2016-08-31  Filip Pizlo  <fpizlo@apple.com>
+
+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
+        https://bugs.webkit.org/show_bug.cgi?id=160125
+
+        Reviewed by Geoffrey Garen and Keith Miller.
+        
+        Most of the things I did properly covered by existing tests, but I found some simple cases of
+        unshifting that had sketchy coverage.
+
+        * stress/array-storage-array-unshift.js: Added.
+        * stress/contiguous-array-unshift.js: Added.
+        * stress/double-array-unshift.js: Added.
+        * stress/int32-array-unshift.js: Added.
+
 2016-09-02  Michael Saboff  <msaboff@apple.com>
 
         Unreviewed fix after importing Chakra test
diff --git a/JSTests/stress/array-storage-array-unshift.js b/JSTests/stress/array-storage-array-unshift.js
new file mode 100644 (file)
index 0000000..a4ae62c
--- /dev/null
@@ -0,0 +1,8 @@
+//@ runDefault
+var x = [2.5, 1.5];
+x.length = 1000000000;
+x.length = 2;
+Array.prototype.unshift.call(x, 3.5);
+if (x.toString() != "3.5,2.5,1.5")
+    throw "Error: bad result: " + describe(x);
+
diff --git a/JSTests/stress/contiguous-array-unshift.js b/JSTests/stress/contiguous-array-unshift.js
new file mode 100644 (file)
index 0000000..2509d9c
--- /dev/null
@@ -0,0 +1,6 @@
+//@ runDefault
+var x = ['b', 'a'];
+Array.prototype.unshift.call(x, 'c');
+if (x.toString() != "c,b,a")
+    throw "Error: bad result: " + describe(x);
+
diff --git a/JSTests/stress/double-array-unshift.js b/JSTests/stress/double-array-unshift.js
new file mode 100644 (file)
index 0000000..fc9aeb8
--- /dev/null
@@ -0,0 +1,6 @@
+//@ runDefault
+var x = [2.5, 1.5];
+Array.prototype.unshift.call(x, 3.5);
+if (x.toString() != "3.5,2.5,1.5")
+    throw "Error: bad result: " + describe(x);
+
diff --git a/JSTests/stress/int32-array-unshift.js b/JSTests/stress/int32-array-unshift.js
new file mode 100644 (file)
index 0000000..4d4c69d
--- /dev/null
@@ -0,0 +1,6 @@
+//@ runDefault
+var x = [2, 1];
+Array.prototype.unshift.call(x, 3);
+if (x.toString() != "3,2,1")
+    throw "Error: bad result: " + describe(x);
+
diff --git a/JSTests/stress/unshift-array-storage.js b/JSTests/stress/unshift-array-storage.js
new file mode 100644 (file)
index 0000000..2c8fc20
--- /dev/null
@@ -0,0 +1,10 @@
+//@ runMiscFTLNoCJITTest("--scribbleFreeCells=true", "--deferGCShouldCollectWithProbability=true", "--deferGCProbability=1")
+
+// Create some array storage.
+var array = [];
+ensureArrayStorage(array);
+
+for (var i = 0; i < 1000; ++i)
+    array.push(i);
+
+array.unshift(1, 2, 3, 4); // This will crash if we aren't careful about when we GC during unshift.
index e788b5c..d62dfa0 100644 (file)
@@ -213,6 +213,7 @@ private:
     m_owners = [[NSMapTable alloc] initWithKeyOptions:weakIDOptions valueOptions:integerOptions capacity:1];
 
     JSC::JSValue jsValue = toJS(exec, [value JSValueRef]);
+    dataLog("Creating managed value with value ", jsValue, "\n");
     if (jsValue.isObject())
         m_weakValue.setObject(JSC::jsCast<JSC::JSObject*>(jsValue.asCell()), self);
     else if (jsValue.isString())
index 87dd8fa..d488cc8 100644 (file)
@@ -32,7 +32,7 @@
 #include "ClassInfo.h"
 #include "Error.h"
 #include "JSArrayBufferViewInlines.h"
-#include "JSCJSValueInlines.h"
+#include "JSCInlines.h"
 #include "JSDataView.h"
 #include "JSGenericTypedArrayViewInlines.h"
 #include "JSTypedArrays.h"
index 10b3097..ffdc0d4 100644 (file)
@@ -31,9 +31,8 @@
 #import "APICallbackFunction.h"
 #import "APICast.h"
 #import "Error.h"
-#import "JSCJSValueInlines.h"
 #import "JSCell.h"
-#import "JSCellInlines.h"
+#import "JSCInlines.h"
 #import "JSContextInternal.h"
 #import "JSWrapperMap.h"
 #import "JSValueInternal.h"
index 6d6ed17..13d4362 100644 (file)
@@ -510,33 +510,6 @@ static void* multiVMThreadMain(void* okPtr)
     return nullptr;
 }
 
-// This test is flaky. Since GC marks C stack and registers as roots conservatively,
-// objects not referenced logically can be accidentally marked and alive.
-// To avoid this situation as possible as we can,
-// 1. run this test first before stack is polluted,
-// 2. extract this test as a function to suppress stack height.
-static void testWeakValue()
-{
-    @autoreleasepool {
-        JSVirtualMachine *vm = [[JSVirtualMachine alloc] init];
-        TestObject *testObject = [TestObject testObject];
-        JSManagedValue *weakValue;
-        @autoreleasepool {
-            JSContext *context = [[JSContext alloc] initWithVirtualMachine:vm];
-            context[@"testObject"] = testObject;
-            weakValue = [[JSManagedValue alloc] initWithValue:context[@"testObject"]];
-        }
-
-        @autoreleasepool {
-            JSContext *context = [[JSContext alloc] initWithVirtualMachine:vm];
-            context[@"testObject"] = testObject;
-            JSSynchronousGarbageCollectForDebugging([context JSGlobalContextRef]);
-            checkResult(@"weak value == nil", ![weakValue value]);
-            checkResult(@"root is still alive", !context[@"testObject"].isUndefined);
-        }
-    }
-}
-
 static void testObjectiveCAPIMain()
 {
     @autoreleasepool {
@@ -1513,7 +1486,6 @@ void testObjectiveCAPI()
 {
     NSLog(@"Testing Objective-C API");
     checkNegativeNSIntegers();
-    testWeakValue();
     testObjectiveCAPIMain();
 }
 
index 5dec317..503af3a 100644 (file)
@@ -66,6 +66,7 @@ set(JavaScriptCore_SOURCES
     assembler/MacroAssembler.cpp
     assembler/MacroAssemblerARM.cpp
     assembler/MacroAssemblerARMv7.cpp
+    assembler/MacroAssemblerCodeRef.cpp
     assembler/MacroAssemblerPrinter.cpp
     assembler/MacroAssemblerX86Common.cpp
 
@@ -447,6 +448,7 @@ set(JavaScriptCore_SOURCES
     heap/DestructionMode.cpp
     heap/EdenGCActivityCallback.cpp
     heap/FullGCActivityCallback.cpp
+    heap/FreeList.cpp
     heap/GCActivityCallback.cpp
     heap/GCLogging.cpp
     heap/HandleSet.cpp
@@ -454,6 +456,7 @@ set(JavaScriptCore_SOURCES
     heap/Heap.cpp
     heap/HeapCell.cpp
     heap/HeapHelperPool.cpp
+    heap/HeapOperation.cpp
     heap/HeapProfiler.cpp
     heap/HeapSnapshot.cpp
     heap/HeapSnapshotBuilder.cpp
@@ -462,6 +465,7 @@ set(JavaScriptCore_SOURCES
     heap/HeapVerifier.cpp
     heap/IncrementalSweeper.cpp
     heap/JITStubRoutineSet.cpp
+    heap/LargeAllocation.cpp
     heap/LiveObjectList.cpp
     heap/MachineStackMarker.cpp
     heap/MarkStack.cpp
@@ -619,6 +623,7 @@ set(JavaScriptCore_SOURCES
     runtime/ArrayBufferNeuteringWatchpoint.cpp
     runtime/ArrayBufferView.cpp
     runtime/ArrayConstructor.cpp
+    runtime/ArrayConventions.cpp
     runtime/ArrayIteratorPrototype.cpp
     runtime/ArrayPrototype.cpp
     runtime/BasicBlockLocation.cpp
@@ -799,6 +804,7 @@ set(JavaScriptCore_SOURCES
     runtime/SimpleTypedArrayController.cpp
     runtime/SmallStrings.cpp
     runtime/SparseArrayValueMap.cpp
+    runtime/StackFrame.cpp
     runtime/StrictEvalActivation.cpp
     runtime/StringConstructor.cpp
     runtime/StringIteratorPrototype.cpp
index 33a81b5..ec7c64f 100644 (file)
+2016-08-31  Filip Pizlo  <fpizlo@apple.com>
+
+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
+        https://bugs.webkit.org/show_bug.cgi?id=160125
+
+        Reviewed by Geoffrey Garen and Keith Miller.
+
+        In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
+        copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
+        from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
+        fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
+        to get rid of copied space. This change moves copied space's biggest client over to marked
+        space.
+        
+        Moving butterflies to marked space means having them use the new Auxiliary HeapCell
+        allocation path. This is a fairly mechanical change, but it caused performance regressions
+        everywhere, so this change also fixes MarkedSpace's performance issues.
+        
+        At a high level the mechanical changes are:
+        
+        - We use AuxiliaryBarrier instead of CopyBarrier.
+        
+        - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
+          CheckedBoolean stuff, since it's so much more trouble than it's worth.
+        
+        - The JITs have to emit inlined marked space allocations instead of inline copy space
+          allocations.
+        
+        - Everyone has to get used to zeroing their butterflies after allocation instead of relying
+          on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
+          space doesn't.
+        
+        That's about 1/3 of this change. But this led to performance problems, which I fixed with
+        optimizations that amounted to a major MarkedSpace rewrite:
+        
+        - MarkedSpace always causes internal fragmentation for array allocations because the vector
+          length we choose when we resize usually leads to a cell size that doesn't correspond to any
+          size class. I got around this by making array allocations usually round up vectorLength to
+          the maximum allowed by the size class that we would have allocated in. Also,
+          ensureLengthSlow() and friends first make sure that the requested length can't just be
+          fulfilled with the current allocation size. This safeguard means that not every array
+          allocation has to do size class queries. For example, the fast path of new Array(length)
+          never does any size class queries, under the assumption that (1) the speed gained from
+          avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
+          size class query, is too small to offset the speed lost by doing the query on every
+          allocation and (2) new Array(length) is a pretty good hint that resizing is not very
+          likely.
+        
+        - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
+          changes MarkedSpace size classes to use a linear progression for very small sizes followed
+          by a geometric progression that naturally transitions to a hyperbolic progression. We want
+          hyperbolic sizes when we get close to blockSize: for example the largest size we want is
+          payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
+          next size down should be payloadSize / 3 rounded down, and so on. After the last precise
+          size (80 bytes), we proceed using a geometric progression, but round up each size to
+          minimize slop at the end of the block. This naturally causes the geometric progression to
+          turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
+          it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
+        
+        - Large allocations caused massive internal fragmentation, since the smallest large
+          allocation had to use exactly blockSize, and the largest small allocation used
+          blockSize / 2. The next size up - the first large allocation size to require two blocks -
+          also had 50% internal fragmentation. This is because we required large allocations to be
+          blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
+          that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
+          owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
+          a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
+          Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
+          out great since (1) typed arrays won't use large allocations anyway since they have their
+          own malloc fallback and (2) large array butterflies already have a 8 byte header, which
+          means that the 8 byte base misalignment aligns the large array payload on a 16 byte
+          boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
+          rare as possible; for example, ExecState::vm() skips the check because we know that callees
+          must be small allocations. It's also possible to use template tricks to do one check for
+          cell container kind, and then invoke a function specialized for MarkedBlock or a function
+          specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
+          that get used from functions that are template-specialized like this. That's mostly to
+          speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
+          directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
+          allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
+          HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
+          cutoff for large allocations is runtime-configurable, so long as you don't choose something
+          so small that callees end up large. I found that 400 bytes is roughly optimal. This means
+          that the MarkedBlock size classes end up being:
+          
+          16, 32, 48, 64, 80, 112, 160, 224, 320
+          
+          The next size class would have been 432, but that's above the 400 byte cutoff. All of this
+          is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
+          size classes you end up with by doing --dumpSizeClasses=true.
+        
+        - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
+          of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
+          lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
+          MarkedBlocks because the list that contains the interesting meta-data is allocated on the
+          side, for better locality during a sequential walk. It's no longer necessary to scan
+          MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
+          maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
+          bits because we now use versioned mark bits: to clear then, just increment the 64-bit
+          heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
+          because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
+          the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
+          together, these optimizations allowed me to reduce block size to 16KB without losing much
+          performance. There is some small perf loss on JetStream/splay, but not enough to hurt
+          JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
+          progression on membuster. That's not possible yet, since there is still enough per-block
+          overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
+          this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
+        
+        - Even after all of that, copying butterflies was still faster because it allowed us to skip
+          sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
+          so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
+          larger than O(live), especially in an eden collection. Copying satisfies this premise while
+          mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
+          MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
+          that we walk the block to build a free list. This means walking dead space. The new
+          allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
+          The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
+          pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
+          we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
+          quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
+          is completely empty, which is the common case because of the generational hypothesis: the
+          number of objects that survive an eden collection is a tiny fraction of the number of
+          objects that had been allocated, and this fraction is so small that there are typically
+          fewer than one survivors per MarkedBlock. This change was enough to make this change a net
+          win over tip-of-tree.
+        
+        - FTL now shares the same allocation fast paths as everything else, which is great, because
+          bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
+          control flow, since it won't be able to improve the machine code we write ourselves. GC
+          fast paths are best written in assembly. So, I've empowered B3 to have even better support
+          for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
+          So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
+          AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
+          size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
+          ability to constant-fold some allocator logic (in case we first assume that we're doing a
+          variable-length allocation but then realize that the length is known). I think it makes
+          sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
+          since this makes lowering easier (you can constant fold during lowering more easily) and it
+          reduces the amount of malloc traffic. In the future, we could teach B3 how to better
+          constant-fold this code. That would require allowing loads to be constant-folded, which is
+          doable but hella tricky.
+        
+        - It used to be that if a logical object allocation required two physical allocations (first
+          the butterfly and then the cell), then the JIT would emit the code in such a way that a
+          failure in the second fast path would cause us to forget the successful first physical
+          allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
+          register to storing either the butterfly or null, because the butterfly register is anyway
+          going to be free inside the first allocation. The only overhead here is zeroing the
+          butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
+          path, which can then either allocate a butterfly or not. So now we never waste a successful
+          allocation. This patch implements such a solution both in DFG (where it's easy to do this
+          since we control registers already) and in FTL (where it's annoying, because mutable
+          "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
+          duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
+          one version of this patch, this wastage would resonate with some Kraken benchmark: the
+          benchmark would always allocate N small things followed by one bigger thing. The problem
+          was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
+          the JSObject size class, which both the small and big thing shared for their cell, could
+          hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
+          it allocated the big thing. So, it would end up having to allocate the big thing's large
+          butterfly twice, every single time! Ouch!
+        
+        - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
+          amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
+          of the allocator to initialize the object to whatever state they need. It used to be that
+          we would just use a dumb loop. I initially changed this so that we would end up in memset
+          for large allocations, but this didn't actually help performance that much. I got a much
+          better result by playing with different memsets written in assembly. First I wrote one
+          using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
+          "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
+          allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
+          memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
+          double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
+          of stores (if the size is known), but those paths in the JIT won't trigger for
+          NewArrayWithSize() if the size is large, since that takes us to the
+          operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
+          optimizations here are all in JSArray::create() - that's the hot place for large arrays
+          that need to be filled with holes.
+        
+        All of this put together gives us neutral perf on JetStream,  membuster, and PLT3, a ~1%
+        regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
+        because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
+        best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
+        difference, but take the hit for now because it's a crazy corner case. By comparison, the
+        alternative approach of using a copy barrier would have cost us 1-2%. That's the real
+        apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
+        finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
+        marking barrier and we simply won't need a copying barrier. This change gets us there for
+        the purposes of our benchmarks, since the remaining clients of copied space are not very
+        important. On the other hand, if we keep copying, then getting barrier-ready would mean
+        adding back the copy barrier, which costs more perf.
+        
+        We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
+        typed arrays and a few other weird things over to Aux MarkedSpace.
+        
+        This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
+        and CellContainer meant that I had to include those files from everywhere. Fortunately,
+        just including JSCInlines.h (instead of manually including the files that includes) is
+        usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
+        that we were already basically doing. In places where JSCInlines.h would be too much, I just
+        included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
+        JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
+        meant having to manually include HeapInlines.h from the places that previously got it
+        implicitly via JSObject.h. But that led to more problems for some reason: I started getting
+        build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
+        since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
+        made it impossible to include it from outside JSC. This was a lot of work, but it was
+        necessary to get the patch to build on all ports. It's also a net win. There were many places
+        in WebCore that were transitively including a *ton* of JSC headers just because of the
+        JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
+        (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
+
+        * API/JSManagedValue.mm:
+        (-[JSManagedValue initWithValue:]):
+        * API/JSTypedArray.cpp:
+        * API/ObjCCallbackFunction.mm:
+        * API/tests/testapi.mm:
+        (testObjectiveCAPI):
+        (testWeakValue): Deleted.
+        * CMakeLists.txt:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * Scripts/builtins/builtins_generate_combined_implementation.py:
+        (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
+        * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
+        (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
+        * Scripts/builtins/builtins_generate_separate_implementation.py:
+        (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::AbstractMacroAssembler::JumpList::link):
+        (JSC::AbstractMacroAssembler::JumpList::linkTo):
+        * assembler/MacroAssembler.h:
+        * assembler/MacroAssemblerARM64.h:
+        (JSC::MacroAssemblerARM64::add32):
+        * assembler/MacroAssemblerCodeRef.cpp: Added.
+        (JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
+        (JSC::MacroAssemblerCodePtr::dumpWithName):
+        (JSC::MacroAssemblerCodePtr::dump):
+        (JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
+        (JSC::MacroAssemblerCodeRef::dump):
+        * assembler/MacroAssemblerCodeRef.h:
+        (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
+        (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
+        (JSC::MacroAssemblerCodePtr::dump): Deleted.
+        (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
+        (JSC::MacroAssemblerCodeRef::dump): Deleted.
+        * b3/B3BasicBlock.cpp:
+        (JSC::B3::BasicBlock::appendBoolConstant):
+        * b3/B3BasicBlock.h:
+        * b3/B3DuplicateTails.cpp:
+        * b3/B3StackmapGenerationParams.h:
+        * b3/testb3.cpp:
+        (JSC::B3::testPatchpointTerminalReturnValue):
+        (JSC::B3::run):
+        * bindings/ScriptValue.cpp:
+        * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
+        * bytecode/BytecodeBasicBlock.cpp:
+        * bytecode/BytecodeLivenessAnalysis.cpp:
+        * bytecode/BytecodeUseDef.h:
+        * bytecode/CallLinkInfo.cpp:
+        (JSC::CallLinkInfo::callTypeFor):
+        * bytecode/CallLinkInfo.h:
+        (JSC::CallLinkInfo::callTypeFor): Deleted.
+        * bytecode/CallLinkStatus.cpp:
+        * bytecode/CodeBlock.cpp:
+        (JSC::CodeBlock::finishCreation):
+        (JSC::CodeBlock::clearLLIntGetByIdCache):
+        (JSC::CodeBlock::predictedMachineCodeSize):
+        * bytecode/CodeBlock.h:
+        (JSC::CodeBlock::jitCodeMap): Deleted.
+        (JSC::clearLLIntGetByIdCache): Deleted.
+        * bytecode/ExecutionCounter.h:
+        * bytecode/Instruction.h:
+        * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
+        (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
+        * bytecode/ObjectAllocationProfile.h:
+        (JSC::ObjectAllocationProfile::isNull):
+        (JSC::ObjectAllocationProfile::initialize):
+        * bytecode/Opcode.h:
+        (JSC::padOpcodeName):
+        * bytecode/PolymorphicAccess.cpp:
+        (JSC::AccessCase::generateImpl):
+        (JSC::PolymorphicAccess::regenerate):
+        * bytecode/PolymorphicAccess.h:
+        * bytecode/PreciseJumpTargets.cpp:
+        * bytecode/StructureStubInfo.cpp:
+        * bytecode/StructureStubInfo.h:
+        * bytecode/UnlinkedCodeBlock.cpp:
+        (JSC::UnlinkedCodeBlock::vm): Deleted.
+        * bytecode/UnlinkedCodeBlock.h:
+        * bytecode/UnlinkedInstructionStream.cpp:
+        * bytecode/UnlinkedInstructionStream.h:
+        * dfg/DFGOperations.cpp:
+        * dfg/DFGSpeculativeJIT.cpp:
+        (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
+        (JSC::DFG::SpeculativeJIT::compileMakeRope):
+        (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
+        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
+        * dfg/DFGSpeculativeJIT.h:
+        (JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
+        (JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::compile):
+        (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
+        * dfg/DFGStrengthReductionPhase.cpp:
+        (JSC::DFG::StrengthReductionPhase::handleNode):
+        * ftl/FTLAbstractHeapRepository.h:
+        * ftl/FTLCompile.cpp:
+        * ftl/FTLJITFinalizer.cpp:
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
+        (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
+        (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
+        (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
+        (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
+        (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
+        (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateCell):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateObject):
+        (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
+        (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
+        (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
+        * ftl/FTLOutput.cpp:
+        (JSC::FTL::Output::constBool):
+        (JSC::FTL::Output::add):
+        (JSC::FTL::Output::shl):
+        (JSC::FTL::Output::aShr):
+        (JSC::FTL::Output::lShr):
+        (JSC::FTL::Output::zeroExt):
+        (JSC::FTL::Output::equal):
+        (JSC::FTL::Output::notEqual):
+        (JSC::FTL::Output::above):
+        (JSC::FTL::Output::aboveOrEqual):
+        (JSC::FTL::Output::below):
+        (JSC::FTL::Output::belowOrEqual):
+        (JSC::FTL::Output::greaterThan):
+        (JSC::FTL::Output::greaterThanOrEqual):
+        (JSC::FTL::Output::lessThan):
+        (JSC::FTL::Output::lessThanOrEqual):
+        (JSC::FTL::Output::select):
+        (JSC::FTL::Output::appendSuccessor):
+        (JSC::FTL::Output::addIncomingToPhi):
+        * ftl/FTLOutput.h:
+        * ftl/FTLValueFromBlock.h:
+        (JSC::FTL::ValueFromBlock::operator bool):
+        (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
+        * ftl/FTLWeightedTarget.h:
+        (JSC::FTL::WeightedTarget::frequentedBlock):
+        * heap/CellContainer.h: Added.
+        (JSC::CellContainer::CellContainer):
+        (JSC::CellContainer::operator bool):
+        (JSC::CellContainer::isMarkedBlock):
+        (JSC::CellContainer::isLargeAllocation):
+        (JSC::CellContainer::markedBlock):
+        (JSC::CellContainer::largeAllocation):
+        * heap/CellContainerInlines.h: Added.
+        (JSC::CellContainer::isMarked):
+        (JSC::CellContainer::isMarkedOrNewlyAllocated):
+        (JSC::CellContainer::noteMarked):
+        (JSC::CellContainer::cellSize):
+        (JSC::CellContainer::weakSet):
+        (JSC::CellContainer::flipIfNecessary):
+        * heap/ConservativeRoots.cpp:
+        (JSC::ConservativeRoots::ConservativeRoots):
+        (JSC::ConservativeRoots::~ConservativeRoots):
+        (JSC::ConservativeRoots::grow):
+        (JSC::ConservativeRoots::genericAddPointer):
+        (JSC::ConservativeRoots::genericAddSpan):
+        * heap/ConservativeRoots.h:
+        (JSC::ConservativeRoots::roots):
+        * heap/CopyToken.h:
+        * heap/FreeList.cpp: Added.
+        (JSC::FreeList::dump):
+        * heap/FreeList.h: Added.
+        (JSC::FreeList::FreeList):
+        (JSC::FreeList::list):
+        (JSC::FreeList::bump):
+        (JSC::FreeList::operator==):
+        (JSC::FreeList::operator!=):
+        (JSC::FreeList::operator bool):
+        (JSC::FreeList::allocationWillFail):
+        (JSC::FreeList::allocationWillSucceed):
+        * heap/GCTypeMap.h: Added.
+        (JSC::GCTypeMap::operator[]):
+        * heap/Heap.cpp:
+        (JSC::Heap::Heap):
+        (JSC::Heap::lastChanceToFinalize):
+        (JSC::Heap::finalizeUnconditionalFinalizers):
+        (JSC::Heap::markRoots):
+        (JSC::Heap::copyBackingStores):
+        (JSC::Heap::gatherStackRoots):
+        (JSC::Heap::gatherJSStackRoots):
+        (JSC::Heap::gatherScratchBufferRoots):
+        (JSC::Heap::clearLivenessData):
+        (JSC::Heap::visitSmallStrings):
+        (JSC::Heap::visitConservativeRoots):
+        (JSC::Heap::removeDeadCompilerWorklistEntries):
+        (JSC::Heap::gatherExtraHeapSnapshotData):
+        (JSC::Heap::removeDeadHeapSnapshotNodes):
+        (JSC::Heap::visitProtectedObjects):
+        (JSC::Heap::visitArgumentBuffers):
+        (JSC::Heap::visitException):
+        (JSC::Heap::visitStrongHandles):
+        (JSC::Heap::visitHandleStack):
+        (JSC::Heap::visitSamplingProfiler):
+        (JSC::Heap::traceCodeBlocksAndJITStubRoutines):
+        (JSC::Heap::converge):
+        (JSC::Heap::visitWeakHandles):
+        (JSC::Heap::updateObjectCounts):
+        (JSC::Heap::clearUnmarkedExecutables):
+        (JSC::Heap::deleteUnmarkedCompiledCode):
+        (JSC::Heap::collectAllGarbage):
+        (JSC::Heap::collect):
+        (JSC::Heap::collectWithoutAnySweep):
+        (JSC::Heap::collectImpl):
+        (JSC::Heap::suspendCompilerThreads):
+        (JSC::Heap::willStartCollection):
+        (JSC::Heap::flushOldStructureIDTables):
+        (JSC::Heap::flushWriteBarrierBuffer):
+        (JSC::Heap::stopAllocation):
+        (JSC::Heap::prepareForMarking):
+        (JSC::Heap::reapWeakHandles):
+        (JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
+        (JSC::Heap::sweepArrayBuffers):
+        (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
+        (JSC::MarkedBlockSnapshotFunctor::operator()):
+        (JSC::Heap::snapshotMarkedSpace):
+        (JSC::Heap::deleteSourceProviderCaches):
+        (JSC::Heap::notifyIncrementalSweeper):
+        (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
+        (JSC::Heap::resetAllocators):
+        (JSC::Heap::updateAllocationLimits):
+        (JSC::Heap::didFinishCollection):
+        (JSC::Heap::resumeCompilerThreads):
+        (JSC::Zombify::visit):
+        (JSC::Heap::forEachCodeBlockImpl):
+        * heap/Heap.h:
+        (JSC::Heap::allocatorForObjectWithoutDestructor):
+        (JSC::Heap::allocatorForObjectWithDestructor):
+        (JSC::Heap::allocatorForAuxiliaryData):
+        (JSC::Heap::jitStubRoutines):
+        (JSC::Heap::codeBlockSet):
+        (JSC::Heap::storageAllocator): Deleted.
+        * heap/HeapCell.h:
+        (JSC::HeapCell::isZapped): Deleted.
+        * heap/HeapCellInlines.h: Added.
+        (JSC::HeapCell::isLargeAllocation):
+        (JSC::HeapCell::cellContainer):
+        (JSC::HeapCell::markedBlock):
+        (JSC::HeapCell::largeAllocation):
+        (JSC::HeapCell::heap):
+        (JSC::HeapCell::vm):
+        (JSC::HeapCell::cellSize):
+        (JSC::HeapCell::allocatorAttributes):
+        (JSC::HeapCell::destructionMode):
+        (JSC::HeapCell::cellKind):
+        * heap/HeapInlines.h:
+        (JSC::Heap::heap):
+        (JSC::Heap::isLive):
+        (JSC::Heap::isMarked):
+        (JSC::Heap::testAndSetMarked):
+        (JSC::Heap::setMarked):
+        (JSC::Heap::cellSize):
+        (JSC::Heap::forEachCodeBlock):
+        (JSC::Heap::allocateObjectOfType):
+        (JSC::Heap::subspaceForObjectOfType):
+        (JSC::Heap::allocatorForObjectOfType):
+        (JSC::Heap::allocateAuxiliary):
+        (JSC::Heap::tryAllocateAuxiliary):
+        (JSC::Heap::tryReallocateAuxiliary):
+        (JSC::Heap::isPointerGCObject): Deleted.
+        (JSC::Heap::isValueGCObject): Deleted.
+        * heap/HeapOperation.cpp: Added.
+        (WTF::printInternal):
+        * heap/HeapOperation.h:
+        * heap/HeapUtil.h: Added.
+        (JSC::HeapUtil::findGCObjectPointersForMarking):
+        (JSC::HeapUtil::isPointerGCObjectJSCell):
+        (JSC::HeapUtil::isValueGCObject):
+        * heap/IncrementalSweeper.cpp:
+        (JSC::IncrementalSweeper::sweepNextBlock):
+        * heap/IncrementalSweeper.h:
+        * heap/LargeAllocation.cpp: Added.
+        (JSC::LargeAllocation::tryCreate):
+        (JSC::LargeAllocation::LargeAllocation):
+        (JSC::LargeAllocation::lastChanceToFinalize):
+        (JSC::LargeAllocation::shrink):
+        (JSC::LargeAllocation::visitWeakSet):
+        (JSC::LargeAllocation::reapWeakSet):
+        (JSC::LargeAllocation::flip):
+        (JSC::LargeAllocation::isEmpty):
+        (JSC::LargeAllocation::sweep):
+        (JSC::LargeAllocation::destroy):
+        (JSC::LargeAllocation::dump):
+        * heap/LargeAllocation.h: Added.
+        (JSC::LargeAllocation::fromCell):
+        (JSC::LargeAllocation::cell):
+        (JSC::LargeAllocation::isLargeAllocation):
+        (JSC::LargeAllocation::heap):
+        (JSC::LargeAllocation::vm):
+        (JSC::LargeAllocation::weakSet):
+        (JSC::LargeAllocation::clearNewlyAllocated):
+        (JSC::LargeAllocation::isNewlyAllocated):
+        (JSC::LargeAllocation::isMarked):
+        (JSC::LargeAllocation::isMarkedOrNewlyAllocated):
+        (JSC::LargeAllocation::isLive):
+        (JSC::LargeAllocation::hasValidCell):
+        (JSC::LargeAllocation::cellSize):
+        (JSC::LargeAllocation::aboveLowerBound):
+        (JSC::LargeAllocation::belowUpperBound):
+        (JSC::LargeAllocation::contains):
+        (JSC::LargeAllocation::attributes):
+        (JSC::LargeAllocation::flipIfNecessary):
+        (JSC::LargeAllocation::flipIfNecessaryConcurrently):
+        (JSC::LargeAllocation::testAndSetMarked):
+        (JSC::LargeAllocation::setMarked):
+        (JSC::LargeAllocation::clearMarked):
+        (JSC::LargeAllocation::noteMarked):
+        (JSC::LargeAllocation::headerSize):
+        * heap/MarkedAllocator.cpp:
+        (JSC::MarkedAllocator::MarkedAllocator):
+        (JSC::MarkedAllocator::isPagedOut):
+        (JSC::MarkedAllocator::retire):
+        (JSC::MarkedAllocator::filterNextBlock):
+        (JSC::MarkedAllocator::setNextBlockToSweep):
+        (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
+        (JSC::MarkedAllocator::tryAllocateWithoutCollecting):
+        (JSC::MarkedAllocator::allocateSlowCase):
+        (JSC::MarkedAllocator::tryAllocateSlowCase):
+        (JSC::MarkedAllocator::allocateSlowCaseImpl):
+        (JSC::blockHeaderSize):
+        (JSC::MarkedAllocator::blockSizeForBytes):
+        (JSC::MarkedAllocator::tryAllocateBlock):
+        (JSC::MarkedAllocator::addBlock):
+        (JSC::MarkedAllocator::removeBlock):
+        (JSC::MarkedAllocator::stopAllocating):
+        (JSC::MarkedAllocator::reset):
+        (JSC::MarkedAllocator::lastChanceToFinalize):
+        (JSC::MarkedAllocator::setFreeList):
+        (JSC::isListPagedOut): Deleted.
+        (JSC::MarkedAllocator::tryAllocateHelper): Deleted.
+        (JSC::MarkedAllocator::tryPopFreeList): Deleted.
+        (JSC::MarkedAllocator::tryAllocate): Deleted.
+        (JSC::MarkedAllocator::allocateBlock): Deleted.
+        * heap/MarkedAllocator.h:
+        (JSC::MarkedAllocator::takeLastActiveBlock):
+        (JSC::MarkedAllocator::offsetOfFreeList):
+        (JSC::MarkedAllocator::offsetOfCellSize):
+        (JSC::MarkedAllocator::tryAllocate):
+        (JSC::MarkedAllocator::allocate):
+        (JSC::MarkedAllocator::forEachBlock):
+        (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
+        (JSC::MarkedAllocator::MarkedAllocator): Deleted.
+        (JSC::MarkedAllocator::init): Deleted.
+        (JSC::MarkedAllocator::stopAllocating): Deleted.
+        * heap/MarkedBlock.cpp:
+        (JSC::MarkedBlock::tryCreate):
+        (JSC::MarkedBlock::Handle::Handle):
+        (JSC::MarkedBlock::Handle::~Handle):
+        (JSC::MarkedBlock::MarkedBlock):
+        (JSC::MarkedBlock::Handle::specializedSweep):
+        (JSC::MarkedBlock::Handle::sweep):
+        (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
+        (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
+        (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
+        (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
+        (JSC::SetNewlyAllocatedFunctor::operator()):
+        (JSC::MarkedBlock::Handle::stopAllocating):
+        (JSC::MarkedBlock::Handle::lastChanceToFinalize):
+        (JSC::MarkedBlock::Handle::resumeAllocating):
+        (JSC::MarkedBlock::Handle::zap):
+        (JSC::MarkedBlock::Handle::forEachFreeCell):
+        (JSC::MarkedBlock::flipIfNecessary):
+        (JSC::MarkedBlock::Handle::flipIfNecessary):
+        (JSC::MarkedBlock::flipIfNecessarySlow):
+        (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
+        (JSC::MarkedBlock::clearMarks):
+        (JSC::MarkedBlock::assertFlipped):
+        (JSC::MarkedBlock::needsFlip):
+        (JSC::MarkedBlock::Handle::needsFlip):
+        (JSC::MarkedBlock::Handle::willRemoveBlock):
+        (JSC::MarkedBlock::Handle::didConsumeFreeList):
+        (JSC::MarkedBlock::markCount):
+        (JSC::MarkedBlock::Handle::isEmpty):
+        (JSC::MarkedBlock::clearHasAnyMarked):
+        (JSC::MarkedBlock::noteMarkedSlow):
+        (WTF::printInternal):
+        (JSC::MarkedBlock::create): Deleted.
+        (JSC::MarkedBlock::destroy): Deleted.
+        (JSC::MarkedBlock::callDestructor): Deleted.
+        (JSC::MarkedBlock::specializedSweep): Deleted.
+        (JSC::MarkedBlock::sweep): Deleted.
+        (JSC::MarkedBlock::sweepHelper): Deleted.
+        (JSC::MarkedBlock::stopAllocating): Deleted.
+        (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
+        (JSC::MarkedBlock::lastChanceToFinalize): Deleted.
+        (JSC::MarkedBlock::resumeAllocating): Deleted.
+        (JSC::MarkedBlock::didRetireBlock): Deleted.
+        * heap/MarkedBlock.h:
+        (JSC::MarkedBlock::VoidFunctor::returnValue):
+        (JSC::MarkedBlock::CountFunctor::CountFunctor):
+        (JSC::MarkedBlock::CountFunctor::count):
+        (JSC::MarkedBlock::CountFunctor::returnValue):
+        (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
+        (JSC::MarkedBlock::Handle::isOnBlocksToSweep):
+        (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
+        (JSC::MarkedBlock::Handle::state):
+        (JSC::MarkedBlock::needsDestruction):
+        (JSC::MarkedBlock::handle):
+        (JSC::MarkedBlock::Handle::block):
+        (JSC::MarkedBlock::firstAtom):
+        (JSC::MarkedBlock::atoms):
+        (JSC::MarkedBlock::isAtomAligned):
+        (JSC::MarkedBlock::Handle::cellAlign):
+        (JSC::MarkedBlock::blockFor):
+        (JSC::MarkedBlock::Handle::allocator):
+        (JSC::MarkedBlock::Handle::heap):
+        (JSC::MarkedBlock::Handle::vm):
+        (JSC::MarkedBlock::vm):
+        (JSC::MarkedBlock::Handle::weakSet):
+        (JSC::MarkedBlock::weakSet):
+        (JSC::MarkedBlock::Handle::shrink):
+        (JSC::MarkedBlock::Handle::visitWeakSet):
+        (JSC::MarkedBlock::Handle::reapWeakSet):
+        (JSC::MarkedBlock::Handle::cellSize):
+        (JSC::MarkedBlock::cellSize):
+        (JSC::MarkedBlock::Handle::attributes):
+        (JSC::MarkedBlock::attributes):
+        (JSC::MarkedBlock::Handle::needsDestruction):
+        (JSC::MarkedBlock::Handle::destruction):
+        (JSC::MarkedBlock::Handle::cellKind):
+        (JSC::MarkedBlock::Handle::markCount):
+        (JSC::MarkedBlock::Handle::size):
+        (JSC::MarkedBlock::atomNumber):
+        (JSC::MarkedBlock::flipIfNecessary):
+        (JSC::MarkedBlock::flipIfNecessaryConcurrently):
+        (JSC::MarkedBlock::Handle::flipIfNecessary):
+        (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
+        (JSC::MarkedBlock::Handle::flipForEdenCollection):
+        (JSC::MarkedBlock::assertFlipped):
+        (JSC::MarkedBlock::Handle::assertFlipped):
+        (JSC::MarkedBlock::isMarked):
+        (JSC::MarkedBlock::testAndSetMarked):
+        (JSC::MarkedBlock::Handle::isNewlyAllocated):
+        (JSC::MarkedBlock::Handle::setNewlyAllocated):
+        (JSC::MarkedBlock::Handle::clearNewlyAllocated):
+        (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
+        (JSC::MarkedBlock::isMarkedOrNewlyAllocated):
+        (JSC::MarkedBlock::Handle::isLive):
+        (JSC::MarkedBlock::isAtom):
+        (JSC::MarkedBlock::Handle::isLiveCell):
+        (JSC::MarkedBlock::Handle::forEachCell):
+        (JSC::MarkedBlock::Handle::forEachLiveCell):
+        (JSC::MarkedBlock::Handle::forEachDeadCell):
+        (JSC::MarkedBlock::Handle::needsSweeping):
+        (JSC::MarkedBlock::Handle::isAllocated):
+        (JSC::MarkedBlock::Handle::isMarked):
+        (JSC::MarkedBlock::Handle::isFreeListed):
+        (JSC::MarkedBlock::hasAnyMarked):
+        (JSC::MarkedBlock::noteMarked):
+        (WTF::MarkedBlockHash::hash):
+        (JSC::MarkedBlock::FreeList::FreeList): Deleted.
+        (JSC::MarkedBlock::allocator): Deleted.
+        (JSC::MarkedBlock::heap): Deleted.
+        (JSC::MarkedBlock::shrink): Deleted.
+        (JSC::MarkedBlock::visitWeakSet): Deleted.
+        (JSC::MarkedBlock::reapWeakSet): Deleted.
+        (JSC::MarkedBlock::willRemoveBlock): Deleted.
+        (JSC::MarkedBlock::didConsumeFreeList): Deleted.
+        (JSC::MarkedBlock::markCount): Deleted.
+        (JSC::MarkedBlock::isEmpty): Deleted.
+        (JSC::MarkedBlock::destruction): Deleted.
+        (JSC::MarkedBlock::cellKind): Deleted.
+        (JSC::MarkedBlock::size): Deleted.
+        (JSC::MarkedBlock::capacity): Deleted.
+        (JSC::MarkedBlock::setMarked): Deleted.
+        (JSC::MarkedBlock::clearMarked): Deleted.
+        (JSC::MarkedBlock::isNewlyAllocated): Deleted.
+        (JSC::MarkedBlock::setNewlyAllocated): Deleted.
+        (JSC::MarkedBlock::clearNewlyAllocated): Deleted.
+        (JSC::MarkedBlock::isLive): Deleted.
+        (JSC::MarkedBlock::isLiveCell): Deleted.
+        (JSC::MarkedBlock::forEachCell): Deleted.
+        (JSC::MarkedBlock::forEachLiveCell): Deleted.
+        (JSC::MarkedBlock::forEachDeadCell): Deleted.
+        (JSC::MarkedBlock::needsSweeping): Deleted.
+        (JSC::MarkedBlock::isAllocated): Deleted.
+        (JSC::MarkedBlock::isMarkedOrRetired): Deleted.
+        * heap/MarkedSpace.cpp:
+        (JSC::MarkedSpace::initializeSizeClassForStepSize):
+        (JSC::MarkedSpace::MarkedSpace):
+        (JSC::MarkedSpace::~MarkedSpace):
+        (JSC::MarkedSpace::lastChanceToFinalize):
+        (JSC::MarkedSpace::allocate):
+        (JSC::MarkedSpace::tryAllocate):
+        (JSC::MarkedSpace::allocateLarge):
+        (JSC::MarkedSpace::tryAllocateLarge):
+        (JSC::MarkedSpace::sweep):
+        (JSC::MarkedSpace::sweepLargeAllocations):
+        (JSC::MarkedSpace::zombifySweep):
+        (JSC::MarkedSpace::resetAllocators):
+        (JSC::MarkedSpace::visitWeakSets):
+        (JSC::MarkedSpace::reapWeakSets):
+        (JSC::MarkedSpace::stopAllocating):
+        (JSC::MarkedSpace::prepareForMarking):
+        (JSC::MarkedSpace::resumeAllocating):
+        (JSC::MarkedSpace::isPagedOut):
+        (JSC::MarkedSpace::freeBlock):
+        (JSC::MarkedSpace::freeOrShrinkBlock):
+        (JSC::MarkedSpace::shrink):
+        (JSC::MarkedSpace::clearNewlyAllocated):
+        (JSC::VerifyMarked::operator()):
+        (JSC::MarkedSpace::flip):
+        (JSC::MarkedSpace::objectCount):
+        (JSC::MarkedSpace::size):
+        (JSC::MarkedSpace::capacity):
+        (JSC::MarkedSpace::addActiveWeakSet):
+        (JSC::MarkedSpace::didAddBlock):
+        (JSC::MarkedSpace::didAllocateInBlock):
+        (JSC::MarkedSpace::forEachAllocator): Deleted.
+        (JSC::VerifyMarkedOrRetired::operator()): Deleted.
+        (JSC::MarkedSpace::clearMarks): Deleted.
+        * heap/MarkedSpace.h:
+        (JSC::MarkedSpace::sizeClassToIndex):
+        (JSC::MarkedSpace::indexToSizeClass):
+        (JSC::MarkedSpace::version):
+        (JSC::MarkedSpace::blocksWithNewObjects):
+        (JSC::MarkedSpace::largeAllocations):
+        (JSC::MarkedSpace::largeAllocationsNurseryOffset):
+        (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
+        (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
+        (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
+        (JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
+        (JSC::MarkedSpace::forEachLiveCell):
+        (JSC::MarkedSpace::forEachDeadCell):
+        (JSC::MarkedSpace::allocatorFor):
+        (JSC::MarkedSpace::destructorAllocatorFor):
+        (JSC::MarkedSpace::auxiliaryAllocatorFor):
+        (JSC::MarkedSpace::allocateWithoutDestructor):
+        (JSC::MarkedSpace::allocateWithDestructor):
+        (JSC::MarkedSpace::allocateAuxiliary):
+        (JSC::MarkedSpace::tryAllocateAuxiliary):
+        (JSC::MarkedSpace::forEachBlock):
+        (JSC::MarkedSpace::forEachAllocator):
+        (JSC::MarkedSpace::optimalSizeFor):
+        (JSC::MarkedSpace::didAddBlock): Deleted.
+        (JSC::MarkedSpace::didAllocateInBlock): Deleted.
+        (JSC::MarkedSpace::objectCount): Deleted.
+        (JSC::MarkedSpace::size): Deleted.
+        (JSC::MarkedSpace::capacity): Deleted.
+        * heap/SlotVisitor.cpp:
+        (JSC::SlotVisitor::SlotVisitor):
+        (JSC::SlotVisitor::didStartMarking):
+        (JSC::SlotVisitor::reset):
+        (JSC::SlotVisitor::append):
+        (JSC::SlotVisitor::appendJSCellOrAuxiliary):
+        (JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
+        (JSC::SlotVisitor::appendToMarkStack):
+        (JSC::SlotVisitor::markAuxiliary):
+        (JSC::SlotVisitor::noteLiveAuxiliaryCell):
+        (JSC::SlotVisitor::visitChildren):
+        * heap/SlotVisitor.h:
+        * heap/WeakBlock.cpp:
+        (JSC::WeakBlock::create):
+        (JSC::WeakBlock::WeakBlock):
+        (JSC::WeakBlock::visit):
+        (JSC::WeakBlock::reap):
+        * heap/WeakBlock.h:
+        (JSC::WeakBlock::disconnectContainer):
+        (JSC::WeakBlock::disconnectMarkedBlock): Deleted.
+        * heap/WeakSet.cpp:
+        (JSC::WeakSet::~WeakSet):
+        (JSC::WeakSet::sweep):
+        (JSC::WeakSet::shrink):
+        (JSC::WeakSet::addAllocator):
+        * heap/WeakSet.h:
+        (JSC::WeakSet::container):
+        (JSC::WeakSet::setContainer):
+        (JSC::WeakSet::WeakSet):
+        (JSC::WeakSet::visit):
+        (JSC::WeakSet::shrink): Deleted.
+        * heap/WeakSetInlines.h:
+        (JSC::WeakSet::allocate):
+        * inspector/InjectedScriptManager.cpp:
+        * inspector/JSGlobalObjectInspectorController.cpp:
+        * inspector/JSJavaScriptCallFrame.cpp:
+        * inspector/ScriptDebugServer.cpp:
+        * inspector/agents/InspectorDebuggerAgent.cpp:
+        * interpreter/CachedCall.h:
+        (JSC::CachedCall::CachedCall):
+        * interpreter/Interpreter.cpp:
+        (JSC::loadVarargs):
+        (JSC::StackFrame::sourceID): Deleted.
+        (JSC::StackFrame::sourceURL): Deleted.
+        (JSC::StackFrame::functionName): Deleted.
+        (JSC::StackFrame::computeLineAndColumn): Deleted.
+        (JSC::StackFrame::toString): Deleted.
+        * interpreter/Interpreter.h:
+        (JSC::StackFrame::isNative): Deleted.
+        * jit/AssemblyHelpers.h:
+        (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
+        (JSC::AssemblyHelpers::emitAllocate):
+        (JSC::AssemblyHelpers::emitAllocateJSCell):
+        (JSC::AssemblyHelpers::emitAllocateJSObject):
+        (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
+        (JSC::AssemblyHelpers::emitAllocateVariableSized):
+        * jit/GCAwareJITStubRoutine.cpp:
+        (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
+        * jit/JIT.cpp:
+        (JSC::JIT::compileCTINativeCall):
+        (JSC::JIT::link):
+        * jit/JIT.h:
+        (JSC::JIT::compileCTINativeCall): Deleted.
+        * jit/JITExceptions.cpp:
+        (JSC::genericUnwind):
+        * jit/JITExceptions.h:
+        * jit/JITOpcodes.cpp:
+        (JSC::JIT::emit_op_new_object):
+        (JSC::JIT::emitSlow_op_new_object):
+        (JSC::JIT::emit_op_create_this):
+        (JSC::JIT::emitSlow_op_create_this):
+        * jit/JITOpcodes32_64.cpp:
+        (JSC::JIT::emit_op_new_object):
+        (JSC::JIT::emitSlow_op_new_object):
+        (JSC::JIT::emit_op_create_this):
+        (JSC::JIT::emitSlow_op_create_this):
+        * jit/JITOperations.cpp:
+        * jit/JITOperations.h:
+        * jit/JITPropertyAccess.cpp:
+        (JSC::JIT::emitWriteBarrier):
+        * jit/JITThunks.cpp:
+        * jit/JITThunks.h:
+        * jsc.cpp:
+        (functionDescribeArray):
+        (main):
+        * llint/LLIntData.cpp:
+        (JSC::LLInt::Data::performAssertions):
+        * llint/LLIntExceptions.cpp:
+        * llint/LLIntThunks.cpp:
+        * llint/LLIntThunks.h:
+        * llint/LowLevelInterpreter.asm:
+        * llint/LowLevelInterpreter.cpp:
+        * llint/LowLevelInterpreter32_64.asm:
+        * llint/LowLevelInterpreter64.asm:
+        * parser/ModuleAnalyzer.cpp:
+        * parser/NodeConstructors.h:
+        * parser/Nodes.h:
+        * profiler/ProfilerBytecode.cpp:
+        * profiler/ProfilerBytecode.h:
+        * profiler/ProfilerBytecodeSequence.cpp:
+        * runtime/ArrayConventions.h:
+        (JSC::indexingHeaderForArrayStorage):
+        (JSC::baseIndexingHeaderForArrayStorage):
+        (JSC::indexingHeaderForArray): Deleted.
+        (JSC::baseIndexingHeaderForArray): Deleted.
+        * runtime/ArrayPrototype.cpp:
+        (JSC::arrayProtoFuncSplice):
+        (JSC::concatAppendOne):
+        (JSC::arrayProtoPrivateFuncConcatMemcpy):
+        * runtime/ArrayStorage.h:
+        (JSC::ArrayStorage::vectorLength):
+        (JSC::ArrayStorage::totalSizeFor):
+        (JSC::ArrayStorage::totalSize):
+        (JSC::ArrayStorage::availableVectorLength):
+        (JSC::ArrayStorage::optimalVectorLength):
+        (JSC::ArrayStorage::sizeFor): Deleted.
+        * runtime/AuxiliaryBarrier.h: Added.
+        (JSC::AuxiliaryBarrier::AuxiliaryBarrier):
+        (JSC::AuxiliaryBarrier::clear):
+        (JSC::AuxiliaryBarrier::get):
+        (JSC::AuxiliaryBarrier::slot):
+        (JSC::AuxiliaryBarrier::operator bool):
+        (JSC::AuxiliaryBarrier::setWithoutBarrier):
+        * runtime/AuxiliaryBarrierInlines.h: Added.
+        (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
+        (JSC::AuxiliaryBarrier<T>::set):
+        * runtime/Butterfly.h:
+        * runtime/ButterflyInlines.h:
+        (JSC::Butterfly::availableContiguousVectorLength):
+        (JSC::Butterfly::optimalContiguousVectorLength):
+        (JSC::Butterfly::createUninitialized):
+        (JSC::Butterfly::growArrayRight):
+        * runtime/ClonedArguments.cpp:
+        (JSC::ClonedArguments::createEmpty):
+        * runtime/CommonSlowPathsExceptions.cpp:
+        * runtime/CommonSlowPathsExceptions.h:
+        * runtime/DataView.cpp:
+        * runtime/DirectArguments.h:
+        * runtime/ECMAScriptSpecInternalFunctions.cpp:
+        * runtime/Error.cpp:
+        * runtime/Error.h:
+        * runtime/ErrorInstance.cpp:
+        * runtime/ErrorInstance.h:
+        * runtime/Exception.cpp:
+        * runtime/Exception.h:
+        * runtime/GeneratorFrame.cpp:
+        * runtime/GeneratorPrototype.cpp:
+        * runtime/InternalFunction.cpp:
+        (JSC::InternalFunction::InternalFunction):
+        * runtime/IntlCollator.cpp:
+        * runtime/IntlCollatorConstructor.cpp:
+        * runtime/IntlCollatorPrototype.cpp:
+        * runtime/IntlDateTimeFormat.cpp:
+        * runtime/IntlDateTimeFormatConstructor.cpp:
+        * runtime/IntlDateTimeFormatPrototype.cpp:
+        * runtime/IntlNumberFormat.cpp:
+        * runtime/IntlNumberFormatConstructor.cpp:
+        * runtime/IntlNumberFormatPrototype.cpp:
+        * runtime/IntlObject.cpp:
+        * runtime/IteratorPrototype.cpp:
+        * runtime/JSArray.cpp:
+        (JSC::JSArray::tryCreateUninitialized):
+        (JSC::JSArray::setLengthWritable):
+        (JSC::JSArray::unshiftCountSlowCase):
+        (JSC::JSArray::setLengthWithArrayStorage):
+        (JSC::JSArray::appendMemcpy):
+        (JSC::JSArray::setLength):
+        (JSC::JSArray::pop):
+        (JSC::JSArray::push):
+        (JSC::JSArray::fastSlice):
+        (JSC::JSArray::shiftCountWithArrayStorage):
+        (JSC::JSArray::shiftCountWithAnyIndexingType):
+        (JSC::JSArray::unshiftCountWithArrayStorage):
+        (JSC::JSArray::fillArgList):
+        (JSC::JSArray::copyToArguments):
+        * runtime/JSArray.h:
+        (JSC::createContiguousArrayButterfly):
+        (JSC::createArrayButterfly):
+        (JSC::JSArray::create):
+        (JSC::JSArray::tryCreateUninitialized): Deleted.
+        * runtime/JSArrayBufferView.h:
+        * runtime/JSCInlines.h:
+        * runtime/JSCJSValue.cpp:
+        (JSC::JSValue::dumpInContextAssumingStructure):
+        * runtime/JSCallee.cpp:
+        (JSC::JSCallee::JSCallee):
+        * runtime/JSCell.cpp:
+        (JSC::JSCell::estimatedSize):
+        * runtime/JSCell.h:
+        (JSC::JSCell::cellStateOffset): Deleted.
+        * runtime/JSCellInlines.h:
+        (JSC::ExecState::vm):
+        (JSC::JSCell::classInfo):
+        (JSC::JSCell::callDestructor):
+        (JSC::JSCell::vm): Deleted.
+        * runtime/JSFunction.cpp:
+        (JSC::JSFunction::create):
+        (JSC::JSFunction::allocateAndInitializeRareData):
+        (JSC::JSFunction::initializeRareData):
+        (JSC::JSFunction::getOwnPropertySlot):
+        (JSC::JSFunction::put):
+        (JSC::JSFunction::deleteProperty):
+        (JSC::JSFunction::defineOwnProperty):
+        (JSC::JSFunction::setFunctionName):
+        (JSC::JSFunction::reifyLength):
+        (JSC::JSFunction::reifyName):
+        (JSC::JSFunction::reifyLazyPropertyIfNeeded):
+        (JSC::JSFunction::reifyBoundNameIfNeeded):
+        * runtime/JSFunction.h:
+        * runtime/JSFunctionInlines.h:
+        (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
+        (JSC::JSFunction::JSFunction):
+        * runtime/JSGenericTypedArrayViewInlines.h:
+        (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
+        * runtime/JSInternalPromise.cpp:
+        * runtime/JSInternalPromiseConstructor.cpp:
+        * runtime/JSInternalPromiseDeferred.cpp:
+        * runtime/JSInternalPromisePrototype.cpp:
+        * runtime/JSJob.cpp:
+        * runtime/JSMapIterator.cpp:
+        * runtime/JSModuleNamespaceObject.cpp:
+        * runtime/JSModuleRecord.cpp:
+        * runtime/JSObject.cpp:
+        (JSC::JSObject::visitButterfly):
+        (JSC::JSObject::notifyPresenceOfIndexedAccessors):
+        (JSC::JSObject::createInitialIndexedStorage):
+        (JSC::JSObject::createInitialUndecided):
+        (JSC::JSObject::createInitialInt32):
+        (JSC::JSObject::createInitialDouble):
+        (JSC::JSObject::createInitialContiguous):
+        (JSC::JSObject::createArrayStorage):
+        (JSC::JSObject::createInitialArrayStorage):
+        (JSC::JSObject::convertUndecidedToInt32):
+        (JSC::JSObject::convertUndecidedToContiguous):
+        (JSC::JSObject::convertUndecidedToArrayStorage):
+        (JSC::JSObject::convertInt32ToDouble):
+        (JSC::JSObject::convertInt32ToArrayStorage):
+        (JSC::JSObject::convertDoubleToArrayStorage):
+        (JSC::JSObject::convertContiguousToArrayStorage):
+        (JSC::JSObject::putByIndexBeyondVectorLength):
+        (JSC::JSObject::putDirectIndexBeyondVectorLength):
+        (JSC::JSObject::getNewVectorLength):
+        (JSC::JSObject::increaseVectorLength):
+        (JSC::JSObject::ensureLengthSlow):
+        (JSC::JSObject::growOutOfLineStorage):
+        (JSC::JSObject::copyButterfly): Deleted.
+        (JSC::JSObject::copyBackingStore): Deleted.
+        * runtime/JSObject.h:
+        (JSC::JSObject::globalObject):
+        (JSC::JSObject::putDirectInternal):
+        (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
+        * runtime/JSObjectInlines.h:
+        * runtime/JSPromise.cpp:
+        * runtime/JSPromiseConstructor.cpp:
+        * runtime/JSPromiseDeferred.cpp:
+        * runtime/JSPromisePrototype.cpp:
+        * runtime/JSPropertyNameIterator.cpp:
+        * runtime/JSScope.cpp:
+        (JSC::JSScope::resolve):
+        * runtime/JSScope.h:
+        (JSC::JSScope::globalObject):
+        (JSC::JSScope::vm): Deleted.
+        * runtime/JSSetIterator.cpp:
+        * runtime/JSStringIterator.cpp:
+        * runtime/JSTemplateRegistryKey.cpp:
+        * runtime/JSTypedArrayViewConstructor.cpp:
+        * runtime/JSTypedArrayViewPrototype.cpp:
+        * runtime/JSWeakMap.cpp:
+        * runtime/JSWeakSet.cpp:
+        * runtime/MapConstructor.cpp:
+        * runtime/MapIteratorPrototype.cpp:
+        * runtime/MapPrototype.cpp:
+        * runtime/NativeErrorConstructor.cpp:
+        * runtime/NativeStdFunctionCell.cpp:
+        * runtime/Operations.h:
+        (JSC::scribbleFreeCells):
+        (JSC::scribble):
+        * runtime/Options.h:
+        * runtime/PropertyTable.cpp:
+        * runtime/ProxyConstructor.cpp:
+        * runtime/ProxyObject.cpp:
+        * runtime/ProxyRevoke.cpp:
+        * runtime/RegExp.cpp:
+        (JSC::RegExp::match):
+        (JSC::RegExp::matchConcurrently):
+        (JSC::RegExp::matchCompareWithInterpreter):
+        * runtime/RegExp.h:
+        * runtime/RegExpConstructor.h:
+        * runtime/RegExpInlines.h:
+        (JSC::RegExp::matchInline):
+        * runtime/RegExpMatchesArray.h:
+        (JSC::tryCreateUninitializedRegExpMatchesArray):
+        (JSC::createRegExpMatchesArray):
+        * runtime/RegExpPrototype.cpp:
+        (JSC::genericSplit):
+        * runtime/RuntimeType.cpp:
+        * runtime/SamplingProfiler.cpp:
+        (JSC::SamplingProfiler::processUnverifiedStackTraces):
+        * runtime/SetConstructor.cpp:
+        * runtime/SetIteratorPrototype.cpp:
+        * runtime/SetPrototype.cpp:
+        * runtime/StackFrame.cpp: Added.
+        (JSC::StackFrame::sourceID):
+        (JSC::StackFrame::sourceURL):
+        (JSC::StackFrame::functionName):
+        (JSC::StackFrame::computeLineAndColumn):
+        (JSC::StackFrame::toString):
+        * runtime/StackFrame.h: Added.
+        (JSC::StackFrame::isNative):
+        * runtime/StringConstructor.cpp:
+        * runtime/StringIteratorPrototype.cpp:
+        * runtime/StructureInlines.h:
+        (JSC::Structure::propertyTable):
+        * runtime/TemplateRegistry.cpp:
+        * runtime/TestRunnerUtils.cpp:
+        (JSC::finalizeStatsAtEndOfTesting):
+        * runtime/TestRunnerUtils.h:
+        * runtime/TypeProfilerLog.cpp:
+        * runtime/TypeSet.cpp:
+        * runtime/VM.cpp:
+        (JSC::VM::VM):
+        (JSC::VM::ensureStackCapacityForCLoop):
+        (JSC::VM::isSafeToRecurseSoftCLoop):
+        * runtime/VM.h:
+        * runtime/VMEntryScope.h:
+        * runtime/VMInlines.h:
+        (JSC::VM::ensureStackCapacityFor):
+        (JSC::VM::isSafeToRecurseSoft):
+        * runtime/WeakMapConstructor.cpp:
+        * runtime/WeakMapData.cpp:
+        * runtime/WeakMapPrototype.cpp:
+        * runtime/WeakSetConstructor.cpp:
+        * runtime/WeakSetPrototype.cpp:
+        * testRegExp.cpp:
+        (testOneRegExp):
+        * tools/JSDollarVM.cpp:
+        * tools/JSDollarVMPrototype.cpp:
+        (JSC::JSDollarVMPrototype::isInObjectSpace):
+
 2016-09-04  Commit Queue  <commit-queue@webkit.org>
 
         Unreviewed, rolling out r205415.
index f49ab60..1c84b78 100644 (file)
                0F04396D1B03DC0B009598B7 /* DFGCombinedLiveness.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F04396B1B03DC0B009598B7 /* DFGCombinedLiveness.cpp */; };
                0F04396E1B03DC0B009598B7 /* DFGCombinedLiveness.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */; };
                0F05C3B41683CF9200BAF45B /* DFGArrayifySlowPathGenerator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */; };
+               0F070A471D543A8B006E7232 /* CellContainer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A421D543A89006E7232 /* CellContainer.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F070A481D543A90006E7232 /* CellContainerInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A431D543A89006E7232 /* CellContainerInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F070A491D543A93006E7232 /* HeapCellInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A441D543A89006E7232 /* HeapCellInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F070A4A1D543A95006E7232 /* LargeAllocation.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F070A451D543A89006E7232 /* LargeAllocation.cpp */; };
+               0F070A4B1D543A98006E7232 /* LargeAllocation.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A461D543A89006E7232 /* LargeAllocation.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F0776BF14FF002B00102332 /* JITCompilationEffort.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0776BD14FF002800102332 /* JITCompilationEffort.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F0A75221B94BFA900110660 /* InferredType.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F0A75201B94BFA900110660 /* InferredType.cpp */; };
                0F0A75231B94BFA900110660 /* InferredType.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0A75211B94BFA900110660 /* InferredType.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F38B01817CFE75500B144D3 /* DFGCompilationKey.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38B01417CFE75500B144D3 /* DFGCompilationKey.h */; };
                0F38B01917CFE75500B144D3 /* DFGCompilationMode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F38B01517CFE75500B144D3 /* DFGCompilationMode.cpp */; };
                0F38B01A17CFE75500B144D3 /* DFGCompilationMode.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38B01617CFE75500B144D3 /* DFGCompilationMode.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F38D2A21D44196800680499 /* AuxiliaryBarrier.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F38D2A31D44196D00680499 /* AuxiliaryBarrierInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F392C891B46188400844728 /* DFGOSRExitFuzz.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F392C871B46188400844728 /* DFGOSRExitFuzz.cpp */; };
                0F392C8A1B46188400844728 /* DFGOSRExitFuzz.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F392C881B46188400844728 /* DFGOSRExitFuzz.h */; };
                0F3A1BF91A9ECB7D000DE01A /* DFGPutStackSinkingPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3A1BF71A9ECB7D000DE01A /* DFGPutStackSinkingPhase.cpp */; };
                0F4680CA14BBB16C00BFE272 /* LLIntCommon.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C514BBB16900BFE272 /* LLIntCommon.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F4680CB14BBB17200BFE272 /* LLIntOfflineAsmConfig.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C614BBB16900BFE272 /* LLIntOfflineAsmConfig.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F4680CC14BBB17A00BFE272 /* LowLevelInterpreter.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680C714BBB16900BFE272 /* LowLevelInterpreter.cpp */; };
-               0F4680CD14BBB17D00BFE272 /* LowLevelInterpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C814BBB16900BFE272 /* LowLevelInterpreter.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F4680CD14BBB17D00BFE272 /* LowLevelInterpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C814BBB16900BFE272 /* LowLevelInterpreter.h */; };
                0F4680D214BBD16500BFE272 /* LLIntData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680CE14BBB3D100BFE272 /* LLIntData.cpp */; };
-               0F4680D314BBD16700BFE272 /* LLIntData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680CF14BBB3D100BFE272 /* LLIntData.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F4680D314BBD16700BFE272 /* LLIntData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680CF14BBB3D100BFE272 /* LLIntData.h */; };
                0F4680D414BBD24900BFE272 /* HostCallReturnValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680D014BBC5F800BFE272 /* HostCallReturnValue.cpp */; };
                0F4680D514BBD24B00BFE272 /* HostCallReturnValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680D114BBC5F800BFE272 /* HostCallReturnValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F485321187750560083B687 /* DFGArithMode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F48531F187750560083B687 /* DFGArithMode.cpp */; };
                0F4F29DF18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4F29DD18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.cpp */; };
                0F4F29E018B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */; };
                0F50AF3C193E8B3900674EE8 /* DFGStructureClobberState.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */; };
+               0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5513A51D5A682A00C32BD8 /* FreeList.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F5513A81D5A68CD00C32BD8 /* FreeList.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */; };
                0F5541B11613C1FB00CE3E25 /* SpecialPointer.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F5541AF1613C1FB00CE3E25 /* SpecialPointer.cpp */; };
                0F5541B21613C1FB00CE3E25 /* SpecialPointer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5541B01613C1FB00CE3E25 /* SpecialPointer.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F55989817C86C5800A1E543 /* ToNativeFromValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F55989717C86C5600A1E543 /* ToNativeFromValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F6B8AE51C4EFE1700969052 /* B3FixSSA.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6B8AE11C4EFE1700969052 /* B3FixSSA.h */; };
                0F6C73501AC9F99F00BE1682 /* VariableWriteFireDetail.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6C734E1AC9F99F00BE1682 /* VariableWriteFireDetail.cpp */; };
                0F6C73511AC9F99F00BE1682 /* VariableWriteFireDetail.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6C734F1AC9F99F00BE1682 /* VariableWriteFireDetail.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F6DB7E91D6124B500CDBF8E /* StackFrame.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6DB7E81D6124B200CDBF8E /* StackFrame.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               0F6DB7EA1D6124B800CDBF8E /* StackFrame.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */; };
+               0F6DB7EC1D617D1100CDBF8E /* MacroAssemblerCodeRef.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */; };
                0F6E845A19030BEF00562741 /* DFGVariableAccessData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6E845919030BEF00562741 /* DFGVariableAccessData.cpp */; };
                0F6FC750196110A800E1D02D /* ComplexGetStatus.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6FC74E196110A800E1D02D /* ComplexGetStatus.cpp */; };
                0F6FC751196110A800E1D02D /* ComplexGetStatus.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6FC74F196110A800E1D02D /* ComplexGetStatus.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F8335B71639C1E6001443B5 /* ArrayAllocationProfile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F8335B41639C1E3001443B5 /* ArrayAllocationProfile.cpp */; };
                0F8335B81639C1EA001443B5 /* ArrayAllocationProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F8364B7164B0C110053329A /* DFGBranchDirection.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8364B5164B0C0E0053329A /* DFGBranchDirection.h */; };
+               0F86A26D1D6F796500CB0C92 /* HeapOperation.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F86A26C1D6F796200CB0C92 /* HeapOperation.cpp */; };
+               0F86A26F1D6F7B3300CB0C92 /* GCTypeMap.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F86A26E1D6F7B3100CB0C92 /* GCTypeMap.h */; };
                0F86AE201C5311C5006BE8EC /* B3ComputeDivisionMagic.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F86AE1F1C5311C5006BE8EC /* B3ComputeDivisionMagic.h */; };
                0F885E111849A3BE00F1E3FA /* BytecodeUseDef.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F885E101849A3BE00F1E3FA /* BytecodeUseDef.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0F893BDB1936E23C001211F4 /* DFGStructureAbstractValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F893BDA1936E23C001211F4 /* DFGStructureAbstractValue.cpp */; };
                0FA7A8EB18B413C80052371D /* Reg.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA7A8E918B413C80052371D /* Reg.cpp */; };
                0FA7A8EC18B413C80052371D /* Reg.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FA7A8EA18B413C80052371D /* Reg.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FA7A8EE18CE4FD80052371D /* ScratchRegisterAllocator.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA7A8ED18CE4FD80052371D /* ScratchRegisterAllocator.cpp */; };
+               0FADE6731D4D23BE00768457 /* HeapUtil.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FADE6721D4D23BC00768457 /* HeapUtil.h */; };
                0FAF7EFD165BA91B000C8455 /* JITDisassembler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FAF7EFA165BA919000C8455 /* JITDisassembler.cpp */; };
                0FAF7EFE165BA91F000C8455 /* JITDisassembler.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FAF7EFB165BA919000C8455 /* JITDisassembler.h */; settings = {ATTRIBUTES = (Private, ); }; };
                0FB105851675480F00F8AB6E /* ExitKind.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB105821675480C00F8AB6E /* ExitKind.cpp */; };
                0FB3878F1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB3878C1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.cpp */; };
                0FB387901BFBC44D00E3AB1E /* AirOptimizeBlockOrder.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB3878D1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.h */; };
                0FB387921BFD31A100E3AB1E /* FTLCompile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB387911BFD31A100E3AB1E /* FTLCompile.cpp */; };
+               0FB415841D78FB4C00DF8D09 /* ArrayConventions.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB415831D78F98200DF8D09 /* ArrayConventions.cpp */; };
                0FB438A319270B1D00E1FBC9 /* StructureSet.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB438A219270B1D00E1FBC9 /* StructureSet.cpp */; };
                0FB4FB731BC843140025CA5A /* FTLLazySlowPath.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB4FB701BC843140025CA5A /* FTLLazySlowPath.cpp */; };
                0FB4FB741BC843140025CA5A /* FTLLazySlowPath.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB4FB711BC843140025CA5A /* FTLLazySlowPath.h */; };
                14280865107EC11A0013E7B2 /* BooleanPrototype.cpp in Sources */ = {isa = PBXBuildFile; fileRef = BC7952340E15EB5600A898AB /* BooleanPrototype.cpp */; };
                14280870107EC1340013E7B2 /* JSWrapperObject.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 65C7A1710A8EAACB00FA37EA /* JSWrapperObject.cpp */; };
                14280875107EC13E0013E7B2 /* JSLock.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 65EA4C99092AF9E20093D800 /* JSLock.cpp */; };
-               1429D77C0ED20D7300B89619 /* Interpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 1429D77B0ED20D7300B89619 /* Interpreter.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               1429D77C0ED20D7300B89619 /* Interpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 1429D77B0ED20D7300B89619 /* Interpreter.h */; };
                1429D7D40ED2128200B89619 /* Interpreter.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D7D30ED2128200B89619 /* Interpreter.cpp */; };
                1429D8780ED21ACD00B89619 /* ExceptionHelpers.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D8770ED21ACD00B89619 /* ExceptionHelpers.cpp */; };
                1429D8DD0ED2205B00B89619 /* CallFrame.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D8DB0ED2205B00B89619 /* CallFrame.cpp */; };
                969A07980ED1D3AE00F1F681 /* EvalCodeCache.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07920ED1D3AE00F1F681 /* EvalCodeCache.h */; settings = {ATTRIBUTES = (Private, ); }; };
                969A07990ED1D3AE00F1F681 /* Instruction.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07930ED1D3AE00F1F681 /* Instruction.h */; settings = {ATTRIBUTES = (Private, ); }; };
                969A079A0ED1D3AE00F1F681 /* Opcode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 969A07940ED1D3AE00F1F681 /* Opcode.cpp */; };
-               969A079B0ED1D3AE00F1F681 /* Opcode.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07950ED1D3AE00F1F681 /* Opcode.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               969A079B0ED1D3AE00F1F681 /* Opcode.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07950ED1D3AE00F1F681 /* Opcode.h */; };
                978801401471AD920041B016 /* JSDateMath.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9788FC221471AD0C0068CE2D /* JSDateMath.cpp */; };
                978801411471AD920041B016 /* JSDateMath.h in Headers */ = {isa = PBXBuildFile; fileRef = 9788FC231471AD0C0068CE2D /* JSDateMath.h */; settings = {ATTRIBUTES = (Private, ); }; };
                990DA67F1C8E316A00295159 /* generate_objc_protocol_type_conversions_implementation.py in Headers */ = {isa = PBXBuildFile; fileRef = 990DA67E1C8E311D00295159 /* generate_objc_protocol_type_conversions_implementation.py */; settings = {ATTRIBUTES = (Private, ); }; };
                0F04396B1B03DC0B009598B7 /* DFGCombinedLiveness.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGCombinedLiveness.cpp; path = dfg/DFGCombinedLiveness.cpp; sourceTree = "<group>"; };
                0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCombinedLiveness.h; path = dfg/DFGCombinedLiveness.h; sourceTree = "<group>"; };
                0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGArrayifySlowPathGenerator.h; path = dfg/DFGArrayifySlowPathGenerator.h; sourceTree = "<group>"; };
+               0F070A421D543A89006E7232 /* CellContainer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CellContainer.h; sourceTree = "<group>"; };
+               0F070A431D543A89006E7232 /* CellContainerInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CellContainerInlines.h; sourceTree = "<group>"; };
+               0F070A441D543A89006E7232 /* HeapCellInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapCellInlines.h; sourceTree = "<group>"; };
+               0F070A451D543A89006E7232 /* LargeAllocation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LargeAllocation.cpp; sourceTree = "<group>"; };
+               0F070A461D543A89006E7232 /* LargeAllocation.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LargeAllocation.h; sourceTree = "<group>"; };
                0F0776BD14FF002800102332 /* JITCompilationEffort.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITCompilationEffort.h; sourceTree = "<group>"; };
                0F0A75201B94BFA900110660 /* InferredType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InferredType.cpp; sourceTree = "<group>"; };
                0F0A75211B94BFA900110660 /* InferredType.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InferredType.h; sourceTree = "<group>"; };
                0F38B01417CFE75500B144D3 /* DFGCompilationKey.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCompilationKey.h; path = dfg/DFGCompilationKey.h; sourceTree = "<group>"; };
                0F38B01517CFE75500B144D3 /* DFGCompilationMode.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGCompilationMode.cpp; path = dfg/DFGCompilationMode.cpp; sourceTree = "<group>"; };
                0F38B01617CFE75500B144D3 /* DFGCompilationMode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCompilationMode.h; path = dfg/DFGCompilationMode.h; sourceTree = "<group>"; };
+               0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AuxiliaryBarrier.h; sourceTree = "<group>"; };
+               0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AuxiliaryBarrierInlines.h; sourceTree = "<group>"; };
                0F392C871B46188400844728 /* DFGOSRExitFuzz.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGOSRExitFuzz.cpp; path = dfg/DFGOSRExitFuzz.cpp; sourceTree = "<group>"; };
                0F392C881B46188400844728 /* DFGOSRExitFuzz.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGOSRExitFuzz.h; path = dfg/DFGOSRExitFuzz.h; sourceTree = "<group>"; };
                0F3A1BF71A9ECB7D000DE01A /* DFGPutStackSinkingPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGPutStackSinkingPhase.cpp; path = dfg/DFGPutStackSinkingPhase.cpp; sourceTree = "<group>"; };
                0F4F29DD18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGStaticExecutionCountEstimationPhase.cpp; path = dfg/DFGStaticExecutionCountEstimationPhase.cpp; sourceTree = "<group>"; };
                0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStaticExecutionCountEstimationPhase.h; path = dfg/DFGStaticExecutionCountEstimationPhase.h; sourceTree = "<group>"; };
                0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStructureClobberState.h; path = dfg/DFGStructureClobberState.h; sourceTree = "<group>"; };
+               0F5513A51D5A682A00C32BD8 /* FreeList.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FreeList.h; sourceTree = "<group>"; };
+               0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FreeList.cpp; sourceTree = "<group>"; };
                0F5541AF1613C1FB00CE3E25 /* SpecialPointer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SpecialPointer.cpp; sourceTree = "<group>"; };
                0F5541B01613C1FB00CE3E25 /* SpecialPointer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpecialPointer.h; sourceTree = "<group>"; };
                0F55989717C86C5600A1E543 /* ToNativeFromValue.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ToNativeFromValue.h; sourceTree = "<group>"; };
                0F6B8AE11C4EFE1700969052 /* B3FixSSA.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3FixSSA.h; path = b3/B3FixSSA.h; sourceTree = "<group>"; };
                0F6C734E1AC9F99F00BE1682 /* VariableWriteFireDetail.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = VariableWriteFireDetail.cpp; sourceTree = "<group>"; };
                0F6C734F1AC9F99F00BE1682 /* VariableWriteFireDetail.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VariableWriteFireDetail.h; sourceTree = "<group>"; };
+               0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StackFrame.cpp; sourceTree = "<group>"; };
+               0F6DB7E81D6124B200CDBF8E /* StackFrame.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StackFrame.h; sourceTree = "<group>"; };
+               0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MacroAssemblerCodeRef.cpp; sourceTree = "<group>"; };
                0F6E845919030BEF00562741 /* DFGVariableAccessData.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGVariableAccessData.cpp; path = dfg/DFGVariableAccessData.cpp; sourceTree = "<group>"; };
                0F6FC74E196110A800E1D02D /* ComplexGetStatus.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ComplexGetStatus.cpp; sourceTree = "<group>"; };
                0F6FC74F196110A800E1D02D /* ComplexGetStatus.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ComplexGetStatus.h; sourceTree = "<group>"; };
                0F8335B41639C1E3001443B5 /* ArrayAllocationProfile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ArrayAllocationProfile.cpp; sourceTree = "<group>"; };
                0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ArrayAllocationProfile.h; sourceTree = "<group>"; };
                0F8364B5164B0C0E0053329A /* DFGBranchDirection.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGBranchDirection.h; path = dfg/DFGBranchDirection.h; sourceTree = "<group>"; };
+               0F86A26C1D6F796200CB0C92 /* HeapOperation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = HeapOperation.cpp; sourceTree = "<group>"; };
+               0F86A26E1D6F7B3100CB0C92 /* GCTypeMap.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = GCTypeMap.h; sourceTree = "<group>"; };
                0F86AE1F1C5311C5006BE8EC /* B3ComputeDivisionMagic.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3ComputeDivisionMagic.h; path = b3/B3ComputeDivisionMagic.h; sourceTree = "<group>"; };
                0F885E101849A3BE00F1E3FA /* BytecodeUseDef.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BytecodeUseDef.h; sourceTree = "<group>"; };
                0F893BDA1936E23C001211F4 /* DFGStructureAbstractValue.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGStructureAbstractValue.cpp; path = dfg/DFGStructureAbstractValue.cpp; sourceTree = "<group>"; };
                0FA7A8E918B413C80052371D /* Reg.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Reg.cpp; sourceTree = "<group>"; };
                0FA7A8EA18B413C80052371D /* Reg.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Reg.h; sourceTree = "<group>"; };
                0FA7A8ED18CE4FD80052371D /* ScratchRegisterAllocator.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ScratchRegisterAllocator.cpp; sourceTree = "<group>"; };
+               0FADE6721D4D23BC00768457 /* HeapUtil.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapUtil.h; sourceTree = "<group>"; };
                0FAF7EFA165BA919000C8455 /* JITDisassembler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JITDisassembler.cpp; sourceTree = "<group>"; };
                0FAF7EFB165BA919000C8455 /* JITDisassembler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITDisassembler.h; sourceTree = "<group>"; };
                0FB105821675480C00F8AB6E /* ExitKind.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ExitKind.cpp; sourceTree = "<group>"; };
                0FB3878C1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = AirOptimizeBlockOrder.cpp; path = b3/air/AirOptimizeBlockOrder.cpp; sourceTree = "<group>"; };
                0FB3878D1BFBC44D00E3AB1E /* AirOptimizeBlockOrder.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = AirOptimizeBlockOrder.h; path = b3/air/AirOptimizeBlockOrder.h; sourceTree = "<group>"; };
                0FB387911BFD31A100E3AB1E /* FTLCompile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = FTLCompile.cpp; path = ftl/FTLCompile.cpp; sourceTree = "<group>"; };
+               0FB415831D78F98200DF8D09 /* ArrayConventions.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ArrayConventions.cpp; sourceTree = "<group>"; };
                0FB438A219270B1D00E1FBC9 /* StructureSet.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StructureSet.cpp; sourceTree = "<group>"; };
                0FB4B51016B3A964003F696B /* DFGMinifiedID.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGMinifiedID.h; path = dfg/DFGMinifiedID.h; sourceTree = "<group>"; };
                0FB4B51916B62772003F696B /* DFGAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGAllocator.h; path = dfg/DFGAllocator.h; sourceTree = "<group>"; };
                        children = (
                                0F9630351D4192C3005609D9 /* AllocatorAttributes.cpp */,
                                0F9630361D4192C3005609D9 /* AllocatorAttributes.h */,
+                               0F070A421D543A89006E7232 /* CellContainer.h */,
+                               0F070A431D543A89006E7232 /* CellContainerInlines.h */,
                                0F1C3DD91BBCE09E00E523E4 /* CellState.h */,
                                0FD8A31117D4326C00CA2C40 /* CodeBlockSet.cpp */,
                                0FD8A31217D4326C00CA2C40 /* CodeBlockSet.h */,
                                0F9630381D4192C3005609D9 /* DestructionMode.h */,
                                2A83638318D7D0EE0000EBCC /* EdenGCActivityCallback.cpp */,
                                2A83638418D7D0EE0000EBCC /* EdenGCActivityCallback.h */,
+                               0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */,
+                               0F5513A51D5A682A00C32BD8 /* FreeList.h */,
                                2A83638718D7D0FE0000EBCC /* FullGCActivityCallback.cpp */,
                                2A83638818D7D0FE0000EBCC /* FullGCActivityCallback.h */,
                                2AACE63A18CA5A0300ED0191 /* GCActivityCallback.cpp */,
                                2AABCDE618EF294200002096 /* GCLogging.h */,
                                2A343F7418A1748B0039B085 /* GCSegmentedArray.h */,
                                2A343F7718A1749D0039B085 /* GCSegmentedArrayInlines.h */,
+                               0F86A26E1D6F7B3100CB0C92 /* GCTypeMap.h */,
                                142E312B134FF0A600AFADB5 /* Handle.h */,
                                C28318FF16FE4B7D00157BFD /* HandleBlock.h */,
                                C283190116FE533E00157BFD /* HandleBlockInlines.h */,
                                14BA7A9613AADFF8005B7C2C /* Heap.h */,
                                DC3D2B0B1D34376E00BA918C /* HeapCell.cpp */,
                                DC3D2B091D34316100BA918C /* HeapCell.h */,
+                               0F070A441D543A89006E7232 /* HeapCellInlines.h */,
                                0F32BD0E1BB34F190093A57F /* HeapHelperPool.cpp */,
                                0F32BD0F1BB34F190093A57F /* HeapHelperPool.h */,
                                C2DA778218E259990066FCB6 /* HeapInlines.h */,
                                2AD8932917E3868F00668276 /* HeapIterationScope.h */,
                                A5339EC81BB4B4510054F005 /* HeapObserver.h */,
+                               0F86A26C1D6F796200CB0C92 /* HeapOperation.cpp */,
                                2A6F462517E959CE00C45C98 /* HeapOperation.h */,
                                A5398FA91C750D950060A963 /* HeapProfiler.cpp */,
                                A5398FAA1C750D950060A963 /* HeapProfiler.h */,
                                C24D31E1161CD695002AA4DB /* HeapStatistics.h */,
                                C2E526BB1590EF000054E48D /* HeapTimer.cpp */,
                                C2E526BC1590EF000054E48D /* HeapTimer.h */,
+                               0FADE6721D4D23BC00768457 /* HeapUtil.h */,
                                FE7BA60D1A1A7CEC00F1F7B4 /* HeapVerifier.cpp */,
                                FE7BA60E1A1A7CEC00F1F7B4 /* HeapVerifier.h */,
                                C25F8BCB157544A900245B71 /* IncrementalSweeper.cpp */,
                                C25F8BCC157544A900245B71 /* IncrementalSweeper.h */,
                                0F766D2915A8CC34008F363E /* JITStubRoutineSet.cpp */,
                                0F766D2A15A8CC34008F363E /* JITStubRoutineSet.h */,
+                               0F070A451D543A89006E7232 /* LargeAllocation.cpp */,
+                               0F070A461D543A89006E7232 /* LargeAllocation.h */,
                                0F431736146BAC65007E3890 /* ListableHandler.h */,
                                FE3913511B794AC900EDAF71 /* LiveObjectData.h */,
                                FE3913521B794AC900EDAF71 /* LiveObjectList.cpp */,
                                A7A8AF2817ADB5F3005AB174 /* ArrayBufferView.h */,
                                BC7952060E15E8A800A898AB /* ArrayConstructor.cpp */,
                                BC7952070E15E8A800A898AB /* ArrayConstructor.h */,
+                               0FB415831D78F98200DF8D09 /* ArrayConventions.cpp */,
                                0FB7F38915ED8E3800F167B2 /* ArrayConventions.h */,
                                A7BDAEC217F4EA1400F6140C /* ArrayIteratorPrototype.cpp */,
                                A7BDAEC317F4EA1400F6140C /* ArrayIteratorPrototype.h */,
                                F692A84D0255597D01FF60F7 /* ArrayPrototype.cpp */,
                                F692A84E0255597D01FF60F7 /* ArrayPrototype.h */,
                                0FB7F38A15ED8E3800F167B2 /* ArrayStorage.h */,
+                               0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */,
+                               0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */,
                                52678F8C1A031009006A306D /* BasicBlockLocation.cpp */,
                                52678F8D1A031009006A306D /* BasicBlockLocation.h */,
                                147B83AA0E6DB8C9004775A4 /* BatchedTransitionOptimizer.h */,
                                70DC3E081B2DF2C700054299 /* IteratorPrototype.h */,
                                93ADFCE60CCBD7AC00D30B08 /* JSArray.cpp */,
                                938772E5038BFE19008635CE /* JSArray.h */,
-                               539FB8B91C99DA7C00940FA1 /* JSArrayInlines.h */,
                                0F2B66B417B6B5AB00A7AE3F /* JSArrayBuffer.cpp */,
                                0F2B66B517B6B5AB00A7AE3F /* JSArrayBuffer.h */,
                                0F2B66B617B6B5AB00A7AE3F /* JSArrayBufferConstructor.cpp */,
                                0F2B66BA17B6B5AB00A7AE3F /* JSArrayBufferView.cpp */,
                                0F2B66BB17B6B5AB00A7AE3F /* JSArrayBufferView.h */,
                                0F2B66BC17B6B5AB00A7AE3F /* JSArrayBufferViewInlines.h */,
+                               539FB8B91C99DA7C00940FA1 /* JSArrayInlines.h */,
                                86FA9E8F142BBB2D001773B7 /* JSBoundFunction.cpp */,
                                86FA9E90142BBB2E001773B7 /* JSBoundFunction.h */,
                                657CF45619BF6662004ACBF2 /* JSCallee.cpp */,
                                A74DEF90182D991400522C22 /* JSMapIterator.h */,
                                E3D239C61B829C1C00BBEF67 /* JSModuleEnvironment.cpp */,
                                E3D239C71B829C1C00BBEF67 /* JSModuleEnvironment.h */,
+                               1879510614C540FFB561C124 /* JSModuleLoader.cpp */,
+                               77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */,
                                E318CBBE1B8AEF5100A2929D /* JSModuleNamespaceObject.cpp */,
                                E318CBBF1B8AEF5100A2929D /* JSModuleNamespaceObject.h */,
                                E39DA4A41B7E8B7C0084F33A /* JSModuleRecord.cpp */,
                                0F0CD4C315F6B6B50032F1C0 /* SparseArrayValueMap.cpp */,
                                0FB7F39215ED8E3800F167B2 /* SparseArrayValueMap.h */,
                                0F3AC751183EA1040032029F /* StackAlignment.h */,
+                               0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */,
+                               0F6DB7E81D6124B200CDBF8E /* StackFrame.h */,
                                A730B6111250068F009D25B1 /* StrictEvalActivation.cpp */,
                                A730B6101250068F009D25B1 /* StrictEvalActivation.h */,
                                BC18C3C00E16EE3300B34460 /* StringConstructor.cpp */,
                                709FB8661AE335C60039D069 /* WeakSetPrototype.h */,
                                A7DCB77912E3D90500911940 /* WriteBarrier.h */,
                                C2B6D75218A33793004A9301 /* WriteBarrierInlines.h */,
-                               77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */,
-                               1879510614C540FFB561C124 /* JSModuleLoader.cpp */,
                        );
                        path = runtime;
                        sourceTree = "<group>";
                                8640923C156EED3B00566CB2 /* MacroAssemblerARM64.h */,
                                A729009B17976C6000317298 /* MacroAssemblerARMv7.cpp */,
                                86ADD1440FDDEA980006EEC2 /* MacroAssemblerARMv7.h */,
+                               0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */,
                                863B23DF0FC60E6200703AA4 /* MacroAssemblerCodeRef.h */,
                                86C568DE11A213EE0007F7F0 /* MacroAssemblerMIPS.h */,
                                FE68C6351B90DDD90042BCB3 /* MacroAssemblerPrinter.cpp */,
                                99DA00A91BD5993100F4575C /* builtins_generate_separate_header.py in Headers */,
                                0F338E111BF0276C0013C88F /* B3OpaqueByproduct.h in Headers */,
                                FEA0C4031CDD7D1D00481991 /* FunctionWhitelist.h in Headers */,
+                               0F6DB7E91D6124B500CDBF8E /* StackFrame.h in Headers */,
                                E3A421431D6F58930007C617 /* PreciseJumpTargetsInlines.h in Headers */,
                                99DA00AA1BD5993100F4575C /* builtins_generate_separate_implementation.py in Headers */,
                                99DA00A31BD5993100F4575C /* builtins_generator.py in Headers */,
                                0F338E1C1BF286EA0013C88F /* B3BlockInsertionSet.h in Headers */,
                                0F9495881C57F47500413A48 /* B3StackSlot.h in Headers */,
                                C4F4B6F31A05C944005CAB76 /* cpp_generator_templates.py in Headers */,
+                               0F38D2A21D44196800680499 /* AuxiliaryBarrier.h in Headers */,
                                5DE6E5B30E1728EC00180407 /* create_hash_table in Headers */,
                                9959E92B1BD17FA4001AA413 /* cssmin.py in Headers */,
                                2A111246192FCE79005EE18D /* CustomGetterSetter.h in Headers */,
                                A7D9A29817A0BC7400EE2618 /* DFGLICMPhase.h in Headers */,
                                99D6A1161BEAD34D00E25C37 /* RemoteAutomationTarget.h in Headers */,
                                79C4B15E1BA2158F00FD592E /* DFGLiveCatchVariablePreservationPhase.h in Headers */,
+                               0F86A26F1D6F7B3300CB0C92 /* GCTypeMap.h in Headers */,
                                A7D89CFC17A0B8CC00773AD8 /* DFGLivenessAnalysisPhase.h in Headers */,
                                0FF0F19B16B729FA005DF95B /* DFGLongLivedState.h in Headers */,
                                0F338DF21BE93AD10013C88F /* B3StackmapValue.h in Headers */,
                                0FFFC96014EF90BD00C72532 /* DFGVirtualRegisterAllocationPhase.h in Headers */,
                                0FC97F4218202119002C9B26 /* DFGWatchpointCollectionPhase.h in Headers */,
                                0FDB2CE8174830A2007B3C1B /* DFGWorklist.h in Headers */,
+                               0F070A491D543A93006E7232 /* HeapCellInlines.h in Headers */,
                                0FE050181AA9091100D33B33 /* DirectArguments.h in Headers */,
                                0FE050161AA9091100D33B33 /* DirectArgumentsOffset.h in Headers */,
                                0FF42731158EBD54004CB9FF /* Disassembler.h in Headers */,
                                0FE0501A1AA9091100D33B33 /* GenericArgumentsInlines.h in Headers */,
                                FE3A06C01C11041A00390FDD /* JITRightShiftGenerator.h in Headers */,
                                708EBE241CE8F35800453146 /* IntlObjectInlines.h in Headers */,
+                               0F070A481D543A90006E7232 /* CellContainerInlines.h in Headers */,
                                FE6029D91D6E1E4F0030204D /* ThrowScopeLocation.h in Headers */,
                                0FE0501B1AA9091100D33B33 /* GenericOffset.h in Headers */,
                                0F2B66E017B6B5AB00A7AE3F /* GenericTypedArrayView.h in Headers */,
                                A1587D6E1B4DC14100D69849 /* IntlDateTimeFormat.h in Headers */,
                                FE187A0F1C030D6C0038BBCA /* SnippetOperand.h in Headers */,
                                A1587D701B4DC14100D69849 /* IntlDateTimeFormatConstructor.h in Headers */,
+                               0FADE6731D4D23BE00768457 /* HeapUtil.h in Headers */,
                                A1587D751B4DC1C600D69849 /* IntlDateTimeFormatConstructor.lut.h in Headers */,
                                A5398FAB1C750DA40060A963 /* HeapProfiler.h in Headers */,
                                A1587D721B4DC14100D69849 /* IntlDateTimeFormatPrototype.h in Headers */,
                                C25D709C16DE99F400FCA6BC /* JSManagedValue.h in Headers */,
                                2A4BB7F318A41179008A0FCD /* JSManagedValueInternal.h in Headers */,
                                A700874217CBE8EB00C3E643 /* JSMap.h in Headers */,
+                               0F38D2A31D44196D00680499 /* AuxiliaryBarrierInlines.h in Headers */,
                                A74DEF96182D991400522C22 /* JSMapIterator.h in Headers */,
                                9959E92D1BD17FA4001AA413 /* jsmin.py in Headers */,
                                E3D239C91B829C1C00BBEF67 /* JSModuleEnvironment.h in Headers */,
                                BC18C4270E16F5CD00B34460 /* JSString.h in Headers */,
                                86E85539111B9968001AF51E /* JSStringBuilder.h in Headers */,
                                70EC0EC31AA0D7DA00B6AAFA /* JSStringIterator.h in Headers */,
+                               0F070A471D543A8B006E7232 /* CellContainer.h in Headers */,
                                2600B5A7152BAAA70091EE5F /* JSStringJoiner.h in Headers */,
                                BC18C4280E16F5CD00B34460 /* JSStringRef.h in Headers */,
                                43AB26C61C1A535900D82AE6 /* B3MathExtras.h in Headers */,
                                14B723B812D7DA6F003BD5ED /* MachineStackMarker.h in Headers */,
                                86C36EEA0EE1289D00B3DF59 /* MacroAssembler.h in Headers */,
                                43422A671C16267800E2EB98 /* B3ReduceDoubleToFloat.h in Headers */,
+                               0F070A4B1D543A98006E7232 /* LargeAllocation.h in Headers */,
                                86D3B2C610156BDE002865E7 /* MacroAssemblerARM.h in Headers */,
                                A1A009C01831A22D00CF8711 /* MacroAssemblerARM64.h in Headers */,
                                86ADD1460FDDEA980006EEC2 /* MacroAssemblerARMv7.h in Headers */,
                                BC18C4440E16F5CD00B34460 /* NumberPrototype.h in Headers */,
                                996B73211BDA08EF00331B84 /* NumberPrototype.lut.h in Headers */,
                                142D3939103E4560007DCB52 /* NumericStrings.h in Headers */,
+                               0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */,
                                A5EA710C19F6DE820098F5EC /* objc_generator.py in Headers */,
                                C4F4B6F61A05C984005CAB76 /* objc_generator_templates.py in Headers */,
                                86F3EEBD168CDE930077B92A /* ObjCCallbackFunction.h in Headers */,
                                0FEC856F1BDACDC70080FF74 /* AirArg.cpp in Sources */,
                                0F4DE1CE1C4C1B54004D6C11 /* AirFixObviousSpills.cpp in Sources */,
                                0FEC85711BDACDC70080FF74 /* AirBasicBlock.cpp in Sources */,
+                               0F070A4A1D543A95006E7232 /* LargeAllocation.cpp in Sources */,
                                0FEC85731BDACDC70080FF74 /* AirCCallSpecial.cpp in Sources */,
                                0FEC85751BDACDC70080FF74 /* AirCode.cpp in Sources */,
                                0F4570381BE44C910062A629 /* AirEliminateDeadCode.cpp in Sources */,
                                A78A977A179738B8009DF744 /* DFGPlan.cpp in Sources */,
                                0FBE0F7416C1DB090082C5E8 /* DFGPredictionInjectionPhase.cpp in Sources */,
                                0FFFC95D14EF90B300C72532 /* DFGPredictionPropagationPhase.cpp in Sources */,
+                               0F86A26D1D6F796500CB0C92 /* HeapOperation.cpp in Sources */,
                                0F3E01AA19D353A500F61B7F /* DFGPrePostNumbering.cpp in Sources */,
                                0F2B9CEC19D0BA7D00B1D1B5 /* DFGPromotedHeapLocation.cpp in Sources */,
                                0FB17662196B8F9E0091052A /* DFGPureValue.cpp in Sources */,
                                0FE34C191C4B39AE0003A512 /* AirLogRegisterPressure.cpp in Sources */,
                                A1B9E2391B4E0D6700BC7FED /* IntlCollator.cpp in Sources */,
                                A1B9E23B1B4E0D6700BC7FED /* IntlCollatorConstructor.cpp in Sources */,
+                               0F6DB7EA1D6124B800CDBF8E /* StackFrame.cpp in Sources */,
                                A1B9E23D1B4E0D6700BC7FED /* IntlCollatorPrototype.cpp in Sources */,
                                A1587D6D1B4DC14100D69849 /* IntlDateTimeFormat.cpp in Sources */,
                                A1587D6F1B4DC14100D69849 /* IntlDateTimeFormatConstructor.cpp in Sources */,
                                146FE51211A710430087AE66 /* JITCall32_64.cpp in Sources */,
                                0F8F94441667635400D61971 /* JITCode.cpp in Sources */,
                                0FAF7EFD165BA91B000C8455 /* JITDisassembler.cpp in Sources */,
+                               0F6DB7EC1D617D1100CDBF8E /* MacroAssemblerCodeRef.cpp in Sources */,
                                0F46808314BA573100BFE272 /* JITExceptions.cpp in Sources */,
                                0FB14E1E18124ACE009B6B4D /* JITInlineCacheGenerator.cpp in Sources */,
                                BCDD51EB0FB8DF74004A8BDC /* JITOpcodes.cpp in Sources */,
                                A71236E51195F33C00BD2174 /* JITOpcodes32_64.cpp in Sources */,
                                0F24E54C17EE274900ABB217 /* JITOperations.cpp in Sources */,
+                               0F5513A81D5A68CD00C32BD8 /* FreeList.cpp in Sources */,
                                FE99B24A1C24C3D700C82159 /* JITNegGenerator.cpp in Sources */,
                                86CC85C40EE7A89400288682 /* JITPropertyAccess.cpp in Sources */,
                                A7C1E8E4112E72EF00A37F98 /* JITPropertyAccess32_64.cpp in Sources */,
+                               0FB415841D78FB4C00DF8D09 /* ArrayConventions.cpp in Sources */,
                                0F766D2815A8CC1E008F363E /* JITStubRoutine.cpp in Sources */,
                                0F766D2B15A8CC38008F363E /* JITStubRoutineSet.cpp in Sources */,
                                FE4238901BE18C3C00514737 /* JITSubGenerator.cpp in Sources */,
index 5331031..c60ee4d 100644 (file)
@@ -72,6 +72,9 @@ class BuiltinsCombinedImplementationGenerator(BuiltinsGenerator):
                 ("JavaScriptCore", "builtins/BuiltinExecutables.h"),
             ),
             (["JavaScriptCore", "WebCore"],
+                ("JavaScriptCore", "heap/HeapInlines.h"),
+            ),
+            (["JavaScriptCore", "WebCore"],
                 ("JavaScriptCore", "runtime/Executable.h"),
             ),
             (["JavaScriptCore", "WebCore"],
index 9f23f49..2597f28 100644 (file)
@@ -68,6 +68,9 @@ class BuiltinsInternalsWrapperImplementationGenerator(BuiltinsGenerator):
                 ("WebCore", "WebCoreJSClientData.h"),
             ),
             (["WebCore"],
+                ("JavaScriptCore", "heap/HeapInlines.h"),
+            ),
+            (["WebCore"],
                 ("JavaScriptCore", "heap/SlotVisitorInlines.h"),
             ),
             (["WebCore"],
index 7789fe3..5c7b554 100644 (file)
@@ -84,6 +84,9 @@ class BuiltinsSeparateImplementationGenerator(BuiltinsGenerator):
                 ("JavaScriptCore", "builtins/BuiltinExecutables.h"),
             ),
             (["JavaScriptCore", "WebCore"],
+                ("JavaScriptCore", "heap/HeapInlines.h"),
+            ),
+            (["JavaScriptCore", "WebCore"],
                 ("JavaScriptCore", "runtime/Executable.h"),
             ),
             (["JavaScriptCore", "WebCore"],
index 50bc264..d05c913 100644 (file)
@@ -725,20 +725,18 @@ public:
                 append(jump);
         }
 
-        void link(AbstractMacroAssemblerType* masm)
+        void link(AbstractMacroAssemblerType* masm) const
         {
             size_t size = m_jumps.size();
             for (size_t i = 0; i < size; ++i)
                 m_jumps[i].link(masm);
-            m_jumps.clear();
         }
         
-        void linkTo(Label label, AbstractMacroAssemblerType* masm)
+        void linkTo(Label label, AbstractMacroAssemblerType* masm) const
         {
             size_t size = m_jumps.size();
             for (size_t i = 0; i < size; ++i)
                 m_jumps[i].linkTo(label, masm);
-            m_jumps.clear();
         }
         
         void append(Jump jump)
index 3e2bb18..bfd483e 100644 (file)
@@ -28,6 +28,8 @@
 
 #if ENABLE(ASSEMBLER)
 
+#include "JSCJSValue.h"
+
 #if CPU(ARM_THUMB2)
 #include "MacroAssemblerARMv7.h"
 namespace JSC { typedef MacroAssemblerARMv7 MacroAssemblerBase; };
index d306f62..92ffdbc 100644 (file)
@@ -166,7 +166,10 @@ public:
             m_assembler.add<32>(dest, src, UInt12(imm.m_value));
         else if (isUInt12(-imm.m_value))
             m_assembler.sub<32>(dest, src, UInt12(-imm.m_value));
-        else {
+        else if (src != dest) {
+            move(imm, dest);
+            add32(src, dest);
+        } else {
             move(imm, getCachedDataTempRegisterIDAndInvalidate());
             m_assembler.add<32>(dest, src, dataTempRegister);
         }
diff --git a/Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.cpp b/Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.cpp
new file mode 100644 (file)
index 0000000..168b328
--- /dev/null
@@ -0,0 +1,68 @@
+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+ */
+
+#include "config.h"
+#include "MacroAssemblerCodeRef.h"
+
+#include "JSCInlines.h"
+#include "LLIntData.h"
+
+namespace JSC {
+
+MacroAssemblerCodePtr MacroAssemblerCodePtr::createLLIntCodePtr(OpcodeID codeId)
+{
+    return createFromExecutableAddress(LLInt::getCodePtr(codeId));
+}
+
+void MacroAssemblerCodePtr::dumpWithName(const char* name, PrintStream& out) const
+{
+    if (!m_value) {
+        out.print(name, "(null)");
+        return;
+    }
+    if (executableAddress() == dataLocation()) {
+        out.print(name, "(", RawPointer(executableAddress()), ")");
+        return;
+    }
+    out.print(name, "(executable = ", RawPointer(executableAddress()), ", dataLocation = ", RawPointer(dataLocation()), ")");
+}
+
+void MacroAssemblerCodePtr::dump(PrintStream& out) const
+{
+    dumpWithName("CodePtr", out);
+}
+
+MacroAssemblerCodeRef MacroAssemblerCodeRef::createLLIntCodeRef(OpcodeID codeId)
+{
+    return createSelfManagedCodeRef(MacroAssemblerCodePtr::createFromExecutableAddress(LLInt::getCodePtr(codeId)));
+}
+
+void MacroAssemblerCodeRef::dump(PrintStream& out) const
+{
+    m_codePtr.dumpWithName("CodeRef", out);
+}
+
+} // namespace JSC
+
index 94cabc7..f155f7d 100644 (file)
@@ -28,7 +28,6 @@
 
 #include "Disassembler.h"
 #include "ExecutableAllocator.h"
-#include "LLIntData.h"
 #include <wtf/DataLog.h>
 #include <wtf/PassRefPtr.h>
 #include <wtf/PrintStream.h>
@@ -53,6 +52,8 @@
 
 namespace JSC {
 
+enum OpcodeID : unsigned;
+
 // FunctionPtr:
 //
 // FunctionPtr should be used to wrap pointers to C/C++ functions in JSC
@@ -273,10 +274,7 @@ public:
         return result;
     }
 
-    static MacroAssemblerCodePtr createLLIntCodePtr(OpcodeID codeId)
-    {
-        return createFromExecutableAddress(LLInt::getCodePtr(codeId));
-    }
+    static MacroAssemblerCodePtr createLLIntCodePtr(OpcodeID codeId);
 
     explicit MacroAssemblerCodePtr(ReturnAddressPtr ra)
         : m_value(ra.value())
@@ -299,23 +297,9 @@ public:
         return m_value == other.m_value;
     }
 
-    void dumpWithName(const char* name, PrintStream& out) const
-    {
-        if (!m_value) {
-            out.print(name, "(null)");
-            return;
-        }
-        if (executableAddress() == dataLocation()) {
-            out.print(name, "(", RawPointer(executableAddress()), ")");
-            return;
-        }
-        out.print(name, "(executable = ", RawPointer(executableAddress()), ", dataLocation = ", RawPointer(dataLocation()), ")");
-    }
+    void dumpWithName(const char* name, PrintStream& out) const;
     
-    void dump(PrintStream& out) const
-    {
-        dumpWithName("CodePtr", out);
-    }
+    void dump(PrintStream& out) const;
     
     enum EmptyValueTag { EmptyValue };
     enum DeletedValueTag { DeletedValue };
@@ -389,10 +373,7 @@ public:
     }
     
     // Helper for creating self-managed code refs from LLInt.
-    static MacroAssemblerCodeRef createLLIntCodeRef(OpcodeID codeId)
-    {
-        return createSelfManagedCodeRef(MacroAssemblerCodePtr::createFromExecutableAddress(LLInt::getCodePtr(codeId)));
-    }
+    static MacroAssemblerCodeRef createLLIntCodeRef(OpcodeID codeId);
 
     ExecutableMemoryHandle* executableMemory() const
     {
@@ -418,10 +399,7 @@ public:
     
     explicit operator bool() const { return !!m_codePtr; }
     
-    void dump(PrintStream& out) const
-    {
-        m_codePtr.dumpWithName("CodeRef", out);
-    }
+    void dump(PrintStream& out) const;
 
 private:
     MacroAssemblerCodePtr m_codePtr;
index 0317c81..63a4e58 100644 (file)
@@ -85,6 +85,11 @@ Value* BasicBlock::appendIntConstant(Procedure& proc, Value* likeValue, int64_t
     return appendIntConstant(proc, likeValue->origin(), likeValue->type(), value);
 }
 
+Value* BasicBlock::appendBoolConstant(Procedure& proc, Origin origin, bool value)
+{
+    return appendIntConstant(proc, origin, Int32, value ? 1 : 0);
+}
+
 void BasicBlock::clearSuccessors()
 {
     m_successors.clear();
index 7d46bb9..9fb666a 100644 (file)
@@ -82,6 +82,7 @@ public:
 
     JS_EXPORT_PRIVATE Value* appendIntConstant(Procedure&, Origin, Type, int64_t value);
     Value* appendIntConstant(Procedure&, Value* likeValue, int64_t value);
+    Value* appendBoolConstant(Procedure&, Origin, bool);
 
     void removeLast(Procedure&);
     
index 95492bc..fe94a60 100644 (file)
@@ -71,7 +71,11 @@ public:
         IndexSet<BasicBlock> candidates;
 
         for (BasicBlock* block : m_proc) {
-            if (block->size() > m_maxSize || block->numSuccessors() > m_maxSuccessors)
+            if (block->size() > m_maxSize)
+                continue;
+            if (block->numSuccessors() > m_maxSuccessors)
+                continue;
+            if (block->last()->type() != Void) // Demoting doesn't handle terminals with values.
                 continue;
 
             candidates.add(block);
index af8c3fb..77e17b8 100644 (file)
@@ -92,7 +92,7 @@ public:
     // This is computed lazily, so it won't work if you capture StackmapGenerationParams by value.
     // Returns true if the successor at the given index is going to be emitted right after the
     // patchpoint.
-    bool fallsThroughToSuccessor(unsigned successorIndex) const;
+    JS_EXPORT_PRIVATE bool fallsThroughToSuccessor(unsigned successorIndex) const;
 
     // This is provided for convenience; it means that you don't have to capture it if you don't want to.
     JS_EXPORT_PRIVATE Procedure& proc() const;
index 6f134dc..e7b8ca8 100644 (file)
@@ -12920,6 +12920,80 @@ void testBranchBitAndImmFusion(
     CHECK(terminal.args[2].kind() == Air::Arg::BitImm || terminal.args[2].kind() == Air::Arg::BitImm64);
 }
 
+void testPatchpointTerminalReturnValue(bool successIsRare)
+{
+    // This is a unit test for how FTL's heap allocation fast paths behave.
+    Procedure proc;
+    
+    BasicBlock* root = proc.addBlock();
+    BasicBlock* success = proc.addBlock();
+    BasicBlock* slowPath = proc.addBlock();
+    BasicBlock* continuation = proc.addBlock();
+    
+    Value* arg = root->appendNew<Value>(
+        proc, Trunc, Origin(),
+        root->appendNew<ArgumentRegValue>(proc, Origin(), GPRInfo::argumentGPR0));
+    
+    PatchpointValue* patchpoint = root->appendNew<PatchpointValue>(proc, Int32, Origin());
+    patchpoint->effects.terminal = true;
+    patchpoint->clobber(RegisterSet::macroScratchRegisters());
+    
+    if (successIsRare) {
+        root->appendSuccessor(FrequentedBlock(success, FrequencyClass::Rare));
+        root->appendSuccessor(slowPath);
+    } else {
+        root->appendSuccessor(success);
+        root->appendSuccessor(FrequentedBlock(slowPath, FrequencyClass::Rare));
+    }
+    
+    patchpoint->appendSomeRegister(arg);
+    
+    patchpoint->setGenerator(
+        [&] (CCallHelpers& jit, const StackmapGenerationParams& params) {
+            AllowMacroScratchRegisterUsage allowScratch(jit);
+            
+            CCallHelpers::Jump jumpToSlow =
+                jit.branch32(CCallHelpers::Above, params[1].gpr(), CCallHelpers::TrustedImm32(42));
+            
+            jit.add32(CCallHelpers::TrustedImm32(31), params[1].gpr(), params[0].gpr());
+            
+            CCallHelpers::Jump jumpToSuccess;
+            if (!params.fallsThroughToSuccessor(0))
+                jumpToSuccess = jit.jump();
+            
+            Vector<Box<CCallHelpers::Label>> labels = params.successorLabels();
+            
+            params.addLatePath(
+                [=] (CCallHelpers& jit) {
+                    jumpToSlow.linkTo(*labels[1], &jit);
+                    if (jumpToSuccess.isSet())
+                        jumpToSuccess.linkTo(*labels[0], &jit);
+                });
+        });
+    
+    UpsilonValue* successUpsilon = success->appendNew<UpsilonValue>(proc, Origin(), patchpoint);
+    success->appendNew<Value>(proc, Jump, Origin());
+    success->setSuccessors(continuation);
+    
+    UpsilonValue* slowPathUpsilon = slowPath->appendNew<UpsilonValue>(
+        proc, Origin(), slowPath->appendNew<Const32Value>(proc, Origin(), 666));
+    slowPath->appendNew<Value>(proc, Jump, Origin());
+    slowPath->setSuccessors(continuation);
+    
+    Value* phi = continuation->appendNew<Value>(proc, Phi, Int32, Origin());
+    successUpsilon->setPhi(phi);
+    slowPathUpsilon->setPhi(phi);
+    continuation->appendNew<Value>(proc, Return, Origin(), phi);
+    
+    auto code = compile(proc);
+    CHECK_EQ(invoke<int>(*code, 0), 31);
+    CHECK_EQ(invoke<int>(*code, 1), 32);
+    CHECK_EQ(invoke<int>(*code, 41), 72);
+    CHECK_EQ(invoke<int>(*code, 42), 73);
+    CHECK_EQ(invoke<int>(*code, 43), 666);
+    CHECK_EQ(invoke<int>(*code, -1), 666);
+}
+
 // Make sure the compiler does not try to optimize anything out.
 NEVER_INLINE double zero()
 {
@@ -14337,6 +14411,8 @@ void run(const char* filter)
     RUN(testEntrySwitchLoop());
 
     RUN(testSomeEarlyRegister());
+    RUN(testPatchpointTerminalReturnValue(true));
+    RUN(testPatchpointTerminalReturnValue(false));
     
     if (isX86()) {
         RUN(testBranchBitAndImmFusion(Identity, Int64, 1, Air::BranchTest32, Air::Arg::Tmp));
index 7da8673..7f6ea16 100644 (file)
@@ -32,9 +32,8 @@
 
 #include "APICast.h"
 #include "InspectorValues.h"
+#include "JSCInlines.h"
 #include "JSLock.h"
-#include "JSObjectInlines.h"
-#include "StructureInlines.h"
 
 using namespace JSC;
 using namespace Inspector;
index 04d98f6..2fd7031 100644 (file)
@@ -26,8 +26,7 @@
 #include "config.h"
 #include "AdaptiveInferredPropertyValueWatchpointBase.h"
 
-#include "JSCellInlines.h"
-#include "StructureInlines.h"
+#include "JSCInlines.h"
 
 namespace JSC {
 
index b13e80b..d7868a8 100644 (file)
@@ -32,6 +32,7 @@
 #include "CodeBlock.h"
 #include "FullBytecodeLiveness.h"
 #include "InterpreterInlines.h"
+#include "PreciseJumpTargets.h"
 
 namespace JSC {
 
index 28dc20b..6dadb6e 100644 (file)
@@ -27,6 +27,7 @@
 #include "config.h"
 #include "BytecodeRewriter.h"
 
+#include "HeapInlines.h"
 #include "PreciseJumpTargetsInlines.h"
 #include <wtf/BubbleSort.h>
 
index 4f14b23..0eb5ec6 100644 (file)
@@ -27,6 +27,7 @@
 #define BytecodeUseDef_h
 
 #include "CodeBlock.h"
+#include "Interpreter.h"
 
 namespace JSC {
 
index c004d8a..67248fe 100644 (file)
 #include "DFGOperations.h"
 #include "DFGThunks.h"
 #include "JSCInlines.h"
+#include "Opcode.h"
 #include "Repatch.h"
 #include <wtf/ListDump.h>
 
 #if ENABLE(JIT)
 namespace JSC {
 
+CallLinkInfo::CallType CallLinkInfo::callTypeFor(OpcodeID opcodeID)
+{
+    if (opcodeID == op_call || opcodeID == op_call_eval)
+        return Call;
+    if (opcodeID == op_call_varargs)
+        return CallVarargs;
+    if (opcodeID == op_construct)
+        return Construct;
+    if (opcodeID == op_construct_varargs)
+        return ConstructVarargs;
+    if (opcodeID == op_tail_call)
+        return TailCall;
+    ASSERT(opcodeID == op_tail_call_varargs || op_tail_call_forward_arguments);
+    return TailCallVarargs;
+}
+
 CallLinkInfo::CallLinkInfo()
     : m_hasSeenShouldRepatch(false)
     , m_hasSeenClosure(false)
index f7e6e73..d5402eb 100644 (file)
@@ -31,7 +31,6 @@
 #include "CodeSpecializationKind.h"
 #include "JITWriteBarrier.h"
 #include "JSFunction.h"
-#include "Opcode.h"
 #include "PolymorphicCallStubRoutine.h"
 #include "WriteBarrier.h"
 #include <wtf/SentinelLinkedList.h>
@@ -40,26 +39,13 @@ namespace JSC {
 
 #if ENABLE(JIT)
 
+enum OpcodeID : unsigned;
 struct CallFrameShuffleData;
 
 class CallLinkInfo : public BasicRawSentinelNode<CallLinkInfo> {
 public:
     enum CallType { None, Call, CallVarargs, Construct, ConstructVarargs, TailCall, TailCallVarargs };
-    static CallType callTypeFor(OpcodeID opcodeID)
-    {
-        if (opcodeID == op_call || opcodeID == op_call_eval)
-            return Call;
-        if (opcodeID == op_call_varargs)
-            return CallVarargs;
-        if (opcodeID == op_construct)
-            return Construct;
-        if (opcodeID == op_construct_varargs)
-            return ConstructVarargs;
-        if (opcodeID == op_tail_call)
-            return TailCall;
-        ASSERT(opcodeID == op_tail_call_varargs || op_tail_call_forward_arguments);
-        return TailCallVarargs;
-    }
+    static CallType callTypeFor(OpcodeID opcodeID);
 
     static bool isVarargsCallType(CallType callType)
     {
index 7ce79ab..d909508 100644 (file)
@@ -30,6 +30,7 @@
 #include "CodeBlock.h"
 #include "DFGJITCode.h"
 #include "InlineCallFrame.h"
+#include "Interpreter.h"
 #include "LLIntCallLinkInfo.h"
 #include "JSCInlines.h"
 #include <wtf/CommaPrinter.h>
index d23a234..043c946 100644 (file)
@@ -52,6 +52,7 @@
 #include "JSFunction.h"
 #include "JSLexicalEnvironment.h"
 #include "JSModuleEnvironment.h"
+#include "LLIntData.h"
 #include "LLIntEntrypoint.h"
 #include "LLIntPrototypeLoadAdaptiveStructureWatchpoint.h"
 #include "LowLevelInterpreter.h"
@@ -70,6 +71,7 @@
 #include "VMInlines.h"
 #include <wtf/BagToHashMap.h>
 #include <wtf/CommaPrinter.h>
+#include <wtf/SimpleStats.h>
 #include <wtf/StringExtras.h>
 #include <wtf/StringPrintStream.h>
 #include <wtf/text/UniquedStringImpl.h>
@@ -1876,7 +1878,7 @@ void CodeBlock::finishCreation(VM& vm, CopyParsedBlockTag, CodeBlock& other)
         m_rareData->m_stringSwitchJumpTables = other.m_rareData->m_stringSwitchJumpTables;
     }
     
-    heap()->m_codeBlocks.add(this);
+    heap()->m_codeBlocks->add(this);
 }
 
 CodeBlock::CodeBlock(VM* vm, Structure* structure, ScriptExecutable* ownerExecutable, UnlinkedCodeBlock* unlinkedCodeBlock,
@@ -2342,7 +2344,7 @@ void CodeBlock::finishCreation(VM& vm, ScriptExecutable* ownerExecutable, Unlink
     if (Options::dumpGeneratedBytecodes())
         dumpBytecode();
     
-    heap()->m_codeBlocks.add(this);
+    heap()->m_codeBlocks->add(this);
     heap()->reportExtraMemoryAllocated(m_instructions.size() * sizeof(Instruction));
 }
 
@@ -2379,7 +2381,7 @@ void CodeBlock::finishCreation(VM& vm, WebAssemblyExecutable*, JSGlobalObject*)
 {
     Base::finishCreation(vm);
 
-    heap()->m_codeBlocks.add(this);
+    heap()->m_codeBlocks->add(this);
 }
 #endif
 
@@ -2783,6 +2785,14 @@ void CodeBlock::WeakReferenceHarvester::visitWeakReferences(SlotVisitor& visitor
     codeBlock->determineLiveness(visitor);
 }
 
+void CodeBlock::clearLLIntGetByIdCache(Instruction* instruction)
+{
+    instruction[0].u.opcode = LLInt::getOpcode(op_get_by_id);
+    instruction[4].u.pointer = nullptr;
+    instruction[5].u.pointer = nullptr;
+    instruction[6].u.pointer = nullptr;
+}
+
 void CodeBlock::finalizeLLIntInlineCaches()
 {
 #if ENABLE(WEBASSEMBLY)
@@ -4184,12 +4194,12 @@ size_t CodeBlock::predictedMachineCodeSize()
     if (!m_vm)
         return 0;
     
-    if (!m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT)
+    if (!*m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT)
         return 0; // It's as good of a prediction as we'll get.
     
     // Be conservative: return a size that will be an overestimation 84% of the time.
-    double multiplier = m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT.mean() +
-        m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT.standardDeviation();
+    double multiplier = m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT->mean() +
+        m_vm->machineCodeBytesPerBytecodeWordForBaselineJIT->standardDeviation();
     
     // Be paranoid: silently reject bogus multipiers. Silently doing the "wrong" thing
     // here is OK, since this whole method is just a heuristic.
index 9e5d607..e3431a3 100644 (file)
@@ -293,6 +293,8 @@ public:
     {
         return m_jitCodeMap.get();
     }
+    
+    static void clearLLIntGetByIdCache(Instruction*);
 
     unsigned bytecodeOffset(Instruction* returnAddress)
     {
@@ -1283,14 +1285,6 @@ private:
 };
 #endif
 
-inline void clearLLIntGetByIdCache(Instruction* instruction)
-{
-    instruction[0].u.opcode = LLInt::getOpcode(op_get_by_id);
-    instruction[4].u.pointer = nullptr;
-    instruction[5].u.pointer = nullptr;
-    instruction[6].u.pointer = nullptr;
-}
-
 inline Register& ExecState::r(int index)
 {
     CodeBlock* codeBlock = this->codeBlock();
index 5002c6c..2ab450a 100644 (file)
@@ -29,7 +29,6 @@
 #include "JSGlobalObject.h"
 #include "Options.h"
 #include <wtf/PrintStream.h>
-#include <wtf/SimpleStats.h>
 
 namespace JSC {
 
index 494b000..4f74320 100644 (file)
@@ -31,7 +31,6 @@
 
 #include "BasicBlockLocation.h"
 #include "MacroAssembler.h"
-#include "Opcode.h"
 #include "PutByIdFlags.h"
 #include "SymbolTable.h"
 #include "TypeLocation.h"
@@ -52,6 +51,12 @@ class WatchpointSet;
 struct LLIntCallLinkInfo;
 struct ValueProfile;
 
+#if ENABLE(COMPUTED_GOTO_OPCODES)
+typedef void* Opcode;
+#else
+typedef OpcodeID Opcode;
+#endif
+
 struct Instruction {
     Instruction()
     {
index 7ae2c0d..9a5ac01 100644 (file)
@@ -28,7 +28,7 @@
 
 #include "CodeBlock.h"
 #include "Instruction.h"
-#include "StructureInlines.h"
+#include "JSCInlines.h"
 
 namespace JSC {
 
@@ -59,7 +59,7 @@ void LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal(const FireDetai
 
     StringFireDetail stringDetail(out.toCString().data());
 
-    clearLLIntGetByIdCache(m_getByIdInstruction);
+    CodeBlock::clearLLIntGetByIdCache(m_getByIdInstruction);
 }
 
 } // namespace JSC
index 5fa706d..2beeaa7 100644 (file)
@@ -45,7 +45,7 @@ public:
     {
     }
 
-    bool isNull() { return !m_allocator; }
+    bool isNull() { return !m_structure; }
 
     void initialize(VM& vm, JSCell* owner, JSObject* prototype, unsigned inferredInlineCapacity)
     {
@@ -80,14 +80,15 @@ public:
         ASSERT(inlineCapacity <= JSFinalObject::maxInlineCapacity());
 
         size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
-        MarkedAllocator* allocator = &vm.heap.allocatorForObjectWithoutDestructor(allocationSize);
-        ASSERT(allocator->cellSize());
-
+        MarkedAllocator* allocator = vm.heap.allocatorForObjectWithoutDestructor(allocationSize);
+        
         // Take advantage of extra inline capacity available in the size class.
-        size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
-        inlineCapacity += slop;
-        if (inlineCapacity > JSFinalObject::maxInlineCapacity())
-            inlineCapacity = JSFinalObject::maxInlineCapacity();
+        if (allocator) {
+            size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
+            inlineCapacity += slop;
+            if (inlineCapacity > JSFinalObject::maxInlineCapacity())
+                inlineCapacity = JSFinalObject::maxInlineCapacity();
+        }
 
         Structure* structure = vm.prototypeMap.emptyObjectStructureForPrototype(prototype, inlineCapacity);
 
index 4db610e..545d8e9 100644 (file)
@@ -55,7 +55,7 @@ namespace JSC {
 
 
 #define OPCODE_ID_ENUM(opcode, length) opcode,
-    typedef enum { FOR_EACH_OPCODE_ID(OPCODE_ID_ENUM) } OpcodeID;
+    enum OpcodeID : unsigned { FOR_EACH_OPCODE_ID(OPCODE_ID_ENUM) };
 #undef OPCODE_ID_ENUM
 
 const int maxOpcodeLength = 9;
index 5f057ee..5caca06 100644 (file)
@@ -1206,34 +1206,25 @@ void AccessCase::generateImpl(AccessGenerationState& state)
             size_t newSize = newStructure()->outOfLineCapacity() * sizeof(JSValue);
             
             if (allocatingInline) {
-                CopiedAllocator* copiedAllocator = &vm.heap.storageAllocator();
-
-                if (!reallocating) {
-                    jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
-                    slowPath.append(
-                        jit.branchSubPtr(
-                            CCallHelpers::Signed, CCallHelpers::TrustedImm32(newSize), scratchGPR));
-                    jit.storePtr(scratchGPR, &copiedAllocator->m_currentRemaining);
-                    jit.negPtr(scratchGPR);
-                    jit.addPtr(
-                        CCallHelpers::AbsoluteAddress(&copiedAllocator->m_currentPayloadEnd), scratchGPR);
-                    jit.addPtr(CCallHelpers::TrustedImm32(sizeof(JSValue)), scratchGPR);
-                } else {
+                MarkedAllocator* allocator = vm.heap.allocatorForAuxiliaryData(newSize);
+                
+                if (!allocator) {
+                    // Yuck, this case would suck!
+                    slowPath.append(jit.jump());
+                }
+                
+                jit.move(CCallHelpers::TrustedImmPtr(allocator), scratchGPR2);
+                jit.emitAllocate(scratchGPR, allocator, scratchGPR2, scratchGPR3, slowPath);
+                jit.addPtr(CCallHelpers::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR);
+                
+                if (reallocating) {
                     // Handle the case where we are reallocating (i.e. the old structure/butterfly
                     // already had out-of-line property storage).
                     size_t oldSize = structure()->outOfLineCapacity() * sizeof(JSValue);
                     ASSERT(newSize > oldSize);
             
                     jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR3);
-                    jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
-                    slowPath.append(
-                        jit.branchSubPtr(
-                            CCallHelpers::Signed, CCallHelpers::TrustedImm32(newSize), scratchGPR));
-                    jit.storePtr(scratchGPR, &copiedAllocator->m_currentRemaining);
-                    jit.negPtr(scratchGPR);
-                    jit.addPtr(
-                        CCallHelpers::AbsoluteAddress(&copiedAllocator->m_currentPayloadEnd), scratchGPR);
-                    jit.addPtr(CCallHelpers::TrustedImm32(sizeof(JSValue)), scratchGPR);
+                    
                     // We have scratchGPR = new storage, scratchGPR3 = old storage,
                     // scratchGPR2 = available
                     for (size_t offset = 0; offset < oldSize; offset += sizeof(void*)) {
@@ -1659,6 +1650,7 @@ AccessGenerationResult PolymorphicAccess::regenerate(
         // Cascade through the list, preferring newer entries.
         for (unsigned i = cases.size(); i--;) {
             fallThrough.link(&jit);
+            fallThrough.clear();
             cases[i]->generateWithGuard(state, fallThrough);
         }
         state.failAndRepatch.append(fallThrough);
index b602c00..2ca4adc 100644 (file)
 #if ENABLE(JIT)
 
 #include "CodeOrigin.h"
+#include "JITStubRoutine.h"
 #include "JSFunctionInlines.h"
 #include "MacroAssembler.h"
 #include "ObjectPropertyConditionSet.h"
-#include "Opcode.h"
 #include "ScratchRegisterAllocator.h"
 #include "Structure.h"
 #include <wtf/Vector.h>
index 30dff72..38e5cef 100644 (file)
@@ -27,6 +27,7 @@
 #include "StructureStubInfo.h"
 
 #include "JSObject.h"
+#include "JSCInlines.h"
 #include "PolymorphicAccess.h"
 #include "Repatch.h"
 
index 0138466..3dcc51a 100644 (file)
@@ -31,7 +31,6 @@
 #include "JITStubRoutine.h"
 #include "MacroAssembler.h"
 #include "ObjectPropertyConditionSet.h"
-#include "Opcode.h"
 #include "Options.h"
 #include "RegisterSet.h"
 #include "Structure.h"
index 919259a..81e51d4 100644 (file)
@@ -36,6 +36,7 @@ namespace JSC {
 
 volatile uint32_t g_superSamplerCount;
 
+static StaticLock lock;
 static double in;
 static double out;
 
@@ -51,10 +52,13 @@ void initializeSuperSampler()
             const int printingPeriod = 1000;
             for (;;) {
                 for (int ms = 0; ms < printingPeriod; ms += sleepQuantum) {
-                    if (g_superSamplerCount)
-                        in++;
-                    else
-                        out++;
+                    {
+                        LockHolder locker(lock);
+                        if (g_superSamplerCount)
+                            in++;
+                        else
+                            out++;
+                    }
                     sleepMS(sleepQuantum);
                 }
                 printSuperSamplerState();
@@ -64,11 +68,19 @@ void initializeSuperSampler()
         });
 }
 
+void resetSuperSamplerState()
+{
+    LockHolder locker(lock);
+    in = 0;
+    out = 0;
+}
+
 void printSuperSamplerState()
 {
     if (!Options::useSuperSampler())
         return;
 
+    LockHolder locker(lock);
     double percentage = 100.0 * in / (in + out);
     if (percentage != percentage)
         percentage = 0.0;
index 85e6ce7..22780ea 100644 (file)
@@ -53,6 +53,7 @@ private:
     bool m_doSample;
 };
 
+JS_EXPORT_PRIVATE void resetSuperSamplerState();
 JS_EXPORT_PRIVATE void printSuperSamplerState();
 
 } // namespace JSC
index a6e0e82..1486d2a 100644 (file)
@@ -90,11 +90,6 @@ UnlinkedCodeBlock::UnlinkedCodeBlock(VM* vm, Structure* structure, CodeType code
     ASSERT(m_constructorKind == static_cast<unsigned>(info.constructorKind()));
 }
 
-VM* UnlinkedCodeBlock::vm() const
-{
-    return MarkedBlock::blockFor(this)->vm();
-}
-
 void UnlinkedCodeBlock::visitChildren(JSCell* cell, SlotVisitor& visitor)
 {
     UnlinkedCodeBlock* thisObject = jsCast<UnlinkedCodeBlock*>(cell);
index 72d1696..9487ba6 100644 (file)
@@ -284,8 +284,6 @@ public:
     void addExceptionHandler(const UnlinkedHandlerInfo& handler) { createRareDataIfNecessary(); return m_rareData->m_exceptionHandlers.append(handler); }
     UnlinkedHandlerInfo& exceptionHandler(int index) { ASSERT(m_rareData); return m_rareData->m_exceptionHandlers[index]; }
 
-    VM* vm() const;
-
     UnlinkedArrayProfile addArrayProfile() { return m_arrayProfileCount++; }
     unsigned numberOfArrayProfiles() { return m_arrayProfileCount; }
     UnlinkedArrayAllocationProfile addArrayAllocationProfile() { return m_arrayAllocationProfileCount++; }
index 6a300d3..e8762ff 100644 (file)
@@ -26,6 +26,8 @@
 #include "config.h"
 #include "UnlinkedInstructionStream.h"
 
+#include "Opcode.h"
+
 namespace JSC {
 
 static void append8(unsigned char*& ptr, unsigned char value)
index a875e49..07dc678 100644 (file)
@@ -27,6 +27,7 @@
 #ifndef UnlinkedInstructionStream_h
 #define UnlinkedInstructionStream_h
 
+#include "Opcode.h"
 #include "UnlinkedCodeBlock.h"
 #include <wtf/RefCountedArray.h>
 
index 01d834c..b537a83 100644 (file)
@@ -38,7 +38,7 @@ namespace JSC { namespace DFG {
 class CallArrayAllocatorSlowPathGenerator : public JumpingSlowPathGenerator<MacroAssembler::JumpList> {
 public:
     CallArrayAllocatorSlowPathGenerator(
-        MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZ function,
+        MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZB function,
         GPRReg resultGPR, GPRReg storageGPR, Structure* structure, size_t size)
         : JumpingSlowPathGenerator<MacroAssembler::JumpList>(from, jit)
         , m_function(function)
@@ -57,7 +57,7 @@ protected:
         linkFrom(jit);
         for (unsigned i = 0; i < m_plans.size(); ++i)
             jit->silentSpill(m_plans[i]);
-        jit->callOperation(m_function, m_resultGPR, m_structure, m_size);
+        jit->callOperation(m_function, m_resultGPR, m_structure, m_size, m_storageGPR);
         GPRReg canTrample = SpeculativeJIT::pickCanTrample(m_resultGPR);
         for (unsigned i = m_plans.size(); i--;)
             jit->silentFill(m_plans[i], canTrample);
@@ -67,7 +67,7 @@ protected:
     }
     
 private:
-    P_JITOperation_EStZ m_function;
+    P_JITOperation_EStZB m_function;
     GPRReg m_resultGPR;
     GPRReg m_storageGPR;
     Structure* m_structure;
@@ -78,14 +78,15 @@ private:
 class CallArrayAllocatorWithVariableSizeSlowPathGenerator : public JumpingSlowPathGenerator<MacroAssembler::JumpList> {
 public:
     CallArrayAllocatorWithVariableSizeSlowPathGenerator(
-        MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZ function,
-        GPRReg resultGPR, Structure* contiguousStructure, Structure* arrayStorageStructure, GPRReg sizeGPR)
+        MacroAssembler::JumpList from, SpeculativeJIT* jit, P_JITOperation_EStZB function,
+        GPRReg resultGPR, Structure* contiguousStructure, Structure* arrayStorageStructure, GPRReg sizeGPR, GPRReg storageGPR)
         : JumpingSlowPathGenerator<MacroAssembler::JumpList>(from, jit)
         , m_function(function)
         , m_resultGPR(resultGPR)
         , m_contiguousStructure(contiguousStructure)
         , m_arrayStorageOrContiguousStructure(arrayStorageStructure)
         , m_sizeGPR(sizeGPR)
+        , m_storageGPR(storageGPR)
     {
         jit->silentSpillAllRegistersImpl(false, m_plans, resultGPR);
     }
@@ -96,7 +97,7 @@ protected:
         linkFrom(jit);
         for (unsigned i = 0; i < m_plans.size(); ++i)
             jit->silentSpill(m_plans[i]);
-        GPRReg scratchGPR = AssemblyHelpers::selectScratchGPR(m_sizeGPR);
+        GPRReg scratchGPR = AssemblyHelpers::selectScratchGPR(m_sizeGPR, m_storageGPR);
         if (m_contiguousStructure != m_arrayStorageOrContiguousStructure) {
             MacroAssembler::Jump bigLength = jit->m_jit.branch32(MacroAssembler::AboveOrEqual, m_sizeGPR, MacroAssembler::TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH));
             jit->m_jit.move(MacroAssembler::TrustedImmPtr(m_contiguousStructure), scratchGPR);
@@ -106,7 +107,7 @@ protected:
             done.link(&jit->m_jit);
         } else
             jit->m_jit.move(MacroAssembler::TrustedImmPtr(m_contiguousStructure), scratchGPR);
-        jit->callOperation(m_function, m_resultGPR, scratchGPR, m_sizeGPR);
+        jit->callOperation(m_function, m_resultGPR, scratchGPR, m_sizeGPR, m_storageGPR);
         GPRReg canTrample = SpeculativeJIT::pickCanTrample(m_resultGPR);
         for (unsigned i = m_plans.size(); i--;)
             jit->silentFill(m_plans[i], canTrample);
@@ -115,11 +116,12 @@ protected:
     }
     
 private:
-    P_JITOperation_EStZ m_function;
+    P_JITOperation_EStZB m_function;
     GPRReg m_resultGPR;
     Structure* m_contiguousStructure;
     Structure* m_arrayStorageOrContiguousStructure;
     GPRReg m_sizeGPR;
+    GPRReg m_storageGPR;
     Vector<SilentRegisterSavePlan, 2> m_plans;
 };
 
index bfa8353..b387aa6 100644 (file)
@@ -933,17 +933,20 @@ char* JIT_OPERATION operationNewEmptyArray(ExecState* exec, Structure* arrayStru
     return bitwise_cast<char*>(JSArray::create(*vm, arrayStructure));
 }
 
-char* JIT_OPERATION operationNewArrayWithSize(ExecState* exec, Structure* arrayStructure, int32_t size)
+char* JIT_OPERATION operationNewArrayWithSize(ExecState* exec, Structure* arrayStructure, int32_t size, Butterfly* butterfly)
 {
-    VM* vm = &exec->vm();
-    NativeCallFrameTracer tracer(vm, exec);
-    auto scope = DECLARE_THROW_SCOPE(*vm);
+    VM& vm = exec->vm();
+    NativeCallFrameTracer tracer(&vm, exec);
+    auto scope = DECLARE_THROW_SCOPE(vm);
 
     if (UNLIKELY(size < 0))
         return bitwise_cast<char*>(throwException(exec, scope, createRangeError(exec, ASCIILiteral("Array size is not a small enough positive integer."))));
 
-    JSArray* result = JSArray::create(*vm, arrayStructure, size);
-    result->butterfly(); // Ensure that the backing store is in to-space.
+    JSArray* result;
+    if (butterfly)
+        result = JSArray::createWithButterfly(vm, arrayStructure, butterfly);
+    else
+        result = JSArray::create(vm, arrayStructure, size);
     return bitwise_cast<char*>(result);
 }
 
@@ -1629,13 +1632,13 @@ size_t JIT_OPERATION operationDefaultHasInstance(ExecState* exec, JSCell* value,
     return 0;
 }
 
-char* JIT_OPERATION operationNewRawObject(ExecState* exec, Structure* structure, int32_t length)
+char* JIT_OPERATION operationNewRawObject(ExecState* exec, Structure* structure, int32_t length, Butterfly* butterfly)
 {
     VM& vm = exec->vm();
     NativeCallFrameTracer tracer(&vm, exec);
 
-    Butterfly* butterfly;
-    if (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType())) {
+    if (!butterfly
+        && (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType()))) {
         IndexingHeader header;
         header.setVectorLength(length);
         header.setPublicLength(0);
@@ -1644,28 +1647,29 @@ char* JIT_OPERATION operationNewRawObject(ExecState* exec, Structure* structure,
             vm, nullptr, 0, structure->outOfLineCapacity(),
             hasIndexedProperties(structure->indexingType()), header,
             length * sizeof(EncodedJSValue));
-    } else
-        butterfly = nullptr;
+    }
 
     JSObject* result = JSObject::createRawObject(exec, structure, butterfly);
     result->butterfly(); // Ensure that the butterfly is in to-space.
     return bitwise_cast<char*>(result);
 }
 
-JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState* exec, Structure* structure)
+JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState* exec, Structure* structure, Butterfly* butterfly)
 {
     VM& vm = exec->vm();
     NativeCallFrameTracer tracer(&vm, exec);
     
-    Butterfly* butterfly = Butterfly::create(
-        vm, nullptr, 0, structure->outOfLineCapacity(), false, IndexingHeader(), 0);
+    if (!butterfly) {
+        butterfly = Butterfly::create(
+            vm, nullptr, 0, structure->outOfLineCapacity(), false, IndexingHeader(), 0);
+    }
     
     JSObject* result = JSObject::createRawObject(exec, structure, butterfly);
     result->butterfly(); // Ensure that the butterfly is in to-space.
     return result;
 }
 
-JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState* exec, Structure* structure, unsigned length)
+JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState* exec, Structure* structure, unsigned length, Butterfly* butterfly)
 {
     VM& vm = exec->vm();
     NativeCallFrameTracer tracer(&vm, exec);
@@ -1673,10 +1677,14 @@ JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVector
     IndexingHeader header;
     header.setVectorLength(length);
     header.setPublicLength(0);
-    Butterfly* butterfly = Butterfly::create(
-        vm, nullptr, 0, structure->outOfLineCapacity(), true, header,
-        sizeof(EncodedJSValue) * length);
-
+    if (butterfly)
+        *butterfly->indexingHeader() = header;
+    else {
+        butterfly = Butterfly::create(
+            vm, nullptr, 0, structure->outOfLineCapacity(), true, header,
+            sizeof(EncodedJSValue) * length);
+    }
+    
     // Paradoxically this may allocate a JSArray. That's totally cool.
     JSObject* result = JSObject::createRawObject(exec, structure, butterfly);
     result->butterfly(); // Ensure that the butterfly is in to-space.
index b66fc1e..2c392c5 100644 (file)
@@ -70,7 +70,7 @@ EncodedJSValue JIT_OPERATION operationGetByValWithThis(ExecState*, EncodedJSValu
 char* JIT_OPERATION operationNewArray(ExecState*, Structure*, void*, size_t) WTF_INTERNAL;
 char* JIT_OPERATION operationNewArrayBuffer(ExecState*, Structure*, size_t, size_t) WTF_INTERNAL;
 char* JIT_OPERATION operationNewEmptyArray(ExecState*, Structure*) WTF_INTERNAL;
-char* JIT_OPERATION operationNewArrayWithSize(ExecState*, Structure*, int32_t) WTF_INTERNAL;
+char* JIT_OPERATION operationNewArrayWithSize(ExecState*, Structure*, int32_t, Butterfly*) WTF_INTERNAL;
 char* JIT_OPERATION operationNewInt8ArrayWithSize(ExecState*, Structure*, int32_t) WTF_INTERNAL;
 char* JIT_OPERATION operationNewInt8ArrayWithOneArgument(ExecState*, Structure*, EncodedJSValue) WTF_INTERNAL;
 char* JIT_OPERATION operationNewInt16ArrayWithSize(ExecState*, Structure*, int32_t) WTF_INTERNAL;
@@ -176,9 +176,9 @@ int64_t JIT_OPERATION operationConvertDoubleToInt52(double);
 
 size_t JIT_OPERATION operationDefaultHasInstance(ExecState*, JSCell* value, JSCell* proto);
 
-char* JIT_OPERATION operationNewRawObject(ExecState*, Structure*, int32_t) WTF_INTERNAL;
-JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState*, Structure*) WTF_INTERNAL;
-JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState*, Structure*, unsigned length) WTF_INTERNAL;
+char* JIT_OPERATION operationNewRawObject(ExecState*, Structure*, int32_t, Butterfly*) WTF_INTERNAL;
+JSCell* JIT_OPERATION operationNewObjectWithButterfly(ExecState*, Structure*, Butterfly*) WTF_INTERNAL;
+JSCell* JIT_OPERATION operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength(ExecState*, Structure*, unsigned length, Butterfly*) WTF_INTERNAL;
 
 void JIT_OPERATION operationProcessTypeProfilerLogDFG(ExecState*) WTF_INTERNAL;
 
index 91dfe82..87fe277 100644 (file)
@@ -94,7 +94,7 @@ void SpeculativeJIT::emitAllocateRawObject(GPRReg resultGPR, Structure* structur
     GPRReg scratch2GPR = scratch2.gpr();
 
     ASSERT(vectorLength >= numElements);
-    vectorLength = std::max(BASE_VECTOR_LEN, vectorLength);
+    vectorLength = Butterfly::optimalContiguousVectorLength(structure, vectorLength);
     
     JITCompiler::JumpList slowCases;
 
@@ -103,23 +103,30 @@ void SpeculativeJIT::emitAllocateRawObject(GPRReg resultGPR, Structure* structur
         size += vectorLength * sizeof(JSValue) + sizeof(IndexingHeader);
     size += outOfLineCapacity * sizeof(JSValue);
 
+    m_jit.move(TrustedImmPtr(0), storageGPR);
+    
     if (size) {
-        slowCases.append(
-            emitAllocateBasicStorage(TrustedImm32(size), storageGPR));
-        if (hasIndexingHeader)
-            m_jit.subPtr(TrustedImm32(vectorLength * sizeof(JSValue)), storageGPR);
-        else
-            m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
-    } else
-        m_jit.move(TrustedImmPtr(0), storageGPR);
+        if (MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(size)) {
+            m_jit.move(TrustedImmPtr(allocator), scratchGPR);
+            m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases);
+            
+            m_jit.addPtr(
+                TrustedImm32(outOfLineCapacity * sizeof(JSValue) + sizeof(IndexingHeader)),
+                storageGPR);
+            
+            if (hasIndexingHeader)
+                m_jit.store32(TrustedImm32(vectorLength), MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+        } else
+            slowCases.append(m_jit.jump());
+    }
 
     size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
-    MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
-    m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
-    emitAllocateJSObject(resultGPR, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
-
-    if (hasIndexingHeader)
-        m_jit.store32(TrustedImm32(vectorLength), MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+    MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
+    if (allocatorPtr) {
+        m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
+        emitAllocateJSObject(resultGPR, allocatorPtr, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
+    } else
+        slowCases.append(m_jit.jump());
 
     // I want a slow path that also loads out the storage pointer, and that's
     // what this custom CallArrayAllocatorSlowPathGenerator gives me. It's a lot
@@ -128,14 +135,20 @@ void SpeculativeJIT::emitAllocateRawObject(GPRReg resultGPR, Structure* structur
         slowCases, this, operationNewRawObject, resultGPR, storageGPR,
         structure, vectorLength));
 
-    if (hasDouble(structure->indexingType()) && numElements < vectorLength) {
+    if (numElements < vectorLength) {
 #if USE(JSVALUE64)
-        m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
+        if (hasDouble(structure->indexingType()))
+            m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
+        else
+            m_jit.move(TrustedImm64(JSValue::encode(JSValue())), scratchGPR);
         for (unsigned i = numElements; i < vectorLength; ++i)
             m_jit.store64(scratchGPR, MacroAssembler::Address(storageGPR, sizeof(double) * i));
 #else
         EncodedValueDescriptor value;
-        value.asInt64 = JSValue::encode(JSValue(JSValue::EncodeAsDouble, PNaN));
+        if (hasDouble(structure->indexingType()))
+            value.asInt64 = JSValue::encode(JSValue(JSValue::EncodeAsDouble, PNaN));
+        else
+            value.asInt64 = JSValue::encode(JSValue());
         for (unsigned i = numElements; i < vectorLength; ++i) {
             m_jit.store32(TrustedImm32(value.asBits.tag), MacroAssembler::Address(storageGPR, sizeof(double) * i + OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
             m_jit.store32(TrustedImm32(value.asBits.payload), MacroAssembler::Address(storageGPR, sizeof(double) * i + OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
@@ -3823,9 +3836,10 @@ void SpeculativeJIT::compileMakeRope(Node* node)
     GPRReg scratchGPR = scratch.gpr();
     
     JITCompiler::JumpList slowPath;
-    MarkedAllocator& markedAllocator = m_jit.vm()->heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
-    m_jit.move(TrustedImmPtr(&markedAllocator), allocatorGPR);
-    emitAllocateJSCell(resultGPR, allocatorGPR, TrustedImmPtr(m_jit.vm()->stringStructure.get()), scratchGPR, slowPath);
+    MarkedAllocator* markedAllocator = m_jit.vm()->heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
+    RELEASE_ASSERT(markedAllocator);
+    m_jit.move(TrustedImmPtr(markedAllocator), allocatorGPR);
+    emitAllocateJSCell(resultGPR, markedAllocator, allocatorGPR, TrustedImmPtr(m_jit.vm()->stringStructure.get()), scratchGPR, slowPath);
         
     m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(resultGPR, JSString::offsetOfValue()));
     for (unsigned i = 0; i < numOpGPRs; ++i)
@@ -6908,7 +6922,14 @@ void SpeculativeJIT::compileCheckStructure(Node* node)
 
 void SpeculativeJIT::compileAllocatePropertyStorage(Node* node)
 {
-    if (node->transition()->previous->couldHaveIndexingHeader()) {
+    ASSERT(!node->transition()->previous->outOfLineCapacity());
+    ASSERT(initialOutOfLineCapacity == node->transition()->next->outOfLineCapacity());
+    
+    size_t size = initialOutOfLineCapacity * sizeof(JSValue);
+
+    MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(size);
+
+    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
         SpeculateCellOperand base(this, node->child1());
         
         GPRReg baseGPR = base.gpr();
@@ -6925,18 +6946,18 @@ void SpeculativeJIT::compileAllocatePropertyStorage(Node* node)
     
     SpeculateCellOperand base(this, node->child1());
     GPRTemporary scratch1(this);
+    GPRTemporary scratch2(this);
+    GPRTemporary scratch3(this);
         
     GPRReg baseGPR = base.gpr();
     GPRReg scratchGPR1 = scratch1.gpr();
+    GPRReg scratchGPR2 = scratch2.gpr();
+    GPRReg scratchGPR3 = scratch3.gpr();
         
-    ASSERT(!node->transition()->previous->outOfLineCapacity());
-    ASSERT(initialOutOfLineCapacity == node->transition()->next->outOfLineCapacity());
-    
-    JITCompiler::Jump slowPath =
-        emitAllocateBasicStorage(
-            TrustedImm32(initialOutOfLineCapacity * sizeof(JSValue)), scratchGPR1);
-
-    m_jit.addPtr(JITCompiler::TrustedImm32(sizeof(IndexingHeader)), scratchGPR1);
+    m_jit.move(JITCompiler::TrustedImmPtr(allocator), scratchGPR2);
+    JITCompiler::JumpList slowPath;
+    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
+    m_jit.addPtr(JITCompiler::TrustedImm32(size + sizeof(IndexingHeader)), scratchGPR1);
         
     addSlowPathGenerator(
         slowPathCall(slowPath, this, operationAllocatePropertyStorageWithInitialCapacity, scratchGPR1));
@@ -6951,8 +6972,10 @@ void SpeculativeJIT::compileReallocatePropertyStorage(Node* node)
     size_t oldSize = node->transition()->previous->outOfLineCapacity() * sizeof(JSValue);
     size_t newSize = oldSize * outOfLineGrowthFactor;
     ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue));
+    
+    MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(newSize);
 
-    if (node->transition()->previous->couldHaveIndexingHeader()) {
+    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
         SpeculateCellOperand base(this, node->child1());
         
         GPRReg baseGPR = base.gpr();
@@ -6971,16 +6994,19 @@ void SpeculativeJIT::compileReallocatePropertyStorage(Node* node)
     StorageOperand oldStorage(this, node->child2());
     GPRTemporary scratch1(this);
     GPRTemporary scratch2(this);
+    GPRTemporary scratch3(this);
         
     GPRReg baseGPR = base.gpr();
     GPRReg oldStorageGPR = oldStorage.gpr();
     GPRReg scratchGPR1 = scratch1.gpr();
     GPRReg scratchGPR2 = scratch2.gpr();
-        
-    JITCompiler::Jump slowPath =
-        emitAllocateBasicStorage(TrustedImm32(newSize), scratchGPR1);
-
-    m_jit.addPtr(JITCompiler::TrustedImm32(sizeof(IndexingHeader)), scratchGPR1);
+    GPRReg scratchGPR3 = scratch3.gpr();
+    
+    JITCompiler::JumpList slowPath;
+    m_jit.move(JITCompiler::TrustedImmPtr(allocator), scratchGPR2);
+    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
+    
+    m_jit.addPtr(JITCompiler::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR1);
         
     addSlowPathGenerator(
         slowPathCall(slowPath, this, operationAllocatePropertyStorage, scratchGPR1, newSize / sizeof(JSValue)));
index b9a8d0f..5be01c9 100644 (file)
@@ -1003,6 +1003,26 @@ public:
         m_jit.setupArgumentsWithExecState(arg1, arg2);
         return appendCallSetResult(operation, result);
     }
+    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, Structure* structure, GPRReg arg2, GPRReg butterfly)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), arg2, butterfly);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, Structure* structure, size_t arg2, GPRReg butterfly)
+    {
+        m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), TrustedImm32(arg2), butterfly);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, GPRReg arg1, GPRReg arg2, GPRReg butterfly)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, butterfly);
+        return appendCallSetResult(operation, result);
+    }
+    JITCompiler::Call callOperation(P_JITOperation_EStZB operation, GPRReg result, GPRReg arg1, GPRReg arg2, Butterfly* butterfly)
+    {
+        m_jit.setupArgumentsWithExecState(arg1, arg2, TrustedImmPtr(butterfly));
+        return appendCallSetResult(operation, result);
+    }
     JITCompiler::Call callOperation(P_JITOperation_EStPS operation, GPRReg result, Structure* structure, void* pointer, size_t size)
     {
         m_jit.setupArgumentsWithExecState(TrustedImmPtr(structure), TrustedImmPtr(pointer), TrustedImmPtr(size));
@@ -2557,18 +2577,21 @@ public:
 
     // Allocator for a cell of a specific size.
     template <typename StructureType> // StructureType can be GPR or ImmPtr.
-    void emitAllocateJSCell(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure,
+    void emitAllocateJSCell(
+        GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure,
         GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
     {
-        m_jit.emitAllocateJSCell(resultGPR, allocatorGPR, structure, scratchGPR, slowPath);
+        m_jit.emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
     }
 
     // Allocator for an object of a specific size.
     template <typename StructureType, typename StorageType> // StructureType and StorageType can be GPR or ImmPtr.
-    void emitAllocateJSObject(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure,
+    void emitAllocateJSObject(
+        GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure,
         StorageType storage, GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
     {
-        m_jit.emitAllocateJSObject(resultGPR, allocatorGPR, structure, storage, scratchGPR, slowPath);
+        m_jit.emitAllocateJSObject(
+            resultGPR, allocator, allocatorGPR, structure, storage, scratchGPR, slowPath);
     }
 
     template <typename ClassType, typename StructureType, typename StorageType> // StructureType and StorageType can be GPR or ImmPtr.
index f5a790d..e37fa13 100644 (file)
@@ -3866,8 +3866,7 @@ void SpeculativeJIT::compile(Node* node)
         bigLength.link(&m_jit);
         m_jit.move(TrustedImmPtr(globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)), structureGPR);
         done.link(&m_jit);
-        callOperation(
-            operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR);
+        callOperation(operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR, nullptr);
         m_jit.exceptionCheck();
         cellResult(resultGPR, node);
         break;
@@ -4033,7 +4032,7 @@ void SpeculativeJIT::compile(Node* node)
         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
         slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
-        emitAllocateJSObject(resultGPR, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
+        emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
 
         addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
         
@@ -4054,10 +4053,10 @@ void SpeculativeJIT::compile(Node* node)
         
         Structure* structure = node->structure();
         size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
-        MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
+        MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
 
         m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
-        emitAllocateJSObject(resultGPR, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
+        emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
 
         addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
         
@@ -5364,41 +5363,47 @@ void SpeculativeJIT::compileAllocateNewArrayWithSize(JSGlobalObject* globalObjec
     GPRReg scratchGPR = scratch.gpr();
     GPRReg scratch2GPR = scratch2.gpr();
     
+    m_jit.move(TrustedImmPtr(0), storageGPR);
+            
     MacroAssembler::JumpList slowCases;
     if (shouldConvertLargeSizeToArrayStorage)
         slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, sizeGPR, TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)));
-    
+            
     ASSERT((1 << 3) == sizeof(JSValue));
     m_jit.move(sizeGPR, scratchGPR);
     m_jit.lshift32(TrustedImm32(3), scratchGPR);
     m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratchGPR, resultGPR);
-    slowCases.append(
-        emitAllocateBasicStorage(resultGPR, storageGPR));
-    m_jit.subPtr(scratchGPR, storageGPR);
-    Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
-    emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
-    
+    m_jit.emitAllocateVariableSized(
+        storageGPR, m_jit.vm()->heap.subspaceForAuxiliaryData(), resultGPR, scratchGPR,
+        scratch2GPR, slowCases);
+    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
+
     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
+            
+    JSValue hole;
+    if (hasDouble(indexingType))
+        hole = JSValue(JSValue::EncodeAsDouble, PNaN);
+    else
+        hole = JSValue();
+            
+    m_jit.move(sizeGPR, scratchGPR);
+    MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratchGPR);
+    MacroAssembler::Label loop = m_jit.label();
+    m_jit.sub32(TrustedImm32(1), scratchGPR);
+    m_jit.store32(TrustedImm32(hole.u.asBits.tag), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
+    m_jit.store32(TrustedImm32(hole.u.asBits.payload), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
+    m_jit.branchTest32(MacroAssembler::NonZero, scratchGPR).linkTo(loop, &m_jit);
+    done.link(&m_jit);
     
-    if (hasDouble(indexingType)) {
-        JSValue nan = JSValue(JSValue::EncodeAsDouble, PNaN);
-        
-        m_jit.move(sizeGPR, scratchGPR);
-        MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratchGPR);
-        MacroAssembler::Label loop = m_jit.label();
-        m_jit.sub32(TrustedImm32(1), scratchGPR);
-        m_jit.store32(TrustedImm32(nan.u.asBits.tag), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
-        m_jit.store32(TrustedImm32(nan.u.asBits.payload), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
-        m_jit.branchTest32(MacroAssembler::NonZero, scratchGPR).linkTo(loop, &m_jit);
-        done.link(&m_jit);
-    }
-    
+    Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
+    emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
+            
     addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
         slowCases, this, operationNewArrayWithSize, resultGPR,
         structure,
         shouldConvertLargeSizeToArrayStorage ? globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage) : structure,
-        sizeGPR));
+        sizeGPR, storageGPR));
 }
 
 #endif
index 45cb64a..bfb61ef 100644 (file)
@@ -3818,7 +3818,7 @@ void SpeculativeJIT::compile(Node* node)
         bigLength.link(&m_jit);
         m_jit.move(TrustedImmPtr(globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)), structureGPR);
         done.link(&m_jit);
-        callOperation(operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR);
+        callOperation(operationNewArrayWithSize, resultGPR, structureGPR, sizeGPR, nullptr);
         m_jit.exceptionCheck();
         cellResult(resultGPR, node);
         break;
@@ -3975,7 +3975,7 @@ void SpeculativeJIT::compile(Node* node)
         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
         slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
-        emitAllocateJSObject(resultGPR, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
+        emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
 
         addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
         
@@ -3996,10 +3996,10 @@ void SpeculativeJIT::compile(Node* node)
 
         Structure* structure = node->structure();
         size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
-        MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
+        MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
 
         m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
-        emitAllocateJSObject(resultGPR, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
+        emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
 
         addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
         
@@ -5236,7 +5236,7 @@ void SpeculativeJIT::compile(Node* node)
 
         unsigned bytecodeIndex = node->origin.semantic.bytecodeIndex;
         auto triggerIterator = m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex);
-        RELEASE_ASSERT(triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
+        DFG_ASSERT(m_jit.graph(), node, triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
         uint8_t* forceEntryTrigger = &(m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex)->value);
 
         MacroAssembler::Jump forceOSREntry = m_jit.branchTest8(MacroAssembler::NonZero, MacroAssembler::AbsoluteAddress(forceEntryTrigger));
@@ -5420,39 +5420,45 @@ void SpeculativeJIT::compileAllocateNewArrayWithSize(JSGlobalObject* globalObjec
     GPRReg scratchGPR = scratch.gpr();
     GPRReg scratch2GPR = scratch2.gpr();
     
+    m_jit.move(TrustedImmPtr(0), storageGPR);
+    
     MacroAssembler::JumpList slowCases;
     if (shouldConvertLargeSizeToArrayStorage)
         slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, sizeGPR, TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)));
-    
+            
     ASSERT((1 << 3) == sizeof(JSValue));
     m_jit.move(sizeGPR, scratchGPR);
     m_jit.lshift32(TrustedImm32(3), scratchGPR);
     m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratchGPR, resultGPR);
-    slowCases.append(
-        emitAllocateBasicStorage(resultGPR, storageGPR));
-    m_jit.subPtr(scratchGPR, storageGPR);
-    Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
-    emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
-    
+    m_jit.emitAllocateVariableSized(
+        storageGPR, m_jit.vm()->heap.subspaceForAuxiliaryData(), resultGPR, scratchGPR,
+        scratch2GPR, slowCases);
+    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
+
     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
-    
-    if (hasDouble(indexingType)) {
+            
+    if (hasDouble(indexingType))
         m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
-        m_jit.move(sizeGPR, scratch2GPR);
-        MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratch2GPR);
-        MacroAssembler::Label loop = m_jit.label();
-        m_jit.sub32(TrustedImm32(1), scratch2GPR);
-        m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
-        m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
-        done.link(&m_jit);
-    }
+    else
+        m_jit.move(TrustedImm64(JSValue::encode(JSValue())), scratchGPR);
+    m_jit.move(sizeGPR, scratch2GPR);
+    MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratch2GPR);
+    MacroAssembler::Label loop = m_jit.label();
+    m_jit.sub32(TrustedImm32(1), scratch2GPR);
+    m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
+    m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
+    done.link(&m_jit);
+            
+    Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
+            
+    emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
     
     addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
         slowCases, this, operationNewArrayWithSize, resultGPR,
         structure,
         shouldConvertLargeSizeToArrayStorage ? globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage) : structure,
-        sizeGPR));
+        sizeGPR, storageGPR));
 }
 
 #endif
index 5002b79..bfd7849 100644 (file)
@@ -40,6 +40,7 @@
 #include "RegExpConstructor.h"
 #include "StringPrototype.h"
 #include <cstdlib>
+#include <wtf/text/StringBuilder.h>
 
 namespace JSC { namespace DFG {
 
@@ -420,7 +421,7 @@ private:
                     dataLog("Giving up because of pattern limit.\n");
                 break;
             }
-
+            
             unsigned lastIndex;
             if (regExp->globalOrSticky()) {
                 // This will only work if we can prove what the value of lastIndex is. To do this
@@ -468,7 +469,7 @@ private:
             FrozenValue* constructorFrozenValue = m_graph.freeze(constructor);
 
             MatchResult result;
-            Vector<int, 32> ovector;
+            Vector<int> ovector;
             // We have to call the kind of match function that the main thread would have called.
             // Otherwise, we might not have the desired Yarr code compiled, and the match will fail.
             if (m_node->op() == RegExpExec) {
@@ -514,7 +515,8 @@ private:
                     }
 
                     unsigned publicLength = resultArray.size();
-                    unsigned vectorLength = std::max(BASE_VECTOR_LEN, publicLength);
+                    unsigned vectorLength =
+                        Butterfly::optimalContiguousVectorLength(structure, publicLength);
 
                     UniquedStringImpl* indexUID = vm().propertyNames->index.impl();
                     UniquedStringImpl* inputUID = vm().propertyNames->input.impl();
@@ -649,7 +651,7 @@ private:
             bool ok = true;
             do {
                 MatchResult result;
-                Vector<int, 32> ovector;
+                Vector<int> ovector;
                 // Model which version of match() is called by the main thread.
                 if (replace.isEmpty() && regExp->global()) {
                     if (!regExp->matchConcurrently(vm(), string, startPosition, result)) {
index cff05a8..d2e16f3 100644 (file)
@@ -76,7 +76,6 @@ namespace JSC { namespace FTL {
     macro(JSString_value, JSString::offsetOfValue()) \
     macro(JSSymbolTableObject_symbolTable, JSSymbolTableObject::offsetOfSymbolTable()) \
     macro(JSWrapperObject_internalValue, JSWrapperObject::internalValueOffset()) \
-    macro(MarkedAllocator_freeListHead, MarkedAllocator::offsetOfFreeListHead()) \
     macro(RegExpConstructor_cachedResult_lastRegExp, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfLastRegExp()) \
     macro(RegExpConstructor_cachedResult_lastInput, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfLastInput()) \
     macro(RegExpConstructor_cachedResult_result_start, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfResult() + OBJECT_OFFSETOF(MatchResult, start)) \
@@ -109,8 +108,7 @@ namespace JSC { namespace FTL {
     macro(JSEnvironmentRecord_variables, JSEnvironmentRecord::offsetOfVariables(), sizeof(EncodedJSValue)) \
     macro(JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, 0, sizeof(WriteBarrier<JSString>)) \
     macro(JSRopeString_fibers, JSRopeString::offsetOfFibers(), sizeof(WriteBarrier<JSString>)) \
-    macro(MarkedSpace_Subspace_impreciseAllocators, OBJECT_OFFSETOF(MarkedSpace::Subspace, impreciseAllocators), sizeof(MarkedAllocator)) \
-    macro(MarkedSpace_Subspace_preciseAllocators, OBJECT_OFFSETOF(MarkedSpace::Subspace, preciseAllocators), sizeof(MarkedAllocator)) \
+    macro(MarkedSpace_Subspace_allocatorForSizeStep, OBJECT_OFFSETOF(MarkedSpace::Subspace, allocatorForSizeStep), sizeof(MarkedAllocator*)) \
     macro(ScopedArguments_overflowStorage, ScopedArguments::overflowStorageOffset(), sizeof(EncodedJSValue)) \
     macro(WriteBarrierBuffer_bufferContents, 0, sizeof(JSCell*)) \
     macro(characters8, 0, sizeof(LChar)) \
index 5f8ae85..39eb666 100644 (file)
@@ -42,6 +42,7 @@
 #include "FTLJITCode.h"
 #include "FTLThunks.h"
 #include "JITSubGenerator.h"
+#include "JSCInlines.h"
 #include "LinkBuffer.h"
 #include "PCToCodeOriginMap.h"
 #include "ScratchRegisterAllocator.h"
index 00a88f0..dcf3dad 100644 (file)
@@ -32,6 +32,7 @@
 #include "DFGPlan.h"
 #include "FTLState.h"
 #include "FTLThunks.h"
+#include "JSCInlines.h"
 #include "ProfilerDatabase.h"
 
 namespace JSC { namespace FTL {
index 28e63ff..e4a6f14 100644 (file)
@@ -3815,7 +3815,7 @@ private:
                 size, m_out.constInt32(DirectArguments::allocationSize(minCapacity)));
             
             fastObject = allocateVariableSizedObject<DirectArguments>(
-                size, structure, m_out.intPtrZero, slowPath);
+                m_out.zeroExtPtr(size), structure, m_out.intPtrZero, slowPath);
         }
         
         m_out.store32(length.value, fastObject, m_heaps.DirectArguments_length);
@@ -3915,10 +3915,11 @@ private:
             LBasicBlock continuation = m_out.newBlock();
             LValue arrayLength = lowInt32(m_node->child1());
             LBasicBlock loopStart = m_out.newBlock();
-            bool shouldLargeArraySizeCreateArrayStorage = false;
-            LValue array = compileAllocateArrayWithSize(arrayLength, ArrayWithContiguous, shouldLargeArraySizeCreateArrayStorage);
-
-            LValue butterfly = m_out.loadPtr(array, m_heaps.JSObject_butterfly);
+            JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
+            Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithContiguous);
+            ArrayValues arrayValues = allocateUninitializedContiguousJSArray(arrayLength, structure);
+            LValue array = arrayValues.array;
+            LValue butterfly = arrayValues.butterfly;
             ValueFromBlock startLength = m_out.anchor(arrayLength);
             LValue argumentRegion = m_out.add(getArgumentsStart(), m_out.constInt64(sizeof(Register) * m_node->numberOfArgumentsToSkip()));
             m_out.branch(m_out.equal(arrayLength, m_out.constInt32(0)),
@@ -3930,7 +3931,7 @@ private:
             m_out.addIncomingToPhi(phiOffset, m_out.anchor(currentOffset));
             LValue loadedValue = m_out.load64(m_out.baseIndex(m_heaps.variables, argumentRegion, m_out.zeroExtPtr(currentOffset)));
             IndexedAbstractHeap& heap = m_heaps.indexedContiguousProperties;
-            m_out.store(loadedValue, m_out.baseIndex(heap, butterfly, m_out.zeroExtPtr(currentOffset)), Output::Store64);
+            m_out.store64(loadedValue, m_out.baseIndex(heap, butterfly, m_out.zeroExtPtr(currentOffset)));
             m_out.branch(m_out.equal(currentOffset, m_out.constInt32(0)), unsure(continuation), unsure(loopStart));
 
             m_out.appendTo(continuation, lastNext);
@@ -3987,7 +3988,8 @@ private:
         if (!globalObject->isHavingABadTime() && !hasAnyArrayStorage(m_node->indexingType())) {
             unsigned numElements = m_node->numChildren();
             
-            ArrayValues arrayValues = allocateJSArray(structure, numElements);
+            ArrayValues arrayValues =
+                allocateUninitializedContiguousJSArray(m_out.constInt32(numElements), structure);
             
             for (unsigned operandIndex = 0; operandIndex < m_node->numChildren(); ++operandIndex) {
                 Edge edge = m_graph.varArgChild(m_node, operandIndex);
@@ -4063,7 +4065,8 @@ private:
         if (!globalObject->isHavingABadTime() && !hasAnyArrayStorage(m_node->indexingType())) {
             unsigned numElements = m_node->numConstants();
             
-            ArrayValues arrayValues = allocateJSArray(structure, numElements);
+            ArrayValues arrayValues =
+                allocateUninitializedContiguousJSArray(m_out.constInt32(numElements), structure);
             
             JSValue* data = codeBlock()->constantBuffer(m_node->startConstant());
             for (unsigned index = 0; index < m_node->numConstants(); ++index) {
@@ -4089,88 +4092,6 @@ private:
             m_out.constIntPtr(m_node->numConstants())));
     }
 
-    LValue compileAllocateArrayWithSize(LValue publicLength, IndexingType indexingType, bool shouldLargeArraySizeCreateArrayStorage = true)
-    {
-        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
-        Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
-        ASSERT(
-            hasUndecided(structure->indexingType())
-            || hasInt32(structure->indexingType())
-            || hasDouble(structure->indexingType())
-            || hasContiguous(structure->indexingType()));
-
-        LBasicBlock fastCase = m_out.newBlock();
-        LBasicBlock largeCase = shouldLargeArraySizeCreateArrayStorage ? m_out.newBlock() : nullptr;
-        LBasicBlock failCase = m_out.newBlock();
-        LBasicBlock continuation = m_out.newBlock();
-        LBasicBlock lastNext = nullptr;
-        if (shouldLargeArraySizeCreateArrayStorage) {
-            m_out.branch(
-                m_out.aboveOrEqual(publicLength, m_out.constInt32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)),
-                rarely(largeCase), usually(fastCase));
-            lastNext = m_out.appendTo(fastCase, largeCase);
-        }
-
-        
-        // We don't round up to BASE_VECTOR_LEN for new Array(blah).
-        LValue vectorLength = publicLength;
-        
-        LValue payloadSize =
-            m_out.shl(m_out.zeroExt(vectorLength, pointerType()), m_out.constIntPtr(3));
-        
-        LValue butterflySize = m_out.add(
-            payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
-        
-        LValue endOfStorage = allocateBasicStorageAndGetEnd(butterflySize, failCase);
-        
-        LValue butterfly = m_out.sub(endOfStorage, payloadSize);
-        
-        LValue object = allocateObject<JSArray>(structure, butterfly, failCase);
-        
-        m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
-        m_out.store32(vectorLength, butterfly, m_heaps.Butterfly_vectorLength);
-
-        initializeArrayElements(indexingType, vectorLength, butterfly);
-        
-        ValueFromBlock fastResult = m_out.anchor(object);
-        m_out.jump(continuation);
-        
-        LValue structureValue;
-        if (shouldLargeArraySizeCreateArrayStorage) {
-            LBasicBlock slowCase = m_out.newBlock();
-
-            m_out.appendTo(largeCase, failCase);
-            ValueFromBlock largeStructure = m_out.anchor(m_out.constIntPtr(
-                globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)));
-            m_out.jump(slowCase);
-
-            m_out.appendTo(failCase, slowCase);
-            ValueFromBlock failStructure = m_out.anchor(m_out.constIntPtr(structure));
-            m_out.jump(slowCase);
-
-            m_out.appendTo(slowCase, continuation);
-            structureValue = m_out.phi(
-                pointerType(), largeStructure, failStructure);
-        } else {
-            ASSERT(!lastNext);
-            lastNext = m_out.appendTo(failCase, continuation);
-            structureValue = m_out.constIntPtr(structure);
-        }
-
-        LValue slowResultValue = lazySlowPath(
-            [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
-                return createLazyCallGenerator(
-                    operationNewArrayWithSize, locations[0].directGPR(),
-                    locations[1].directGPR(), locations[2].directGPR());
-            },
-            structureValue, publicLength);
-        ValueFromBlock slowResult = m_out.anchor(slowResultValue);
-        m_out.jump(continuation);
-        
-        m_out.appendTo(continuation, lastNext);
-        return m_out.phi(pointerType(), fastResult, slowResult);
-    }
-    
     void compileNewArrayWithSize()
     {
         LValue publicLength = lowInt32(m_node->child1());
@@ -4180,7 +4101,11 @@ private:
             m_node->indexingType());
         
         if (!globalObject->isHavingABadTime() && !hasAnyArrayStorage(m_node->indexingType())) {
-            setJSValue(compileAllocateArrayWithSize(publicLength, m_node->indexingType()));
+            setJSValue(
+                allocateJSArray(
+                    publicLength,
+                    globalObject->arrayStructureForIndexingTypeDuringAllocation(
+                        m_node->indexingType())).array);
             return;
         }
         
@@ -4189,7 +4114,7 @@ private:
             m_out.constIntPtr(
                 globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)),
             m_out.constIntPtr(structure));
-        setJSValue(vmCall(Int64, m_out.operation(operationNewArrayWithSize), m_callFrame, structureValue, publicLength));
+        setJSValue(vmCall(Int64, m_out.operation(operationNewArrayWithSize), m_callFrame, structureValue, publicLength, m_out.intPtrZero));
     }
 
     void compileNewTypedArray()
@@ -4449,13 +4374,12 @@ private:
         
         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
         
-        MarkedAllocator& allocator =
+        MarkedAllocator* allocator =
             vm().heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
+        DFG_ASSERT(m_graph, m_node, allocator);
         
         LValue result = allocateCell(
-            m_out.constIntPtr(&allocator),
-            vm().stringStructure.get(),
-            slowPath);
+            m_out.constIntPtr(allocator), vm().stringStructure.get(), slowPath);
         
         m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value);
         for (unsigned i = 0; i < numKids; ++i)
@@ -7004,7 +6928,8 @@ private:
             
             if (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType())) {
                 size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
-                MarkedAllocator* allocator = &vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
+                MarkedAllocator* cellAllocator = vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
+                DFG_ASSERT(m_graph, m_node, cellAllocator);
 
                 bool hasIndexingHeader = hasIndexedProperties(structure->indexingType());
                 unsigned indexingHeaderSize = 0;
@@ -7029,7 +6954,7 @@ private:
                     indexingPayloadSizeInBytes =
                         m_out.mul(m_out.zeroExtPtr(vectorLength), m_out.intPtrEight);
                 }
-
+                
                 LValue butterflySize = m_out.add(
                     m_out.constIntPtr(
                         structure->outOfLineCapacity() * sizeof(JSValue) + indexingHeaderSize),
@@ -7040,22 +6965,31 @@ private:
                 
                 LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
                 
-                LValue endOfStorage = allocateBasicStorageAndGetEnd(butterflySize, slowPath);
+                ValueFromBlock noButterfly = m_out.anchor(m_out.intPtrZero);
+                
+                LValue startOfStorage = allocateHeapCell(
+                    allocatorForSize(vm().heap.subspaceForAuxiliaryData(), butterflySize, slowPath),
+                    slowPath);
 
                 LValue fastButterflyValue = m_out.add(
-                    m_out.sub(endOfStorage, indexingPayloadSizeInBytes),
-                    m_out.constIntPtr(sizeof(IndexingHeader) - indexingHeaderSize));
+                    startOfStorage,
+                    m_out.constIntPtr(
+                        structure->outOfLineCapacity() * sizeof(JSValue) + sizeof(IndexingHeader)));
+                
+                ValueFromBlock haveButterfly = m_out.anchor(fastButterflyValue);
 
                 m_out.store32(vectorLength, fastButterflyValue, m_heaps.Butterfly_vectorLength);
                 
                 LValue fastObjectValue = allocateObject(
-                    m_out.constIntPtr(allocator), structure, fastButterflyValue, slowPath);
+                    m_out.constIntPtr(cellAllocator), structure, fastButterflyValue, slowPath);
 
                 ValueFromBlock fastObject = m_out.anchor(fastObjectValue);
                 ValueFromBlock fastButterfly = m_out.anchor(fastButterflyValue);
                 m_out.jump(continuation);
                 
                 m_out.appendTo(slowPath, continuation);
+                
+                LValue butterflyValue = m_out.phi(pointerType(), noButterfly, haveButterfly);
 
                 LValue slowObjectValue;
                 if (hasIndexingHeader) {
@@ -7064,16 +6998,17 @@ private:
                             return createLazyCallGenerator(
                                 operationNewObjectWithButterflyWithIndexingHeaderAndVectorLength,
                                 locations[0].directGPR(), CCallHelpers::TrustedImmPtr(structure),
-                                locations[1].directGPR());
+                                locations[1].directGPR(), locations[2].directGPR());
                         },
-                        vectorLength);
+                        vectorLength, butterflyValue);
                 } else {
                     slowObjectValue = lazySlowPath(
                         [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
                             return createLazyCallGenerator(
                                 operationNewObjectWithButterfly, locations[0].directGPR(),
-                                CCallHelpers::TrustedImmPtr(structure));
-                        });
+                                CCallHelpers::TrustedImmPtr(structure), locations[1].directGPR());
+                        },
+                        butterflyValue);
                 }
                 ValueFromBlock slowObject = m_out.anchor(slowObjectValue);
                 ValueFromBlock slowButterfly = m_out.anchor(
@@ -7088,7 +7023,7 @@ private:
 
                 m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
 
-                initializeArrayElements(structure->indexingType(), vectorLength, butterfly);
+                initializeArrayElements(structure->indexingType(), m_out.int32Zero, vectorLength, butterfly);
 
                 HashMap<int32_t, LValue, DefaultHash<int32_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<int32_t>> indexMap;
                 Vector<int32_t> indices;
@@ -7833,35 +7768,54 @@ private:
         return result;
     }
 
-    void initializeArrayElements(IndexingType indexingType, LValue vectorLength, LValue butterfly)
+    void initializeArrayElements(IndexingType indexingType, LValue begin, LValue end, LValue butterfly)
     {
-        if (!hasDouble(indexingType)) {
-            // The GC already initialized everything to JSValue() for us.
+        if (hasUndecided(indexingType))
             return;
+        
+        if (begin == end)
+            return;
+        
+        IndexedAbstractHeap* heap = m_heaps.forIndexingType(indexingType);
+        DFG_ASSERT(m_graph, m_node, heap);
+        
+        LValue hole;
+        if (hasDouble(indexingType))
+            hole = m_out.constInt64(bitwise_cast<int64_t>(PNaN));
+        else
+            hole = m_out.constInt64(JSValue::encode(JSValue()));
+        
+        const uint64_t unrollingLimit = 10;
+        if (begin->hasInt() && end->hasInt()) {
+            uint64_t beginConst = static_cast<uint64_t>(begin->asInt());
+            uint64_t endConst = static_cast<uint64_t>(end->asInt());
+            
+            if (endConst - beginConst <= unrollingLimit) {
+                for (uint64_t i = beginConst; i < endConst; ++i)
+                    m_out.store64(hole, butterfly, heap->at(i));
+                return;
+            }
         }
 
         // Doubles must be initialized to PNaN.
         LBasicBlock initLoop = m_out.newBlock();
         LBasicBlock initDone = m_out.newBlock();
         
-        ValueFromBlock originalIndex = m_out.anchor(vectorLength);
+        ValueFromBlock originalIndex = m_out.anchor(end);
         ValueFromBlock originalPointer = m_out.anchor(butterfly);
-        m_out.branch(
-            m_out.notZero32(vectorLength), unsure(initLoop), unsure(initDone));
+        m_out.branch(m_out.notEqual(end, begin), unsure(initLoop), unsure(initDone));
         
         LBasicBlock initLastNext = m_out.appendTo(initLoop, initDone);
         LValue index = m_out.phi(Int32, originalIndex);
         LValue pointer = m_out.phi(pointerType(), originalPointer);
         
-        m_out.store64(
-            m_out.constInt64(bitwise_cast<int64_t>(PNaN)),
-            TypedPointer(m_heaps.indexedDoubleProperties.atAnyIndex(), pointer));
+        m_out.store64(hole, TypedPointer(heap->atAnyIndex(), pointer));
         
         LValue nextIndex = m_out.sub(index, m_out.int32One);
         m_out.addIncomingToPhi(index, m_out.anchor(nextIndex));
         m_out.addIncomingToPhi(pointer, m_out.anchor(m_out.add(pointer, m_out.intPtrEight)));
         m_out.branch(
-            m_out.notZero32(nextIndex), unsure(initLoop), unsure(initDone));
+            m_out.notEqual(nextIndex, begin), unsure(initLoop), unsure(initDone));
         
         m_out.appendTo(initDone, initLastNext);
     }
@@ -7916,13 +7870,12 @@ private:
         LBasicBlock continuation = m_out.newBlock();
         
         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
-        
-        LValue endOfStorage = allocateBasicStorageAndGetEnd(
-            m_out.constIntPtr(sizeInValues * sizeof(JSValue)), slowPath);
-        
+
+        size_t sizeInBytes = sizeInValues * sizeof(JSValue);
+        MarkedAllocator* allocator = vm().heap.allocatorForAuxiliaryData(sizeInBytes);
+        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
         ValueFromBlock fastButterfly = m_out.anchor(
-            m_out.add(m_out.constIntPtr(sizeof(IndexingHeader)), endOfStorage));
-        
+            m_out.add(m_out.constIntPtr(sizeInBytes + sizeof(IndexingHeader)), startOfStorage));
         m_out.jump(continuation);
         
         m_out.appendTo(slowPath, continuation);
@@ -8487,29 +8440,64 @@ private:
         setJSValue(patchpoint);
     }
 
-    LValue allocateCell(LValue allocator, LBasicBlock slowPath)
+    LValue allocateHeapCell(LValue allocator, LBasicBlock slowPath)
     {
-        LBasicBlock success = m_out.newBlock();
-    
-        LValue result;
-        LValue condition;
-        if (Options::forceGCSlowPaths()) {
-            result = m_out.intPtrZero;
-            condition = m_out.booleanFalse;
-        } else {
-            result = m_out.loadPtr(
-                allocator, m_heaps.MarkedAllocator_freeListHead);
-            condition = m_out.notNull(result);
+        MarkedAllocator* actualAllocator = nullptr;
+        if (allocator->hasIntPtr())
+            actualAllocator = bitwise_cast<MarkedAllocator*>(allocator->asIntPtr());
+        
+        if (!actualAllocator) {
+            // This means that either we know that the allocator is null or we don't know what the
+            // allocator is. In either case, we need the null check.
+            LBasicBlock haveAllocator = m_out.newBlock();
+            LBasicBlock lastNext = m_out.insertNewBlocksBefore(haveAllocator);
+            m_out.branch(allocator, usually(haveAllocator), rarely(slowPath));
+            m_out.appendTo(haveAllocator, lastNext);
         }
-        m_out.branch(condition, usually(success), rarely(slowPath));
         
-        m_out.appendTo(success);
+        LBasicBlock continuation = m_out.newBlock();
         
-        m_out.storePtr(
-            m_out.loadPtr(result, m_heaps.JSCell_freeListNext),
-            allocator, m_heaps.MarkedAllocator_freeListHead);
-
-        return result;
+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
+        
+        PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
+        patchpoint->effects.terminal = true;
+        patchpoint->appendSomeRegister(allocator);
+        patchpoint->numGPScratchRegisters++;
+        patchpoint->resultConstraint = ValueRep::SomeEarlyRegister;
+        
+        m_out.appendSuccessor(usually(continuation));
+        m_out.appendSuccessor(rarely(slowPath));
+        
+        patchpoint->setGenerator(
+            [=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
+                CCallHelpers::JumpList jumpToSlowPath;
+                
+                // We use a patchpoint to emit the allocation path because whenever we mess with
+                // allocation paths, we already reason about them at the machine code level. We know
+                // exactly what instruction sequence we want. We're confident that no compiler
+                // optimization could make this code better. So, it's best to have the code in
+                // AssemblyHelpers::emitAllocate(). That way, the same optimized path is shared by
+                // all of the compiler tiers.
+                jit.emitAllocateWithNonNullAllocator(
+                    params[0].gpr(), actualAllocator, params[1].gpr(), params.gpScratch(0),
+                    jumpToSlowPath);
+                
+                CCallHelpers::Jump jumpToSuccess;
+                if (!params.fallsThroughToSuccessor(0))
+                    jumpToSuccess = jit.jump();
+                
+                Vector<Box<CCallHelpers::Label>> labels = params.successorLabels();
+                
+                params.addLatePath(
+                    [=] (CCallHelpers& jit) {
+                        jumpToSlowPath.linkTo(*labels[1], &jit);
+                        if (jumpToSuccess.isSet())
+                            jumpToSuccess.linkTo(*labels[0], &jit);
+                    });
+            });
+        
+        m_out.appendTo(continuation, lastNext);
+        return patchpoint;
     }
     
     void storeStructure(LValue object, Structure* structure)
@@ -8522,7 +8510,7 @@ private:
 
     LValue allocateCell(LValue allocator, Structure* structure, LBasicBlock slowPath)
     {
-        LValue result = allocateCell(allocator, slowPath);
+        LValue result = allocateHeapCell(allocator, slowPath);
         storeStructure(result, structure);
         return result;
     }
@@ -8539,7 +8527,7 @@ private:
     LValue allocateObject(
         size_t size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
     {
-        MarkedAllocator* allocator = &vm().heap.allocatorForObjectOfType<ClassType>(size);
+        MarkedAllocator* allocator = vm().heap.allocatorForObjectOfType<ClassType>(size);
         return allocateObject(m_out.constIntPtr(allocator), structure, butterfly, slowPath);
     }
     
@@ -8550,46 +8538,60 @@ private:
             ClassType::allocationSize(0), structure, butterfly, slowPath);
     }
     
-    template<typename ClassType>
-    LValue allocateVariableSizedObject(
-        LValue size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
+    LValue allocatorForSize(LValue subspace, LValue size, LBasicBlock slowPath)
     {
-        static_assert(!(MarkedSpace::preciseStep & (MarkedSpace::preciseStep - 1)), "MarkedSpace::preciseStep must be a power of two.");
-        static_assert(!(MarkedSpace::impreciseStep & (MarkedSpace::impreciseStep - 1)), "MarkedSpace::impreciseStep must be a power of two.");
-
-        LValue subspace = m_out.constIntPtr(&vm().heap.subspaceForObjectOfType<ClassType>());
+        static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
+        
+        // Try to do some constant-folding here.
+        if (subspace->hasIntPtr() && size->hasIntPtr()) {
+            MarkedSpace::Subspace* actualSubspace = bitwise_cast<MarkedSpace::Subspace*>(subspace->asIntPtr());
+            size_t actualSize = size->asIntPtr();
+            
+            MarkedAllocator* actualAllocator = MarkedSpace::allocatorFor(*actualSubspace, actualSize);
+            if (!actualAllocator) {
+                LBasicBlock continuation = m_out.newBlock();
+                LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
+                m_out.jump(slowPath);
+                m_out.appendTo(continuation, lastNext);
+                return m_out.intPtrZero;
+            }
+            
+            return m_out.constIntPtr(actualAllocator);
+        }
+        
+        unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
         
-        LBasicBlock smallCaseBlock = m_out.newBlock();
-        LBasicBlock largeOrOversizeCaseBlock = m_out.newBlock();
-        LBasicBlock largeCaseBlock = m_out.newBlock();
         LBasicBlock continuation = m_out.newBlock();
         
-        LValue uproundedSize = m_out.add(size, m_out.constInt32(MarkedSpace::preciseStep - 1));
-        LValue isSmall = m_out.below(uproundedSize, m_out.constInt32(MarkedSpace::preciseCutoff));
-        m_out.branch(isSmall, unsure(smallCaseBlock), unsure(largeOrOversizeCaseBlock));
+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
         
-        LBasicBlock lastNext = m_out.appendTo(smallCaseBlock, largeOrOversizeCaseBlock);
-        TypedPointer address = m_out.baseIndex(
-            m_heaps.MarkedSpace_Subspace_preciseAllocators, subspace,
-            m_out.zeroExtPtr(m_out.lShr(uproundedSize, m_out.constInt32(getLSBSet(MarkedSpace::preciseStep)))));
-        ValueFromBlock smallAllocator = m_out.anchor(address.value());
-        m_out.jump(continuation);
+        LValue sizeClassIndex = m_out.lShr(
+            m_out.add(size, m_out.constIntPtr(MarkedSpace::sizeStep - 1)),
+            m_out.constInt32(stepShift));
         
-        m_out.appendTo(largeOrOversizeCaseBlock, largeCaseBlock);
         m_out.branch(
-            m_out.below(uproundedSize, m_out.constInt32(MarkedSpace::impreciseCutoff)),
-            usually(largeCaseBlock), rarely(slowPath));
-        
-        m_out.appendTo(largeCaseBlock, continuation);
-        address = m_out.baseIndex(
-            m_heaps.MarkedSpace_Subspace_impreciseAllocators, subspace,
-            m_out.zeroExtPtr(m_out.lShr(uproundedSize, m_out.constInt32(getLSBSet(MarkedSpace::impreciseStep)))));
-        ValueFromBlock largeAllocator = m_out.anchor(address.value());
-        m_out.jump(continuation);
+            m_out.above(sizeClassIndex, m_out.constIntPtr(MarkedSpace::largeCutoff >> stepShift)),
+            rarely(slowPath), usually(continuation));
         
         m_out.appendTo(continuation, lastNext);
-        LValue allocator = m_out.phi(pointerType(), smallAllocator, largeAllocator);
         
+        return m_out.loadPtr(
+            m_out.baseIndex(
+                m_heaps.MarkedSpace_Subspace_allocatorForSizeStep,
+                subspace, m_out.sub(sizeClassIndex, m_out.intPtrOne)));
+    }
+    
+    LValue allocatorForSize(MarkedSpace::Subspace& subspace, LValue size, LBasicBlock slowPath)
+    {
+        return allocatorForSize(m_out.constIntPtr(&subspace), size, slowPath);
+    }
+    
+    template<typename ClassType>
+    LValue allocateVariableSizedObject(
+        LValue size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
+    {
+        LValue allocator = allocatorForSize(
+            vm().heap.subspaceForObjectOfType<ClassType>(), size, slowPath);
         return allocateObject(allocator, structure, butterfly, slowPath);
     }
     
@@ -8622,7 +8624,11 @@ private:
     LValue allocateObject(Structure* structure)
     {
         size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
-        MarkedAllocator* allocator = &vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
+        MarkedAllocator* allocator = vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
+        
+        // FIXME: If the allocator is null, we could simply emit a normal C call to the allocator
+        // instead of putting it on the slow path.
+        // https://bugs.webkit.org/show_bug.cgi?id=161062
         
         LBasicBlock slowPath = m_out.newBlock();
         LBasicBlock continuation = m_out.newBlock();
@@ -8665,75 +8671,122 @@ private:
         LValue array;
         LValue butterfly;
     };
-    ArrayValues allocateJSArray(
-        Structure* structure, unsigned numElements, LBasicBlock slowPath)
+
+    ArrayValues allocateJSArray(LValue publicLength, Structure* structure, bool shouldInitializeElements = true, bool shouldLargeArraySizeCreateArrayStorage = true)
     {
+        JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
+        IndexingType indexingType = structure->indexingType();
         ASSERT(
-            hasUndecided(structure->indexingType())
-            || hasInt32(structure->indexingType())
-            || hasDouble(structure->indexingType())
-            || hasContiguous(structure->indexingType()));
-        
-        unsigned vectorLength = std::max(BASE_VECTOR_LEN, numElements);
+            hasUndecided(indexingType)
+            || hasInt32(indexingType)
+            || hasDouble(indexingType)
+            || hasContiguous(indexingType));
+
+        LBasicBlock fastCase = m_out.newBlock();
+        LBasicBlock largeCase = m_out.newBlock();
+        LBasicBlock failCase = m_out.newBlock();
+        LBasicBlock continuation = m_out.newBlock();
+        LBasicBlock slowCase = m_out.newBlock();
         
-        LValue endOfStorage = allocateBasicStorageAndGetEnd(
-            m_out.constIntPtr(sizeof(JSValue) * vectorLength + sizeof(IndexingHeader)),
-            slowPath);
+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(fastCase);
         
-        LValue butterfly = m_out.sub(
-            endOfStorage, m_out.constIntPtr(sizeof(JSValue) * vectorLength));
+        ValueFromBlock noButterfly = m_out.anchor(m_out.intPtrZero);
         
-        LValue object = allocateObject<JSArray>(
-            structure, butterfly, slowPath);
+        LValue predicate;
+        if (shouldLargeArraySizeCreateArrayStorage)
+            predicate = m_out.aboveOrEqual(publicLength, m_out.constInt32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH));
+        else
+            predicate = m_out.booleanFalse;
         
-        m_out.store32(m_out.constInt32(numElements), butterfly, m_heaps.Butterfly_publicLength);
-        m_out.store32(m_out.constInt32(vectorLength), butterfly, m_heaps.Butterfly_vectorLength);
+        m_out.branch(predicate, rarely(largeCase), usually(fastCase));
         
-        if (hasDouble(structure->indexingType())) {
-            for (unsigned i = numElements; i < vectorLength; ++i) {
-                m_out.store64(
-                    m_out.constInt64(bitwise_cast<int64_t>(PNaN)),
-                    butterfly, m_heaps.indexedDoubleProperties[i]);
+        m_out.appendTo(fastCase, largeCase);
+
+        LValue vectorLength = nullptr;
+        if (publicLength->hasInt32()) {
+            unsigned publicLengthConst = static_cast<unsigned>(publicLength->asInt32());
+            if (publicLengthConst <= MAX_STORAGE_VECTOR_LENGTH) {
+                vectorLength = m_out.constInt32(
+                    Butterfly::optimalContiguousVectorLength(
+                        structure->outOfLineCapacity(), publicLengthConst));
             }
         }
         
-        return ArrayValues(object, butterfly);
-    }
-    
-    ArrayValues allocateJSArray(Structure* structure, unsigned numElements)
-    {
-        LBasicBlock slowPath = m_out.newBlock();
-        LBasicBlock continuation = m_out.newBlock();
+        if (!vectorLength) {
+            // We don't compute the optimal vector length for new Array(blah) where blah is not
+            // statically known, since the compute effort of doing it here is probably not worth it.
+            vectorLength = publicLength;
+        }
+            
+        LValue payloadSize =
+            m_out.shl(m_out.zeroExt(vectorLength, pointerType()), m_out.constIntPtr(3));
+            
+        LValue butterflySize = m_out.add(
+            payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
+            
+        LValue allocator = allocatorForSize(
+            vm().heap.subspaceForAuxiliaryData(), butterflySize, failCase);
+        LValue startOfStorage = allocateHeapCell(allocator, failCase);
+            
+        LValue butterfly = m_out.add(startOfStorage, m_out.constIntPtr(sizeof(IndexingHeader)));
         
-        LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
+        m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
+        m_out.store32(vectorLength, butterfly, m_heaps.Butterfly_vectorLength);
+    
+        initializeArrayElements(
+            indexingType,
+            shouldInitializeElements ? m_out.int32Zero : publicLength, vectorLength,
+            butterfly);
         
-        ArrayValues fastValues = allocateJSArray(structure, numElements, slowPath);
-        ValueFromBlock fastArray = m_out.anchor(fastValues.array);
-        ValueFromBlock fastButterfly = m_out.anchor(fastValues.butterfly);
+        ValueFromBlock haveButterfly = m_out.anchor(butterfly);
         
+        LValue object = allocateObject<JSArray>(structure, butterfly, failCase);
+            
+        ValueFromBlock fastResult = m_out.anchor(object);
+        ValueFromBlock fastButterfly = m_out.anchor(butterfly);
         m_out.jump(continuation);
         
-        m_out.appendTo(slowPath, continuation);
+        m_out.appendTo(largeCase, failCase);
+        ValueFromBlock largeStructure = m_out.anchor(
+            m_out.constIntPtr(
+                globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)));
+        m_out.jump(slowCase);
+        
+        m_out.appendTo(failCase, slowCase);
+        ValueFromBlock failStructure = m_out.anchor(m_out.constIntPtr(structure));
+        m_out.jump(slowCase);
+        
+        m_out.appendTo(slowCase, continuation);
+        LValue structureValue = m_out.phi(pointerType(), largeStructure, failStructure);
+        LValue butterflyValue = m_out.phi(pointerType(), noButterfly, haveButterfly);
 
-        LValue slowArrayValue = lazySlowPath(
+        LValue slowResultValue = lazySlowPath(
             [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
                 return createLazyCallGenerator(
                     operationNewArrayWithSize, locations[0].directGPR(),
-                    CCallHelpers::TrustedImmPtr(structure), CCallHelpers::TrustedImm32(numElements));
-            });
-        ValueFromBlock slowArray = m_out.anchor(slowArrayValue);
+                    locations[1].directGPR(), locations[2].directGPR(), locations[3].directGPR());
+            },
+            structureValue, publicLength, butterflyValue);
+        ValueFromBlock slowResult = m_out.anchor(slowResultValue);
         ValueFromBlock slowButterfly = m_out.anchor(
-            m_out.loadPtr(slowArrayValue, m_heaps.JSObject_butterfly));
-
+            m_out.loadPtr(slowResultValue, m_heaps.JSObject_butterfly));
         m_out.jump(continuation);
         
         m_out.appendTo(continuation, lastNext);
-        
         return ArrayValues(
-            m_out.phi(pointerType(), fastArray, slowArray),
+            m_out.phi(pointerType(), fastResult, slowResult),
             m_out.phi(pointerType(), fastButterfly, slowButterfly));
     }
     
+    ArrayValues allocateUninitializedContiguousJSArray(LValue publicLength, Structure* structure)
+    {
+        bool shouldInitializeElements = false;
+        bool shouldLargeArraySizeCreateArrayStorage = false;
+        return allocateJSArray(
+            publicLength, structure, shouldInitializeElements,
+            shouldLargeArraySizeCreateArrayStorage);
+    }
+    
     LValue ensureShadowChickenPacket()
     {
         LBasicBlock slowCase = m_out.newBlock();
index fea12dc..2b1f163 100644 (file)
@@ -100,7 +100,9 @@ SlotBaseValue* Output::lockedStackSlot(size_t bytes)
 
 LValue Output::constBool(bool value)
 {
-    return m_block->appendNew<B3::Const32Value>(m_proc, origin(), value);
+    if (value)
+        return booleanTrue;
+    return booleanFalse;
 }
 
 LValue Output::constInt32(int32_t value)
@@ -125,6 +127,10 @@ LValue Output::phi(LType type)
 
 LValue Output::add(LValue left, LValue right)
 {
+    if (Value* result = left->addConstant(m_proc, right)) {
+        m_block->append(result);
+        return result;
+    }
     return m_block->appendNew<B3::Value>(m_proc, B3::Add, origin(), left, right);
 }
 
@@ -205,17 +211,32 @@ LValue Output::bitXor(LValue left, LValue right)
 
 LValue Output::shl(LValue left, LValue right)
 {
-    return m_block->appendNew<B3::Value>(m_proc, B3::Shl,&nbs