592d6829345b403d30da249a3b64710a5b06b0d7
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-03-02  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc: Eagerly remove allocated objects from the free list
4         https://bugs.webkit.org/show_bug.cgi?id=142194
5
6         Reviewed by Andreas Kling.
7
8         This reduces the pressure to garbage collect the free list.
9
10         Might be a 1% speedup on MallocBench.
11
12         * bmalloc/FreeList.cpp: Put this comment at the top of the file instead
13         of repeating it inside of each function. Tried to clarify the details.
14
15         (bmalloc::FreeList::takeGreedy): Matched the other iteration code in this
16         file for consistency -- even though either direction works fine in this
17         function.
18
19         (bmalloc::FreeList::take): Change to iterate from low to high so that we
20         can maintain an index into the vector that is not disturbed even if we
21         pop from the middle (which invalidates the last index in the vector).
22
23         Decrement i when popping from the middle to make sure that we don't
24         skip the next item after popping.
25
26         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): Ditto.
27
28 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
29
30         Fixed a typo in the previous commit.
31
32         * bmalloc/BoundaryTag.h:
33         (bmalloc::BoundaryTag::setOwner):
34
35 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
36
37         EFL build fix after r180797.
38
39         * bmalloc/BoundaryTag.h:
40         (bmalloc::BoundaryTag::owner):
41         (bmalloc::BoundaryTag::setOwner):
42
43 2015-02-27  Geoffrey Garen  <ggaren@apple.com>
44
45         bmalloc: Pathological madvise churn on the free(malloc(x)) benchmark
46         https://bugs.webkit.org/show_bug.cgi?id=142058
47
48         Reviewed by Andreas Kling.
49
50         The churn was caused by repeatedly splitting an object with physical
51         pages from an object without, and then merging them back together again.
52         The merge would conservatively forget that we had physical pages, forcing
53         a new call to madvise on the next allocation.
54
55         This patch more strictly segregates objects in the heap from objects in
56         the VM heap, with these changes:
57
58         (1) Objects in the heap are not allowed to merge with objects in the VM
59         heap, and vice versa -- since that would erase our precise knowledge of
60         which physical pages had been allocated.
61
62         (2) The VM heap is exclusively responsible for allocating and deallocating
63         physical pages.
64
65         (3) The heap free list must consider entries for objects that are in the
66         VM heap to be invalid, and vice versa. (This condition can arise
67         because the free list does not eagerly remove items.)
68
69         With these changes, we can know that any valid object in the heap's free
70         list already has physical pages, and does not need to call madvise.
71
72         Note that the VM heap -- as before -- might sometimes contain ranges
73         or pieces of ranges that have physical pages, since we allow splitting
74         of ranges at granularities smaller than the VM page size. These ranges
75         can eventually merge with ranges in the heap during scavenging.
76
77         * bmalloc.xcodeproj/project.pbxproj:
78
79         * bmalloc/BoundaryTag.h:
80         (bmalloc::BoundaryTag::owner):
81         (bmalloc::BoundaryTag::setOwner):
82         (bmalloc::BoundaryTag::initSentinel):
83         (bmalloc::BoundaryTag::hasPhysicalPages): Deleted.
84         (bmalloc::BoundaryTag::setHasPhysicalPages): Deleted. Replaced the concept
85         of "has physical pages" with a bit indicating which heap owns the large
86         object. This is a more precise concept, since the old bit was really a
87         Yes / Maybe bit.
88
89         * bmalloc/Deallocator.cpp:
90
91         * bmalloc/FreeList.cpp: Adopt
92         (bmalloc::FreeList::takeGreedy):
93         (bmalloc::FreeList::take):
94         (bmalloc::FreeList::removeInvalidAndDuplicateEntries):
95         * bmalloc/FreeList.h:
96         (bmalloc::FreeList::push): Added API for considering the owner when
97         deciding if a free list entry is valid.
98
99         * bmalloc/Heap.cpp:
100         (bmalloc::Heap::Heap): Adopt new API.
101
102         (bmalloc::Heap::scavengeLargeRanges): Scavenge all ranges with no minimum,
103         since some ranges might be able to merge with ranges in the VM heap, and
104         they won't be allowed to until we scavenge them.
105
106         (bmalloc::Heap::allocateSmallPage):
107         (bmalloc::Heap::allocateMediumPage):
108         (bmalloc::Heap::allocateLarge): New VM heap API makes this function
109         simpler, since we always get back physical pages now.
110
111         * bmalloc/Heap.h:
112         * bmalloc/LargeObject.h:
113         (bmalloc::LargeObject::end):
114         (bmalloc::LargeObject::owner):
115         (bmalloc::LargeObject::setOwner):
116         (bmalloc::LargeObject::isValidAndFree):
117         (bmalloc::LargeObject::merge): Do not merge objects across heaps since
118         that causes madvise churn.
119         (bmalloc::LargeObject::validateSelf):
120         (bmalloc::LargeObject::init):
121         (bmalloc::LargeObject::hasPhysicalPages): Deleted.
122         (bmalloc::LargeObject::setHasPhysicalPages): Deleted. Propogate the Owner API.
123
124         * bmalloc/Owner.h: Added.
125
126         * bmalloc/SegregatedFreeList.cpp:
127         (bmalloc::SegregatedFreeList::SegregatedFreeList):
128         (bmalloc::SegregatedFreeList::insert):
129         (bmalloc::SegregatedFreeList::takeGreedy):
130         (bmalloc::SegregatedFreeList::take):
131         * bmalloc/SegregatedFreeList.h: Propogate the owner API.
132
133         * bmalloc/VMAllocate.h:
134         (bmalloc::vmDeallocatePhysicalPagesSloppy):
135         (bmalloc::vmAllocatePhysicalPagesSloppy): Clarified these functions and
136         removed an edge case.
137
138         * bmalloc/VMHeap.cpp:
139         (bmalloc::VMHeap::VMHeap):
140         * bmalloc/VMHeap.h:
141         (bmalloc::VMHeap::allocateSmallPage):
142         (bmalloc::VMHeap::allocateMediumPage):
143         (bmalloc::VMHeap::allocateLargeObject):
144         (bmalloc::VMHeap::deallocateLargeObject): Be sure to give each object
145         a new chance to merge, since it might have been prohibited from merging
146         before by virtue of not being in the VM heap.
147
148         (bmalloc::VMHeap::allocateLargeRange): Deleted.
149         (bmalloc::VMHeap::deallocateLargeRange): Deleted.
150
151 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
152
153         bmalloc: Large object free list can grow infinitely
154         https://bugs.webkit.org/show_bug.cgi?id=142055
155
156         Reviewed by Andreas Kling.
157
158         By design, we don't eagerly remove large objects from the free list.
159         This creates two simple pathologies:
160
161             (1) If you free and then allocate the same object repeatedly, it will
162             duplicate itself in the free list repeatedly. Since it is never
163             invalid at the time of allocation, it will never be removed.
164
165             (2) If you split and then merge the same object repeatedly, it will
166             duplicate its split sibling in the free list repeatedly. If its
167             sibling is in a separate free list size class, it will never be
168             consulted at the time of allocation, so it will never be removed.
169
170         So, a simple "while (1) { free(malloc(x)); }" causes infinite memory
171         use in the free list.
172
173         The solution in this patch is a simple helper to remove garbage from the
174         free list if it grows too large. This pathology is not common, so the
175         cost is OK.
176
177         Long-term, perhaps we should rethink the laziness of these free lists.
178
179         * bmalloc/BoundaryTag.h:
180         (bmalloc::BoundaryTag::isMarked):
181         (bmalloc::BoundaryTag::setMarked): New bit, used by free list GC.
182
183         * bmalloc/FreeList.cpp:
184         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): The GC algorithm.
185
186         * bmalloc/FreeList.h:
187         (bmalloc::FreeList::FreeList):
188         (bmalloc::FreeList::push): Invoke the GC if we're getting huge.
189
190         * bmalloc/LargeObject.h:
191         (bmalloc::LargeObject::isMarked):
192         (bmalloc::LargeObject::setMarked):
193         (bmalloc::LargeObject::validateSelf): Expose the new bit.
194
195         * bmalloc/Sizes.h: New constant to control GC frequency.
196
197 2015-02-26  Csaba Osztrogonác  <ossy@webkit.org>
198
199         URTBF after r180693.
200
201         * CMakeLists.txt:
202
203 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
204
205         Try to fix the Mac build.
206
207         Unreviewed.
208
209         * bmalloc.xcodeproj/project.pbxproj: Make FreeList.h available.
210
211 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
212
213         bmalloc: Refactored SegregatedFreeList and BoundaryTag::init
214         https://bugs.webkit.org/show_bug.cgi?id=142049
215
216         Reviewed by Anders Carlsson.
217
218         Split out a FreeList class from SegregatedFreeList. This will make it
219         easier to add behaviors on free list insertion and removal -- and it's
220         probably how I should have designed things at the start.
221
222         Moved BoundaryTag::init into LargeObject, since all the related logic
223         lives in LargeObject now too, and this allows us to remove BoundaryTagInlines.h.
224
225         * bmalloc.xcodeproj/project.pbxproj:
226         * bmalloc/BoundaryTagInlines.h: Removed.
227         * bmalloc/FreeList.cpp: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.cpp.
228         (bmalloc::FreeList::takeGreedy):
229         (bmalloc::FreeList::take):
230         (bmalloc::SegregatedFreeList::SegregatedFreeList): Deleted.
231         (bmalloc::SegregatedFreeList::insert): Deleted.
232         (bmalloc::SegregatedFreeList::takeGreedy): Deleted.
233         (bmalloc::SegregatedFreeList::take): Deleted.
234         * bmalloc/FreeList.h: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.h.
235         (bmalloc::FreeList::push):
236         * bmalloc/LargeObject.h:
237         (bmalloc::LargeObject::init):
238         * bmalloc/SegregatedFreeList.cpp:
239         (bmalloc::SegregatedFreeList::SegregatedFreeList):
240         (bmalloc::SegregatedFreeList::insert):
241         (bmalloc::SegregatedFreeList::takeGreedy):
242         (bmalloc::SegregatedFreeList::take):
243         * bmalloc/SegregatedFreeList.h:
244         * bmalloc/Sizes.h:
245         * bmalloc/VMHeap.cpp:
246         (bmalloc::VMHeap::grow):
247
248 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
249
250         bmalloc: free up a bit in BoundaryTag
251         https://bugs.webkit.org/show_bug.cgi?id=142048
252
253         Reviewed by Brady Eidson.
254
255         We were wasting a bit by accident, and I need one now.
256
257         * bmalloc/Algorithm.h:
258         (bmalloc::rightShift): Deleted. Not needed, now that I've simplified
259         the math.
260
261         * bmalloc/BoundaryTag.h: Since each boundary tag bucket is 1024 bytes
262         long, the maximum offset into a bucket is 1023.
263
264         You need 5 bits to count up to 1024, but only 4 to count up to 1023.
265         
266         Math is hard.
267
268         (bmalloc::BoundaryTag::compactBegin): Switched to division because it
269         is simpler, and easier to match up with our ASSERT. The compiler will
270         turn division by constant power of two into a shift for us.
271
272         (bmalloc::BoundaryTag::setRange): Added an ASSERT for compactBegin
273         because we do encode it, so we should ASSERT that encoding did not
274         lose information.
275
276         * bmalloc/Sizes.h: Shifting is no longer used since we use division
277         instead.
278
279 2015-02-24  Stephanie Lewis  <slewis@apple.com>
280
281         Rolling out http://trac.webkit.org/changeset/180430 as it causes the PLT to crash.
282         <rdar://problem/19948015>
283
284         Unreviewed.
285
286         * bmalloc/VMHeap.cpp:
287         (bmalloc::VMHeap::grow):
288         * bmalloc/VMHeap.h:
289         * bmalloc/Zone.cpp:
290         (bmalloc::Zone::Zone):
291         (bmalloc::Zone::size): Deleted.
292         * bmalloc/Zone.h:
293
294 2015-02-24  Geoffrey Garen  <ggaren@apple.com>
295
296         bmalloc: Added a little more abstraction for large objects
297         https://bugs.webkit.org/show_bug.cgi?id=141978
298
299         Reviewed by Sam Weinig.
300
301         Previously, each client needed to manage the boundary tags of
302         a large object using free functions. This patch introduces a LargeObject
303         class that does things a little more automatically.
304
305         * bmalloc.xcodeproj/project.pbxproj:
306
307         * bmalloc/Allocator.cpp:
308         (bmalloc::Allocator::reallocate): Use the new LargeObject class.
309
310         * bmalloc/BeginTag.h:
311         (bmalloc::BeginTag::isInFreeList): Deleted. Moved this logic into the
312         LargeObject class.
313
314         * bmalloc/BoundaryTag.h:
315         (bmalloc::BoundaryTag::isSentinel):
316         (bmalloc::BoundaryTag::compactBegin):
317         (bmalloc::BoundaryTag::setRange):
318         (bmalloc::BoundaryTag::initSentinel): Added an explicit API for sentinels,
319         which we used to create and test for implicitly.
320
321         * bmalloc/BoundaryTagInlines.h:
322         (bmalloc::BoundaryTag::init):
323         (bmalloc::validate): Deleted.
324         (bmalloc::validatePrev): Deleted.
325         (bmalloc::validateNext): Deleted.
326         (bmalloc::BoundaryTag::mergeLeft): Deleted.
327         (bmalloc::BoundaryTag::mergeRight): Deleted.
328         (bmalloc::BoundaryTag::merge): Deleted.
329         (bmalloc::BoundaryTag::deallocate): Deleted.
330         (bmalloc::BoundaryTag::split): Deleted.
331         (bmalloc::BoundaryTag::allocate): Deleted. Moved this logic into the
332         LargeObject class.
333
334         * bmalloc/EndTag.h:
335         (bmalloc::EndTag::init):
336         (bmalloc::EndTag::operator=): Deleted. Re-reading this code, I found
337         special behavior in the assignment operator to be a surprising API.
338         So, I replaced the assignment operation with an explicit initializing
339         function.
340
341         * bmalloc/Heap.cpp:
342         (bmalloc::Heap::scavengeLargeRanges):
343         (bmalloc::Heap::allocateXLarge):
344         (bmalloc::Heap::findXLarge):
345         (bmalloc::Heap::deallocateXLarge):
346         (bmalloc::Heap::allocateLarge):
347         (bmalloc::Heap::deallocateLarge):
348         * bmalloc/Heap.h: No behavior changes here -- just adopting the
349         LargeObject interface.
350
351         * bmalloc/LargeObject.h: Added.
352         (bmalloc::LargeObject::operator!):
353         (bmalloc::LargeObject::begin):
354         (bmalloc::LargeObject::size):
355         (bmalloc::LargeObject::range):
356         (bmalloc::LargeObject::LargeObject):
357         (bmalloc::LargeObject::setFree):
358         (bmalloc::LargeObject::isFree):
359         (bmalloc::LargeObject::hasPhysicalPages):
360         (bmalloc::LargeObject::setHasPhysicalPages):
361         (bmalloc::LargeObject::isValidAndFree):
362         (bmalloc::LargeObject::merge):
363         (bmalloc::LargeObject::split):
364         (bmalloc::LargeObject::validateSelf):
365         (bmalloc::LargeObject::validate): Moved this code into a class, out of
366         BoundaryTag free functions.
367
368         New to the class are these features:
369
370             (1) Every reference to an object is validated upon creation and use.
371
372             (2) There's an explicit API for "This is a reference to an object
373             that might be stale (the DoNotValidate API)".
374
375             (3) The begin and end tags are kept in sync automatically.
376
377         * bmalloc/SegregatedFreeList.cpp:
378         (bmalloc::SegregatedFreeList::insert):
379         (bmalloc::SegregatedFreeList::takeGreedy):
380         (bmalloc::SegregatedFreeList::take):
381         * bmalloc/SegregatedFreeList.h: Adopt the LargeObject interface.
382
383         * bmalloc/VMHeap.cpp:
384         (bmalloc::VMHeap::grow):
385         * bmalloc/VMHeap.h:
386         (bmalloc::VMHeap::allocateLargeRange):
387         (bmalloc::VMHeap::deallocateLargeRange): Adopt the LargeObject interface.
388
389 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
390
391         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
392         https://bugs.webkit.org/show_bug.cgi?id=141802
393
394         Reviewed by Andreas Kling.
395
396         Rolling back in with a fix for a crash seen while using GuardMalloc.
397
398         * bmalloc/VMHeap.cpp:
399         (bmalloc::VMHeap::grow):
400         * bmalloc/VMHeap.h:
401         * bmalloc/Zone.cpp: Re-land the old patch.
402
403         (bmalloc::Zone::size): Be sure to implement the size() function since
404         it's accessible indirectly via the malloc_zone_from_ptr public API --
405         and GuardMalloc calls it all the time.
406
407         (bmalloc::Zone::Zone):
408         * bmalloc/Zone.h: Re-land the old patch.
409
410 2015-02-19  Commit Queue  <commit-queue@webkit.org>
411
412         Unreviewed, rolling out r180363.
413         https://bugs.webkit.org/show_bug.cgi?id=141814
414
415         Caused >50 crashes when running LayoutTests in GuardMalloc or
416         ASAN modes. (Requested by jernoble on #webkit).
417
418         Reverted changeset:
419
420         "bmalloc should implement malloc introspection (to stop false-
421         positive leaks when MallocStackLogging is off)"
422         https://bugs.webkit.org/show_bug.cgi?id=141802
423         http://trac.webkit.org/changeset/180363
424
425 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
426
427         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
428         https://bugs.webkit.org/show_bug.cgi?id=141802
429
430         Reviewed by Andreas Kling.
431
432         Fixed a last-minute type.
433
434         The macro is OS, not PLATFORM.
435
436         * bmalloc/VMHeap.cpp:
437         (bmalloc::VMHeap::grow):
438         * bmalloc/VMHeap.h:
439         * bmalloc/Zone.h:
440
441 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
442
443         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
444         https://bugs.webkit.org/show_bug.cgi?id=141802
445
446         Reviewed by Andreas Kling.
447
448         This patch does the bare minimum to stop false positive leaks from
449         being reported by the Darwin leaks tool. We register each super chunk
450         as a single object, and then request that the leaks tool scan it.
451
452         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
453         zone introspection API.
454
455         * bmalloc/Algorithm.h: Missing #include.
456
457         * bmalloc/VMHeap.cpp:
458         (bmalloc::VMHeap::grow):
459         * bmalloc/VMHeap.h: Adopt the new abstraction.
460
461         * bmalloc/Zone.cpp: Added.
462         (bmalloc::remoteRead): Helper for reading an object out of another process.
463         (bmalloc::Zone::enumerator):
464         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
465         in introspection.
466
467         * bmalloc/Zone.h: Added.
468         (bmalloc::Zone::superChunks):
469         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
470         since our dynamic allocations will not be scanned by leaks since they
471         will have the malloc VM tag.
472
473 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
474
475         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
476         https://bugs.webkit.org/show_bug.cgi?id=141759
477
478         Reviewed by Andreas Kling.
479
480         * bmalloc.xcodeproj/project.pbxproj:
481         * bmalloc/SuperChunk.h: Added.
482         (bmalloc::SuperChunk::create):
483         (bmalloc::SuperChunk::SuperChunk):
484         (bmalloc::SuperChunk::smallChunk):
485         (bmalloc::SuperChunk::mediumChunk):
486         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
487         into a separate class, for clarity and type safety.
488
489         * bmalloc/VMHeap.cpp:
490         (bmalloc::VMHeap::grow):
491         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
492         "grow" because Andreas found "allocateSuperChunk" to be unclear.
493
494         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
495         for malloc introspection.
496
497         (bmalloc::VMHeap::allocateSmallPage):
498         (bmalloc::VMHeap::allocateMediumPage):
499         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
500
501 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
502
503         Build bmalloc through CMake as a static library. It's then linked either
504         into the WTF library (if built as a shared library) or into the JSC and
505         WebKit2 libraries. There's no need to build it as a standalone shared library.
506
507         Rubber-stamped by Carlos Garcia Campos.
508
509         * CMakeLists.txt:
510
511 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
512
513         [BMalloc] Add a FIXME comment for memory alignas
514         https://bugs.webkit.org/show_bug.cgi?id=141556
515
516         Reviewed by Csaba Osztrogonác.
517
518         * bmalloc/Chunk.h: Add a FIXME comment.
519         * bmalloc/LargeChunk.h: ditto.
520
521 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
522
523         bmalloc buildfix on 32 bit Linux (x86/ARM)
524         https://bugs.webkit.org/show_bug.cgi?id=141472
525
526         Reviewed by Gyuyoung Kim.
527
528         * bmalloc/Algorithm.h:
529         (bmalloc::roundUpToMultipleOf):
530         * bmalloc/FixedVector.h:
531         (bmalloc::FixedVector::clear):
532         * bmalloc/Sizes.h:
533         (bmalloc::Sizes::sizeClass):
534
535 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
536
537         [EFL][GTK] Use bmalloc instead of tcmalloc
538         https://bugs.webkit.org/show_bug.cgi?id=140162
539
540         Reviewed by Carlos Garcia Campos.
541
542         Support to use bmalloc on EFL and GTK ports.
543
544         * CMakeLists.txt: Added.
545         * bmalloc/Allocator.cpp:
546         (bmalloc::Allocator::allocate):
547             Fix unused return value caused by posix_memalign().
548         * bmalloc/AsyncTask.h:
549         * bmalloc/BoundaryTag.h:
550         (bmalloc::BoundaryTag::clear):
551         * bmalloc/Chunk.h:
552             Change Traits::Page with Traits::PageType in order to fix
553             -fpermitive build error on EFL and GTK port. 
554         * bmalloc/EndTag.h:
555         (bmalloc::EndTag::operator=):
556         * bmalloc/Line.h: ditto.
557         * bmalloc/MediumTraits.h:
558         * bmalloc/Page.h: ditto.
559         * bmalloc/PerThread.h:
560             EFL port doesn't support __has_include definition yet.
561             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
562         * bmalloc/SmallTraits.h: ditto.
563         * bmalloc/VMAllocate.h:
564         (bmalloc::vmDeallocatePhysicalPages):
565         (bmalloc::vmAllocatePhysicalPages):
566         * bmalloc/Vector.h:
567         (bmalloc::Vector<T>::push):
568         (bmalloc::Vector<T>::reallocateBuffer):
569
570 2015-01-31  Sam Weinig  <sam@webkit.org>
571
572         Remove even more Mountain Lion support
573         https://bugs.webkit.org/show_bug.cgi?id=141124
574
575         Reviewed by Alexey Proskuryakov.
576
577         * Configurations/Base.xcconfig:
578         * Configurations/DebugRelease.xcconfig:
579
580 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
581
582         GC marking threads should clear malloc caches
583         https://bugs.webkit.org/show_bug.cgi?id=141097
584
585         Reviewed by Andreas Kling.
586
587         Split the scavenging API into per-thread vs global, so that you can
588         request to scavenge your own thread without scavenging the whole heap.
589
590         * bmalloc/Cache.cpp:
591         (bmalloc::Cache::scavenge):
592         * bmalloc/bmalloc.h:
593         (bmalloc::api::scavengeThisThread):
594         (bmalloc::api::scavenge):
595
596 2015-01-28  Dana Burkart  <dburkart@apple.com>
597
598         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
599         https://bugs.webkit.org/show_bug.cgi?id=136765
600
601         Reviewed by Alexey Proskuryakov.
602
603         * Configurations/Base.xcconfig:
604         * Configurations/DebugRelease.xcconfig:
605
606 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
607
608         bmalloc: support aligned allocation
609         https://bugs.webkit.org/show_bug.cgi?id=140732
610
611         Reviewed by Andreas Kling.
612
613         * bmalloc/Allocator.cpp:
614         (bmalloc::Allocator::allocate): New function for aligned allocation.
615
616         Small and medium requests just allocate and free until they find an
617         aligned pointer. This is slightly inefficient in the worst case, but
618         still constant-time with little-to-no space overhead.
619
620         Large requests use a new API that requires the client to specify both
621         its ideal size and alignment, and the worst-case size you would have to
622         allocate in order to produce some interior pointer of the requested size
623         and alignment. We put the burden of this calculation on the client
624         because it simplifies things if we guarantee that allocation won't fail.
625
626         XLarge requests are easy: we just forward them to vmAllocate, which
627         already supported aligned requests.
628
629         * bmalloc/BoundaryTag.h:
630         * bmalloc/BoundaryTagInlines.h:
631         (bmalloc::BoundaryTag::mergeLeft):
632         (bmalloc::BoundaryTag::mergeRight):
633         (bmalloc::BoundaryTag::merge):
634         (bmalloc::BoundaryTag::deallocate):
635         (bmalloc::BoundaryTag::split):
636         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
637         refactored the interface to remove some reference out parameters in
638         order to clarify what changes and what doesn't.
639
640         * bmalloc/Heap.cpp:
641         (bmalloc::Heap::allocateXLarge): Added an alignment API.
642
643         (bmalloc::Heap::allocateLarge):
644         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
645         a few variants, so aligned and unaligned allocation could share some code.
646
647         * bmalloc/SegregatedFreeList.cpp:
648         (bmalloc::SegregatedFreeList::take):
649         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
650         for aligned allocation. It turns out that the aligned path is pretty
651         different, since it ends up searching for two potential ways to satisfy
652         an allocation: either large enough and aligned, or large enough to split
653         into something not aligned and something large enough and aligned.
654
655         * bmalloc/VMAllocate.h:
656         (bmalloc::vmAllocate): Switched alignment to come before size because
657         that's how the memalign API specifies it.
658
659         * bmalloc/VMHeap.h:
660         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
661
662 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
663
664         bmalloc: a little bit of cleanup
665         https://bugs.webkit.org/show_bug.cgi?id=140687
666
667         Reviewed by Anders Carlsson.
668
669         * bmalloc/Algorithm.h:
670         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
671         of code.
672
673         * bmalloc/BoundaryTag.h:
674         * bmalloc/BoundaryTagInlines.h:
675         (bmalloc::BoundaryTag::mergeLeft):
676         (bmalloc::BoundaryTag::mergeRight):
677         (bmalloc::BoundaryTag::merge):
678         (bmalloc::BoundaryTag::deallocate):
679         (bmalloc::BoundaryTag::split):
680         (bmalloc::BoundaryTag::allocate):
681         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
682         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
683         (bmalloc::BoundaryTag::mergeLarge): Deleted.
684         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
685         from all these functions, since boundary tags always pertain to large
686         objects, and putting the word "Large" everywhere wasn't helping to
687         explain that.
688
689         * bmalloc/Heap.cpp:
690         (bmalloc::Heap::allocateXLarge):
691         (bmalloc::Heap::findXLarge):
692         (bmalloc::Heap::deallocateXLarge):
693         * bmalloc/Heap.h:
694         * bmalloc/VMHeap.h:
695         (bmalloc::VMHeap::allocateXLarge): Deleted.
696         (bmalloc::VMHeap::findXLarge): Deleted.
697         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
698         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
699         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
700         really make sense for the VMHeap to be involved.
701
702 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
703
704         bmalloc: refactored XLarge allocation for better alignment
705         https://bugs.webkit.org/show_bug.cgi?id=140582
706
707         Reviewed by Andreas Kling.
708
709         XLarge objects used to be Large objects with an extra bit of metadata
710         that said "actually, I'm not large -- I'm extra large".
711
712         The metadata header in an XLarge allocation made it impossible for the
713         XLarge object to honor a very large alignment request.
714
715         The solution is to stop using a metadata header for XLarge objects, and
716         instead to store explicit metadata on the side.
717
718         This is a bit less astonishing, which is also nice.
719
720         Finding XLarge metadata is now a linear search. That's probably OK, since
721         it was always so in TCMalloc, and the usual number of XLarge allocations
722         in a process is 0.
723
724         This design makes it possible for the heap to cache XLarge allocations
725         with and/or without physical pages. I haven't actually done that yet
726         because the tradeoffs are subtle, so I don't want to do anything without
727         a motivating test case.
728
729         * bmalloc.xcodeproj/project.pbxproj:
730         * bmalloc/Allocator.cpp:
731         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
732         since an XLarge allocation is now just a naked buffer without a header.
733
734         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
735         XLarge alignment, since XLargeChunk won't give this to us implicitly
736         anymore.
737
738         * bmalloc/BoundaryTag.h:
739         (bmalloc::BoundaryTag::setRange):
740         (bmalloc::BoundaryTag::isXLarge): Deleted.
741         (bmalloc::BoundaryTag::setXLarge): Deleted.
742         * bmalloc/BoundaryTagInlines.h:
743         (bmalloc::validate):
744         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
745
746         * bmalloc/Deallocator.cpp:
747         (bmalloc::Deallocator::deallocateXLarge):
748         (bmalloc::Deallocator::deallocateSlowCase):
749         * bmalloc/Heap.cpp:
750         (bmalloc::Heap::findXLarge):
751         (bmalloc::Heap::allocateXLarge):
752         (bmalloc::Heap::deallocateXLarge):
753         * bmalloc/Heap.h: Updated for interface changes.
754
755         * bmalloc/ObjectType.cpp:
756         (bmalloc::objectType):
757         * bmalloc/ObjectType.h:
758         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
759         examining its bit pattern -- just like we do for other kinds of
760         allocations -- which is nice.
761
762         * bmalloc/Sizes.h:
763         * bmalloc/VMHeap.h:
764         (bmalloc::VMHeap::allocateXLarge):
765         (bmalloc::VMHeap::findXLarge):
766         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
767         for XLarge allocations.
768
769         * bmalloc/XLargeChunk.h: Removed.
770
771 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
772
773         bmalloc: added some infrastructure for aligned allocation
774         https://bugs.webkit.org/show_bug.cgi?id=140572
775
776         Reviewed by Andreas Kling.
777
778         * bmalloc/Algorithm.h:
779         (bmalloc::isPowerOfTwo):
780         (bmalloc::roundUpToMultipleOf):
781         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
782         isPowerOfTwo helper function.
783
784         * bmalloc/Allocator.cpp:
785         (bmalloc::Allocator::allocate):
786         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
787         Doesn't do anything yet, but does correctly forward to system malloc
788         when bmalloc is disabled.
789
790         * bmalloc/Cache.cpp:
791         (bmalloc::Cache::allocateSlowCaseNullCache):
792         * bmalloc/Cache.h:
793         (bmalloc::Cache::allocate):
794         * bmalloc/bmalloc.h:
795         (bmalloc::api::memalign):
796         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
797
798 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
799
800         Consider alignment when allocating from a SegregatedFreeList
801         https://bugs.webkit.org/show_bug.cgi?id=140408
802
803         Reviewed by Sam Weinig.
804
805         In preparation for supporting aligned allocation.
806
807         No performance change.
808
809         Since this is just one extra branch in an already expensive function,
810         I decided not to duplicate the function just to avoid the branch in
811         the un-aligned case.
812
813         * bmalloc/SegregatedFreeList.cpp:
814         (bmalloc::SegregatedFreeList::take):
815         * bmalloc/SegregatedFreeList.h:
816
817 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
818
819         Renamed minimum to size in SegregatedFreeList
820         https://bugs.webkit.org/show_bug.cgi?id=140406
821
822         Reviewed by Sam Weinig.
823
824         In preparation for supporting aligned allocation.
825
826         * bmalloc/SegregatedFreeList.cpp:
827         (bmalloc::SegregatedFreeList::takeGreedy):
828         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
829         really just a minimum. Let's not imply that this value is special.
830
831 2015-01-11  Dan Bernstein  <mitz@apple.com>
832
833         Geoff is organized, but he is not an organization.
834
835         Rubber-stamped by Anders Carlsson.
836
837         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
838
839 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
840
841         Make bmalloc work with ASan
842         https://bugs.webkit.org/show_bug.cgi?id=140194
843
844         Reviewed by Mark Lam.
845
846         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
847         an OS-specific API to test for loaded runtime libraries.
848
849         * bmalloc/Environment.cpp:
850         (bmalloc::isASanEnabled):
851         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
852         ASan is enabled, since system malloc has the Asan hooks we need.
853
854         You could check for the ASan compile-time flag instead, but doing this
855         check at runtime prepares bmalloc for a world where it is a dynamic
856         library that might be loaded into projects it did not compile with.
857
858 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
859
860         Fix up bmalloc's PerThread for use on Linux
861         https://bugs.webkit.org/show_bug.cgi?id=139804
862
863         Reviewed by Anders Carlsson.
864
865         The previous implementation was a bit slow.
866
867         * bmalloc/PerThread.h:
868         (bmalloc::PerThreadStorage<Cache>::get):
869         (bmalloc::PerThreadStorage::get):
870         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
871         way to do fast per-thread access without taking a lock every time. This
872         probably works on all the platforms we care about, and it matches other
873         techniques we use elsewhere in WebKit.
874
875         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
876         this class because PerThreadStorage now encapsulates everything that
877         needs to be conditional.
878
879         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
880
881 2014-12-26  Dan Bernstein  <mitz@apple.com>
882
883         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
884         https://bugs.webkit.org/show_bug.cgi?id=139950
885
886         Reviewed by David Kilzer.
887
888         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
889         in a manner that works with Xcode 5.1.1.
890
891 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
892
893         Safari crashes when you set Malloc environment variables
894         https://bugs.webkit.org/show_bug.cgi?id=139656
895
896         Reviewed by Michael Saboff.
897
898         I forgot to cover the realloc() case. Whoops. (OoPS?)
899
900         This time around, I ran the full MallocBench test suite in Malloc=1
901         mode, and it passed.
902
903         * bmalloc/Allocator.cpp:
904         (bmalloc::Allocator::reallocate):
905         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
906         It needs to be down there so that we can do the short-circuiting check
907         for whether bmalloc is enabled first.
908
909         Also added the check.
910
911         * bmalloc/Cache.cpp:
912         (bmalloc::Cache::scavenge):
913         (bmalloc::Cache::Cache):
914         (bmalloc::Cache::reallocateSlowCaseNullCache):
915         * bmalloc/Cache.h:
916         (bmalloc::Cache::deallocator):
917         (bmalloc::Cache::reallocate): Ditto.
918
919         * bmalloc/bmalloc.h:
920         (bmalloc::api::free):
921         (bmalloc::api::realloc): Ditto.
922
923         (bmalloc::api::scavenge): Pushed this down into Cache to match the
924         surrounding functions.
925
926 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
927
928         bmalloc should support system memory analysis tools (part 2)
929         https://bugs.webkit.org/show_bug.cgi?id=139565
930
931         Reviewed by Mark Lam.
932
933         This patch actually queries the environment to see if memory analysis
934         tools have been enabled.
935
936         * bmalloc/Deallocator.cpp:
937         (bmalloc::Deallocator::scavenge): Don't process the object log if
938         we've disabled bmalloc because it will be full of invalid nullptrs.
939
940         * bmalloc/Environment.cpp:
941         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
942         Malloc debugging flags. I also added a plain "Malloc" catch-all for
943         when you want to disable bmalloc without enabling any kind of funny
944         business.
945
946         It would be slightly nicer just to iterate the list of environment
947         variables and strstr them, but getenv is the more portable option,
948         and performance here doesn't really matter.
949
950         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
951         environment variable.
952
953         (bmalloc::Environment::computeIsBmallocEnabled):
954
955 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
956
957         Try to fix the iOS simulator build.
958
959         #include the declaration of malloc / free.
960
961         * bmalloc/Allocator.cpp:
962         * bmalloc/Deallocator.cpp:
963
964 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
965
966         Try to fix the build.
967
968         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
969
970 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
971
972         bmalloc should support system memory analysis tools (part 1)
973         https://bugs.webkit.org/show_bug.cgi?id=139559
974
975         Reviewed by Mark Lam.
976
977         This patch adds the hooks to disable bmalloc at runtime if certain
978         environment variables are set, but doesn't actually read from the
979         environment yet.
980
981         No performance change.
982
983         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
984         we'll use to read environment variables and see if memory analysis tools
985         have been enabled.
986
987         * bmalloc/Allocator.cpp:
988         (bmalloc::Allocator::Allocator):
989         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
990         on the allocation path. We cache the setting to make the check fast.
991
992         * bmalloc/Allocator.h: Interface changes.
993
994         * bmalloc/Cache.cpp:
995         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
996         and deallocator. This main purpose is to enable them to query the
997         environment for whether bmalloc is enabled; but this is also a slightly
998         cleaner way to guarantee to them that the Heap has been pre-initialized.
999
1000         * bmalloc/Deallocator.cpp:
1001         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
1002         fill the object log to force us to take the slow path on all deallocations.
1003
1004         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
1005
1006         * bmalloc/Deallocator.h: Interface changes.
1007
1008         * bmalloc/Environment.cpp: Added.
1009         (bmalloc::Environment::Environment):
1010         (bmalloc::Environment::computeIsBmallocEnabled):
1011         * bmalloc/Environment.h: Added.
1012         (bmalloc::Environment::isBmallocEnabled): This is the class that will
1013         encapsulate looking for environment variables that turn on heap
1014         analysis tools.
1015
1016         * bmalloc/Heap.h:
1017         (bmalloc::Heap::environment):
1018
1019         * bmalloc/Mutex.h:
1020         (bmalloc::Mutex::Mutex):
1021         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
1022         since I got super confused about them while writing this patch.
1023
1024         * bmalloc/VMHeap.cpp: Fixed an #include.
1025
1026 2014-12-09  David Kilzer  <ddkilzer@apple.com>
1027
1028         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
1029         <http://webkit.org/b/139212>
1030
1031         Reviewed by Joseph Pecoraro.
1032
1033         * Configurations/Base.xcconfig:
1034         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
1035           on OS X.
1036         * Configurations/DebugRelease.xcconfig:
1037         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
1038
1039 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
1040
1041         bmalloc uses 8X more virtual memory than necessary
1042         https://bugs.webkit.org/show_bug.cgi?id=138495
1043
1044         Reviewed by Mark Lam.
1045
1046         iOS has a per-process virtual memory cap around 1GB, so there's some
1047         value to not going totally ham with virtual memory.
1048
1049         We currently use about 8X the necessary amount:
1050             - 2X to align our VM allocation
1051             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
1052
1053         We can cut that down:
1054             - Return the unaligned portion of our VM allocation (-2X)
1055             - Use all the chunks in a superchunk, instead of allocating one
1056               chunk per superchunk (-4X)
1057
1058         * bmalloc/Algorithm.h:
1059         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
1060         function so we can call it with getpagesize() at runtime.
1061
1062         * bmalloc/Chunk.h:
1063         * bmalloc/LargeChunk.h:
1064         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
1065         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
1066
1067         * bmalloc/VMAllocate.h:
1068         (bmalloc::vmValidate):
1069         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
1070         if it does not succeed. Allocate precisely, and give back the extra.
1071
1072         * bmalloc/VMHeap.cpp:
1073         (bmalloc::VMHeap::allocateSuperChunk):
1074         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
1075         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
1076         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
1077         in a superchunk, instead of just one.
1078
1079         * bmalloc/VMHeap.h:
1080         (bmalloc::VMHeap::allocateSmallPage):
1081         (bmalloc::VMHeap::allocateMediumPage):
1082         (bmalloc::VMHeap::allocateLargeRange):
1083         * bmalloc/XLargeChunk.h:
1084         (bmalloc::XLargeChunk::create): Updated to match changes above.
1085
1086 2014-11-01  David Kilzer  <ddkilzer@apple.com>
1087
1088         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
1089         <https://webkit.org/b/138286>
1090         <rdar://problem/18847087>
1091
1092         Reviewed by Dan Bernstein.
1093
1094         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
1095         to NO for the target that produces libbmalloc.a so that the
1096         debug symbols will be linked into JavaScriptCore and end up in
1097         its dSYM file.
1098
1099 2014-10-30  Dana Burkart  <dburkart@apple.com>
1100
1101         <rdar://problem/18821260> Prepare for the mysterious future
1102
1103         Reviewed by Lucas Forschler.
1104
1105         * Configurations/Base.xcconfig:
1106         * Configurations/DebugRelease.xcconfig:
1107
1108 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
1109
1110         bmalloc: cleaned up fast path vs slow path
1111         https://bugs.webkit.org/show_bug.cgi?id=137081
1112
1113         Reviewed by Sam Weinig.
1114
1115         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
1116
1117         * bmalloc/Allocator.cpp:
1118         (bmalloc::Allocator::Allocator): Merged the small and medium range
1119         caches, just like the small and medium allocators. Ranges are abstract
1120         objects that don't really care whether they hold small or medium objects,
1121         so they don't need to be segregated.
1122
1123         (bmalloc::Allocator::scavenge): Ditto.
1124
1125         (bmalloc::Allocator::allocateBumpRangeSlowCase):
1126         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
1127         we do care a tiny bit, because we need to specify small vs medium when
1128         allocating new ranges from the heap, to ensure that the heap allocates
1129         from the right segment of VM.
1130
1131         (bmalloc::Allocator::allocateLarge):
1132         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
1133         up the fast path. Large allocation performance is dominated by allocation
1134         logic and initialization, so inlining it doesn't help.
1135
1136         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
1137         it doesn't need to distinguish small vs medium objects.
1138
1139         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
1140         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
1141
1142         * bmalloc/Allocator.h:
1143         * bmalloc/BumpRange.h:
1144
1145         * bmalloc/Cache.cpp:
1146         (bmalloc::Cache::allocateSlowCase): Deleted.
1147         (bmalloc::Cache::deallocateSlowCase): Deleted.
1148         * bmalloc/Cache.h:
1149         (bmalloc::Cache::allocate):
1150         (bmalloc::Cache::deallocate):
1151         (bmalloc::Cache::allocateFastCase): Deleted.
1152         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
1153         paths. The downside to this change is that the fast path branches to two
1154         distinct failure cases instead of one. The upside is that the slow path
1155         doesn't need to re-read the segment register, which is not as cheap as a
1156         normal register, and it doesn't need to do an extra level of function 
1157         call. Seems to be worth it.
1158
1159         * bmalloc/Deallocator.h:
1160         * bmalloc/Heap.cpp:
1161         (bmalloc::Heap::refillSmallBumpRangeCache):
1162         (bmalloc::Heap::refillMediumBumpRangeCache):
1163         * bmalloc/Heap.h: Updated for interface changes.
1164
1165         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
1166         small lines in a page / 2, since any other free lines will coalesce
1167         with their neighbors.
1168
1169 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1170
1171         Rolled out r173346.
1172
1173             bmalloc should honor the FastMalloc statistics API
1174             https://bugs.webkit.org/show_bug.cgi?id=136592
1175
1176         This didn't really work. Because we allow ranges with and without
1177         physical pages to merge, and we allow double-committing and
1178         double-decommitting, we can't rely on commit actions to track memory
1179         footprint.
1180
1181         * bmalloc/Heap.cpp:
1182         (bmalloc::Heap::size): Deleted.
1183         (bmalloc::Heap::capacity): Deleted.
1184         * bmalloc/Heap.h:
1185         * bmalloc/VMHeap.cpp:
1186         (bmalloc::VMHeap::VMHeap):
1187         (bmalloc::VMHeap::allocateSmallChunk):
1188         (bmalloc::VMHeap::allocateMediumChunk):
1189         (bmalloc::VMHeap::allocateLargeChunk):
1190         * bmalloc/VMHeap.h:
1191         (bmalloc::VMHeap::allocateSmallPage):
1192         (bmalloc::VMHeap::allocateMediumPage):
1193         (bmalloc::VMHeap::allocateLargeRange):
1194         (bmalloc::VMHeap::deallocateSmallPage):
1195         (bmalloc::VMHeap::deallocateMediumPage):
1196         (bmalloc::VMHeap::deallocateLargeRange):
1197         (bmalloc::VMHeap::size): Deleted.
1198         (bmalloc::VMHeap::capacity): Deleted.
1199         * bmalloc/bmalloc.h:
1200         (bmalloc::api::heapSize): Deleted.
1201         (bmalloc::api::heapCapacity): Deleted.
1202
1203 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1204
1205         bmalloc: Allocation should be more precise
1206         https://bugs.webkit.org/show_bug.cgi?id=136993
1207
1208         Reviewed by Gavin Barraclough.
1209
1210         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1211
1212         This patch teaches the allocator to merge adjacent free lines into a
1213         single allocatable range. This allows us to shrink the size of an
1214         individual line without increasing fragmentation or the rate of allocator
1215         slow paths.
1216
1217         We'll only take more slow paths when available memory is sparse, which
1218         is exactly when it's worth it. When available memory is dense, we'll
1219         take fewer slow paths.
1220
1221         * bmalloc.xcodeproj/project.pbxproj:
1222         * bmalloc/Algorithm.h:
1223         (bmalloc::divideRoundingUp):
1224
1225         * bmalloc/Allocator.cpp:
1226         (bmalloc::Allocator::Allocator): Updated for interface changes.
1227
1228         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
1229         Now that we merge lines, it's not convenient to scavenge by line.
1230
1231         (bmalloc::Allocator::allocateSmallBumpRange):
1232         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
1233         instead of individual lines.
1234
1235         (bmalloc::Allocator::allocateSlowCase):
1236         (bmalloc::Allocator::allocateSmallLine): Deleted.
1237         (bmalloc::Allocator::allocateMediumLine): Deleted.
1238         (bmalloc::Allocator::allocateMedium): Deleted.
1239         * bmalloc/Allocator.h:
1240         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
1241         into the standard fast path with small allocations. Since a BumpAllocator
1242         just allocates out of an arbitrary range, it doesn't need to distinguish
1243         between small and medium lines.
1244
1245         * bmalloc/BumpAllocator.h:
1246         (bmalloc::BumpAllocator::size):
1247         (bmalloc::BumpAllocator::BumpAllocator):
1248         (bmalloc::BumpAllocator::init):
1249         (bmalloc::BumpAllocator::refill):
1250         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
1251         anymore: the heap just gives us a pointer and a pre-computed number of
1252         objects, and we allocate them.
1253
1254         * bmalloc/Deallocator.cpp:
1255         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
1256
1257         * bmalloc/Heap.cpp:
1258         (bmalloc::Heap::Heap):
1259         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
1260         detailing where all objects will lie in memory. After we merge two lines,
1261         we might allocate an object that spans from one line to the next. This
1262         metadata details which bits of memory overlap in that way, and how they
1263         overlap.
1264
1265         (bmalloc::Heap::refillSmallBumpRangeCache):
1266         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
1267         and merge adjacent free lines into BumpRanges.
1268
1269         (bmalloc::Heap::allocateSmallPage):
1270         (bmalloc::Heap::allocateMediumPage):
1271         (bmalloc::Heap::deallocateSmallLine):
1272         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
1273         since we scan for free memory a page at a time.
1274
1275         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
1276         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
1277         fast path.
1278
1279         * bmalloc/Heap.h:
1280         (bmalloc::Heap::derefSmallLine):
1281         (bmalloc::Heap::derefMediumLine):
1282         (bmalloc::Heap::deallocateSmallLine): Deleted.
1283         (bmalloc::Heap::allocateSmallLine): Deleted.
1284         (bmalloc::Heap::deallocateMediumLine): Deleted.
1285         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
1286
1287         * bmalloc/Line.h:
1288         (bmalloc::Line<Traits>::ref):
1289         (bmalloc::Line<Traits>::deref):
1290         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
1291         anymore, since we only ever deref by 1 now.
1292
1293         * bmalloc/MediumAllocator.h:
1294         (bmalloc::MediumAllocator::isNull): Deleted.
1295         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
1296         (bmalloc::MediumAllocator::line): Deleted.
1297         (bmalloc::MediumAllocator::allocate): Deleted.
1298         (bmalloc::MediumAllocator::derefCount): Deleted.
1299         (bmalloc::MediumAllocator::refill): Deleted.
1300         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
1301         been dead for a while, since it doesn't build anymore with this patch.
1302
1303         * bmalloc/Page.h:
1304         (bmalloc::Page::sizeClass):
1305         (bmalloc::Page::setSizeClass):
1306         (bmalloc::Page::smallSizeClass): Deleted.
1307         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
1308         to sizeClass, since we use it for medium sizes too.
1309
1310         * bmalloc/Sizes.h:
1311         (bmalloc::Sizes::sizeClass):
1312         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
1313
1314         (bmalloc::Sizes::smallSizeClassFor): Deleted.
1315         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
1316
1317         * bmalloc/bmalloc.h:
1318         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
1319         can be a bit more precise. It also has to be, since we can't guarantee
1320         that an object ends at the end of a line anymore.
1321
1322 2014-09-19  Daniel Bates  <dabates@apple.com>
1323
1324         Always assume internal SDK when building configuration Production
1325         https://bugs.webkit.org/show_bug.cgi?id=136925
1326         <rdar://problem/18362399>
1327
1328         Reviewed by Dan Bernstein.
1329
1330         * Configurations/Base.xcconfig:
1331
1332 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
1333
1334         bmalloc: moved line caches from the deallocator to the allocator
1335         https://bugs.webkit.org/show_bug.cgi?id=136868
1336
1337         Reviewed by Gavin Barraclough.
1338
1339         I did this mostly as a simplification, to make it easier to change the
1340         allocation strategy.
1341
1342         No throughput change on MallocBench. Saves about 50kB.
1343
1344         Since the deallocator needs to lock the heap when freeing lines anyway,
1345         there isn't much benefit to giving the deallocator a local cache of
1346         deallocated lines.
1347
1348         We still give the allocator a local cache of lines because that does
1349         reduce the frequency at which it needs to lock the heap in order to
1350         acquire more lines.
1351
1352         * bmalloc/Allocator.cpp:
1353         (bmalloc::Allocator::scavenge):
1354         (bmalloc::Allocator::allocateSmallLine):
1355         (bmalloc::Allocator::allocateMediumLine):
1356         (bmalloc::Allocator::allocateMedium):
1357         (bmalloc::Allocator::allocateSlowCase):
1358         * bmalloc/Allocator.h:
1359         * bmalloc/Deallocator.cpp:
1360         (bmalloc::Deallocator::Deallocator):
1361         (bmalloc::Deallocator::scavenge):
1362         (bmalloc::Deallocator::processObjectLog):
1363         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
1364         (bmalloc::Deallocator::allocateSmallLine): Deleted.
1365         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
1366         (bmalloc::Deallocator::allocateMediumLine): Deleted.
1367         * bmalloc/Deallocator.h:
1368
1369         * bmalloc/Sizes.h:
1370         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
1371         exactly one page in size. That's about what we were shooting for anyway,
1372         and it may make it easier to switch to per-page allocation in future.
1373
1374 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
1375
1376         bmalloc: allocate small and medium objects using the same bump pointer class
1377         https://bugs.webkit.org/show_bug.cgi?id=136843
1378
1379         Reviewed by Gavin Barraclough.
1380
1381         4% speedup on MallocBench.
1382
1383         Now that medium-sized objects have dedicated per-size allocators, they
1384         don't need to use an arbitrary bump pointer allocator. This means that
1385         every allocator knows how many objects it will allocate from the start,
1386         and we don't need a post-processing step to adjust refcounts based on
1387         real allocation count.
1388
1389         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1390         since it's used for small and medium objects now.
1391
1392         * bmalloc/Allocator.cpp:
1393         (bmalloc::Allocator::Allocator): Updated to use new interface.
1394         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1395         to make sure that we finish allocating all the objects in it.
1396
1397         (bmalloc::Allocator::allocateMedium):
1398         (bmalloc::Allocator::allocateSlowCase):
1399         (bmalloc::Allocator::retire): Deleted.
1400         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1401         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1402         * bmalloc/Allocator.h:
1403         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1404         used to post-process an allocator based on how many objects it allocated.
1405
1406         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1407         (bmalloc::BumpAllocator::BumpAllocator):
1408         (bmalloc::BumpAllocator::init):
1409         (bmalloc::BumpAllocator::line):
1410         (bmalloc::BumpAllocator::validate):
1411         (bmalloc::BumpAllocator::allocate):
1412         (bmalloc::BumpAllocator::refill):
1413         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1414         about the kinds of lines they allocate into. In some cases, the line
1415         type must be provided as a template parameter by the caller.
1416
1417         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1418         (bmalloc::SmallAllocator::line): Deleted.
1419         (bmalloc::SmallAllocator::allocate): Deleted.
1420         (bmalloc::SmallAllocator::objectCount): Deleted.
1421         (bmalloc::SmallAllocator::derefCount): Deleted.
1422         (bmalloc::SmallAllocator::refill): Deleted.
1423         (bmalloc::SmallAllocator::clear): Deleted.
1424
1425         * bmalloc/ObjectType.h:
1426         (bmalloc::isMedium):
1427
1428         * bmalloc/SmallAllocator.h:
1429         (bmalloc::SmallAllocator::isNull): Deleted.
1430         (bmalloc::SmallAllocator::canAllocate): Deleted.
1431         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1432         (bmalloc::SmallAllocator::line): Deleted.
1433         (bmalloc::SmallAllocator::allocate): Deleted.
1434         (bmalloc::SmallAllocator::objectCount): Deleted.
1435         (bmalloc::SmallAllocator::derefCount): Deleted.
1436         (bmalloc::SmallAllocator::refill): Deleted.
1437         (bmalloc::SmallAllocator::clear): Deleted.
1438
1439 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1440
1441         Fixed a goof in bmalloc Vector sizing
1442         https://bugs.webkit.org/show_bug.cgi?id=136795
1443
1444         Reviewed by Gavin Barraclough and Sam Weinig.
1445
1446         We want our minimum vector to be page-sized since the OS will give us
1447         a page no matter what -- but we want that many bytes, and not enough
1448         bytes to store that many elements.
1449
1450         * bmalloc/Vector.h: Math is hard.
1451
1452 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1453
1454         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1455         https://bugs.webkit.org/show_bug.cgi?id=136693
1456
1457         Reviewed by Gavin Barraclough.
1458
1459         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1460
1461         No throughput change.
1462
1463         We keep an array of medium allocators, just like our array of small
1464         allocators.
1465
1466         In future, we can simplify the allocation fast path by merging the small
1467         and medium allocator arrays. For now, this is the simplest change that
1468         gets the win.
1469
1470         * bmalloc/Allocator.cpp:
1471         (bmalloc::Allocator::Allocator):
1472         (bmalloc::Allocator::scavenge):
1473         (bmalloc::Allocator::allocateMedium):
1474         * bmalloc/Allocator.h:
1475         * bmalloc/Sizes.h:
1476         (bmalloc::Sizes::mediumSizeClassFor):
1477
1478 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1479
1480         Reviewed by Sam Weinig.
1481
1482         Renamed log => retire for clarity.
1483
1484         * bmalloc/Allocator.cpp:
1485         (bmalloc::Allocator::scavenge):
1486         (bmalloc::Allocator::retire):
1487         (bmalloc::Allocator::allocateMedium):
1488         (bmalloc::Allocator::allocateSlowCase):
1489         (bmalloc::Allocator::log): Deleted.
1490         * bmalloc/Allocator.h:
1491
1492 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1493
1494         bmalloc: eager scavenge leaves behind a bogus allocator
1495         https://bugs.webkit.org/show_bug.cgi?id=136743
1496
1497         Reviewed by Sam Weinig.
1498
1499         Be sure to clear the allocator after logging it in the eager scavenge
1500         case, so that we don't later try to allocate out of the lines that we
1501         have thrown away.
1502
1503         We didn't need to do this previously because scavenge would only happen
1504         at thread exit time, after which no further allocation from the per-thread
1505         cache would take place.
1506
1507         * bmalloc/Allocator.cpp:
1508         (bmalloc::Allocator::scavenge):
1509         * bmalloc/MediumAllocator.h:
1510         (bmalloc::MediumAllocator::clear):
1511         * bmalloc/SmallAllocator.h:
1512         (bmalloc::SmallAllocator::clear):
1513
1514 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1515
1516         bmalloc should honor the FastMalloc statistics API
1517         https://bugs.webkit.org/show_bug.cgi?id=136592
1518
1519         Reviewed by Gavin Barraclough.
1520
1521         We do this by tracking "size" and "capacity" in the VM heap.
1522
1523         The VM heap's "capacity" is all the VM we ever allocated.
1524
1525         The VM heap's "size" the subset of VM currently held onto by the
1526         VM heap (and therefore not in use by the regular heap).
1527
1528         Somewhat ironically, reducing the process's memory footprint, increases
1529         the size of the VM heap, since the VM heap holds the pages that are
1530         purely virtual and not physical.
1531
1532         * bmalloc/Heap.cpp:
1533         (bmalloc::Heap::size):
1534         (bmalloc::Heap::capacity):
1535         * bmalloc/Heap.h:
1536         * bmalloc/VMHeap.cpp:
1537         (bmalloc::VMHeap::VMHeap):
1538         (bmalloc::VMHeap::allocateSmallChunk):
1539         (bmalloc::VMHeap::allocateMediumChunk):
1540         (bmalloc::VMHeap::allocateLargeChunk):
1541         * bmalloc/VMHeap.h:
1542         (bmalloc::VMHeap::size):
1543         (bmalloc::VMHeap::capacity):
1544         (bmalloc::VMHeap::allocateSmallPage):
1545         (bmalloc::VMHeap::allocateMediumPage):
1546         (bmalloc::VMHeap::allocateLargeRange):
1547         (bmalloc::VMHeap::deallocateSmallPage):
1548         (bmalloc::VMHeap::deallocateMediumPage):
1549         (bmalloc::VMHeap::deallocateLargeRange):
1550         * bmalloc/bmalloc.h:
1551         (bmalloc::api::heapSize):
1552         (bmalloc::api::heapCapacity):
1553
1554 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1555
1556         bmalloc crashes on the EWS bots (due to bad large object allocation)
1557         https://bugs.webkit.org/show_bug.cgi?id=136469
1558
1559         Reviewed by Andreas Kling.
1560
1561         It's possible to convince bmalloc to perform a bad large object allocation,
1562         through these steps:
1563
1564         (1) Insert object A into freelist F0.
1565
1566         (2) Split, merge and split again A's neighbors such that object B is
1567         inserted into freelist F0, with boundary tag and size equal to object A,
1568         but pointer not completely equal to object A. Put object B at the head of F0.
1569
1570         (3) Allocate some other object from F0, swapping its position in the
1571         freelist with object B, such that object A is now ahead of object B.
1572
1573         --> Now, the next allocation for size A/B will allocate object A, which
1574         has a slightly wrong idea about where the object actually begins.
1575         Immediately, you'll corrupt a little memory, and over time, you'll also
1576         corrupt boundary tag metadata.
1577
1578         The solution is to store the begin pointer in the boundary tag. Luckily,
1579         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1580         on MallocBench.
1581
1582         * bmalloc/Algorithm.h:
1583         (bmalloc::rightShift):
1584         * bmalloc/BeginTag.h:
1585         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1586         validate the start pointer when popping off the free list. Through a
1587         very uncommon set of steps, it is possible to have an item in the free
1588         list that is valid by all accounts except for its start pointer.
1589
1590         * bmalloc/BoundaryTag.h:
1591         (bmalloc::BoundaryTag::compactBegin):
1592         (bmalloc::BoundaryTag::setRange):
1593         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1594         start pointer. We don't need the whole pointer -- just the offset, in
1595         largeAlignment increments, into the relevant boundary tag bucket.
1596
1597         * bmalloc/BoundaryTagInlines.h:
1598         (bmalloc::validateNext):
1599         (bmalloc::BoundaryTag::init):
1600         (bmalloc::BoundaryTag::mergeLarge):
1601         (bmalloc::BoundaryTag::splitLarge):
1602         * bmalloc/SegregatedFreeList.cpp:
1603         (bmalloc::SegregatedFreeList::insert):
1604         (bmalloc::SegregatedFreeList::takeGreedy):
1605         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1606         the size when establishing a boundary tag, as required by the new
1607         interface.
1608
1609         * bmalloc/Sizes.h:
1610
1611 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1612
1613         Fixed a bmalloc crash seen on the EWS bot
1614         https://bugs.webkit.org/show_bug.cgi?id=135955
1615
1616         Reviewed by Andreas Kling.
1617
1618         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1619         input buffer is a malloc region, that region will get marked Copy-On-Write
1620         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1621         on older OS X's. In 10.10, they still fail, but they do not return
1622         EINVAL.
1623
1624         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1625
1626 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1627
1628         Fixed the bmalloc build
1629         https://bugs.webkit.org/show_bug.cgi?id=135953
1630
1631         Reviewed by Andreas Kling.
1632
1633         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1634         These headers are used, so they must be available outside the project.
1635
1636 2014-08-13  Daniel Bates  <dabates@apple.com>
1637
1638         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1639         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1640
1641         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1642         implementation of PerThread<T>::getFastCase().
1643
1644         * bmalloc/PerThread.h:
1645         (bmalloc::PerThread<T>::getFastCase):
1646
1647 2014-08-13  Daniel Bates  <dabates@apple.com>
1648
1649         Make bmalloc::PerThread work without C++ thread local storage
1650         https://bugs.webkit.org/show_bug.cgi?id=135895
1651
1652         Reviewed by Geoffrey Garen.
1653
1654         Implement support for building bmalloc without C++ thread local storage.
1655
1656         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1657         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1658         to determine whether the compiler supports C++ thread local storage.
1659         * bmalloc/PerThread.h:
1660         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1661         without C++ thread local storage.
1662         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1663         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1664         PerThreadStorage::initSharedKeyIfNeeded().
1665         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1666         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1667         ensure that the shared key has been initialized.
1668         (_pthread_setspecific_direct): Deleted.
1669         (_pthread_getspecific_direct): Deleted.
1670
1671 2014-08-13  Daniel Bates  <dabates@apple.com>
1672
1673         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1674         https://bugs.webkit.org/show_bug.cgi?id=135848
1675
1676         Reviewed by Geoffrey Garen.
1677
1678         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1679         when building for the iOS Simulator.
1680         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1681         using the public SDK.
1682         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1683         using the public SDK.
1684         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1685         using the public SDK.
1686
1687 2014-08-12  Daniel Bates  <dabates@apple.com>
1688
1689         BPLATFORM(IOS) always evaluates to false
1690         https://bugs.webkit.org/show_bug.cgi?id=135843
1691
1692         Reviewed by Geoffrey Garen.
1693
1694         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1695         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1696         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1697         building for iOS.
1698
1699         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1700         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1701
1702         * bmalloc/BPlatform.h:
1703
1704 2014-08-11  Andy Estes  <aestes@apple.com>
1705
1706         [iOS] Get rid of iOS.xcconfig
1707         https://bugs.webkit.org/show_bug.cgi?id=135809
1708
1709         Reviewed by Joseph Pecoraro.
1710
1711         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1712
1713         * Configurations/Base.xcconfig:
1714         * Configurations/iOS.xcconfig: Removed.
1715         * bmalloc.xcodeproj/project.pbxproj:
1716
1717 2014-05-01  Dan Bernstein  <mitz@apple.com>
1718
1719         Fixed production builds for the iOS Simulator.
1720         <rdar://problem/16792221>
1721
1722         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1723         PRIVATE_HEADERS_FOLDER_PATH when installing.
1724
1725 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1726
1727         bmalloc: Segregate pages by objects size
1728         https://bugs.webkit.org/show_bug.cgi?id=131909
1729
1730         Reviewed by Andreas Kling.
1731
1732         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1733
1734         * bmalloc/Allocator.cpp:
1735         (bmalloc::Allocator::allocateSlowCase):
1736         * bmalloc/Allocator.h:
1737         (bmalloc::Allocator::allocateFastCase):
1738         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1739         function for size class calculation.
1740
1741         * bmalloc/Deallocator.cpp:
1742         (bmalloc::Deallocator::Deallocator):
1743         (bmalloc::Deallocator::scavenge):
1744         (bmalloc::Deallocator::deallocateSmallLine):
1745         (bmalloc::Deallocator::allocateSmallLine):
1746         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1747         cache can't be shared anymore.
1748
1749         * bmalloc/Heap.cpp:
1750         (bmalloc::Heap::allocateSmallLineSlowCase):
1751         * bmalloc/Heap.h:
1752         (bmalloc::Heap::deallocateSmallLine): Ditto.
1753
1754         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1755         page refcount when allocating a line because we might have deallocated
1756         the page and the recycled it for another size class.
1757
1758         (bmalloc::Heap::deallocateMediumLine):
1759         (bmalloc::Heap::allocateMediumLine):
1760         * bmalloc/Line.h:
1761         (bmalloc::Line::refCount):
1762         * bmalloc/Page.h:
1763         (bmalloc::Page::refCount):
1764         (bmalloc::Page::smallSizeClass):
1765         (bmalloc::Page::setSmallSizeClass):
1766         (bmalloc::Page<Traits>::refCount): Deleted.
1767         * bmalloc/Sizes.h:
1768         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1769         an index into an array from a size.
1770
1771 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1772
1773         bmalloc: Improved alignment in LargeChunk
1774         https://bugs.webkit.org/show_bug.cgi?id=131895
1775
1776         Reviewed by Andreas Kling.
1777
1778         * bmalloc/Chunk.h:
1779         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1780         Technically, the previous alignment was harmless, but I would prefer,
1781         dear reader, not to have to explain the interlocking set of
1782         circumstances that made it so.
1783
1784 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1785
1786         Rolled out r167502 because it caused a crash on the facebook benchmark.
1787
1788         Unreviewed.
1789
1790             bmalloc: Added an XSmall line size
1791             https://bugs.webkit.org/show_bug.cgi?id=131851
1792
1793             Reviewed by Sam Weinig.
1794
1795 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1796
1797         bmalloc: Mutex should be harder to use wrong
1798         https://bugs.webkit.org/show_bug.cgi?id=131879
1799
1800         Reviewed by Andreas Kling.
1801
1802         Mutex now has a proper constructor, so you can't deadlock by forgetting
1803         to initialize it.
1804
1805         * bmalloc.xcodeproj/project.pbxproj:
1806         * bmalloc/Allocator.cpp:
1807         (bmalloc::Allocator::processXSmallAllocatorLog):
1808         (bmalloc::Allocator::processSmallAllocatorLog):
1809         (bmalloc::Allocator::processMediumAllocatorLog):
1810         (bmalloc::Allocator::allocateLarge):
1811         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1812         since the Heap mutex is a static.
1813
1814         * bmalloc/AsyncTask.h:
1815         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1816         need for explicit initialization anymore.
1817
1818         * bmalloc/Deallocator.cpp:
1819         (bmalloc::Deallocator::scavenge):
1820         (bmalloc::Deallocator::deallocateLarge):
1821         (bmalloc::Deallocator::deallocateXLarge):
1822         (bmalloc::Deallocator::processObjectLog):
1823         (bmalloc::Deallocator::deallocateSmallLine):
1824         (bmalloc::Deallocator::deallocateXSmallLine):
1825         (bmalloc::Deallocator::allocateSmallLine):
1826         (bmalloc::Deallocator::allocateXSmallLine):
1827         (bmalloc::Deallocator::deallocateMediumLine):
1828         (bmalloc::Deallocator::allocateMediumLine):
1829         * bmalloc/Deallocator.h:
1830         * bmalloc/Heap.cpp:
1831         (bmalloc::sleep):
1832         (bmalloc::Heap::Heap):
1833         (bmalloc::Heap::concurrentScavenge):
1834         (bmalloc::Heap::scavenge):
1835         (bmalloc::Heap::scavengeSmallPages):
1836         (bmalloc::Heap::scavengeXSmallPages):
1837         (bmalloc::Heap::scavengeMediumPages):
1838         (bmalloc::Heap::scavengeLargeRanges):
1839         (bmalloc::Heap::allocateXSmallLineSlowCase):
1840         (bmalloc::Heap::allocateSmallLineSlowCase):
1841         (bmalloc::Heap::allocateMediumLineSlowCase):
1842         (bmalloc::Heap::allocateXLarge):
1843         (bmalloc::Heap::deallocateXLarge):
1844         (bmalloc::Heap::allocateLarge):
1845         (bmalloc::Heap::deallocateLarge):
1846         * bmalloc/Heap.h:
1847         (bmalloc::Heap::deallocateXSmallLine):
1848         (bmalloc::Heap::allocateXSmallLine):
1849         (bmalloc::Heap::deallocateSmallLine):
1850         (bmalloc::Heap::allocateSmallLine):
1851         (bmalloc::Heap::deallocateMediumLine):
1852         (bmalloc::Heap::allocateMediumLine):
1853         * bmalloc/Line.h:
1854         (bmalloc::Line<Traits>::deref):
1855         * bmalloc/Mutex.cpp: Removed.
1856         * bmalloc/Mutex.h:
1857         (bmalloc::Mutex::Mutex):
1858         (bmalloc::Mutex::init): Deleted.
1859         (bmalloc::Mutex::try_lock): Deleted.
1860         (bmalloc::Mutex::lock): Deleted.
1861         (bmalloc::Mutex::unlock): Deleted.
1862         * bmalloc/Page.h:
1863         (bmalloc::Page<Traits>::ref):
1864         (bmalloc::Page<Traits>::deref):
1865         (bmalloc::Page<Traits>::refCount):
1866         * bmalloc/PerProcess.h:
1867         (bmalloc::PerProcess::mutex):
1868         (bmalloc::PerProcess<T>::getSlowCase):
1869         * bmalloc/StaticMutex.cpp: Added.
1870         (bmalloc::StaticMutex::lockSlowCase):
1871         * bmalloc/StaticMutex.h: Added.
1872         (bmalloc::StaticMutex::init):
1873         (bmalloc::StaticMutex::try_lock):
1874         (bmalloc::StaticMutex::lock):
1875         (bmalloc::StaticMutex::unlock):
1876         * bmalloc/VMHeap.h:
1877         (bmalloc::VMHeap::deallocateXSmallPage):
1878         (bmalloc::VMHeap::deallocateSmallPage):
1879         (bmalloc::VMHeap::deallocateMediumPage):
1880         (bmalloc::VMHeap::deallocateLargeRange):
1881         * bmalloc/bmalloc.h:
1882         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1883         since the Heap mutex is a static.
1884
1885 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1886
1887         bmalloc: AsyncTask should use Mutex instead of std::mutex
1888         https://bugs.webkit.org/show_bug.cgi?id=131865
1889
1890         Reviewed by Gavin Barraclough.
1891
1892         std::mutex is so slow that it makes parallelizing simple tasks through
1893         AsyncTask a net regression. Mutex fixes this.
1894
1895         * bmalloc/AsyncTask.h:
1896         (bmalloc::Function>::AsyncTask):
1897         (bmalloc::Function>::join):
1898         (bmalloc::Function>::runSlowCase):
1899         (bmalloc::Function>::entryPoint):
1900         * bmalloc/Mutex.h:
1901         (bmalloc::Mutex::init):
1902
1903 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1904
1905         bmalloc: Added an XSmall line size
1906         https://bugs.webkit.org/show_bug.cgi?id=131851
1907
1908         Reviewed by Sam Weinig.
1909
1910         Reduces malloc footprint on Membuster recordings by 10%.
1911
1912         This is a throughput regression, but we're still way ahead of TCMalloc.
1913         I have some ideas for how to recover the regression -- but I wanted to
1914         get this win in first.
1915
1916         Full set of benchmark results:
1917
1918                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1919                                                                                                 
1920                                                                nopatch                      patch                                Δ
1921                 Peak Memory:
1922                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1923                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1924                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1925
1926                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1927                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1928                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1929
1930                 Memory at End:
1931                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1932                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1933                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1934
1935                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1936                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1937                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1938
1939                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1940                                                                                                 
1941                                                    nopatch                patch                         Δ
1942                 Execution Time:
1943                     churn                            127ms                151ms            ! 1.19x slower
1944                     list_allocate                    130ms                164ms            ! 1.26x slower
1945                     tree_allocate                    109ms                127ms            ! 1.17x slower
1946                     tree_churn                       115ms                120ms            ! 1.04x slower
1947                     facebook                         240ms                259ms            ! 1.08x slower
1948                     fragment                          91ms                131ms            ! 1.44x slower
1949                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1950                     message_one                      260ms                259ms             ^ 1.0x faster
1951                     message_many                     149ms                154ms            ! 1.03x slower
1952                     medium                           194ms                248ms            ! 1.28x slower
1953                     big                              157ms                160ms            ! 1.02x slower
1954
1955                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1956                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1957                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1958
1959                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1960                                                                                                 
1961                                                                        nopatch                          patch                                     Δ
1962                 Execution Time:
1963                     churn                                                126ms                          148ms                        ! 1.17x slower
1964                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1965                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1966                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1967                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1968                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1969                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1970                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1971                     facebook                                             240ms                          276ms                        ! 1.15x slower
1972                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1973                     fragment                                              92ms                          130ms                        ! 1.41x slower
1974                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1975                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1976                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1977                     message_one                                          260ms                          260ms                                      
1978                     message_many                                         170ms                          238ms                         ! 1.4x slower
1979                     medium                                               185ms                          250ms                        ! 1.35x slower
1980                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1981                     big                                                  150ms                          169ms                        ! 1.13x slower
1982                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1983
1984                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1985                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1986                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1987
1988                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1989
1990                                                                     TC                      patch                                Δ
1991                 Peak Memory:
1992                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1993                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1994                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1995
1996                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1997                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1998                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1999
2000                 Memory at End:
2001                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
2002                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
2003                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
2004
2005                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
2006                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
2007                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
2008
2009                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2010                                                                                                 
2011                                                         TC                patch                         Δ
2012                 Execution Time:
2013                     churn                            416ms                148ms            ^ 2.81x faster
2014                     list_allocate                    463ms                164ms            ^ 2.82x faster
2015                     tree_allocate                    292ms                127ms             ^ 2.3x faster
2016                     tree_churn                       157ms                120ms            ^ 1.31x faster
2017                     facebook                         327ms                276ms            ^ 1.18x faster
2018                     fragment                         335ms                129ms             ^ 2.6x faster
2019                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
2020                     message_one                      386ms                258ms             ^ 1.5x faster
2021                     message_many                     410ms                154ms            ^ 2.66x faster
2022                     medium                           391ms                245ms             ^ 1.6x faster
2023                     big                              261ms                167ms            ^ 1.56x faster
2024
2025                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
2026                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
2027                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
2028
2029         * bmalloc.xcodeproj/project.pbxproj:
2030         * bmalloc/Allocator.cpp:
2031         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
2032         index corresponds with its size. Instead, use the size selection function
2033         explicitly. Now that we have XSmall, some small allocator entries are
2034         unused.
2035
2036         (bmalloc::Allocator::scavenge):
2037         (bmalloc::Allocator::log):
2038         (bmalloc::Allocator::processXSmallAllocatorLog):
2039         (bmalloc::Allocator::allocateSlowCase):
2040         * bmalloc/Allocator.h:
2041         (bmalloc::Allocator::xSmallAllocatorFor):
2042         (bmalloc::Allocator::allocateFastCase):
2043         * bmalloc/Chunk.h:
2044         * bmalloc/Deallocator.cpp:
2045         (bmalloc::Deallocator::scavenge):
2046         (bmalloc::Deallocator::processObjectLog):
2047         (bmalloc::Deallocator::deallocateSlowCase):
2048         (bmalloc::Deallocator::deallocateXSmallLine):
2049         (bmalloc::Deallocator::allocateXSmallLine):
2050         * bmalloc/Deallocator.h:
2051         (bmalloc::Deallocator::deallocateFastCase):
2052         * bmalloc/Heap.cpp:
2053         (bmalloc::Heap::scavenge):
2054         (bmalloc::Heap::scavengeXSmallPages):
2055         (bmalloc::Heap::allocateXSmallLineSlowCase):
2056         * bmalloc/Heap.h:
2057         (bmalloc::Heap::deallocateXSmallLine):
2058         (bmalloc::Heap::allocateXSmallLine):
2059         * bmalloc/LargeChunk.h:
2060         (bmalloc::LargeChunk::get):
2061         (bmalloc::LargeChunk::endTag):
2062         * bmalloc/Line.h:
2063         * bmalloc/MediumAllocator.h:
2064         (bmalloc::MediumAllocator::allocate):
2065         (bmalloc::MediumAllocator::refill):
2066         * bmalloc/ObjectType.cpp:
2067         (bmalloc::objectType):
2068         * bmalloc/ObjectType.h:
2069         (bmalloc::isXSmall):
2070         (bmalloc::isSmall):
2071         (bmalloc::isMedium):
2072         (bmalloc::isLarge):
2073         (bmalloc::isSmallOrMedium): Deleted.
2074         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
2075         handling small objects. There's probably a reasonable way to share this
2076         code in the future -- I'll look into that once it's stopped changing.
2077
2078         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
2079         is the main reason things got slower.
2080
2081         * bmalloc/SmallAllocator.h:
2082         (bmalloc::SmallAllocator::allocate):
2083         * bmalloc/SmallTraits.h:
2084         * bmalloc/VMHeap.cpp:
2085         (bmalloc::VMHeap::allocateXSmallChunk):
2086         * bmalloc/VMHeap.h:
2087         (bmalloc::VMHeap::allocateXSmallPage):
2088         (bmalloc::VMHeap::deallocateXSmallPage):
2089         * bmalloc/XSmallAllocator.h: Added.
2090         (bmalloc::XSmallAllocator::isNull):
2091         (bmalloc::XSmallAllocator::canAllocate):
2092         (bmalloc::XSmallAllocator::XSmallAllocator):
2093         (bmalloc::XSmallAllocator::line):
2094         (bmalloc::XSmallAllocator::allocate):
2095         (bmalloc::XSmallAllocator::objectCount):
2096         (bmalloc::XSmallAllocator::derefCount):
2097         (bmalloc::XSmallAllocator::refill):
2098         * bmalloc/XSmallChunk.h: Added.
2099         * bmalloc/XSmallLine.h: Added.
2100         * bmalloc/XSmallPage.h: Added.
2101         * bmalloc/XSmallTraits.h: Added.
2102         * bmalloc/bmalloc.h:
2103         (bmalloc::api::realloc): Boiler-plate copy, as above.
2104
2105 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2106
2107         MallocBench should scavenge explicitly instead of waiting
2108         https://bugs.webkit.org/show_bug.cgi?id=131661
2109
2110         Reviewed by Andreas Kling.
2111
2112         Added explicit scavenge support to bmalloc. This isn't a memory win,
2113         since bmalloc's per-thread cache is so small. But it makes testing
2114         simpler.
2115
2116         * bmalloc/Allocator.cpp:
2117         (bmalloc::Allocator::~Allocator):
2118         (bmalloc::Allocator::scavenge):
2119         * bmalloc/Allocator.h:
2120         * bmalloc/Cache.cpp:
2121         (bmalloc::Cache::operator new):
2122         (bmalloc::Cache::operator delete):
2123         (bmalloc::Cache::Cache):
2124         (bmalloc::Cache::scavenge):
2125         * bmalloc/Cache.h:
2126         * bmalloc/Deallocator.cpp:
2127         (bmalloc::Deallocator::~Deallocator):
2128         (bmalloc::Deallocator::scavenge):
2129         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
2130         functions, for reuse.
2131
2132         * bmalloc/Heap.cpp:
2133         (bmalloc::sleep):
2134         (bmalloc::Heap::concurrentScavenge):
2135         (bmalloc::Heap::scavenge):
2136         (bmalloc::Heap::scavengeSmallPages):
2137         (bmalloc::Heap::scavengeMediumPages):
2138         (bmalloc::Heap::scavengeLargeRanges):
2139         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
2140         scavenging -- in response to a benchmark or a low memory warning --
2141         wants to complete as soon as possible, so its sleep duration is 0.
2142
2143         * bmalloc/bmalloc.h:
2144         (bmalloc::api::scavenge):
2145         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
2146
2147 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2148
2149         Use 4kB pages on Mac
2150         https://bugs.webkit.org/show_bug.cgi?id=131658
2151
2152         Reviewed by Sam Weinig.
2153
2154         This reduces memory use a lot on Membuster:
2155
2156                                                                   base                      patch                                Δ
2157                 Execution Time:
2158                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
2159                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
2160                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
2161
2162                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
2163                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
2164                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
2165
2166                 Peak Memory:
2167                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
2168                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
2169                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
2170
2171                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
2172                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
2173                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
2174
2175                 Memory at End:
2176                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
2177                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
2178                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
2179
2180                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
2181                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
2182                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
2183
2184
2185         * bmalloc.xcodeproj/project.pbxproj:
2186         * bmalloc/BPlatform.h: Added.
2187         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
2188         is 4kB on Mac.
2189
2190 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
2191
2192         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
2193
2194         * bmalloc.xcodeproj: Modified property svn:ignore.
2195
2196 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
2197
2198         Fixed some mbmalloc exports
2199         https://bugs.webkit.org/show_bug.cgi?id=131599
2200
2201         Reviewed by Ryosuke Niwa.
2202
2203         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
2204         of the project, so we can call them from API.
2205
2206         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
2207         visibility, so they show up as exported in the .dylib.
2208
2209 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
2210
2211         Put bmalloc headers in the right place
2212         https://bugs.webkit.org/show_bug.cgi?id=131464
2213
2214         Reviewed by Mark Rowe.
2215
2216         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
2217         specify that we don't just want to dump all of our generically-named
2218         headers into /usr/local/include.
2219
2220 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2221
2222         Made bmalloc more #include friendly
2223         https://bugs.webkit.org/show_bug.cgi?id=131386
2224
2225         Reviewed by Andreas Kling.
2226
2227         Marked a bunch of headers private so they can be used from client code
2228         that #includes bmalloc.h.
2229
2230         Renamed ASSERT macros to BASSERT. This matches their header, which already
2231         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
2232
2233         * bmalloc.xcodeproj/project.pbxproj:
2234         * bmalloc/Allocator.cpp:
2235         (bmalloc::Allocator::allocateSlowCase):
2236         * bmalloc/AsyncTask.h:
2237         (bmalloc::Function>::runSlowCase):
2238         * bmalloc/BAssert.h:
2239         * bmalloc/BoundaryTag.h:
2240         (bmalloc::BoundaryTag::setSize):
2241         * bmalloc/BoundaryTagInlines.h:
2242         (bmalloc::validate):
2243         (bmalloc::BoundaryTag::init):
2244         (bmalloc::BoundaryTag::deallocate):
2245         (bmalloc::BoundaryTag::splitLarge):
2246         (bmalloc::BoundaryTag::allocate):
2247         * bmalloc/Chunk.h:
2248         * bmalloc/Deallocator.cpp:
2249         (bmalloc::Deallocator::processObjectLog):
2250         (bmalloc::Deallocator::deallocateSlowCase):
2251         * bmalloc/Deallocator.h:
2252         (bmalloc::Deallocator::deallocateFastCase):
2253         * bmalloc/FixedVector.h:
2254         (bmalloc::Capacity>::operator):
2255         (bmalloc::Capacity>::push):
2256         (bmalloc::Capacity>::pop):
2257         (bmalloc::Capacity>::shrink):
2258         * bmalloc/Heap.cpp:
2259         (bmalloc::Heap::allocateLarge):
2260         * bmalloc/LargeChunk.h:
2261         (bmalloc::LargeChunk::get):
2262         (bmalloc::LargeChunk::endTag):
2263         * bmalloc/Line.h:
2264         (bmalloc::Line<Traits>::concurrentRef):
2265         (bmalloc::Line<Traits>::deref):
2266         * bmalloc/MediumAllocator.h:
2267         (bmalloc::MediumAllocator::allocate):
2268         * bmalloc/ObjectType.h:
2269         (bmalloc::isSmall):
2270         * bmalloc/Page.h:
2271         (bmalloc::Page<Traits>::ref):
2272         (bmalloc::Page<Traits>::deref):
2273         * bmalloc/PerThread.h:
2274         (bmalloc::PerThread<T>::getSlowCase):
2275         * bmalloc/SegregatedFreeList.cpp:
2276         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2277         (bmalloc::SegregatedFreeList::insert):
2278         * bmalloc/SmallAllocator.h:
2279         (bmalloc::SmallAllocator::allocate):
2280         (bmalloc::SmallAllocator::refill):
2281         * bmalloc/Syscall.h:
2282         * bmalloc/VMAllocate.h:
2283         (bmalloc::vmValidate):
2284         (bmalloc::vmAllocate):
2285         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2286         * bmalloc/Vector.h:
2287         (bmalloc::Vector<T>::operator):
2288         (bmalloc::Vector<T>::pop):
2289         (bmalloc::Vector<T>::shrink):
2290         * bmalloc/XLargeChunk.h:
2291         (bmalloc::XLargeChunk::range):
2292         (bmalloc::XLargeChunk::size):
2293
2294 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2295
2296         Removed an unused file.
2297
2298         Unreviewed.
2299
2300         * bmalloc/AsyncTask.cpp: Removed.
2301
2302 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2303
2304         Build bmalloc on Mac
2305         https://bugs.webkit.org/show_bug.cgi?id=131333
2306
2307         Reviewed by Mark Rowe.
2308
2309         * Makefile: Added. For make clients.
2310
2311         These files are required for building any project in WebKit. I copied
2312         them from WTF:
2313         * Configurations: Added.
2314         * Configurations/Base.xcconfig: Added.
2315         * Configurations/DebugRelease.xcconfig: Added.
2316         * Configurations/bmalloc.xcconfig: Added.
2317         * Configurations/iOS.xcconfig: Added.
2318         * Configurations/mbmalloc.xcconfig: Added.
2319
2320         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
2321         from here because everything is in .xcconfig files now.
2322
2323         I had to fix a bunch of minor warnings, since they're enabled in our
2324         .xcconfig files:
2325
2326         * bmalloc/AsyncTask.h:
2327         (bmalloc::Function>::AsyncTask):
2328         * bmalloc/BAssert.h:
2329         * bmalloc/BoundaryTagInlines.h:
2330         (bmalloc::validate):
2331         * bmalloc/Heap.cpp:
2332         (bmalloc::Heap::Heap):
2333         (bmalloc::Heap::allocateLarge):
2334         (bmalloc::Heap::deallocateLarge):
2335         * bmalloc/Mutex.h:
2336         (bmalloc::Mutex::Mutex): Deleted.
2337         * bmalloc/VMAllocate.h:
2338         (bmalloc::vmValidate):
2339         * bmalloc/mbmalloc.cpp:
2340
2341 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2342
2343         bmalloc: Fixed a leak in the per-thread cache
2344         https://bugs.webkit.org/show_bug.cgi?id=131330
2345
2346         Reviewed by Andreas Kling.
2347
2348         Remember to deallocate our line caches upon thread exit.
2349
2350         * bmalloc/Deallocator.cpp:
2351         (bmalloc::Deallocator::~Deallocator):
2352
2353 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2354
2355         bmalloc: rolled out the tryLock experiment
2356         https://bugs.webkit.org/show_bug.cgi?id=131328
2357
2358         Reviewed by Andreas Kling.
2359
2360         It wasn't a speedup.
2361
2362         * bmalloc.xcodeproj/project.pbxproj:
2363         * bmalloc/Allocator.cpp:
2364         (bmalloc::Allocator::processSmallAllocatorLog):
2365         (bmalloc::Allocator::processMediumAllocatorLog):
2366         * bmalloc/Deallocator.cpp:
2367         (bmalloc::Deallocator::processObjectLog):
2368         (bmalloc::Deallocator::deallocateSlowCase):
2369         (bmalloc::Deallocator::deallocateSmallLine):
2370         (bmalloc::Deallocator::deallocateMediumLine):
2371         * bmalloc/Deallocator.h:
2372         (bmalloc::Deallocator::deallocateFastCase):
2373         * bmalloc/Heap.h:
2374         (bmalloc::Heap::deallocateSmallLine):
2375         (bmalloc::Heap::deallocateMediumLine):
2376         * bmalloc/Line.h:
2377         (bmalloc::Line<Traits>::deref):
2378         * bmalloc/Page.h:
2379         (bmalloc::Page<Traits>::deref):
2380
2381 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2382
2383         bmalloc
2384         https://bugs.webkit.org/show_bug.cgi?id=131170
2385
2386         Reviewed by Andreas Kling.
2387
2388         Initial commit.
2389
2390         * bmalloc: Added.
2391         * bmalloc.xcodeproj: Added.
2392         * bmalloc.xcodeproj/project.pbxproj: Added.
2393         * bmalloc/Algorithm.h: Added.
2394         (bmalloc::max):
2395         (bmalloc::min):
2396         (bmalloc::mask):
2397         (bmalloc::test):
2398         (bmalloc::roundUpToMultipleOf):
2399         (bmalloc::roundDownToMultipleOf):
2400         (bmalloc::sizeOf):
2401         (bmalloc::bitCount):
2402         (bmalloc::isPowerOfTwo):
2403         * bmalloc/Allocator.cpp: Added.
2404         (bmalloc::Allocator::Allocator):
2405         (bmalloc::Allocator::~Allocator):
2406         (bmalloc::Allocator::log):
2407         (bmalloc::Allocator::processSmallAllocatorLog):
2408         (bmalloc::Allocator::processMediumAllocatorLog):
2409         (bmalloc::Allocator::allocateLarge):
2410         (bmalloc::Allocator::allocateXLarge):
2411         (bmalloc::Allocator::allocateMedium):
2412         (bmalloc::Allocator::allocateSlowCase):
2413         * bmalloc/Allocator.h: Added.
2414         (bmalloc::Allocator::smallAllocatorFor):
2415         (bmalloc::Allocator::allocateFastCase):
2416         (bmalloc::Allocator::allocate):
2417         * bmalloc/AsyncTask.cpp: Added.
2418         (bmalloc::AsyncTask<Function>::runSlowCase):
2419         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2420         (bmalloc::AsyncTask<Function>::entryPoint):
2421         * bmalloc/AsyncTask.h: Added.
2422         (bmalloc::Function>::AsyncTask):
2423         (bmalloc::Function>::join):
2424         (bmalloc::Function>::run):
2425         (bmalloc::Function>::runSlowCase):
2426         (bmalloc::Function>::pthreadEntryPoint):
2427         (bmalloc::Function>::entryPoint):
2428         * bmalloc/BAssert.h: Added.
2429         * bmalloc/BeginTag.h: Added.
2430         (bmalloc::BeginTag::isInFreeList):
2431         * bmalloc/BoundaryTag.h: Added.
2432         (bmalloc::BoundaryTag::isXLarge):
2433         (bmalloc::BoundaryTag::setXLarge):
2434         (bmalloc::BoundaryTag::isFree):
2435         (bmalloc::BoundaryTag::setFree):
2436         (bmalloc::BoundaryTag::isEnd):
2437         (bmalloc::BoundaryTag::setEnd):
2438         (bmalloc::BoundaryTag::hasPhysicalPages):
2439         (bmalloc::BoundaryTag::setHasPhysicalPages):
2440         (bmalloc::BoundaryTag::isNull):
2441         (bmalloc::BoundaryTag::clear):
2442         (bmalloc::BoundaryTag::size):
2443         (bmalloc::BoundaryTag::setSize):
2444         (bmalloc::BoundaryTag::prev):
2445         (bmalloc::BoundaryTag::next):
2446         * bmalloc/BoundaryTagInlines.h: Added.
2447         (bmalloc::validate):
2448         (bmalloc::validatePrev):
2449         (bmalloc::validateNext):
2450         (bmalloc::BoundaryTag::init):
2451         (bmalloc::BoundaryTag::mergeLargeLeft):
2452         (bmalloc::BoundaryTag::mergeLargeRight):
2453         (bmalloc::BoundaryTag::mergeLarge):
2454         (bmalloc::BoundaryTag::deallocate):
2455         (bmalloc::BoundaryTag::splitLarge):
2456         (bmalloc::BoundaryTag::allocate):
2457         * bmalloc/Cache.cpp: Added.
2458         (bmalloc::Cache::operator new):
2459         (bmalloc::Cache::operator delete):
2460         (bmalloc::Cache::Cache):
2461         (bmalloc::Cache::allocateSlowCase):
2462         (bmalloc::Cache::allocateSlowCaseNullCache):
2463         (bmalloc::Cache::deallocateSlowCase):
2464         (bmalloc::Cache::deallocateSlowCaseNullCache):
2465         * bmalloc/Cache.h: Added.
2466         (bmalloc::Cache::allocator):
2467         (bmalloc::Cache::deallocator):
2468         (bmalloc::Cache::allocateFastCase):
2469         (bmalloc::Cache::deallocateFastCase):
2470         (bmalloc::Cache::allocate):
2471         (bmalloc::Cache::deallocate):
2472         * bmalloc/Chunk.h: Added.
2473         (bmalloc::Chunk::begin):
2474         (bmalloc::Chunk::end):
2475         (bmalloc::Chunk::lines):
2476         (bmalloc::Chunk::pages):
2477         * bmalloc/Deallocator.cpp: Added.
2478         (bmalloc::Deallocator::Deallocator):
2479         (bmalloc::Deallocator::~Deallocator):
2480         (bmalloc::Deallocator::deallocateLarge):
2481         (bmalloc::Deallocator::deallocateXLarge):
2482         (bmalloc::Deallocator::processObjectLog):
2483         (bmalloc::Deallocator::deallocateSlowCase):
2484         (bmalloc::Deallocator::deallocateSmallLine):
2485         (bmalloc::Deallocator::allocateSmallLine):
2486         (bmalloc::Deallocator::deallocateMediumLine):
2487         (bmalloc::Deallocator::allocateMediumLine):
2488         * bmalloc/Deallocator.h: Added.
2489         (bmalloc::Deallocator::deallocateFastCase):
2490         (bmalloc::Deallocator::deallocate):
2491         * bmalloc/EndTag.h: Added.
2492         (bmalloc::EndTag::operator=):
2493         * bmalloc/FixedVector.h: Added.
2494         (bmalloc::FixedVector::begin):
2495         (bmalloc::FixedVector::end):
2496         (bmalloc::FixedVector::size):
2497         (bmalloc::FixedVector::capacity):
2498         (bmalloc::FixedVector::clear):
2499         (bmalloc::FixedVector::isEmpty):
2500         (bmalloc::Capacity>::FixedVector):
2501         (bmalloc::Capacity>::operator):
2502         (bmalloc::Capacity>::push):
2503         (bmalloc::Capacity>::pop):
2504         (bmalloc::Capacity>::shrink):
2505         * bmalloc/Heap.cpp: Added.
2506         (bmalloc::sleep):
2507         (bmalloc::Heap::Heap):
2508         (bmalloc::Heap::concurrentScavenge):
2509         (bmalloc::Heap::scavengeSmallPages):
2510         (bmalloc::Heap::scavengeMediumPages):
2511         (bmalloc::Heap::scavengeLargeRanges):
2512         (bmalloc::Heap::allocateSmallLineSlowCase):
2513         (bmalloc::Heap::allocateMediumLineSlowCase):
2514         (bmalloc::Heap::allocateXLarge):
2515         (bmalloc::Heap::deallocateXLarge):
2516         (bmalloc::Heap::allocateLarge):
2517         (bmalloc::Heap::deallocateLarge):
2518         * bmalloc/Heap.h: Added.
2519         (bmalloc::Heap::deallocateSmallLine):
2520         (bmalloc::Heap::allocateSmallLine):
2521         (bmalloc::Heap::deallocateMediumLine):
2522         (bmalloc::Heap::allocateMediumLine):
2523         * bmalloc/Inline.h: Added.
2524         * bmalloc/LargeChunk.h: Added.
2525         (bmalloc::LargeChunk::begin):
2526         (bmalloc::LargeChunk::end):
2527         (bmalloc::LargeChunk::create):
2528         (bmalloc::LargeChunk::get):
2529         (bmalloc::LargeChunk::beginTag):
2530         (bmalloc::LargeChunk::endTag):
2531         * bmalloc/Line.h: Added.
2532         (bmalloc::Line<Traits>::begin):
2533         (bmalloc::Line<Traits>::end):
2534         (bmalloc::Line<Traits>::concurrentRef):
2535         (bmalloc::Line<Traits>::deref):
2536         * bmalloc/MediumAllocator.h: Added.
2537         (bmalloc::MediumAllocator::isNull):
2538         (bmalloc::MediumAllocator::MediumAllocator):
2539         (bmalloc::MediumAllocator::line):
2540         (bmalloc::MediumAllocator::allocate):
2541         (bmalloc::MediumAllocator::derefCount):
2542         (bmalloc::MediumAllocator::refill):
2543         * bmalloc/MediumChunk.h: Added.
2544         * bmalloc/MediumLine.h: Added.
2545         * bmalloc/MediumPage.h: Added.
2546         * bmalloc/MediumTraits.h: Added.
2547         * bmalloc/Mutex.cpp: Added.
2548         (bmalloc::Mutex::lockSlowCase):
2549         * bmalloc/Mutex.h: Added.
2550         (bmalloc::Mutex::Mutex):
2551         (bmalloc::Mutex::try_lock):
2552         (bmalloc::Mutex::lock):
2553         (bmalloc::Mutex::unlock):
2554         * bmalloc/ObjectType.cpp: Added.
2555         (bmalloc::objectType):
2556         * bmalloc/ObjectType.h: Added.
2557         (bmalloc::isSmallOrMedium):
2558         (bmalloc::isSmall):
2559         * bmalloc/Page.h: Added.
2560         (bmalloc::Page<Traits>::ref):
2561         (bmalloc::Page<Traits>::deref):
2562         (bmalloc::Page<Traits>::refCount):
2563         * bmalloc/PerProcess.h: Added.
2564         (bmalloc::PerProcess::mutex):
2565         (bmalloc::PerProcess<T>::getFastCase):
2566         (bmalloc::PerProcess<T>::get):
2567         (bmalloc::PerProcess<T>::getSlowCase):
2568         * bmalloc/PerThread.h: Added.
2569         (bmalloc::PerThreadStorage<Cache>::get):
2570         (bmalloc::PerThreadStorage<Cache>::init):
2571         (bmalloc::PerThreadStorage::get):
2572         (bmalloc::PerThreadStorage::init):
2573         (bmalloc::PerThread<T>::getFastCase):
2574         (bmalloc::PerThread<T>::get):
2575         (bmalloc::PerThread<T>::destructor):
2576         (bmalloc::PerThread<T>::getSlowCase):
2577         * bmalloc/Range.h: Added.
2578         (bmalloc::Range::Range):
2579         (bmalloc::Range::begin):
2580         (bmalloc::Range::end):
2581         (bmalloc::Range::size):
2582         (bmalloc::Range::operator!):
2583         (bmalloc::Range::operator<):
2584         * bmalloc/SegregatedFreeList.cpp: Added.
2585         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2586         (bmalloc::SegregatedFreeList::insert):
2587         (bmalloc::SegregatedFreeList::takeGreedy):
2588         (bmalloc::SegregatedFreeList::take):
2589         * bmalloc/SegregatedFreeList.h: Added.
2590         * bmalloc/Sizes.h: Added.
2591         * bmalloc/SmallAllocator.h: Added.
2592         (bmalloc::SmallAllocator::isNull):
2593         (bmalloc::SmallAllocator::canAllocate):
2594         (bmalloc::SmallAllocator::SmallAllocator):
2595         (bmalloc::SmallAllocator::line):
2596         (bmalloc::SmallAllocator::allocate):
2597         (bmalloc::SmallAllocator::objectCount):
2598         (bmalloc::SmallAllocator::derefCount):
2599         (bmalloc::SmallAllocator::refill):
2600         * bmalloc/SmallChunk.h: Added.
2601         * bmalloc/SmallLine.h: Added.
2602         * bmalloc/SmallPage.h: Added.
2603         * bmalloc/SmallTraits.h: Added.
2604         * bmalloc/Syscall.h: Added.
2605         * bmalloc/VMAllocate.h: Added.
2606         (bmalloc::vmSize):
2607         (bmalloc::vmValidate):
2608         (bmalloc::vmAllocate):
2609         (bmalloc::vmDeallocate):
2610         (bmalloc::vmDeallocatePhysicalPages):
2611         (bmalloc::vmAllocatePhysicalPages):
2612         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2613         (bmalloc::vmAllocatePhysicalPagesSloppy):
2614         * bmalloc/VMHeap.cpp: Added.
2615         (bmalloc::VMHeap::VMHeap):
2616         (bmalloc::VMHeap::allocateSmallChunk):
2617         (bmalloc::VMHeap::allocateMediumChunk):
2618         (bmalloc::VMHeap::allocateLargeChunk):
2619         * bmalloc/VMHeap.h: Added.
2620         (bmalloc::VMHeap::allocateSmallPage):
2621         (bmalloc::VMHeap::allocateMediumPage):
2622         (bmalloc::VMHeap::allocateLargeRange):
2623         (bmalloc::VMHeap::deallocateSmallPage):
2624         (bmalloc::VMHeap::deallocateMediumPage):
2625         (bmalloc::VMHeap::deallocateLargeRange):
2626         * bmalloc/Vector.h: Added.
2627         (bmalloc::Vector::begin):
2628         (bmalloc::Vector::end):
2629         (bmalloc::Vector::size):
2630         (bmalloc::Vector::capacity):
2631         (bmalloc::Vector::last):
2632         (bmalloc::Vector::pop):
2633         (bmalloc::Vector<T>::Vector):
2634         (bmalloc::Vector<T>::~Vector):
2635         (bmalloc::Vector<T>::operator):
2636         (bmalloc::Vector<T>::push):
2637         (bmalloc::Vector<T>::pop):
2638         (bmalloc::Vector<T>::shrink):
2639         (bmalloc::Vector<T>::reallocateBuffer):
2640         (bmalloc::Vector<T>::shrinkCapacity):
2641         (bmalloc::Vector<T>::growCapacity):
2642         * bmalloc/XLargeChunk.h: Added.
2643         (bmalloc::XLargeChunk::get):
2644         (bmalloc::XLargeChunk::begin):
2645         (bmalloc::XLargeChunk::XLargeChunk):
2646         (bmalloc::XLargeChunk::create):
2647         (bmalloc::XLargeChunk::destroy):
2648         (bmalloc::XLargeChunk::range):
2649         (bmalloc::XLargeChunk::size):
2650         * bmalloc/bmalloc.h: Added.
2651         (bmalloc::api::malloc):
2652         (bmalloc::api::free):
2653         (bmalloc::api::realloc):
2654         * bmalloc/mbmalloc.cpp: Added.
2655