bmalloc: Miscellaneous cleanup
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc: Miscellaneous cleanup
4         https://bugs.webkit.org/show_bug.cgi?id=142231
5
6         Reviewed by Andreas Kling.
7
8         No performance change -- maybe a tiny reduction in memory use.
9
10         * bmalloc/Heap.cpp: Moved the sleep function into StaticMutex, since
11         it's a helper for working with mutexes.
12
13         (bmalloc::Heap::scavenge): Make sure to wait before we start any
14         scavenging, since individual scavenging functions now always scavenge
15         at least one page before waiting themselves.
16
17         (bmalloc::Heap::scavengeSmallPages):
18         (bmalloc::Heap::scavengeMediumPages):
19         (bmalloc::Heap::scavengeLargeObjects): Use the new wait helper to
20         simplify this code. Also, we now require our caller to wait until at
21         least one deallocation is desirable. This simplifies our loop.
22
23         (bmalloc::Heap::allocateSmallPage):
24         (bmalloc::Heap::allocateMediumPage):
25         (bmalloc::Heap::allocateXLarge):
26         (bmalloc::Heap::allocateLarge): Don't freak out any time the heap does
27         an allocation. Only consider the heap to be growing if it actually needs
28         to allocate new VM. This allows us to shrink the heap back down from a
29         high water mark more reliably even if heap activity continues.
30
31         (bmalloc::sleep): Deleted.
32         (bmalloc::Heap::scavengeLargeRanges): Renamed to match our use of
33         "LargeObject".
34
35         * bmalloc/Heap.h:
36
37         * bmalloc/LargeObject.h:
38         (bmalloc::LargeObject::operator bool): Added to simplify a while loop.
39
40         * bmalloc/StaticMutex.h:
41         (bmalloc::sleep):
42         (bmalloc::waitUntilFalse): New helper for waiting until a condition
43         becomes reliably false.
44
45         * bmalloc/Vector.h:
46         (bmalloc::Vector<T>::~Vector): Oops! Don't deallocate the null pointer.
47         We don't actually run any Vector destructors, but an iteration of this
48         patch did, and then crashed. So, let's fix that.
49
50 2015-03-02  Geoffrey Garen  <ggaren@apple.com>
51
52         bmalloc: Eagerly remove allocated objects from the free list
53         https://bugs.webkit.org/show_bug.cgi?id=142194
54
55         Reviewed by Andreas Kling.
56
57         This reduces the pressure to garbage collect the free list.
58
59         Might be a 1% speedup on MallocBench.
60
61         * bmalloc/FreeList.cpp: Put this comment at the top of the file instead
62         of repeating it inside of each function. Tried to clarify the details.
63
64         (bmalloc::FreeList::takeGreedy): Matched the other iteration code in this
65         file for consistency -- even though either direction works fine in this
66         function.
67
68         (bmalloc::FreeList::take): Change to iterate from low to high so that we
69         can maintain an index into the vector that is not disturbed even if we
70         pop from the middle (which invalidates the last index in the vector).
71
72         Decrement i when popping from the middle to make sure that we don't
73         skip the next item after popping.
74
75         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): Ditto.
76
77 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
78
79         Fixed a typo in the previous commit.
80
81         * bmalloc/BoundaryTag.h:
82         (bmalloc::BoundaryTag::setOwner):
83
84 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
85
86         EFL build fix after r180797.
87
88         * bmalloc/BoundaryTag.h:
89         (bmalloc::BoundaryTag::owner):
90         (bmalloc::BoundaryTag::setOwner):
91
92 2015-02-27  Geoffrey Garen  <ggaren@apple.com>
93
94         bmalloc: Pathological madvise churn on the free(malloc(x)) benchmark
95         https://bugs.webkit.org/show_bug.cgi?id=142058
96
97         Reviewed by Andreas Kling.
98
99         The churn was caused by repeatedly splitting an object with physical
100         pages from an object without, and then merging them back together again.
101         The merge would conservatively forget that we had physical pages, forcing
102         a new call to madvise on the next allocation.
103
104         This patch more strictly segregates objects in the heap from objects in
105         the VM heap, with these changes:
106
107         (1) Objects in the heap are not allowed to merge with objects in the VM
108         heap, and vice versa -- since that would erase our precise knowledge of
109         which physical pages had been allocated.
110
111         (2) The VM heap is exclusively responsible for allocating and deallocating
112         physical pages.
113
114         (3) The heap free list must consider entries for objects that are in the
115         VM heap to be invalid, and vice versa. (This condition can arise
116         because the free list does not eagerly remove items.)
117
118         With these changes, we can know that any valid object in the heap's free
119         list already has physical pages, and does not need to call madvise.
120
121         Note that the VM heap -- as before -- might sometimes contain ranges
122         or pieces of ranges that have physical pages, since we allow splitting
123         of ranges at granularities smaller than the VM page size. These ranges
124         can eventually merge with ranges in the heap during scavenging.
125
126         * bmalloc.xcodeproj/project.pbxproj:
127
128         * bmalloc/BoundaryTag.h:
129         (bmalloc::BoundaryTag::owner):
130         (bmalloc::BoundaryTag::setOwner):
131         (bmalloc::BoundaryTag::initSentinel):
132         (bmalloc::BoundaryTag::hasPhysicalPages): Deleted.
133         (bmalloc::BoundaryTag::setHasPhysicalPages): Deleted. Replaced the concept
134         of "has physical pages" with a bit indicating which heap owns the large
135         object. This is a more precise concept, since the old bit was really a
136         Yes / Maybe bit.
137
138         * bmalloc/Deallocator.cpp:
139
140         * bmalloc/FreeList.cpp: Adopt
141         (bmalloc::FreeList::takeGreedy):
142         (bmalloc::FreeList::take):
143         (bmalloc::FreeList::removeInvalidAndDuplicateEntries):
144         * bmalloc/FreeList.h:
145         (bmalloc::FreeList::push): Added API for considering the owner when
146         deciding if a free list entry is valid.
147
148         * bmalloc/Heap.cpp:
149         (bmalloc::Heap::Heap): Adopt new API.
150
151         (bmalloc::Heap::scavengeLargeRanges): Scavenge all ranges with no minimum,
152         since some ranges might be able to merge with ranges in the VM heap, and
153         they won't be allowed to until we scavenge them.
154
155         (bmalloc::Heap::allocateSmallPage):
156         (bmalloc::Heap::allocateMediumPage):
157         (bmalloc::Heap::allocateLarge): New VM heap API makes this function
158         simpler, since we always get back physical pages now.
159
160         * bmalloc/Heap.h:
161         * bmalloc/LargeObject.h:
162         (bmalloc::LargeObject::end):
163         (bmalloc::LargeObject::owner):
164         (bmalloc::LargeObject::setOwner):
165         (bmalloc::LargeObject::isValidAndFree):
166         (bmalloc::LargeObject::merge): Do not merge objects across heaps since
167         that causes madvise churn.
168         (bmalloc::LargeObject::validateSelf):
169         (bmalloc::LargeObject::init):
170         (bmalloc::LargeObject::hasPhysicalPages): Deleted.
171         (bmalloc::LargeObject::setHasPhysicalPages): Deleted. Propogate the Owner API.
172
173         * bmalloc/Owner.h: Added.
174
175         * bmalloc/SegregatedFreeList.cpp:
176         (bmalloc::SegregatedFreeList::SegregatedFreeList):
177         (bmalloc::SegregatedFreeList::insert):
178         (bmalloc::SegregatedFreeList::takeGreedy):
179         (bmalloc::SegregatedFreeList::take):
180         * bmalloc/SegregatedFreeList.h: Propogate the owner API.
181
182         * bmalloc/VMAllocate.h:
183         (bmalloc::vmDeallocatePhysicalPagesSloppy):
184         (bmalloc::vmAllocatePhysicalPagesSloppy): Clarified these functions and
185         removed an edge case.
186
187         * bmalloc/VMHeap.cpp:
188         (bmalloc::VMHeap::VMHeap):
189         * bmalloc/VMHeap.h:
190         (bmalloc::VMHeap::allocateSmallPage):
191         (bmalloc::VMHeap::allocateMediumPage):
192         (bmalloc::VMHeap::allocateLargeObject):
193         (bmalloc::VMHeap::deallocateLargeObject): Be sure to give each object
194         a new chance to merge, since it might have been prohibited from merging
195         before by virtue of not being in the VM heap.
196
197         (bmalloc::VMHeap::allocateLargeRange): Deleted.
198         (bmalloc::VMHeap::deallocateLargeRange): Deleted.
199
200 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
201
202         bmalloc: Large object free list can grow infinitely
203         https://bugs.webkit.org/show_bug.cgi?id=142055
204
205         Reviewed by Andreas Kling.
206
207         By design, we don't eagerly remove large objects from the free list.
208         This creates two simple pathologies:
209
210             (1) If you free and then allocate the same object repeatedly, it will
211             duplicate itself in the free list repeatedly. Since it is never
212             invalid at the time of allocation, it will never be removed.
213
214             (2) If you split and then merge the same object repeatedly, it will
215             duplicate its split sibling in the free list repeatedly. If its
216             sibling is in a separate free list size class, it will never be
217             consulted at the time of allocation, so it will never be removed.
218
219         So, a simple "while (1) { free(malloc(x)); }" causes infinite memory
220         use in the free list.
221
222         The solution in this patch is a simple helper to remove garbage from the
223         free list if it grows too large. This pathology is not common, so the
224         cost is OK.
225
226         Long-term, perhaps we should rethink the laziness of these free lists.
227
228         * bmalloc/BoundaryTag.h:
229         (bmalloc::BoundaryTag::isMarked):
230         (bmalloc::BoundaryTag::setMarked): New bit, used by free list GC.
231
232         * bmalloc/FreeList.cpp:
233         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): The GC algorithm.
234
235         * bmalloc/FreeList.h:
236         (bmalloc::FreeList::FreeList):
237         (bmalloc::FreeList::push): Invoke the GC if we're getting huge.
238
239         * bmalloc/LargeObject.h:
240         (bmalloc::LargeObject::isMarked):
241         (bmalloc::LargeObject::setMarked):
242         (bmalloc::LargeObject::validateSelf): Expose the new bit.
243
244         * bmalloc/Sizes.h: New constant to control GC frequency.
245
246 2015-02-26  Csaba Osztrogonác  <ossy@webkit.org>
247
248         URTBF after r180693.
249
250         * CMakeLists.txt:
251
252 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
253
254         Try to fix the Mac build.
255
256         Unreviewed.
257
258         * bmalloc.xcodeproj/project.pbxproj: Make FreeList.h available.
259
260 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
261
262         bmalloc: Refactored SegregatedFreeList and BoundaryTag::init
263         https://bugs.webkit.org/show_bug.cgi?id=142049
264
265         Reviewed by Anders Carlsson.
266
267         Split out a FreeList class from SegregatedFreeList. This will make it
268         easier to add behaviors on free list insertion and removal -- and it's
269         probably how I should have designed things at the start.
270
271         Moved BoundaryTag::init into LargeObject, since all the related logic
272         lives in LargeObject now too, and this allows us to remove BoundaryTagInlines.h.
273
274         * bmalloc.xcodeproj/project.pbxproj:
275         * bmalloc/BoundaryTagInlines.h: Removed.
276         * bmalloc/FreeList.cpp: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.cpp.
277         (bmalloc::FreeList::takeGreedy):
278         (bmalloc::FreeList::take):
279         (bmalloc::SegregatedFreeList::SegregatedFreeList): Deleted.
280         (bmalloc::SegregatedFreeList::insert): Deleted.
281         (bmalloc::SegregatedFreeList::takeGreedy): Deleted.
282         (bmalloc::SegregatedFreeList::take): Deleted.
283         * bmalloc/FreeList.h: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.h.
284         (bmalloc::FreeList::push):
285         * bmalloc/LargeObject.h:
286         (bmalloc::LargeObject::init):
287         * bmalloc/SegregatedFreeList.cpp:
288         (bmalloc::SegregatedFreeList::SegregatedFreeList):
289         (bmalloc::SegregatedFreeList::insert):
290         (bmalloc::SegregatedFreeList::takeGreedy):
291         (bmalloc::SegregatedFreeList::take):
292         * bmalloc/SegregatedFreeList.h:
293         * bmalloc/Sizes.h:
294         * bmalloc/VMHeap.cpp:
295         (bmalloc::VMHeap::grow):
296
297 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
298
299         bmalloc: free up a bit in BoundaryTag
300         https://bugs.webkit.org/show_bug.cgi?id=142048
301
302         Reviewed by Brady Eidson.
303
304         We were wasting a bit by accident, and I need one now.
305
306         * bmalloc/Algorithm.h:
307         (bmalloc::rightShift): Deleted. Not needed, now that I've simplified
308         the math.
309
310         * bmalloc/BoundaryTag.h: Since each boundary tag bucket is 1024 bytes
311         long, the maximum offset into a bucket is 1023.
312
313         You need 5 bits to count up to 1024, but only 4 to count up to 1023.
314         
315         Math is hard.
316
317         (bmalloc::BoundaryTag::compactBegin): Switched to division because it
318         is simpler, and easier to match up with our ASSERT. The compiler will
319         turn division by constant power of two into a shift for us.
320
321         (bmalloc::BoundaryTag::setRange): Added an ASSERT for compactBegin
322         because we do encode it, so we should ASSERT that encoding did not
323         lose information.
324
325         * bmalloc/Sizes.h: Shifting is no longer used since we use division
326         instead.
327
328 2015-02-24  Stephanie Lewis  <slewis@apple.com>
329
330         Rolling out http://trac.webkit.org/changeset/180430 as it causes the PLT to crash.
331         <rdar://problem/19948015>
332
333         Unreviewed.
334
335         * bmalloc/VMHeap.cpp:
336         (bmalloc::VMHeap::grow):
337         * bmalloc/VMHeap.h:
338         * bmalloc/Zone.cpp:
339         (bmalloc::Zone::Zone):
340         (bmalloc::Zone::size): Deleted.
341         * bmalloc/Zone.h:
342
343 2015-02-24  Geoffrey Garen  <ggaren@apple.com>
344
345         bmalloc: Added a little more abstraction for large objects
346         https://bugs.webkit.org/show_bug.cgi?id=141978
347
348         Reviewed by Sam Weinig.
349
350         Previously, each client needed to manage the boundary tags of
351         a large object using free functions. This patch introduces a LargeObject
352         class that does things a little more automatically.
353
354         * bmalloc.xcodeproj/project.pbxproj:
355
356         * bmalloc/Allocator.cpp:
357         (bmalloc::Allocator::reallocate): Use the new LargeObject class.
358
359         * bmalloc/BeginTag.h:
360         (bmalloc::BeginTag::isInFreeList): Deleted. Moved this logic into the
361         LargeObject class.
362
363         * bmalloc/BoundaryTag.h:
364         (bmalloc::BoundaryTag::isSentinel):
365         (bmalloc::BoundaryTag::compactBegin):
366         (bmalloc::BoundaryTag::setRange):
367         (bmalloc::BoundaryTag::initSentinel): Added an explicit API for sentinels,
368         which we used to create and test for implicitly.
369
370         * bmalloc/BoundaryTagInlines.h:
371         (bmalloc::BoundaryTag::init):
372         (bmalloc::validate): Deleted.
373         (bmalloc::validatePrev): Deleted.
374         (bmalloc::validateNext): Deleted.
375         (bmalloc::BoundaryTag::mergeLeft): Deleted.
376         (bmalloc::BoundaryTag::mergeRight): Deleted.
377         (bmalloc::BoundaryTag::merge): Deleted.
378         (bmalloc::BoundaryTag::deallocate): Deleted.
379         (bmalloc::BoundaryTag::split): Deleted.
380         (bmalloc::BoundaryTag::allocate): Deleted. Moved this logic into the
381         LargeObject class.
382
383         * bmalloc/EndTag.h:
384         (bmalloc::EndTag::init):
385         (bmalloc::EndTag::operator=): Deleted. Re-reading this code, I found
386         special behavior in the assignment operator to be a surprising API.
387         So, I replaced the assignment operation with an explicit initializing
388         function.
389
390         * bmalloc/Heap.cpp:
391         (bmalloc::Heap::scavengeLargeRanges):
392         (bmalloc::Heap::allocateXLarge):
393         (bmalloc::Heap::findXLarge):
394         (bmalloc::Heap::deallocateXLarge):
395         (bmalloc::Heap::allocateLarge):
396         (bmalloc::Heap::deallocateLarge):
397         * bmalloc/Heap.h: No behavior changes here -- just adopting the
398         LargeObject interface.
399
400         * bmalloc/LargeObject.h: Added.
401         (bmalloc::LargeObject::operator!):
402         (bmalloc::LargeObject::begin):
403         (bmalloc::LargeObject::size):
404         (bmalloc::LargeObject::range):
405         (bmalloc::LargeObject::LargeObject):
406         (bmalloc::LargeObject::setFree):
407         (bmalloc::LargeObject::isFree):
408         (bmalloc::LargeObject::hasPhysicalPages):
409         (bmalloc::LargeObject::setHasPhysicalPages):
410         (bmalloc::LargeObject::isValidAndFree):
411         (bmalloc::LargeObject::merge):
412         (bmalloc::LargeObject::split):
413         (bmalloc::LargeObject::validateSelf):
414         (bmalloc::LargeObject::validate): Moved this code into a class, out of
415         BoundaryTag free functions.
416
417         New to the class are these features:
418
419             (1) Every reference to an object is validated upon creation and use.
420
421             (2) There's an explicit API for "This is a reference to an object
422             that might be stale (the DoNotValidate API)".
423
424             (3) The begin and end tags are kept in sync automatically.
425
426         * bmalloc/SegregatedFreeList.cpp:
427         (bmalloc::SegregatedFreeList::insert):
428         (bmalloc::SegregatedFreeList::takeGreedy):
429         (bmalloc::SegregatedFreeList::take):
430         * bmalloc/SegregatedFreeList.h: Adopt the LargeObject interface.
431
432         * bmalloc/VMHeap.cpp:
433         (bmalloc::VMHeap::grow):
434         * bmalloc/VMHeap.h:
435         (bmalloc::VMHeap::allocateLargeRange):
436         (bmalloc::VMHeap::deallocateLargeRange): Adopt the LargeObject interface.
437
438 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
439
440         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
441         https://bugs.webkit.org/show_bug.cgi?id=141802
442
443         Reviewed by Andreas Kling.
444
445         Rolling back in with a fix for a crash seen while using GuardMalloc.
446
447         * bmalloc/VMHeap.cpp:
448         (bmalloc::VMHeap::grow):
449         * bmalloc/VMHeap.h:
450         * bmalloc/Zone.cpp: Re-land the old patch.
451
452         (bmalloc::Zone::size): Be sure to implement the size() function since
453         it's accessible indirectly via the malloc_zone_from_ptr public API --
454         and GuardMalloc calls it all the time.
455
456         (bmalloc::Zone::Zone):
457         * bmalloc/Zone.h: Re-land the old patch.
458
459 2015-02-19  Commit Queue  <commit-queue@webkit.org>
460
461         Unreviewed, rolling out r180363.
462         https://bugs.webkit.org/show_bug.cgi?id=141814
463
464         Caused >50 crashes when running LayoutTests in GuardMalloc or
465         ASAN modes. (Requested by jernoble on #webkit).
466
467         Reverted changeset:
468
469         "bmalloc should implement malloc introspection (to stop false-
470         positive leaks when MallocStackLogging is off)"
471         https://bugs.webkit.org/show_bug.cgi?id=141802
472         http://trac.webkit.org/changeset/180363
473
474 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
475
476         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
477         https://bugs.webkit.org/show_bug.cgi?id=141802
478
479         Reviewed by Andreas Kling.
480
481         Fixed a last-minute type.
482
483         The macro is OS, not PLATFORM.
484
485         * bmalloc/VMHeap.cpp:
486         (bmalloc::VMHeap::grow):
487         * bmalloc/VMHeap.h:
488         * bmalloc/Zone.h:
489
490 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
491
492         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
493         https://bugs.webkit.org/show_bug.cgi?id=141802
494
495         Reviewed by Andreas Kling.
496
497         This patch does the bare minimum to stop false positive leaks from
498         being reported by the Darwin leaks tool. We register each super chunk
499         as a single object, and then request that the leaks tool scan it.
500
501         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
502         zone introspection API.
503
504         * bmalloc/Algorithm.h: Missing #include.
505
506         * bmalloc/VMHeap.cpp:
507         (bmalloc::VMHeap::grow):
508         * bmalloc/VMHeap.h: Adopt the new abstraction.
509
510         * bmalloc/Zone.cpp: Added.
511         (bmalloc::remoteRead): Helper for reading an object out of another process.
512         (bmalloc::Zone::enumerator):
513         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
514         in introspection.
515
516         * bmalloc/Zone.h: Added.
517         (bmalloc::Zone::superChunks):
518         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
519         since our dynamic allocations will not be scanned by leaks since they
520         will have the malloc VM tag.
521
522 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
523
524         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
525         https://bugs.webkit.org/show_bug.cgi?id=141759
526
527         Reviewed by Andreas Kling.
528
529         * bmalloc.xcodeproj/project.pbxproj:
530         * bmalloc/SuperChunk.h: Added.
531         (bmalloc::SuperChunk::create):
532         (bmalloc::SuperChunk::SuperChunk):
533         (bmalloc::SuperChunk::smallChunk):
534         (bmalloc::SuperChunk::mediumChunk):
535         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
536         into a separate class, for clarity and type safety.
537
538         * bmalloc/VMHeap.cpp:
539         (bmalloc::VMHeap::grow):
540         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
541         "grow" because Andreas found "allocateSuperChunk" to be unclear.
542
543         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
544         for malloc introspection.
545
546         (bmalloc::VMHeap::allocateSmallPage):
547         (bmalloc::VMHeap::allocateMediumPage):
548         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
549
550 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
551
552         Build bmalloc through CMake as a static library. It's then linked either
553         into the WTF library (if built as a shared library) or into the JSC and
554         WebKit2 libraries. There's no need to build it as a standalone shared library.
555
556         Rubber-stamped by Carlos Garcia Campos.
557
558         * CMakeLists.txt:
559
560 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
561
562         [BMalloc] Add a FIXME comment for memory alignas
563         https://bugs.webkit.org/show_bug.cgi?id=141556
564
565         Reviewed by Csaba Osztrogonác.
566
567         * bmalloc/Chunk.h: Add a FIXME comment.
568         * bmalloc/LargeChunk.h: ditto.
569
570 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
571
572         bmalloc buildfix on 32 bit Linux (x86/ARM)
573         https://bugs.webkit.org/show_bug.cgi?id=141472
574
575         Reviewed by Gyuyoung Kim.
576
577         * bmalloc/Algorithm.h:
578         (bmalloc::roundUpToMultipleOf):
579         * bmalloc/FixedVector.h:
580         (bmalloc::FixedVector::clear):
581         * bmalloc/Sizes.h:
582         (bmalloc::Sizes::sizeClass):
583
584 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
585
586         [EFL][GTK] Use bmalloc instead of tcmalloc
587         https://bugs.webkit.org/show_bug.cgi?id=140162
588
589         Reviewed by Carlos Garcia Campos.
590
591         Support to use bmalloc on EFL and GTK ports.
592
593         * CMakeLists.txt: Added.
594         * bmalloc/Allocator.cpp:
595         (bmalloc::Allocator::allocate):
596             Fix unused return value caused by posix_memalign().
597         * bmalloc/AsyncTask.h:
598         * bmalloc/BoundaryTag.h:
599         (bmalloc::BoundaryTag::clear):
600         * bmalloc/Chunk.h:
601             Change Traits::Page with Traits::PageType in order to fix
602             -fpermitive build error on EFL and GTK port. 
603         * bmalloc/EndTag.h:
604         (bmalloc::EndTag::operator=):
605         * bmalloc/Line.h: ditto.
606         * bmalloc/MediumTraits.h:
607         * bmalloc/Page.h: ditto.
608         * bmalloc/PerThread.h:
609             EFL port doesn't support __has_include definition yet.
610             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
611         * bmalloc/SmallTraits.h: ditto.
612         * bmalloc/VMAllocate.h:
613         (bmalloc::vmDeallocatePhysicalPages):
614         (bmalloc::vmAllocatePhysicalPages):
615         * bmalloc/Vector.h:
616         (bmalloc::Vector<T>::push):
617         (bmalloc::Vector<T>::reallocateBuffer):
618
619 2015-01-31  Sam Weinig  <sam@webkit.org>
620
621         Remove even more Mountain Lion support
622         https://bugs.webkit.org/show_bug.cgi?id=141124
623
624         Reviewed by Alexey Proskuryakov.
625
626         * Configurations/Base.xcconfig:
627         * Configurations/DebugRelease.xcconfig:
628
629 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
630
631         GC marking threads should clear malloc caches
632         https://bugs.webkit.org/show_bug.cgi?id=141097
633
634         Reviewed by Andreas Kling.
635
636         Split the scavenging API into per-thread vs global, so that you can
637         request to scavenge your own thread without scavenging the whole heap.
638
639         * bmalloc/Cache.cpp:
640         (bmalloc::Cache::scavenge):
641         * bmalloc/bmalloc.h:
642         (bmalloc::api::scavengeThisThread):
643         (bmalloc::api::scavenge):
644
645 2015-01-28  Dana Burkart  <dburkart@apple.com>
646
647         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
648         https://bugs.webkit.org/show_bug.cgi?id=136765
649
650         Reviewed by Alexey Proskuryakov.
651
652         * Configurations/Base.xcconfig:
653         * Configurations/DebugRelease.xcconfig:
654
655 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
656
657         bmalloc: support aligned allocation
658         https://bugs.webkit.org/show_bug.cgi?id=140732
659
660         Reviewed by Andreas Kling.
661
662         * bmalloc/Allocator.cpp:
663         (bmalloc::Allocator::allocate): New function for aligned allocation.
664
665         Small and medium requests just allocate and free until they find an
666         aligned pointer. This is slightly inefficient in the worst case, but
667         still constant-time with little-to-no space overhead.
668
669         Large requests use a new API that requires the client to specify both
670         its ideal size and alignment, and the worst-case size you would have to
671         allocate in order to produce some interior pointer of the requested size
672         and alignment. We put the burden of this calculation on the client
673         because it simplifies things if we guarantee that allocation won't fail.
674
675         XLarge requests are easy: we just forward them to vmAllocate, which
676         already supported aligned requests.
677
678         * bmalloc/BoundaryTag.h:
679         * bmalloc/BoundaryTagInlines.h:
680         (bmalloc::BoundaryTag::mergeLeft):
681         (bmalloc::BoundaryTag::mergeRight):
682         (bmalloc::BoundaryTag::merge):
683         (bmalloc::BoundaryTag::deallocate):
684         (bmalloc::BoundaryTag::split):
685         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
686         refactored the interface to remove some reference out parameters in
687         order to clarify what changes and what doesn't.
688
689         * bmalloc/Heap.cpp:
690         (bmalloc::Heap::allocateXLarge): Added an alignment API.
691
692         (bmalloc::Heap::allocateLarge):
693         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
694         a few variants, so aligned and unaligned allocation could share some code.
695
696         * bmalloc/SegregatedFreeList.cpp:
697         (bmalloc::SegregatedFreeList::take):
698         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
699         for aligned allocation. It turns out that the aligned path is pretty
700         different, since it ends up searching for two potential ways to satisfy
701         an allocation: either large enough and aligned, or large enough to split
702         into something not aligned and something large enough and aligned.
703
704         * bmalloc/VMAllocate.h:
705         (bmalloc::vmAllocate): Switched alignment to come before size because
706         that's how the memalign API specifies it.
707
708         * bmalloc/VMHeap.h:
709         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
710
711 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
712
713         bmalloc: a little bit of cleanup
714         https://bugs.webkit.org/show_bug.cgi?id=140687
715
716         Reviewed by Anders Carlsson.
717
718         * bmalloc/Algorithm.h:
719         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
720         of code.
721
722         * bmalloc/BoundaryTag.h:
723         * bmalloc/BoundaryTagInlines.h:
724         (bmalloc::BoundaryTag::mergeLeft):
725         (bmalloc::BoundaryTag::mergeRight):
726         (bmalloc::BoundaryTag::merge):
727         (bmalloc::BoundaryTag::deallocate):
728         (bmalloc::BoundaryTag::split):
729         (bmalloc::BoundaryTag::allocate):
730         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
731         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
732         (bmalloc::BoundaryTag::mergeLarge): Deleted.
733         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
734         from all these functions, since boundary tags always pertain to large
735         objects, and putting the word "Large" everywhere wasn't helping to
736         explain that.
737
738         * bmalloc/Heap.cpp:
739         (bmalloc::Heap::allocateXLarge):
740         (bmalloc::Heap::findXLarge):
741         (bmalloc::Heap::deallocateXLarge):
742         * bmalloc/Heap.h:
743         * bmalloc/VMHeap.h:
744         (bmalloc::VMHeap::allocateXLarge): Deleted.
745         (bmalloc::VMHeap::findXLarge): Deleted.
746         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
747         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
748         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
749         really make sense for the VMHeap to be involved.
750
751 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
752
753         bmalloc: refactored XLarge allocation for better alignment
754         https://bugs.webkit.org/show_bug.cgi?id=140582
755
756         Reviewed by Andreas Kling.
757
758         XLarge objects used to be Large objects with an extra bit of metadata
759         that said "actually, I'm not large -- I'm extra large".
760
761         The metadata header in an XLarge allocation made it impossible for the
762         XLarge object to honor a very large alignment request.
763
764         The solution is to stop using a metadata header for XLarge objects, and
765         instead to store explicit metadata on the side.
766
767         This is a bit less astonishing, which is also nice.
768
769         Finding XLarge metadata is now a linear search. That's probably OK, since
770         it was always so in TCMalloc, and the usual number of XLarge allocations
771         in a process is 0.
772
773         This design makes it possible for the heap to cache XLarge allocations
774         with and/or without physical pages. I haven't actually done that yet
775         because the tradeoffs are subtle, so I don't want to do anything without
776         a motivating test case.
777
778         * bmalloc.xcodeproj/project.pbxproj:
779         * bmalloc/Allocator.cpp:
780         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
781         since an XLarge allocation is now just a naked buffer without a header.
782
783         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
784         XLarge alignment, since XLargeChunk won't give this to us implicitly
785         anymore.
786
787         * bmalloc/BoundaryTag.h:
788         (bmalloc::BoundaryTag::setRange):
789         (bmalloc::BoundaryTag::isXLarge): Deleted.
790         (bmalloc::BoundaryTag::setXLarge): Deleted.
791         * bmalloc/BoundaryTagInlines.h:
792         (bmalloc::validate):
793         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
794
795         * bmalloc/Deallocator.cpp:
796         (bmalloc::Deallocator::deallocateXLarge):
797         (bmalloc::Deallocator::deallocateSlowCase):
798         * bmalloc/Heap.cpp:
799         (bmalloc::Heap::findXLarge):
800         (bmalloc::Heap::allocateXLarge):
801         (bmalloc::Heap::deallocateXLarge):
802         * bmalloc/Heap.h: Updated for interface changes.
803
804         * bmalloc/ObjectType.cpp:
805         (bmalloc::objectType):
806         * bmalloc/ObjectType.h:
807         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
808         examining its bit pattern -- just like we do for other kinds of
809         allocations -- which is nice.
810
811         * bmalloc/Sizes.h:
812         * bmalloc/VMHeap.h:
813         (bmalloc::VMHeap::allocateXLarge):
814         (bmalloc::VMHeap::findXLarge):
815         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
816         for XLarge allocations.
817
818         * bmalloc/XLargeChunk.h: Removed.
819
820 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
821
822         bmalloc: added some infrastructure for aligned allocation
823         https://bugs.webkit.org/show_bug.cgi?id=140572
824
825         Reviewed by Andreas Kling.
826
827         * bmalloc/Algorithm.h:
828         (bmalloc::isPowerOfTwo):
829         (bmalloc::roundUpToMultipleOf):
830         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
831         isPowerOfTwo helper function.
832
833         * bmalloc/Allocator.cpp:
834         (bmalloc::Allocator::allocate):
835         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
836         Doesn't do anything yet, but does correctly forward to system malloc
837         when bmalloc is disabled.
838
839         * bmalloc/Cache.cpp:
840         (bmalloc::Cache::allocateSlowCaseNullCache):
841         * bmalloc/Cache.h:
842         (bmalloc::Cache::allocate):
843         * bmalloc/bmalloc.h:
844         (bmalloc::api::memalign):
845         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
846
847 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
848
849         Consider alignment when allocating from a SegregatedFreeList
850         https://bugs.webkit.org/show_bug.cgi?id=140408
851
852         Reviewed by Sam Weinig.
853
854         In preparation for supporting aligned allocation.
855
856         No performance change.
857
858         Since this is just one extra branch in an already expensive function,
859         I decided not to duplicate the function just to avoid the branch in
860         the un-aligned case.
861
862         * bmalloc/SegregatedFreeList.cpp:
863         (bmalloc::SegregatedFreeList::take):
864         * bmalloc/SegregatedFreeList.h:
865
866 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
867
868         Renamed minimum to size in SegregatedFreeList
869         https://bugs.webkit.org/show_bug.cgi?id=140406
870
871         Reviewed by Sam Weinig.
872
873         In preparation for supporting aligned allocation.
874
875         * bmalloc/SegregatedFreeList.cpp:
876         (bmalloc::SegregatedFreeList::takeGreedy):
877         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
878         really just a minimum. Let's not imply that this value is special.
879
880 2015-01-11  Dan Bernstein  <mitz@apple.com>
881
882         Geoff is organized, but he is not an organization.
883
884         Rubber-stamped by Anders Carlsson.
885
886         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
887
888 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
889
890         Make bmalloc work with ASan
891         https://bugs.webkit.org/show_bug.cgi?id=140194
892
893         Reviewed by Mark Lam.
894
895         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
896         an OS-specific API to test for loaded runtime libraries.
897
898         * bmalloc/Environment.cpp:
899         (bmalloc::isASanEnabled):
900         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
901         ASan is enabled, since system malloc has the Asan hooks we need.
902
903         You could check for the ASan compile-time flag instead, but doing this
904         check at runtime prepares bmalloc for a world where it is a dynamic
905         library that might be loaded into projects it did not compile with.
906
907 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
908
909         Fix up bmalloc's PerThread for use on Linux
910         https://bugs.webkit.org/show_bug.cgi?id=139804
911
912         Reviewed by Anders Carlsson.
913
914         The previous implementation was a bit slow.
915
916         * bmalloc/PerThread.h:
917         (bmalloc::PerThreadStorage<Cache>::get):
918         (bmalloc::PerThreadStorage::get):
919         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
920         way to do fast per-thread access without taking a lock every time. This
921         probably works on all the platforms we care about, and it matches other
922         techniques we use elsewhere in WebKit.
923
924         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
925         this class because PerThreadStorage now encapsulates everything that
926         needs to be conditional.
927
928         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
929
930 2014-12-26  Dan Bernstein  <mitz@apple.com>
931
932         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
933         https://bugs.webkit.org/show_bug.cgi?id=139950
934
935         Reviewed by David Kilzer.
936
937         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
938         in a manner that works with Xcode 5.1.1.
939
940 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
941
942         Safari crashes when you set Malloc environment variables
943         https://bugs.webkit.org/show_bug.cgi?id=139656
944
945         Reviewed by Michael Saboff.
946
947         I forgot to cover the realloc() case. Whoops. (OoPS?)
948
949         This time around, I ran the full MallocBench test suite in Malloc=1
950         mode, and it passed.
951
952         * bmalloc/Allocator.cpp:
953         (bmalloc::Allocator::reallocate):
954         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
955         It needs to be down there so that we can do the short-circuiting check
956         for whether bmalloc is enabled first.
957
958         Also added the check.
959
960         * bmalloc/Cache.cpp:
961         (bmalloc::Cache::scavenge):
962         (bmalloc::Cache::Cache):
963         (bmalloc::Cache::reallocateSlowCaseNullCache):
964         * bmalloc/Cache.h:
965         (bmalloc::Cache::deallocator):
966         (bmalloc::Cache::reallocate): Ditto.
967
968         * bmalloc/bmalloc.h:
969         (bmalloc::api::free):
970         (bmalloc::api::realloc): Ditto.
971
972         (bmalloc::api::scavenge): Pushed this down into Cache to match the
973         surrounding functions.
974
975 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
976
977         bmalloc should support system memory analysis tools (part 2)
978         https://bugs.webkit.org/show_bug.cgi?id=139565
979
980         Reviewed by Mark Lam.
981
982         This patch actually queries the environment to see if memory analysis
983         tools have been enabled.
984
985         * bmalloc/Deallocator.cpp:
986         (bmalloc::Deallocator::scavenge): Don't process the object log if
987         we've disabled bmalloc because it will be full of invalid nullptrs.
988
989         * bmalloc/Environment.cpp:
990         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
991         Malloc debugging flags. I also added a plain "Malloc" catch-all for
992         when you want to disable bmalloc without enabling any kind of funny
993         business.
994
995         It would be slightly nicer just to iterate the list of environment
996         variables and strstr them, but getenv is the more portable option,
997         and performance here doesn't really matter.
998
999         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
1000         environment variable.
1001
1002         (bmalloc::Environment::computeIsBmallocEnabled):
1003
1004 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1005
1006         Try to fix the iOS simulator build.
1007
1008         #include the declaration of malloc / free.
1009
1010         * bmalloc/Allocator.cpp:
1011         * bmalloc/Deallocator.cpp:
1012
1013 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1014
1015         Try to fix the build.
1016
1017         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
1018
1019 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1020
1021         bmalloc should support system memory analysis tools (part 1)
1022         https://bugs.webkit.org/show_bug.cgi?id=139559
1023
1024         Reviewed by Mark Lam.
1025
1026         This patch adds the hooks to disable bmalloc at runtime if certain
1027         environment variables are set, but doesn't actually read from the
1028         environment yet.
1029
1030         No performance change.
1031
1032         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
1033         we'll use to read environment variables and see if memory analysis tools
1034         have been enabled.
1035
1036         * bmalloc/Allocator.cpp:
1037         (bmalloc::Allocator::Allocator):
1038         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
1039         on the allocation path. We cache the setting to make the check fast.
1040
1041         * bmalloc/Allocator.h: Interface changes.
1042
1043         * bmalloc/Cache.cpp:
1044         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
1045         and deallocator. This main purpose is to enable them to query the
1046         environment for whether bmalloc is enabled; but this is also a slightly
1047         cleaner way to guarantee to them that the Heap has been pre-initialized.
1048
1049         * bmalloc/Deallocator.cpp:
1050         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
1051         fill the object log to force us to take the slow path on all deallocations.
1052
1053         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
1054
1055         * bmalloc/Deallocator.h: Interface changes.
1056
1057         * bmalloc/Environment.cpp: Added.
1058         (bmalloc::Environment::Environment):
1059         (bmalloc::Environment::computeIsBmallocEnabled):
1060         * bmalloc/Environment.h: Added.
1061         (bmalloc::Environment::isBmallocEnabled): This is the class that will
1062         encapsulate looking for environment variables that turn on heap
1063         analysis tools.
1064
1065         * bmalloc/Heap.h:
1066         (bmalloc::Heap::environment):
1067
1068         * bmalloc/Mutex.h:
1069         (bmalloc::Mutex::Mutex):
1070         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
1071         since I got super confused about them while writing this patch.
1072
1073         * bmalloc/VMHeap.cpp: Fixed an #include.
1074
1075 2014-12-09  David Kilzer  <ddkilzer@apple.com>
1076
1077         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
1078         <http://webkit.org/b/139212>
1079
1080         Reviewed by Joseph Pecoraro.
1081
1082         * Configurations/Base.xcconfig:
1083         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
1084           on OS X.
1085         * Configurations/DebugRelease.xcconfig:
1086         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
1087
1088 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
1089
1090         bmalloc uses 8X more virtual memory than necessary
1091         https://bugs.webkit.org/show_bug.cgi?id=138495
1092
1093         Reviewed by Mark Lam.
1094
1095         iOS has a per-process virtual memory cap around 1GB, so there's some
1096         value to not going totally ham with virtual memory.
1097
1098         We currently use about 8X the necessary amount:
1099             - 2X to align our VM allocation
1100             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
1101
1102         We can cut that down:
1103             - Return the unaligned portion of our VM allocation (-2X)
1104             - Use all the chunks in a superchunk, instead of allocating one
1105               chunk per superchunk (-4X)
1106
1107         * bmalloc/Algorithm.h:
1108         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
1109         function so we can call it with getpagesize() at runtime.
1110
1111         * bmalloc/Chunk.h:
1112         * bmalloc/LargeChunk.h:
1113         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
1114         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
1115
1116         * bmalloc/VMAllocate.h:
1117         (bmalloc::vmValidate):
1118         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
1119         if it does not succeed. Allocate precisely, and give back the extra.
1120
1121         * bmalloc/VMHeap.cpp:
1122         (bmalloc::VMHeap::allocateSuperChunk):
1123         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
1124         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
1125         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
1126         in a superchunk, instead of just one.
1127
1128         * bmalloc/VMHeap.h:
1129         (bmalloc::VMHeap::allocateSmallPage):
1130         (bmalloc::VMHeap::allocateMediumPage):
1131         (bmalloc::VMHeap::allocateLargeRange):
1132         * bmalloc/XLargeChunk.h:
1133         (bmalloc::XLargeChunk::create): Updated to match changes above.
1134
1135 2014-11-01  David Kilzer  <ddkilzer@apple.com>
1136
1137         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
1138         <https://webkit.org/b/138286>
1139         <rdar://problem/18847087>
1140
1141         Reviewed by Dan Bernstein.
1142
1143         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
1144         to NO for the target that produces libbmalloc.a so that the
1145         debug symbols will be linked into JavaScriptCore and end up in
1146         its dSYM file.
1147
1148 2014-10-30  Dana Burkart  <dburkart@apple.com>
1149
1150         <rdar://problem/18821260> Prepare for the mysterious future
1151
1152         Reviewed by Lucas Forschler.
1153
1154         * Configurations/Base.xcconfig:
1155         * Configurations/DebugRelease.xcconfig:
1156
1157 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
1158
1159         bmalloc: cleaned up fast path vs slow path
1160         https://bugs.webkit.org/show_bug.cgi?id=137081
1161
1162         Reviewed by Sam Weinig.
1163
1164         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
1165
1166         * bmalloc/Allocator.cpp:
1167         (bmalloc::Allocator::Allocator): Merged the small and medium range
1168         caches, just like the small and medium allocators. Ranges are abstract
1169         objects that don't really care whether they hold small or medium objects,
1170         so they don't need to be segregated.
1171
1172         (bmalloc::Allocator::scavenge): Ditto.
1173
1174         (bmalloc::Allocator::allocateBumpRangeSlowCase):
1175         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
1176         we do care a tiny bit, because we need to specify small vs medium when
1177         allocating new ranges from the heap, to ensure that the heap allocates
1178         from the right segment of VM.
1179
1180         (bmalloc::Allocator::allocateLarge):
1181         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
1182         up the fast path. Large allocation performance is dominated by allocation
1183         logic and initialization, so inlining it doesn't help.
1184
1185         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
1186         it doesn't need to distinguish small vs medium objects.
1187
1188         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
1189         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
1190
1191         * bmalloc/Allocator.h:
1192         * bmalloc/BumpRange.h:
1193
1194         * bmalloc/Cache.cpp:
1195         (bmalloc::Cache::allocateSlowCase): Deleted.
1196         (bmalloc::Cache::deallocateSlowCase): Deleted.
1197         * bmalloc/Cache.h:
1198         (bmalloc::Cache::allocate):
1199         (bmalloc::Cache::deallocate):
1200         (bmalloc::Cache::allocateFastCase): Deleted.
1201         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
1202         paths. The downside to this change is that the fast path branches to two
1203         distinct failure cases instead of one. The upside is that the slow path
1204         doesn't need to re-read the segment register, which is not as cheap as a
1205         normal register, and it doesn't need to do an extra level of function 
1206         call. Seems to be worth it.
1207
1208         * bmalloc/Deallocator.h:
1209         * bmalloc/Heap.cpp:
1210         (bmalloc::Heap::refillSmallBumpRangeCache):
1211         (bmalloc::Heap::refillMediumBumpRangeCache):
1212         * bmalloc/Heap.h: Updated for interface changes.
1213
1214         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
1215         small lines in a page / 2, since any other free lines will coalesce
1216         with their neighbors.
1217
1218 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1219
1220         Rolled out r173346.
1221
1222             bmalloc should honor the FastMalloc statistics API
1223             https://bugs.webkit.org/show_bug.cgi?id=136592
1224
1225         This didn't really work. Because we allow ranges with and without
1226         physical pages to merge, and we allow double-committing and
1227         double-decommitting, we can't rely on commit actions to track memory
1228         footprint.
1229
1230         * bmalloc/Heap.cpp:
1231         (bmalloc::Heap::size): Deleted.
1232         (bmalloc::Heap::capacity): Deleted.
1233         * bmalloc/Heap.h:
1234         * bmalloc/VMHeap.cpp:
1235         (bmalloc::VMHeap::VMHeap):
1236         (bmalloc::VMHeap::allocateSmallChunk):
1237         (bmalloc::VMHeap::allocateMediumChunk):
1238         (bmalloc::VMHeap::allocateLargeChunk):
1239         * bmalloc/VMHeap.h:
1240         (bmalloc::VMHeap::allocateSmallPage):
1241         (bmalloc::VMHeap::allocateMediumPage):
1242         (bmalloc::VMHeap::allocateLargeRange):
1243         (bmalloc::VMHeap::deallocateSmallPage):
1244         (bmalloc::VMHeap::deallocateMediumPage):
1245         (bmalloc::VMHeap::deallocateLargeRange):
1246         (bmalloc::VMHeap::size): Deleted.
1247         (bmalloc::VMHeap::capacity): Deleted.
1248         * bmalloc/bmalloc.h:
1249         (bmalloc::api::heapSize): Deleted.
1250         (bmalloc::api::heapCapacity): Deleted.
1251
1252 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1253
1254         bmalloc: Allocation should be more precise
1255         https://bugs.webkit.org/show_bug.cgi?id=136993
1256
1257         Reviewed by Gavin Barraclough.
1258
1259         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1260
1261         This patch teaches the allocator to merge adjacent free lines into a
1262         single allocatable range. This allows us to shrink the size of an
1263         individual line without increasing fragmentation or the rate of allocator
1264         slow paths.
1265
1266         We'll only take more slow paths when available memory is sparse, which
1267         is exactly when it's worth it. When available memory is dense, we'll
1268         take fewer slow paths.
1269
1270         * bmalloc.xcodeproj/project.pbxproj:
1271         * bmalloc/Algorithm.h:
1272         (bmalloc::divideRoundingUp):
1273
1274         * bmalloc/Allocator.cpp:
1275         (bmalloc::Allocator::Allocator): Updated for interface changes.
1276
1277         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
1278         Now that we merge lines, it's not convenient to scavenge by line.
1279
1280         (bmalloc::Allocator::allocateSmallBumpRange):
1281         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
1282         instead of individual lines.
1283
1284         (bmalloc::Allocator::allocateSlowCase):
1285         (bmalloc::Allocator::allocateSmallLine): Deleted.
1286         (bmalloc::Allocator::allocateMediumLine): Deleted.
1287         (bmalloc::Allocator::allocateMedium): Deleted.
1288         * bmalloc/Allocator.h:
1289         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
1290         into the standard fast path with small allocations. Since a BumpAllocator
1291         just allocates out of an arbitrary range, it doesn't need to distinguish
1292         between small and medium lines.
1293
1294         * bmalloc/BumpAllocator.h:
1295         (bmalloc::BumpAllocator::size):
1296         (bmalloc::BumpAllocator::BumpAllocator):
1297         (bmalloc::BumpAllocator::init):
1298         (bmalloc::BumpAllocator::refill):
1299         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
1300         anymore: the heap just gives us a pointer and a pre-computed number of
1301         objects, and we allocate them.
1302
1303         * bmalloc/Deallocator.cpp:
1304         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
1305
1306         * bmalloc/Heap.cpp:
1307         (bmalloc::Heap::Heap):
1308         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
1309         detailing where all objects will lie in memory. After we merge two lines,
1310         we might allocate an object that spans from one line to the next. This
1311         metadata details which bits of memory overlap in that way, and how they
1312         overlap.
1313
1314         (bmalloc::Heap::refillSmallBumpRangeCache):
1315         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
1316         and merge adjacent free lines into BumpRanges.
1317
1318         (bmalloc::Heap::allocateSmallPage):
1319         (bmalloc::Heap::allocateMediumPage):
1320         (bmalloc::Heap::deallocateSmallLine):
1321         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
1322         since we scan for free memory a page at a time.
1323
1324         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
1325         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
1326         fast path.
1327
1328         * bmalloc/Heap.h:
1329         (bmalloc::Heap::derefSmallLine):
1330         (bmalloc::Heap::derefMediumLine):
1331         (bmalloc::Heap::deallocateSmallLine): Deleted.
1332         (bmalloc::Heap::allocateSmallLine): Deleted.
1333         (bmalloc::Heap::deallocateMediumLine): Deleted.
1334         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
1335
1336         * bmalloc/Line.h:
1337         (bmalloc::Line<Traits>::ref):
1338         (bmalloc::Line<Traits>::deref):
1339         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
1340         anymore, since we only ever deref by 1 now.
1341
1342         * bmalloc/MediumAllocator.h:
1343         (bmalloc::MediumAllocator::isNull): Deleted.
1344         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
1345         (bmalloc::MediumAllocator::line): Deleted.
1346         (bmalloc::MediumAllocator::allocate): Deleted.
1347         (bmalloc::MediumAllocator::derefCount): Deleted.
1348         (bmalloc::MediumAllocator::refill): Deleted.
1349         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
1350         been dead for a while, since it doesn't build anymore with this patch.
1351
1352         * bmalloc/Page.h:
1353         (bmalloc::Page::sizeClass):
1354         (bmalloc::Page::setSizeClass):
1355         (bmalloc::Page::smallSizeClass): Deleted.
1356         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
1357         to sizeClass, since we use it for medium sizes too.
1358
1359         * bmalloc/Sizes.h:
1360         (bmalloc::Sizes::sizeClass):
1361         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
1362
1363         (bmalloc::Sizes::smallSizeClassFor): Deleted.
1364         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
1365
1366         * bmalloc/bmalloc.h:
1367         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
1368         can be a bit more precise. It also has to be, since we can't guarantee
1369         that an object ends at the end of a line anymore.
1370
1371 2014-09-19  Daniel Bates  <dabates@apple.com>
1372
1373         Always assume internal SDK when building configuration Production
1374         https://bugs.webkit.org/show_bug.cgi?id=136925
1375         <rdar://problem/18362399>
1376
1377         Reviewed by Dan Bernstein.
1378
1379         * Configurations/Base.xcconfig:
1380
1381 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
1382
1383         bmalloc: moved line caches from the deallocator to the allocator
1384         https://bugs.webkit.org/show_bug.cgi?id=136868
1385
1386         Reviewed by Gavin Barraclough.
1387
1388         I did this mostly as a simplification, to make it easier to change the
1389         allocation strategy.
1390
1391         No throughput change on MallocBench. Saves about 50kB.
1392
1393         Since the deallocator needs to lock the heap when freeing lines anyway,
1394         there isn't much benefit to giving the deallocator a local cache of
1395         deallocated lines.
1396
1397         We still give the allocator a local cache of lines because that does
1398         reduce the frequency at which it needs to lock the heap in order to
1399         acquire more lines.
1400
1401         * bmalloc/Allocator.cpp:
1402         (bmalloc::Allocator::scavenge):
1403         (bmalloc::Allocator::allocateSmallLine):
1404         (bmalloc::Allocator::allocateMediumLine):
1405         (bmalloc::Allocator::allocateMedium):
1406         (bmalloc::Allocator::allocateSlowCase):
1407         * bmalloc/Allocator.h:
1408         * bmalloc/Deallocator.cpp:
1409         (bmalloc::Deallocator::Deallocator):
1410         (bmalloc::Deallocator::scavenge):
1411         (bmalloc::Deallocator::processObjectLog):
1412         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
1413         (bmalloc::Deallocator::allocateSmallLine): Deleted.
1414         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
1415         (bmalloc::Deallocator::allocateMediumLine): Deleted.
1416         * bmalloc/Deallocator.h:
1417
1418         * bmalloc/Sizes.h:
1419         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
1420         exactly one page in size. That's about what we were shooting for anyway,
1421         and it may make it easier to switch to per-page allocation in future.
1422
1423 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
1424
1425         bmalloc: allocate small and medium objects using the same bump pointer class
1426         https://bugs.webkit.org/show_bug.cgi?id=136843
1427
1428         Reviewed by Gavin Barraclough.
1429
1430         4% speedup on MallocBench.
1431
1432         Now that medium-sized objects have dedicated per-size allocators, they
1433         don't need to use an arbitrary bump pointer allocator. This means that
1434         every allocator knows how many objects it will allocate from the start,
1435         and we don't need a post-processing step to adjust refcounts based on
1436         real allocation count.
1437
1438         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1439         since it's used for small and medium objects now.
1440
1441         * bmalloc/Allocator.cpp:
1442         (bmalloc::Allocator::Allocator): Updated to use new interface.
1443         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1444         to make sure that we finish allocating all the objects in it.
1445
1446         (bmalloc::Allocator::allocateMedium):
1447         (bmalloc::Allocator::allocateSlowCase):
1448         (bmalloc::Allocator::retire): Deleted.
1449         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1450         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1451         * bmalloc/Allocator.h:
1452         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1453         used to post-process an allocator based on how many objects it allocated.
1454
1455         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1456         (bmalloc::BumpAllocator::BumpAllocator):
1457         (bmalloc::BumpAllocator::init):
1458         (bmalloc::BumpAllocator::line):
1459         (bmalloc::BumpAllocator::validate):
1460         (bmalloc::BumpAllocator::allocate):
1461         (bmalloc::BumpAllocator::refill):
1462         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1463         about the kinds of lines they allocate into. In some cases, the line
1464         type must be provided as a template parameter by the caller.
1465
1466         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1467         (bmalloc::SmallAllocator::line): Deleted.
1468         (bmalloc::SmallAllocator::allocate): Deleted.
1469         (bmalloc::SmallAllocator::objectCount): Deleted.
1470         (bmalloc::SmallAllocator::derefCount): Deleted.
1471         (bmalloc::SmallAllocator::refill): Deleted.
1472         (bmalloc::SmallAllocator::clear): Deleted.
1473
1474         * bmalloc/ObjectType.h:
1475         (bmalloc::isMedium):
1476
1477         * bmalloc/SmallAllocator.h:
1478         (bmalloc::SmallAllocator::isNull): Deleted.
1479         (bmalloc::SmallAllocator::canAllocate): Deleted.
1480         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1481         (bmalloc::SmallAllocator::line): Deleted.
1482         (bmalloc::SmallAllocator::allocate): Deleted.
1483         (bmalloc::SmallAllocator::objectCount): Deleted.
1484         (bmalloc::SmallAllocator::derefCount): Deleted.
1485         (bmalloc::SmallAllocator::refill): Deleted.
1486         (bmalloc::SmallAllocator::clear): Deleted.
1487
1488 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1489
1490         Fixed a goof in bmalloc Vector sizing
1491         https://bugs.webkit.org/show_bug.cgi?id=136795
1492
1493         Reviewed by Gavin Barraclough and Sam Weinig.
1494
1495         We want our minimum vector to be page-sized since the OS will give us
1496         a page no matter what -- but we want that many bytes, and not enough
1497         bytes to store that many elements.
1498
1499         * bmalloc/Vector.h: Math is hard.
1500
1501 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1502
1503         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1504         https://bugs.webkit.org/show_bug.cgi?id=136693
1505
1506         Reviewed by Gavin Barraclough.
1507
1508         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1509
1510         No throughput change.
1511
1512         We keep an array of medium allocators, just like our array of small
1513         allocators.
1514
1515         In future, we can simplify the allocation fast path by merging the small
1516         and medium allocator arrays. For now, this is the simplest change that
1517         gets the win.
1518
1519         * bmalloc/Allocator.cpp:
1520         (bmalloc::Allocator::Allocator):
1521         (bmalloc::Allocator::scavenge):
1522         (bmalloc::Allocator::allocateMedium):
1523         * bmalloc/Allocator.h:
1524         * bmalloc/Sizes.h:
1525         (bmalloc::Sizes::mediumSizeClassFor):
1526
1527 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1528
1529         Reviewed by Sam Weinig.
1530
1531         Renamed log => retire for clarity.
1532
1533         * bmalloc/Allocator.cpp:
1534         (bmalloc::Allocator::scavenge):
1535         (bmalloc::Allocator::retire):
1536         (bmalloc::Allocator::allocateMedium):
1537         (bmalloc::Allocator::allocateSlowCase):
1538         (bmalloc::Allocator::log): Deleted.
1539         * bmalloc/Allocator.h:
1540
1541 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1542
1543         bmalloc: eager scavenge leaves behind a bogus allocator
1544         https://bugs.webkit.org/show_bug.cgi?id=136743
1545
1546         Reviewed by Sam Weinig.
1547
1548         Be sure to clear the allocator after logging it in the eager scavenge
1549         case, so that we don't later try to allocate out of the lines that we
1550         have thrown away.
1551
1552         We didn't need to do this previously because scavenge would only happen
1553         at thread exit time, after which no further allocation from the per-thread
1554         cache would take place.
1555
1556         * bmalloc/Allocator.cpp:
1557         (bmalloc::Allocator::scavenge):
1558         * bmalloc/MediumAllocator.h:
1559         (bmalloc::MediumAllocator::clear):
1560         * bmalloc/SmallAllocator.h:
1561         (bmalloc::SmallAllocator::clear):
1562
1563 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1564
1565         bmalloc should honor the FastMalloc statistics API
1566         https://bugs.webkit.org/show_bug.cgi?id=136592
1567
1568         Reviewed by Gavin Barraclough.
1569
1570         We do this by tracking "size" and "capacity" in the VM heap.
1571
1572         The VM heap's "capacity" is all the VM we ever allocated.
1573
1574         The VM heap's "size" the subset of VM currently held onto by the
1575         VM heap (and therefore not in use by the regular heap).
1576
1577         Somewhat ironically, reducing the process's memory footprint, increases
1578         the size of the VM heap, since the VM heap holds the pages that are
1579         purely virtual and not physical.
1580
1581         * bmalloc/Heap.cpp:
1582         (bmalloc::Heap::size):
1583         (bmalloc::Heap::capacity):
1584         * bmalloc/Heap.h:
1585         * bmalloc/VMHeap.cpp:
1586         (bmalloc::VMHeap::VMHeap):
1587         (bmalloc::VMHeap::allocateSmallChunk):
1588         (bmalloc::VMHeap::allocateMediumChunk):
1589         (bmalloc::VMHeap::allocateLargeChunk):
1590         * bmalloc/VMHeap.h:
1591         (bmalloc::VMHeap::size):
1592         (bmalloc::VMHeap::capacity):
1593         (bmalloc::VMHeap::allocateSmallPage):
1594         (bmalloc::VMHeap::allocateMediumPage):
1595         (bmalloc::VMHeap::allocateLargeRange):
1596         (bmalloc::VMHeap::deallocateSmallPage):
1597         (bmalloc::VMHeap::deallocateMediumPage):
1598         (bmalloc::VMHeap::deallocateLargeRange):
1599         * bmalloc/bmalloc.h:
1600         (bmalloc::api::heapSize):
1601         (bmalloc::api::heapCapacity):
1602
1603 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1604
1605         bmalloc crashes on the EWS bots (due to bad large object allocation)
1606         https://bugs.webkit.org/show_bug.cgi?id=136469
1607
1608         Reviewed by Andreas Kling.
1609
1610         It's possible to convince bmalloc to perform a bad large object allocation,
1611         through these steps:
1612
1613         (1) Insert object A into freelist F0.
1614
1615         (2) Split, merge and split again A's neighbors such that object B is
1616         inserted into freelist F0, with boundary tag and size equal to object A,
1617         but pointer not completely equal to object A. Put object B at the head of F0.
1618
1619         (3) Allocate some other object from F0, swapping its position in the
1620         freelist with object B, such that object A is now ahead of object B.
1621
1622         --> Now, the next allocation for size A/B will allocate object A, which
1623         has a slightly wrong idea about where the object actually begins.
1624         Immediately, you'll corrupt a little memory, and over time, you'll also
1625         corrupt boundary tag metadata.
1626
1627         The solution is to store the begin pointer in the boundary tag. Luckily,
1628         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1629         on MallocBench.
1630
1631         * bmalloc/Algorithm.h:
1632         (bmalloc::rightShift):
1633         * bmalloc/BeginTag.h:
1634         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1635         validate the start pointer when popping off the free list. Through a
1636         very uncommon set of steps, it is possible to have an item in the free
1637         list that is valid by all accounts except for its start pointer.
1638
1639         * bmalloc/BoundaryTag.h:
1640         (bmalloc::BoundaryTag::compactBegin):
1641         (bmalloc::BoundaryTag::setRange):
1642         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1643         start pointer. We don't need the whole pointer -- just the offset, in
1644         largeAlignment increments, into the relevant boundary tag bucket.
1645
1646         * bmalloc/BoundaryTagInlines.h:
1647         (bmalloc::validateNext):
1648         (bmalloc::BoundaryTag::init):
1649         (bmalloc::BoundaryTag::mergeLarge):
1650         (bmalloc::BoundaryTag::splitLarge):
1651         * bmalloc/SegregatedFreeList.cpp:
1652         (bmalloc::SegregatedFreeList::insert):
1653         (bmalloc::SegregatedFreeList::takeGreedy):
1654         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1655         the size when establishing a boundary tag, as required by the new
1656         interface.
1657
1658         * bmalloc/Sizes.h:
1659
1660 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1661
1662         Fixed a bmalloc crash seen on the EWS bot
1663         https://bugs.webkit.org/show_bug.cgi?id=135955
1664
1665         Reviewed by Andreas Kling.
1666
1667         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1668         input buffer is a malloc region, that region will get marked Copy-On-Write
1669         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1670         on older OS X's. In 10.10, they still fail, but they do not return
1671         EINVAL.
1672
1673         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1674
1675 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1676
1677         Fixed the bmalloc build
1678         https://bugs.webkit.org/show_bug.cgi?id=135953
1679
1680         Reviewed by Andreas Kling.
1681
1682         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1683         These headers are used, so they must be available outside the project.
1684
1685 2014-08-13  Daniel Bates  <dabates@apple.com>
1686
1687         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1688         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1689
1690         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1691         implementation of PerThread<T>::getFastCase().
1692
1693         * bmalloc/PerThread.h:
1694         (bmalloc::PerThread<T>::getFastCase):
1695
1696 2014-08-13  Daniel Bates  <dabates@apple.com>
1697
1698         Make bmalloc::PerThread work without C++ thread local storage
1699         https://bugs.webkit.org/show_bug.cgi?id=135895
1700
1701         Reviewed by Geoffrey Garen.
1702
1703         Implement support for building bmalloc without C++ thread local storage.
1704
1705         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1706         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1707         to determine whether the compiler supports C++ thread local storage.
1708         * bmalloc/PerThread.h:
1709         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1710         without C++ thread local storage.
1711         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1712         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1713         PerThreadStorage::initSharedKeyIfNeeded().
1714         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1715         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1716         ensure that the shared key has been initialized.
1717         (_pthread_setspecific_direct): Deleted.
1718         (_pthread_getspecific_direct): Deleted.
1719
1720 2014-08-13  Daniel Bates  <dabates@apple.com>
1721
1722         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1723         https://bugs.webkit.org/show_bug.cgi?id=135848
1724
1725         Reviewed by Geoffrey Garen.
1726
1727         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1728         when building for the iOS Simulator.
1729         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1730         using the public SDK.
1731         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1732         using the public SDK.
1733         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1734         using the public SDK.
1735
1736 2014-08-12  Daniel Bates  <dabates@apple.com>
1737
1738         BPLATFORM(IOS) always evaluates to false
1739         https://bugs.webkit.org/show_bug.cgi?id=135843
1740
1741         Reviewed by Geoffrey Garen.
1742
1743         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1744         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1745         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1746         building for iOS.
1747
1748         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1749         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1750
1751         * bmalloc/BPlatform.h:
1752
1753 2014-08-11  Andy Estes  <aestes@apple.com>
1754
1755         [iOS] Get rid of iOS.xcconfig
1756         https://bugs.webkit.org/show_bug.cgi?id=135809
1757
1758         Reviewed by Joseph Pecoraro.
1759
1760         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1761
1762         * Configurations/Base.xcconfig:
1763         * Configurations/iOS.xcconfig: Removed.
1764         * bmalloc.xcodeproj/project.pbxproj:
1765
1766 2014-05-01  Dan Bernstein  <mitz@apple.com>
1767
1768         Fixed production builds for the iOS Simulator.
1769         <rdar://problem/16792221>
1770
1771         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1772         PRIVATE_HEADERS_FOLDER_PATH when installing.
1773
1774 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1775
1776         bmalloc: Segregate pages by objects size
1777         https://bugs.webkit.org/show_bug.cgi?id=131909
1778
1779         Reviewed by Andreas Kling.
1780
1781         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1782
1783         * bmalloc/Allocator.cpp:
1784         (bmalloc::Allocator::allocateSlowCase):
1785         * bmalloc/Allocator.h:
1786         (bmalloc::Allocator::allocateFastCase):
1787         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1788         function for size class calculation.
1789
1790         * bmalloc/Deallocator.cpp:
1791         (bmalloc::Deallocator::Deallocator):
1792         (bmalloc::Deallocator::scavenge):
1793         (bmalloc::Deallocator::deallocateSmallLine):
1794         (bmalloc::Deallocator::allocateSmallLine):
1795         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1796         cache can't be shared anymore.
1797
1798         * bmalloc/Heap.cpp:
1799         (bmalloc::Heap::allocateSmallLineSlowCase):
1800         * bmalloc/Heap.h:
1801         (bmalloc::Heap::deallocateSmallLine): Ditto.
1802
1803         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1804         page refcount when allocating a line because we might have deallocated
1805         the page and the recycled it for another size class.
1806
1807         (bmalloc::Heap::deallocateMediumLine):
1808         (bmalloc::Heap::allocateMediumLine):
1809         * bmalloc/Line.h:
1810         (bmalloc::Line::refCount):
1811         * bmalloc/Page.h:
1812         (bmalloc::Page::refCount):
1813         (bmalloc::Page::smallSizeClass):
1814         (bmalloc::Page::setSmallSizeClass):
1815         (bmalloc::Page<Traits>::refCount): Deleted.
1816         * bmalloc/Sizes.h:
1817         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1818         an index into an array from a size.
1819
1820 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1821
1822         bmalloc: Improved alignment in LargeChunk
1823         https://bugs.webkit.org/show_bug.cgi?id=131895
1824
1825         Reviewed by Andreas Kling.
1826
1827         * bmalloc/Chunk.h:
1828         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1829         Technically, the previous alignment was harmless, but I would prefer,
1830         dear reader, not to have to explain the interlocking set of
1831         circumstances that made it so.
1832
1833 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1834
1835         Rolled out r167502 because it caused a crash on the facebook benchmark.
1836
1837         Unreviewed.
1838
1839             bmalloc: Added an XSmall line size
1840             https://bugs.webkit.org/show_bug.cgi?id=131851
1841
1842             Reviewed by Sam Weinig.
1843
1844 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1845
1846         bmalloc: Mutex should be harder to use wrong
1847         https://bugs.webkit.org/show_bug.cgi?id=131879
1848
1849         Reviewed by Andreas Kling.
1850
1851         Mutex now has a proper constructor, so you can't deadlock by forgetting
1852         to initialize it.
1853
1854         * bmalloc.xcodeproj/project.pbxproj:
1855         * bmalloc/Allocator.cpp:
1856         (bmalloc::Allocator::processXSmallAllocatorLog):
1857         (bmalloc::Allocator::processSmallAllocatorLog):
1858         (bmalloc::Allocator::processMediumAllocatorLog):
1859         (bmalloc::Allocator::allocateLarge):
1860         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1861         since the Heap mutex is a static.
1862
1863         * bmalloc/AsyncTask.h:
1864         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1865         need for explicit initialization anymore.
1866
1867         * bmalloc/Deallocator.cpp:
1868         (bmalloc::Deallocator::scavenge):
1869         (bmalloc::Deallocator::deallocateLarge):
1870         (bmalloc::Deallocator::deallocateXLarge):
1871         (bmalloc::Deallocator::processObjectLog):
1872         (bmalloc::Deallocator::deallocateSmallLine):
1873         (bmalloc::Deallocator::deallocateXSmallLine):
1874         (bmalloc::Deallocator::allocateSmallLine):
1875         (bmalloc::Deallocator::allocateXSmallLine):
1876         (bmalloc::Deallocator::deallocateMediumLine):
1877         (bmalloc::Deallocator::allocateMediumLine):
1878         * bmalloc/Deallocator.h:
1879         * bmalloc/Heap.cpp:
1880         (bmalloc::sleep):
1881         (bmalloc::Heap::Heap):
1882         (bmalloc::Heap::concurrentScavenge):
1883         (bmalloc::Heap::scavenge):
1884         (bmalloc::Heap::scavengeSmallPages):
1885         (bmalloc::Heap::scavengeXSmallPages):
1886         (bmalloc::Heap::scavengeMediumPages):
1887         (bmalloc::Heap::scavengeLargeRanges):
1888         (bmalloc::Heap::allocateXSmallLineSlowCase):
1889         (bmalloc::Heap::allocateSmallLineSlowCase):
1890         (bmalloc::Heap::allocateMediumLineSlowCase):
1891         (bmalloc::Heap::allocateXLarge):
1892         (bmalloc::Heap::deallocateXLarge):
1893         (bmalloc::Heap::allocateLarge):
1894         (bmalloc::Heap::deallocateLarge):
1895         * bmalloc/Heap.h:
1896         (bmalloc::Heap::deallocateXSmallLine):
1897         (bmalloc::Heap::allocateXSmallLine):
1898         (bmalloc::Heap::deallocateSmallLine):
1899         (bmalloc::Heap::allocateSmallLine):
1900         (bmalloc::Heap::deallocateMediumLine):
1901         (bmalloc::Heap::allocateMediumLine):
1902         * bmalloc/Line.h:
1903         (bmalloc::Line<Traits>::deref):
1904         * bmalloc/Mutex.cpp: Removed.
1905         * bmalloc/Mutex.h:
1906         (bmalloc::Mutex::Mutex):
1907         (bmalloc::Mutex::init): Deleted.
1908         (bmalloc::Mutex::try_lock): Deleted.
1909         (bmalloc::Mutex::lock): Deleted.
1910         (bmalloc::Mutex::unlock): Deleted.
1911         * bmalloc/Page.h:
1912         (bmalloc::Page<Traits>::ref):
1913         (bmalloc::Page<Traits>::deref):
1914         (bmalloc::Page<Traits>::refCount):
1915         * bmalloc/PerProcess.h:
1916         (bmalloc::PerProcess::mutex):
1917         (bmalloc::PerProcess<T>::getSlowCase):
1918         * bmalloc/StaticMutex.cpp: Added.
1919         (bmalloc::StaticMutex::lockSlowCase):
1920         * bmalloc/StaticMutex.h: Added.
1921         (bmalloc::StaticMutex::init):
1922         (bmalloc::StaticMutex::try_lock):
1923         (bmalloc::StaticMutex::lock):
1924         (bmalloc::StaticMutex::unlock):
1925         * bmalloc/VMHeap.h:
1926         (bmalloc::VMHeap::deallocateXSmallPage):
1927         (bmalloc::VMHeap::deallocateSmallPage):
1928         (bmalloc::VMHeap::deallocateMediumPage):
1929         (bmalloc::VMHeap::deallocateLargeRange):
1930         * bmalloc/bmalloc.h:
1931         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1932         since the Heap mutex is a static.
1933
1934 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1935
1936         bmalloc: AsyncTask should use Mutex instead of std::mutex
1937         https://bugs.webkit.org/show_bug.cgi?id=131865
1938
1939         Reviewed by Gavin Barraclough.
1940
1941         std::mutex is so slow that it makes parallelizing simple tasks through
1942         AsyncTask a net regression. Mutex fixes this.
1943
1944         * bmalloc/AsyncTask.h:
1945         (bmalloc::Function>::AsyncTask):
1946         (bmalloc::Function>::join):
1947         (bmalloc::Function>::runSlowCase):
1948         (bmalloc::Function>::entryPoint):
1949         * bmalloc/Mutex.h:
1950         (bmalloc::Mutex::init):
1951
1952 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1953
1954         bmalloc: Added an XSmall line size
1955         https://bugs.webkit.org/show_bug.cgi?id=131851
1956
1957         Reviewed by Sam Weinig.
1958
1959         Reduces malloc footprint on Membuster recordings by 10%.
1960
1961         This is a throughput regression, but we're still way ahead of TCMalloc.
1962         I have some ideas for how to recover the regression -- but I wanted to
1963         get this win in first.
1964
1965         Full set of benchmark results:
1966
1967                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1968                                                                                                 
1969                                                                nopatch                      patch                                Δ
1970                 Peak Memory:
1971                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1972                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1973                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1974
1975                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1976                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1977                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1978
1979                 Memory at End:
1980                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1981                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1982                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1983
1984                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1985                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1986                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1987
1988                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1989                                                                                                 
1990                                                    nopatch                patch                         Δ
1991                 Execution Time:
1992                     churn                            127ms                151ms            ! 1.19x slower
1993                     list_allocate                    130ms                164ms            ! 1.26x slower
1994                     tree_allocate                    109ms                127ms            ! 1.17x slower
1995                     tree_churn                       115ms                120ms            ! 1.04x slower
1996                     facebook                         240ms                259ms            ! 1.08x slower
1997                     fragment                          91ms                131ms            ! 1.44x slower
1998                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1999                     message_one                      260ms                259ms             ^ 1.0x faster
2000                     message_many                     149ms                154ms            ! 1.03x slower
2001                     medium                           194ms                248ms            ! 1.28x slower
2002                     big                              157ms                160ms            ! 1.02x slower
2003
2004                     <geometric mean>                 144ms                163ms            ! 1.13x slower
2005                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
2006                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
2007
2008                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2009                                                                                                 
2010                                                                        nopatch                          patch                                     Δ
2011                 Execution Time:
2012                     churn                                                126ms                          148ms                        ! 1.17x slower
2013                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
2014                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
2015                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
2016                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
2017                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
2018                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
2019                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
2020                     facebook                                             240ms                          276ms                        ! 1.15x slower
2021                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
2022                     fragment                                              92ms                          130ms                        ! 1.41x slower
2023                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
2024                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
2025                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
2026                     message_one                                          260ms                          260ms                                      
2027                     message_many                                         170ms                          238ms                         ! 1.4x slower
2028                     medium                                               185ms                          250ms                        ! 1.35x slower
2029                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
2030                     big                                                  150ms                          169ms                        ! 1.13x slower
2031                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
2032
2033                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
2034                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
2035                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
2036
2037                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2038
2039                                                                     TC                      patch                                Δ
2040                 Peak Memory:
2041                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
2042                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
2043                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
2044
2045                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
2046                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
2047                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
2048
2049                 Memory at End:
2050                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
2051                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
2052                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
2053
2054                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
2055                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
2056                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
2057
2058                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2059                                                                                                 
2060                                                         TC                patch                         Δ
2061                 Execution Time:
2062                     churn                            416ms                148ms            ^ 2.81x faster
2063                     list_allocate                    463ms                164ms            ^ 2.82x faster
2064                     tree_allocate                    292ms                127ms             ^ 2.3x faster
2065                     tree_churn                       157ms                120ms            ^ 1.31x faster
2066                     facebook                         327ms                276ms            ^ 1.18x faster
2067                     fragment                         335ms                129ms             ^ 2.6x faster
2068                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
2069                     message_one                      386ms                258ms             ^ 1.5x faster
2070                     message_many                     410ms                154ms            ^ 2.66x faster
2071                     medium                           391ms                245ms             ^ 1.6x faster
2072                     big                              261ms                167ms            ^ 1.56x faster
2073
2074                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
2075                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
2076                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
2077
2078         * bmalloc.xcodeproj/project.pbxproj:
2079         * bmalloc/Allocator.cpp:
2080         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
2081         index corresponds with its size. Instead, use the size selection function
2082         explicitly. Now that we have XSmall, some small allocator entries are
2083         unused.
2084
2085         (bmalloc::Allocator::scavenge):
2086         (bmalloc::Allocator::log):
2087         (bmalloc::Allocator::processXSmallAllocatorLog):
2088         (bmalloc::Allocator::allocateSlowCase):
2089         * bmalloc/Allocator.h:
2090         (bmalloc::Allocator::xSmallAllocatorFor):
2091         (bmalloc::Allocator::allocateFastCase):
2092         * bmalloc/Chunk.h:
2093         * bmalloc/Deallocator.cpp:
2094         (bmalloc::Deallocator::scavenge):
2095         (bmalloc::Deallocator::processObjectLog):
2096         (bmalloc::Deallocator::deallocateSlowCase):
2097         (bmalloc::Deallocator::deallocateXSmallLine):
2098         (bmalloc::Deallocator::allocateXSmallLine):
2099         * bmalloc/Deallocator.h:
2100         (bmalloc::Deallocator::deallocateFastCase):
2101         * bmalloc/Heap.cpp:
2102         (bmalloc::Heap::scavenge):
2103         (bmalloc::Heap::scavengeXSmallPages):
2104         (bmalloc::Heap::allocateXSmallLineSlowCase):
2105         * bmalloc/Heap.h:
2106         (bmalloc::Heap::deallocateXSmallLine):
2107         (bmalloc::Heap::allocateXSmallLine):
2108         * bmalloc/LargeChunk.h:
2109         (bmalloc::LargeChunk::get):
2110         (bmalloc::LargeChunk::endTag):
2111         * bmalloc/Line.h:
2112         * bmalloc/MediumAllocator.h:
2113         (bmalloc::MediumAllocator::allocate):
2114         (bmalloc::MediumAllocator::refill):
2115         * bmalloc/ObjectType.cpp:
2116         (bmalloc::objectType):
2117         * bmalloc/ObjectType.h:
2118         (bmalloc::isXSmall):
2119         (bmalloc::isSmall):
2120         (bmalloc::isMedium):
2121         (bmalloc::isLarge):
2122         (bmalloc::isSmallOrMedium): Deleted.
2123         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
2124         handling small objects. There's probably a reasonable way to share this
2125         code in the future -- I'll look into that once it's stopped changing.
2126
2127         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
2128         is the main reason things got slower.
2129
2130         * bmalloc/SmallAllocator.h:
2131         (bmalloc::SmallAllocator::allocate):
2132         * bmalloc/SmallTraits.h:
2133         * bmalloc/VMHeap.cpp:
2134         (bmalloc::VMHeap::allocateXSmallChunk):
2135         * bmalloc/VMHeap.h:
2136         (bmalloc::VMHeap::allocateXSmallPage):
2137         (bmalloc::VMHeap::deallocateXSmallPage):
2138         * bmalloc/XSmallAllocator.h: Added.
2139         (bmalloc::XSmallAllocator::isNull):
2140         (bmalloc::XSmallAllocator::canAllocate):
2141         (bmalloc::XSmallAllocator::XSmallAllocator):
2142         (bmalloc::XSmallAllocator::line):
2143         (bmalloc::XSmallAllocator::allocate):
2144         (bmalloc::XSmallAllocator::objectCount):
2145         (bmalloc::XSmallAllocator::derefCount):
2146         (bmalloc::XSmallAllocator::refill):
2147         * bmalloc/XSmallChunk.h: Added.
2148         * bmalloc/XSmallLine.h: Added.
2149         * bmalloc/XSmallPage.h: Added.
2150         * bmalloc/XSmallTraits.h: Added.
2151         * bmalloc/bmalloc.h:
2152         (bmalloc::api::realloc): Boiler-plate copy, as above.
2153
2154 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2155
2156         MallocBench should scavenge explicitly instead of waiting
2157         https://bugs.webkit.org/show_bug.cgi?id=131661
2158
2159         Reviewed by Andreas Kling.
2160
2161         Added explicit scavenge support to bmalloc. This isn't a memory win,
2162         since bmalloc's per-thread cache is so small. But it makes testing
2163         simpler.
2164
2165         * bmalloc/Allocator.cpp:
2166         (bmalloc::Allocator::~Allocator):
2167         (bmalloc::Allocator::scavenge):
2168         * bmalloc/Allocator.h:
2169         * bmalloc/Cache.cpp:
2170         (bmalloc::Cache::operator new):
2171         (bmalloc::Cache::operator delete):
2172         (bmalloc::Cache::Cache):
2173         (bmalloc::Cache::scavenge):
2174         * bmalloc/Cache.h:
2175         * bmalloc/Deallocator.cpp:
2176         (bmalloc::Deallocator::~Deallocator):
2177         (bmalloc::Deallocator::scavenge):
2178         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
2179         functions, for reuse.
2180
2181         * bmalloc/Heap.cpp:
2182         (bmalloc::sleep):
2183         (bmalloc::Heap::concurrentScavenge):
2184         (bmalloc::Heap::scavenge):
2185         (bmalloc::Heap::scavengeSmallPages):
2186         (bmalloc::Heap::scavengeMediumPages):
2187         (bmalloc::Heap::scavengeLargeRanges):
2188         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
2189         scavenging -- in response to a benchmark or a low memory warning --
2190         wants to complete as soon as possible, so its sleep duration is 0.
2191
2192         * bmalloc/bmalloc.h:
2193         (bmalloc::api::scavenge):
2194         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
2195
2196 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2197
2198         Use 4kB pages on Mac
2199         https://bugs.webkit.org/show_bug.cgi?id=131658
2200
2201         Reviewed by Sam Weinig.
2202
2203         This reduces memory use a lot on Membuster:
2204
2205                                                                   base                      patch                                Δ
2206                 Execution Time:
2207                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
2208                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
2209                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
2210
2211                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
2212                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
2213                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
2214
2215                 Peak Memory:
2216                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
2217                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
2218                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
2219
2220                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
2221                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
2222                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
2223
2224                 Memory at End:
2225                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
2226                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
2227                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
2228
2229                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
2230                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
2231                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
2232
2233
2234         * bmalloc.xcodeproj/project.pbxproj:
2235         * bmalloc/BPlatform.h: Added.
2236         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
2237         is 4kB on Mac.
2238
2239 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
2240
2241         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
2242
2243         * bmalloc.xcodeproj: Modified property svn:ignore.
2244
2245 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
2246
2247         Fixed some mbmalloc exports
2248         https://bugs.webkit.org/show_bug.cgi?id=131599
2249
2250         Reviewed by Ryosuke Niwa.
2251
2252         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
2253         of the project, so we can call them from API.
2254
2255         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
2256         visibility, so they show up as exported in the .dylib.
2257
2258 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
2259
2260         Put bmalloc headers in the right place
2261         https://bugs.webkit.org/show_bug.cgi?id=131464
2262
2263         Reviewed by Mark Rowe.
2264
2265         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
2266         specify that we don't just want to dump all of our generically-named
2267         headers into /usr/local/include.
2268
2269 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2270
2271         Made bmalloc more #include friendly
2272         https://bugs.webkit.org/show_bug.cgi?id=131386
2273
2274         Reviewed by Andreas Kling.
2275
2276         Marked a bunch of headers private so they can be used from client code
2277         that #includes bmalloc.h.
2278
2279         Renamed ASSERT macros to BASSERT. This matches their header, which already
2280         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
2281
2282         * bmalloc.xcodeproj/project.pbxproj:
2283         * bmalloc/Allocator.cpp:
2284         (bmalloc::Allocator::allocateSlowCase):
2285         * bmalloc/AsyncTask.h:
2286         (bmalloc::Function>::runSlowCase):
2287         * bmalloc/BAssert.h:
2288         * bmalloc/BoundaryTag.h:
2289         (bmalloc::BoundaryTag::setSize):
2290         * bmalloc/BoundaryTagInlines.h:
2291         (bmalloc::validate):
2292         (bmalloc::BoundaryTag::init):
2293         (bmalloc::BoundaryTag::deallocate):
2294         (bmalloc::BoundaryTag::splitLarge):
2295         (bmalloc::BoundaryTag::allocate):
2296         * bmalloc/Chunk.h:
2297         * bmalloc/Deallocator.cpp:
2298         (bmalloc::Deallocator::processObjectLog):
2299         (bmalloc::Deallocator::deallocateSlowCase):
2300         * bmalloc/Deallocator.h:
2301         (bmalloc::Deallocator::deallocateFastCase):
2302         * bmalloc/FixedVector.h:
2303         (bmalloc::Capacity>::operator):
2304         (bmalloc::Capacity>::push):
2305         (bmalloc::Capacity>::pop):
2306         (bmalloc::Capacity>::shrink):
2307         * bmalloc/Heap.cpp:
2308         (bmalloc::Heap::allocateLarge):
2309         * bmalloc/LargeChunk.h:
2310         (bmalloc::LargeChunk::get):
2311         (bmalloc::LargeChunk::endTag):
2312         * bmalloc/Line.h:
2313         (bmalloc::Line<Traits>::concurrentRef):
2314         (bmalloc::Line<Traits>::deref):
2315         * bmalloc/MediumAllocator.h:
2316         (bmalloc::MediumAllocator::allocate):
2317         * bmalloc/ObjectType.h:
2318         (bmalloc::isSmall):
2319         * bmalloc/Page.h:
2320         (bmalloc::Page<Traits>::ref):
2321         (bmalloc::Page<Traits>::deref):
2322         * bmalloc/PerThread.h:
2323         (bmalloc::PerThread<T>::getSlowCase):
2324         * bmalloc/SegregatedFreeList.cpp:
2325         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2326         (bmalloc::SegregatedFreeList::insert):
2327         * bmalloc/SmallAllocator.h:
2328         (bmalloc::SmallAllocator::allocate):
2329         (bmalloc::SmallAllocator::refill):
2330         * bmalloc/Syscall.h:
2331         * bmalloc/VMAllocate.h:
2332         (bmalloc::vmValidate):
2333         (bmalloc::vmAllocate):
2334         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2335         * bmalloc/Vector.h:
2336         (bmalloc::Vector<T>::operator):
2337         (bmalloc::Vector<T>::pop):
2338         (bmalloc::Vector<T>::shrink):
2339         * bmalloc/XLargeChunk.h:
2340         (bmalloc::XLargeChunk::range):
2341         (bmalloc::XLargeChunk::size):
2342
2343 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2344
2345         Removed an unused file.
2346
2347         Unreviewed.
2348
2349         * bmalloc/AsyncTask.cpp: Removed.
2350
2351 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2352
2353         Build bmalloc on Mac
2354         https://bugs.webkit.org/show_bug.cgi?id=131333
2355
2356         Reviewed by Mark Rowe.
2357
2358         * Makefile: Added. For make clients.
2359
2360         These files are required for building any project in WebKit. I copied
2361         them from WTF:
2362         * Configurations: Added.
2363         * Configurations/Base.xcconfig: Added.
2364         * Configurations/DebugRelease.xcconfig: Added.
2365         * Configurations/bmalloc.xcconfig: Added.
2366         * Configurations/iOS.xcconfig: Added.
2367         * Configurations/mbmalloc.xcconfig: Added.
2368
2369         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
2370         from here because everything is in .xcconfig files now.
2371
2372         I had to fix a bunch of minor warnings, since they're enabled in our
2373         .xcconfig files:
2374
2375         * bmalloc/AsyncTask.h:
2376         (bmalloc::Function>::AsyncTask):
2377         * bmalloc/BAssert.h:
2378         * bmalloc/BoundaryTagInlines.h:
2379         (bmalloc::validate):
2380         * bmalloc/Heap.cpp:
2381         (bmalloc::Heap::Heap):
2382         (bmalloc::Heap::allocateLarge):
2383         (bmalloc::Heap::deallocateLarge):
2384         * bmalloc/Mutex.h:
2385         (bmalloc::Mutex::Mutex): Deleted.
2386         * bmalloc/VMAllocate.h:
2387         (bmalloc::vmValidate):
2388         * bmalloc/mbmalloc.cpp:
2389
2390 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2391
2392         bmalloc: Fixed a leak in the per-thread cache
2393         https://bugs.webkit.org/show_bug.cgi?id=131330
2394
2395         Reviewed by Andreas Kling.
2396
2397         Remember to deallocate our line caches upon thread exit.
2398
2399         * bmalloc/Deallocator.cpp:
2400         (bmalloc::Deallocator::~Deallocator):
2401
2402 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2403
2404         bmalloc: rolled out the tryLock experiment
2405         https://bugs.webkit.org/show_bug.cgi?id=131328
2406
2407         Reviewed by Andreas Kling.
2408
2409         It wasn't a speedup.
2410
2411         * bmalloc.xcodeproj/project.pbxproj:
2412         * bmalloc/Allocator.cpp:
2413         (bmalloc::Allocator::processSmallAllocatorLog):
2414         (bmalloc::Allocator::processMediumAllocatorLog):
2415         * bmalloc/Deallocator.cpp:
2416         (bmalloc::Deallocator::processObjectLog):
2417         (bmalloc::Deallocator::deallocateSlowCase):
2418         (bmalloc::Deallocator::deallocateSmallLine):
2419         (bmalloc::Deallocator::deallocateMediumLine):
2420         * bmalloc/Deallocator.h:
2421         (bmalloc::Deallocator::deallocateFastCase):
2422         * bmalloc/Heap.h:
2423         (bmalloc::Heap::deallocateSmallLine):
2424         (bmalloc::Heap::deallocateMediumLine):
2425         * bmalloc/Line.h:
2426         (bmalloc::Line<Traits>::deref):
2427         * bmalloc/Page.h:
2428         (bmalloc::Page<Traits>::deref):
2429
2430 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2431
2432         bmalloc
2433         https://bugs.webkit.org/show_bug.cgi?id=131170
2434
2435         Reviewed by Andreas Kling.
2436
2437         Initial commit.
2438
2439         * bmalloc: Added.
2440         * bmalloc.xcodeproj: Added.
2441         * bmalloc.xcodeproj/project.pbxproj: Added.
2442         * bmalloc/Algorithm.h: Added.
2443         (bmalloc::max):
2444         (bmalloc::min):
2445         (bmalloc::mask):
2446         (bmalloc::test):
2447         (bmalloc::roundUpToMultipleOf):
2448         (bmalloc::roundDownToMultipleOf):
2449         (bmalloc::sizeOf):
2450         (bmalloc::bitCount):
2451         (bmalloc::isPowerOfTwo):
2452         * bmalloc/Allocator.cpp: Added.
2453         (bmalloc::Allocator::Allocator):
2454         (bmalloc::Allocator::~Allocator):
2455         (bmalloc::Allocator::log):
2456         (bmalloc::Allocator::processSmallAllocatorLog):
2457         (bmalloc::Allocator::processMediumAllocatorLog):
2458         (bmalloc::Allocator::allocateLarge):
2459         (bmalloc::Allocator::allocateXLarge):
2460         (bmalloc::Allocator::allocateMedium):
2461         (bmalloc::Allocator::allocateSlowCase):
2462         * bmalloc/Allocator.h: Added.
2463         (bmalloc::Allocator::smallAllocatorFor):
2464         (bmalloc::Allocator::allocateFastCase):
2465         (bmalloc::Allocator::allocate):
2466         * bmalloc/AsyncTask.cpp: Added.
2467         (bmalloc::AsyncTask<Function>::runSlowCase):
2468         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2469         (bmalloc::AsyncTask<Function>::entryPoint):
2470         * bmalloc/AsyncTask.h: Added.
2471         (bmalloc::Function>::AsyncTask):
2472         (bmalloc::Function>::join):
2473         (bmalloc::Function>::run):
2474         (bmalloc::Function>::runSlowCase):
2475         (bmalloc::Function>::pthreadEntryPoint):
2476         (bmalloc::Function>::entryPoint):
2477         * bmalloc/BAssert.h: Added.
2478         * bmalloc/BeginTag.h: Added.
2479         (bmalloc::BeginTag::isInFreeList):
2480         * bmalloc/BoundaryTag.h: Added.
2481         (bmalloc::BoundaryTag::isXLarge):
2482         (bmalloc::BoundaryTag::setXLarge):
2483         (bmalloc::BoundaryTag::isFree):
2484         (bmalloc::BoundaryTag::setFree):
2485         (bmalloc::BoundaryTag::isEnd):
2486         (bmalloc::BoundaryTag::setEnd):
2487         (bmalloc::BoundaryTag::hasPhysicalPages):
2488         (bmalloc::BoundaryTag::setHasPhysicalPages):
2489         (bmalloc::BoundaryTag::isNull):
2490         (bmalloc::BoundaryTag::clear):
2491         (bmalloc::BoundaryTag::size):
2492         (bmalloc::BoundaryTag::setSize):
2493         (bmalloc::BoundaryTag::prev):
2494         (bmalloc::BoundaryTag::next):
2495         * bmalloc/BoundaryTagInlines.h: Added.
2496         (bmalloc::validate):
2497         (bmalloc::validatePrev):
2498         (bmalloc::validateNext):
2499         (bmalloc::BoundaryTag::init):
2500         (bmalloc::BoundaryTag::mergeLargeLeft):
2501         (bmalloc::BoundaryTag::mergeLargeRight):
2502         (bmalloc::BoundaryTag::mergeLarge):
2503         (bmalloc::BoundaryTag::deallocate):
2504         (bmalloc::BoundaryTag::splitLarge):
2505         (bmalloc::BoundaryTag::allocate):
2506         * bmalloc/Cache.cpp: Added.
2507         (bmalloc::Cache::operator new):
2508         (bmalloc::Cache::operator delete):
2509         (bmalloc::Cache::Cache):
2510         (bmalloc::Cache::allocateSlowCase):
2511         (bmalloc::Cache::allocateSlowCaseNullCache):
2512         (bmalloc::Cache::deallocateSlowCase):
2513         (bmalloc::Cache::deallocateSlowCaseNullCache):
2514         * bmalloc/Cache.h: Added.
2515         (bmalloc::Cache::allocator):
2516         (bmalloc::Cache::deallocator):
2517         (bmalloc::Cache::allocateFastCase):
2518         (bmalloc::Cache::deallocateFastCase):
2519         (bmalloc::Cache::allocate):
2520         (bmalloc::Cache::deallocate):
2521         * bmalloc/Chunk.h: Added.
2522         (bmalloc::Chunk::begin):
2523         (bmalloc::Chunk::end):
2524         (bmalloc::Chunk::lines):
2525         (bmalloc::Chunk::pages):
2526         * bmalloc/Deallocator.cpp: Added.
2527         (bmalloc::Deallocator::Deallocator):
2528         (bmalloc::Deallocator::~Deallocator):
2529         (bmalloc::Deallocator::deallocateLarge):
2530         (bmalloc::Deallocator::deallocateXLarge):
2531         (bmalloc::Deallocator::processObjectLog):
2532         (bmalloc::Deallocator::deallocateSlowCase):
2533         (bmalloc::Deallocator::deallocateSmallLine):
2534         (bmalloc::Deallocator::allocateSmallLine):
2535         (bmalloc::Deallocator::deallocateMediumLine):
2536         (bmalloc::Deallocator::allocateMediumLine):
2537         * bmalloc/Deallocator.h: Added.
2538         (bmalloc::Deallocator::deallocateFastCase):
2539         (bmalloc::Deallocator::deallocate):
2540         * bmalloc/EndTag.h: Added.
2541         (bmalloc::EndTag::operator=):
2542         * bmalloc/FixedVector.h: Added.
2543         (bmalloc::FixedVector::begin):
2544         (bmalloc::FixedVector::end):
2545         (bmalloc::FixedVector::size):
2546         (bmalloc::FixedVector::capacity):
2547         (bmalloc::FixedVector::clear):
2548         (bmalloc::FixedVector::isEmpty):
2549         (bmalloc::Capacity>::FixedVector):
2550         (bmalloc::Capacity>::operator):
2551         (bmalloc::Capacity>::push):
2552         (bmalloc::Capacity>::pop):
2553         (bmalloc::Capacity>::shrink):
2554         * bmalloc/Heap.cpp: Added.
2555         (bmalloc::sleep):
2556         (bmalloc::Heap::Heap):
2557         (bmalloc::Heap::concurrentScavenge):
2558         (bmalloc::Heap::scavengeSmallPages):
2559         (bmalloc::Heap::scavengeMediumPages):
2560         (bmalloc::Heap::scavengeLargeRanges):
2561         (bmalloc::Heap::allocateSmallLineSlowCase):
2562         (bmalloc::Heap::allocateMediumLineSlowCase):
2563         (bmalloc::Heap::allocateXLarge):
2564         (bmalloc::Heap::deallocateXLarge):
2565         (bmalloc::Heap::allocateLarge):
2566         (bmalloc::Heap::deallocateLarge):
2567         * bmalloc/Heap.h: Added.
2568         (bmalloc::Heap::deallocateSmallLine):
2569         (bmalloc::Heap::allocateSmallLine):
2570         (bmalloc::Heap::deallocateMediumLine):
2571         (bmalloc::Heap::allocateMediumLine):
2572         * bmalloc/Inline.h: Added.
2573         * bmalloc/LargeChunk.h: Added.
2574         (bmalloc::LargeChunk::begin):
2575         (bmalloc::LargeChunk::end):
2576         (bmalloc::LargeChunk::create):
2577         (bmalloc::LargeChunk::get):
2578         (bmalloc::LargeChunk::beginTag):
2579         (bmalloc::LargeChunk::endTag):
2580         * bmalloc/Line.h: Added.
2581         (bmalloc::Line<Traits>::begin):
2582         (bmalloc::Line<Traits>::end):
2583         (bmalloc::Line<Traits>::concurrentRef):
2584         (bmalloc::Line<Traits>::deref):
2585         * bmalloc/MediumAllocator.h: Added.
2586         (bmalloc::MediumAllocator::isNull):
2587         (bmalloc::MediumAllocator::MediumAllocator):
2588         (bmalloc::MediumAllocator::line):
2589         (bmalloc::MediumAllocator::allocate):
2590         (bmalloc::MediumAllocator::derefCount):
2591         (bmalloc::MediumAllocator::refill):
2592         * bmalloc/MediumChunk.h: Added.
2593         * bmalloc/MediumLine.h: Added.
2594         * bmalloc/MediumPage.h: Added.
2595         * bmalloc/MediumTraits.h: Added.
2596         * bmalloc/Mutex.cpp: Added.
2597         (bmalloc::Mutex::lockSlowCase):
2598         * bmalloc/Mutex.h: Added.
2599         (bmalloc::Mutex::Mutex):
2600         (bmalloc::Mutex::try_lock):
2601         (bmalloc::Mutex::lock):
2602         (bmalloc::Mutex::unlock):
2603         * bmalloc/ObjectType.cpp: Added.
2604         (bmalloc::objectType):
2605         * bmalloc/ObjectType.h: Added.
2606         (bmalloc::isSmallOrMedium):
2607         (bmalloc::isSmall):
2608         * bmalloc/Page.h: Added.
2609         (bmalloc::Page<Traits>::ref):
2610         (bmalloc::Page<Traits>::deref):
2611         (bmalloc::Page<Traits>::refCount):
2612         * bmalloc/PerProcess.h: Added.
2613         (bmalloc::PerProcess::mutex):
2614         (bmalloc::PerProcess<T>::getFastCase):
2615         (bmalloc::PerProcess<T>::get):
2616         (bmalloc::PerProcess<T>::getSlowCase):
2617         * bmalloc/PerThread.h: Added.
2618         (bmalloc::PerThreadStorage<Cache>::get):
2619         (bmalloc::PerThreadStorage<Cache>::init):
2620         (bmalloc::PerThreadStorage::get):
2621         (bmalloc::PerThreadStorage::init):
2622         (bmalloc::PerThread<T>::getFastCase):
2623         (bmalloc::PerThread<T>::get):
2624         (bmalloc::PerThread<T>::destructor):
2625         (bmalloc::PerThread<T>::getSlowCase):
2626         * bmalloc/Range.h: Added.
2627         (bmalloc::Range::Range):
2628         (bmalloc::Range::begin):
2629         (bmalloc::Range::end):
2630         (bmalloc::Range::size):
2631         (bmalloc::Range::operator!):
2632         (bmalloc::Range::operator<):
2633         * bmalloc/SegregatedFreeList.cpp: Added.
2634         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2635         (bmalloc::SegregatedFreeList::insert):
2636         (bmalloc::SegregatedFreeList::takeGreedy):
2637         (bmalloc::SegregatedFreeList::take):
2638         * bmalloc/SegregatedFreeList.h: Added.
2639         * bmalloc/Sizes.h: Added.
2640         * bmalloc/SmallAllocator.h: Added.
2641         (bmalloc::SmallAllocator::isNull):
2642         (bmalloc::SmallAllocator::canAllocate):
2643         (bmalloc::SmallAllocator::SmallAllocator):
2644         (bmalloc::SmallAllocator::line):
2645         (bmalloc::SmallAllocator::allocate):
2646         (bmalloc::SmallAllocator::objectCount):
2647         (bmalloc::SmallAllocator::derefCount):
2648         (bmalloc::SmallAllocator::refill):
2649         * bmalloc/SmallChunk.h: Added.
2650         * bmalloc/SmallLine.h: Added.
2651         * bmalloc/SmallPage.h: Added.
2652         * bmalloc/SmallTraits.h: Added.
2653         * bmalloc/Syscall.h: Added.
2654         * bmalloc/VMAllocate.h: Added.
2655         (bmalloc::vmSize):
2656         (bmalloc::vmValidate):
2657         (bmalloc::vmAllocate):
2658         (bmalloc::vmDeallocate):
2659         (bmalloc::vmDeallocatePhysicalPages):
2660         (bmalloc::vmAllocatePhysicalPages):
2661         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2662         (bmalloc::vmAllocatePhysicalPagesSloppy):
2663         * bmalloc/VMHeap.cpp: Added.
2664         (bmalloc::VMHeap::VMHeap):
2665         (bmalloc::VMHeap::allocateSmallChunk):
2666         (bmalloc::VMHeap::allocateMediumChunk):
2667         (bmalloc::VMHeap::allocateLargeChunk):
2668         * bmalloc/VMHeap.h: Added.
2669         (bmalloc::VMHeap::allocateSmallPage):
2670         (bmalloc::VMHeap::allocateMediumPage):
2671         (bmalloc::VMHeap::allocateLargeRange):
2672         (bmalloc::VMHeap::deallocateSmallPage):
2673         (bmalloc::VMHeap::deallocateMediumPage):
2674         (bmalloc::VMHeap::deallocateLargeRange):
2675         * bmalloc/Vector.h: Added.
2676         (bmalloc::Vector::begin):
2677         (bmalloc::Vector::end):
2678         (bmalloc::Vector::size):
2679         (bmalloc::Vector::capacity):
2680         (bmalloc::Vector::last):
2681         (bmalloc::Vector::pop):
2682         (bmalloc::Vector<T>::Vector):
2683         (bmalloc::Vector<T>::~Vector):
2684         (bmalloc::Vector<T>::operator):
2685         (bmalloc::Vector<T>::push):
2686         (bmalloc::Vector<T>::pop):
2687         (bmalloc::Vector<T>::shrink):
2688         (bmalloc::Vector<T>::reallocateBuffer):
2689         (bmalloc::Vector<T>::shrinkCapacity):
2690         (bmalloc::Vector<T>::growCapacity):
2691         * bmalloc/XLargeChunk.h: Added.
2692         (bmalloc::XLargeChunk::get):
2693         (bmalloc::XLargeChunk::begin):
2694         (bmalloc::XLargeChunk::XLargeChunk):
2695         (bmalloc::XLargeChunk::create):
2696         (bmalloc::XLargeChunk::destroy):
2697         (bmalloc::XLargeChunk::range):
2698         (bmalloc::XLargeChunk::size):
2699         * bmalloc/bmalloc.h: Added.
2700         (bmalloc::api::malloc):
2701         (bmalloc::api::free):
2702         (bmalloc::api::realloc):
2703         * bmalloc/mbmalloc.cpp: Added.
2704