bmalloc: Added missing features to the malloc zone introspection API
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc: Added missing features to the malloc zone introspection API
4         https://bugs.webkit.org/show_bug.cgi?id=142235
5
6         Reviewed by Andreas Kling.
7
8         This should fix the crash we saw on the iOS PLT bot
9         (c.f. http://trac.webkit.org/changeset/180604).
10
11         * bmalloc/Zone.cpp:
12         (bmalloc::good_size):
13         (bmalloc::check):
14         (bmalloc::print):
15         (bmalloc::log):
16         (bmalloc::force_lock):
17         (bmalloc::force_unlock):
18         (bmalloc::statistics):
19         (bmalloc::size):
20         (bmalloc::enumerator): Provide all of these functions since they are called
21         indiscriminately on all zones.
22
23         (bmalloc::Zone::Zone):
24         (bmalloc::Zone::size): Deleted.
25         (bmalloc::Zone::enumerator): Deleted. Moved these functions out of the
26         Zone class since they can stand alone.
27
28         * bmalloc/Zone.h:
29
30 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
31
32         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
33         https://bugs.webkit.org/show_bug.cgi?id=141802
34
35         Reviewed by Andreas Kling.
36
37         Rolling back in but disabled on iOS until I can debug why the iOS PLT crashes.
38
39         * bmalloc/VMHeap.cpp:
40         (bmalloc::VMHeap::grow):
41         * bmalloc/VMHeap.h:
42         * bmalloc/Zone.cpp:
43         (bmalloc::Zone::size):
44         (bmalloc::Zone::Zone):
45         * bmalloc/Zone.h:
46
47 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
48
49         bmalloc: Miscellaneous cleanup
50         https://bugs.webkit.org/show_bug.cgi?id=142231
51
52         Reviewed by Andreas Kling.
53
54         No performance change -- maybe a tiny reduction in memory use.
55
56         * bmalloc/Heap.cpp: Moved the sleep function into StaticMutex, since
57         it's a helper for working with mutexes.
58
59         (bmalloc::Heap::scavenge): Make sure to wait before we start any
60         scavenging, since individual scavenging functions now always scavenge
61         at least one page before waiting themselves.
62
63         (bmalloc::Heap::scavengeSmallPages):
64         (bmalloc::Heap::scavengeMediumPages):
65         (bmalloc::Heap::scavengeLargeObjects): Use the new wait helper to
66         simplify this code. Also, we now require our caller to wait until at
67         least one deallocation is desirable. This simplifies our loop.
68
69         (bmalloc::Heap::allocateSmallPage):
70         (bmalloc::Heap::allocateMediumPage):
71         (bmalloc::Heap::allocateXLarge):
72         (bmalloc::Heap::allocateLarge): Don't freak out any time the heap does
73         an allocation. Only consider the heap to be growing if it actually needs
74         to allocate new VM. This allows us to shrink the heap back down from a
75         high water mark more reliably even if heap activity continues.
76
77         (bmalloc::sleep): Deleted.
78         (bmalloc::Heap::scavengeLargeRanges): Renamed to match our use of
79         "LargeObject".
80
81         * bmalloc/Heap.h:
82
83         * bmalloc/LargeObject.h:
84         (bmalloc::LargeObject::operator bool): Added to simplify a while loop.
85
86         * bmalloc/StaticMutex.h:
87         (bmalloc::sleep):
88         (bmalloc::waitUntilFalse): New helper for waiting until a condition
89         becomes reliably false.
90
91         * bmalloc/Vector.h:
92         (bmalloc::Vector<T>::~Vector): Oops! Don't deallocate the null pointer.
93         We don't actually run any Vector destructors, but an iteration of this
94         patch did, and then crashed. So, let's fix that.
95
96 2015-03-02  Geoffrey Garen  <ggaren@apple.com>
97
98         bmalloc: Eagerly remove allocated objects from the free list
99         https://bugs.webkit.org/show_bug.cgi?id=142194
100
101         Reviewed by Andreas Kling.
102
103         This reduces the pressure to garbage collect the free list.
104
105         Might be a 1% speedup on MallocBench.
106
107         * bmalloc/FreeList.cpp: Put this comment at the top of the file instead
108         of repeating it inside of each function. Tried to clarify the details.
109
110         (bmalloc::FreeList::takeGreedy): Matched the other iteration code in this
111         file for consistency -- even though either direction works fine in this
112         function.
113
114         (bmalloc::FreeList::take): Change to iterate from low to high so that we
115         can maintain an index into the vector that is not disturbed even if we
116         pop from the middle (which invalidates the last index in the vector).
117
118         Decrement i when popping from the middle to make sure that we don't
119         skip the next item after popping.
120
121         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): Ditto.
122
123 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
124
125         Fixed a typo in the previous commit.
126
127         * bmalloc/BoundaryTag.h:
128         (bmalloc::BoundaryTag::setOwner):
129
130 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
131
132         EFL build fix after r180797.
133
134         * bmalloc/BoundaryTag.h:
135         (bmalloc::BoundaryTag::owner):
136         (bmalloc::BoundaryTag::setOwner):
137
138 2015-02-27  Geoffrey Garen  <ggaren@apple.com>
139
140         bmalloc: Pathological madvise churn on the free(malloc(x)) benchmark
141         https://bugs.webkit.org/show_bug.cgi?id=142058
142
143         Reviewed by Andreas Kling.
144
145         The churn was caused by repeatedly splitting an object with physical
146         pages from an object without, and then merging them back together again.
147         The merge would conservatively forget that we had physical pages, forcing
148         a new call to madvise on the next allocation.
149
150         This patch more strictly segregates objects in the heap from objects in
151         the VM heap, with these changes:
152
153         (1) Objects in the heap are not allowed to merge with objects in the VM
154         heap, and vice versa -- since that would erase our precise knowledge of
155         which physical pages had been allocated.
156
157         (2) The VM heap is exclusively responsible for allocating and deallocating
158         physical pages.
159
160         (3) The heap free list must consider entries for objects that are in the
161         VM heap to be invalid, and vice versa. (This condition can arise
162         because the free list does not eagerly remove items.)
163
164         With these changes, we can know that any valid object in the heap's free
165         list already has physical pages, and does not need to call madvise.
166
167         Note that the VM heap -- as before -- might sometimes contain ranges
168         or pieces of ranges that have physical pages, since we allow splitting
169         of ranges at granularities smaller than the VM page size. These ranges
170         can eventually merge with ranges in the heap during scavenging.
171
172         * bmalloc.xcodeproj/project.pbxproj:
173
174         * bmalloc/BoundaryTag.h:
175         (bmalloc::BoundaryTag::owner):
176         (bmalloc::BoundaryTag::setOwner):
177         (bmalloc::BoundaryTag::initSentinel):
178         (bmalloc::BoundaryTag::hasPhysicalPages): Deleted.
179         (bmalloc::BoundaryTag::setHasPhysicalPages): Deleted. Replaced the concept
180         of "has physical pages" with a bit indicating which heap owns the large
181         object. This is a more precise concept, since the old bit was really a
182         Yes / Maybe bit.
183
184         * bmalloc/Deallocator.cpp:
185
186         * bmalloc/FreeList.cpp: Adopt
187         (bmalloc::FreeList::takeGreedy):
188         (bmalloc::FreeList::take):
189         (bmalloc::FreeList::removeInvalidAndDuplicateEntries):
190         * bmalloc/FreeList.h:
191         (bmalloc::FreeList::push): Added API for considering the owner when
192         deciding if a free list entry is valid.
193
194         * bmalloc/Heap.cpp:
195         (bmalloc::Heap::Heap): Adopt new API.
196
197         (bmalloc::Heap::scavengeLargeRanges): Scavenge all ranges with no minimum,
198         since some ranges might be able to merge with ranges in the VM heap, and
199         they won't be allowed to until we scavenge them.
200
201         (bmalloc::Heap::allocateSmallPage):
202         (bmalloc::Heap::allocateMediumPage):
203         (bmalloc::Heap::allocateLarge): New VM heap API makes this function
204         simpler, since we always get back physical pages now.
205
206         * bmalloc/Heap.h:
207         * bmalloc/LargeObject.h:
208         (bmalloc::LargeObject::end):
209         (bmalloc::LargeObject::owner):
210         (bmalloc::LargeObject::setOwner):
211         (bmalloc::LargeObject::isValidAndFree):
212         (bmalloc::LargeObject::merge): Do not merge objects across heaps since
213         that causes madvise churn.
214         (bmalloc::LargeObject::validateSelf):
215         (bmalloc::LargeObject::init):
216         (bmalloc::LargeObject::hasPhysicalPages): Deleted.
217         (bmalloc::LargeObject::setHasPhysicalPages): Deleted. Propogate the Owner API.
218
219         * bmalloc/Owner.h: Added.
220
221         * bmalloc/SegregatedFreeList.cpp:
222         (bmalloc::SegregatedFreeList::SegregatedFreeList):
223         (bmalloc::SegregatedFreeList::insert):
224         (bmalloc::SegregatedFreeList::takeGreedy):
225         (bmalloc::SegregatedFreeList::take):
226         * bmalloc/SegregatedFreeList.h: Propogate the owner API.
227
228         * bmalloc/VMAllocate.h:
229         (bmalloc::vmDeallocatePhysicalPagesSloppy):
230         (bmalloc::vmAllocatePhysicalPagesSloppy): Clarified these functions and
231         removed an edge case.
232
233         * bmalloc/VMHeap.cpp:
234         (bmalloc::VMHeap::VMHeap):
235         * bmalloc/VMHeap.h:
236         (bmalloc::VMHeap::allocateSmallPage):
237         (bmalloc::VMHeap::allocateMediumPage):
238         (bmalloc::VMHeap::allocateLargeObject):
239         (bmalloc::VMHeap::deallocateLargeObject): Be sure to give each object
240         a new chance to merge, since it might have been prohibited from merging
241         before by virtue of not being in the VM heap.
242
243         (bmalloc::VMHeap::allocateLargeRange): Deleted.
244         (bmalloc::VMHeap::deallocateLargeRange): Deleted.
245
246 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
247
248         bmalloc: Large object free list can grow infinitely
249         https://bugs.webkit.org/show_bug.cgi?id=142055
250
251         Reviewed by Andreas Kling.
252
253         By design, we don't eagerly remove large objects from the free list.
254         This creates two simple pathologies:
255
256             (1) If you free and then allocate the same object repeatedly, it will
257             duplicate itself in the free list repeatedly. Since it is never
258             invalid at the time of allocation, it will never be removed.
259
260             (2) If you split and then merge the same object repeatedly, it will
261             duplicate its split sibling in the free list repeatedly. If its
262             sibling is in a separate free list size class, it will never be
263             consulted at the time of allocation, so it will never be removed.
264
265         So, a simple "while (1) { free(malloc(x)); }" causes infinite memory
266         use in the free list.
267
268         The solution in this patch is a simple helper to remove garbage from the
269         free list if it grows too large. This pathology is not common, so the
270         cost is OK.
271
272         Long-term, perhaps we should rethink the laziness of these free lists.
273
274         * bmalloc/BoundaryTag.h:
275         (bmalloc::BoundaryTag::isMarked):
276         (bmalloc::BoundaryTag::setMarked): New bit, used by free list GC.
277
278         * bmalloc/FreeList.cpp:
279         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): The GC algorithm.
280
281         * bmalloc/FreeList.h:
282         (bmalloc::FreeList::FreeList):
283         (bmalloc::FreeList::push): Invoke the GC if we're getting huge.
284
285         * bmalloc/LargeObject.h:
286         (bmalloc::LargeObject::isMarked):
287         (bmalloc::LargeObject::setMarked):
288         (bmalloc::LargeObject::validateSelf): Expose the new bit.
289
290         * bmalloc/Sizes.h: New constant to control GC frequency.
291
292 2015-02-26  Csaba Osztrogonác  <ossy@webkit.org>
293
294         URTBF after r180693.
295
296         * CMakeLists.txt:
297
298 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
299
300         Try to fix the Mac build.
301
302         Unreviewed.
303
304         * bmalloc.xcodeproj/project.pbxproj: Make FreeList.h available.
305
306 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
307
308         bmalloc: Refactored SegregatedFreeList and BoundaryTag::init
309         https://bugs.webkit.org/show_bug.cgi?id=142049
310
311         Reviewed by Anders Carlsson.
312
313         Split out a FreeList class from SegregatedFreeList. This will make it
314         easier to add behaviors on free list insertion and removal -- and it's
315         probably how I should have designed things at the start.
316
317         Moved BoundaryTag::init into LargeObject, since all the related logic
318         lives in LargeObject now too, and this allows us to remove BoundaryTagInlines.h.
319
320         * bmalloc.xcodeproj/project.pbxproj:
321         * bmalloc/BoundaryTagInlines.h: Removed.
322         * bmalloc/FreeList.cpp: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.cpp.
323         (bmalloc::FreeList::takeGreedy):
324         (bmalloc::FreeList::take):
325         (bmalloc::SegregatedFreeList::SegregatedFreeList): Deleted.
326         (bmalloc::SegregatedFreeList::insert): Deleted.
327         (bmalloc::SegregatedFreeList::takeGreedy): Deleted.
328         (bmalloc::SegregatedFreeList::take): Deleted.
329         * bmalloc/FreeList.h: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.h.
330         (bmalloc::FreeList::push):
331         * bmalloc/LargeObject.h:
332         (bmalloc::LargeObject::init):
333         * bmalloc/SegregatedFreeList.cpp:
334         (bmalloc::SegregatedFreeList::SegregatedFreeList):
335         (bmalloc::SegregatedFreeList::insert):
336         (bmalloc::SegregatedFreeList::takeGreedy):
337         (bmalloc::SegregatedFreeList::take):
338         * bmalloc/SegregatedFreeList.h:
339         * bmalloc/Sizes.h:
340         * bmalloc/VMHeap.cpp:
341         (bmalloc::VMHeap::grow):
342
343 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
344
345         bmalloc: free up a bit in BoundaryTag
346         https://bugs.webkit.org/show_bug.cgi?id=142048
347
348         Reviewed by Brady Eidson.
349
350         We were wasting a bit by accident, and I need one now.
351
352         * bmalloc/Algorithm.h:
353         (bmalloc::rightShift): Deleted. Not needed, now that I've simplified
354         the math.
355
356         * bmalloc/BoundaryTag.h: Since each boundary tag bucket is 1024 bytes
357         long, the maximum offset into a bucket is 1023.
358
359         You need 5 bits to count up to 1024, but only 4 to count up to 1023.
360         
361         Math is hard.
362
363         (bmalloc::BoundaryTag::compactBegin): Switched to division because it
364         is simpler, and easier to match up with our ASSERT. The compiler will
365         turn division by constant power of two into a shift for us.
366
367         (bmalloc::BoundaryTag::setRange): Added an ASSERT for compactBegin
368         because we do encode it, so we should ASSERT that encoding did not
369         lose information.
370
371         * bmalloc/Sizes.h: Shifting is no longer used since we use division
372         instead.
373
374 2015-02-24  Stephanie Lewis  <slewis@apple.com>
375
376         Rolling out http://trac.webkit.org/changeset/180430 as it causes the PLT to crash.
377         <rdar://problem/19948015>
378
379         Unreviewed.
380
381         * bmalloc/VMHeap.cpp:
382         (bmalloc::VMHeap::grow):
383         * bmalloc/VMHeap.h:
384         * bmalloc/Zone.cpp:
385         (bmalloc::Zone::Zone):
386         (bmalloc::Zone::size): Deleted.
387         * bmalloc/Zone.h:
388
389 2015-02-24  Geoffrey Garen  <ggaren@apple.com>
390
391         bmalloc: Added a little more abstraction for large objects
392         https://bugs.webkit.org/show_bug.cgi?id=141978
393
394         Reviewed by Sam Weinig.
395
396         Previously, each client needed to manage the boundary tags of
397         a large object using free functions. This patch introduces a LargeObject
398         class that does things a little more automatically.
399
400         * bmalloc.xcodeproj/project.pbxproj:
401
402         * bmalloc/Allocator.cpp:
403         (bmalloc::Allocator::reallocate): Use the new LargeObject class.
404
405         * bmalloc/BeginTag.h:
406         (bmalloc::BeginTag::isInFreeList): Deleted. Moved this logic into the
407         LargeObject class.
408
409         * bmalloc/BoundaryTag.h:
410         (bmalloc::BoundaryTag::isSentinel):
411         (bmalloc::BoundaryTag::compactBegin):
412         (bmalloc::BoundaryTag::setRange):
413         (bmalloc::BoundaryTag::initSentinel): Added an explicit API for sentinels,
414         which we used to create and test for implicitly.
415
416         * bmalloc/BoundaryTagInlines.h:
417         (bmalloc::BoundaryTag::init):
418         (bmalloc::validate): Deleted.
419         (bmalloc::validatePrev): Deleted.
420         (bmalloc::validateNext): Deleted.
421         (bmalloc::BoundaryTag::mergeLeft): Deleted.
422         (bmalloc::BoundaryTag::mergeRight): Deleted.
423         (bmalloc::BoundaryTag::merge): Deleted.
424         (bmalloc::BoundaryTag::deallocate): Deleted.
425         (bmalloc::BoundaryTag::split): Deleted.
426         (bmalloc::BoundaryTag::allocate): Deleted. Moved this logic into the
427         LargeObject class.
428
429         * bmalloc/EndTag.h:
430         (bmalloc::EndTag::init):
431         (bmalloc::EndTag::operator=): Deleted. Re-reading this code, I found
432         special behavior in the assignment operator to be a surprising API.
433         So, I replaced the assignment operation with an explicit initializing
434         function.
435
436         * bmalloc/Heap.cpp:
437         (bmalloc::Heap::scavengeLargeRanges):
438         (bmalloc::Heap::allocateXLarge):
439         (bmalloc::Heap::findXLarge):
440         (bmalloc::Heap::deallocateXLarge):
441         (bmalloc::Heap::allocateLarge):
442         (bmalloc::Heap::deallocateLarge):
443         * bmalloc/Heap.h: No behavior changes here -- just adopting the
444         LargeObject interface.
445
446         * bmalloc/LargeObject.h: Added.
447         (bmalloc::LargeObject::operator!):
448         (bmalloc::LargeObject::begin):
449         (bmalloc::LargeObject::size):
450         (bmalloc::LargeObject::range):
451         (bmalloc::LargeObject::LargeObject):
452         (bmalloc::LargeObject::setFree):
453         (bmalloc::LargeObject::isFree):
454         (bmalloc::LargeObject::hasPhysicalPages):
455         (bmalloc::LargeObject::setHasPhysicalPages):
456         (bmalloc::LargeObject::isValidAndFree):
457         (bmalloc::LargeObject::merge):
458         (bmalloc::LargeObject::split):
459         (bmalloc::LargeObject::validateSelf):
460         (bmalloc::LargeObject::validate): Moved this code into a class, out of
461         BoundaryTag free functions.
462
463         New to the class are these features:
464
465             (1) Every reference to an object is validated upon creation and use.
466
467             (2) There's an explicit API for "This is a reference to an object
468             that might be stale (the DoNotValidate API)".
469
470             (3) The begin and end tags are kept in sync automatically.
471
472         * bmalloc/SegregatedFreeList.cpp:
473         (bmalloc::SegregatedFreeList::insert):
474         (bmalloc::SegregatedFreeList::takeGreedy):
475         (bmalloc::SegregatedFreeList::take):
476         * bmalloc/SegregatedFreeList.h: Adopt the LargeObject interface.
477
478         * bmalloc/VMHeap.cpp:
479         (bmalloc::VMHeap::grow):
480         * bmalloc/VMHeap.h:
481         (bmalloc::VMHeap::allocateLargeRange):
482         (bmalloc::VMHeap::deallocateLargeRange): Adopt the LargeObject interface.
483
484 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
485
486         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
487         https://bugs.webkit.org/show_bug.cgi?id=141802
488
489         Reviewed by Andreas Kling.
490
491         Rolling back in with a fix for a crash seen while using GuardMalloc.
492
493         * bmalloc/VMHeap.cpp:
494         (bmalloc::VMHeap::grow):
495         * bmalloc/VMHeap.h:
496         * bmalloc/Zone.cpp: Re-land the old patch.
497
498         (bmalloc::Zone::size): Be sure to implement the size() function since
499         it's accessible indirectly via the malloc_zone_from_ptr public API --
500         and GuardMalloc calls it all the time.
501
502         (bmalloc::Zone::Zone):
503         * bmalloc/Zone.h: Re-land the old patch.
504
505 2015-02-19  Commit Queue  <commit-queue@webkit.org>
506
507         Unreviewed, rolling out r180363.
508         https://bugs.webkit.org/show_bug.cgi?id=141814
509
510         Caused >50 crashes when running LayoutTests in GuardMalloc or
511         ASAN modes. (Requested by jernoble on #webkit).
512
513         Reverted changeset:
514
515         "bmalloc should implement malloc introspection (to stop false-
516         positive leaks when MallocStackLogging is off)"
517         https://bugs.webkit.org/show_bug.cgi?id=141802
518         http://trac.webkit.org/changeset/180363
519
520 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
521
522         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
523         https://bugs.webkit.org/show_bug.cgi?id=141802
524
525         Reviewed by Andreas Kling.
526
527         Fixed a last-minute type.
528
529         The macro is OS, not PLATFORM.
530
531         * bmalloc/VMHeap.cpp:
532         (bmalloc::VMHeap::grow):
533         * bmalloc/VMHeap.h:
534         * bmalloc/Zone.h:
535
536 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
537
538         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
539         https://bugs.webkit.org/show_bug.cgi?id=141802
540
541         Reviewed by Andreas Kling.
542
543         This patch does the bare minimum to stop false positive leaks from
544         being reported by the Darwin leaks tool. We register each super chunk
545         as a single object, and then request that the leaks tool scan it.
546
547         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
548         zone introspection API.
549
550         * bmalloc/Algorithm.h: Missing #include.
551
552         * bmalloc/VMHeap.cpp:
553         (bmalloc::VMHeap::grow):
554         * bmalloc/VMHeap.h: Adopt the new abstraction.
555
556         * bmalloc/Zone.cpp: Added.
557         (bmalloc::remoteRead): Helper for reading an object out of another process.
558         (bmalloc::Zone::enumerator):
559         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
560         in introspection.
561
562         * bmalloc/Zone.h: Added.
563         (bmalloc::Zone::superChunks):
564         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
565         since our dynamic allocations will not be scanned by leaks since they
566         will have the malloc VM tag.
567
568 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
569
570         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
571         https://bugs.webkit.org/show_bug.cgi?id=141759
572
573         Reviewed by Andreas Kling.
574
575         * bmalloc.xcodeproj/project.pbxproj:
576         * bmalloc/SuperChunk.h: Added.
577         (bmalloc::SuperChunk::create):
578         (bmalloc::SuperChunk::SuperChunk):
579         (bmalloc::SuperChunk::smallChunk):
580         (bmalloc::SuperChunk::mediumChunk):
581         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
582         into a separate class, for clarity and type safety.
583
584         * bmalloc/VMHeap.cpp:
585         (bmalloc::VMHeap::grow):
586         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
587         "grow" because Andreas found "allocateSuperChunk" to be unclear.
588
589         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
590         for malloc introspection.
591
592         (bmalloc::VMHeap::allocateSmallPage):
593         (bmalloc::VMHeap::allocateMediumPage):
594         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
595
596 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
597
598         Build bmalloc through CMake as a static library. It's then linked either
599         into the WTF library (if built as a shared library) or into the JSC and
600         WebKit2 libraries. There's no need to build it as a standalone shared library.
601
602         Rubber-stamped by Carlos Garcia Campos.
603
604         * CMakeLists.txt:
605
606 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
607
608         [BMalloc] Add a FIXME comment for memory alignas
609         https://bugs.webkit.org/show_bug.cgi?id=141556
610
611         Reviewed by Csaba Osztrogonác.
612
613         * bmalloc/Chunk.h: Add a FIXME comment.
614         * bmalloc/LargeChunk.h: ditto.
615
616 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
617
618         bmalloc buildfix on 32 bit Linux (x86/ARM)
619         https://bugs.webkit.org/show_bug.cgi?id=141472
620
621         Reviewed by Gyuyoung Kim.
622
623         * bmalloc/Algorithm.h:
624         (bmalloc::roundUpToMultipleOf):
625         * bmalloc/FixedVector.h:
626         (bmalloc::FixedVector::clear):
627         * bmalloc/Sizes.h:
628         (bmalloc::Sizes::sizeClass):
629
630 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
631
632         [EFL][GTK] Use bmalloc instead of tcmalloc
633         https://bugs.webkit.org/show_bug.cgi?id=140162
634
635         Reviewed by Carlos Garcia Campos.
636
637         Support to use bmalloc on EFL and GTK ports.
638
639         * CMakeLists.txt: Added.
640         * bmalloc/Allocator.cpp:
641         (bmalloc::Allocator::allocate):
642             Fix unused return value caused by posix_memalign().
643         * bmalloc/AsyncTask.h:
644         * bmalloc/BoundaryTag.h:
645         (bmalloc::BoundaryTag::clear):
646         * bmalloc/Chunk.h:
647             Change Traits::Page with Traits::PageType in order to fix
648             -fpermitive build error on EFL and GTK port. 
649         * bmalloc/EndTag.h:
650         (bmalloc::EndTag::operator=):
651         * bmalloc/Line.h: ditto.
652         * bmalloc/MediumTraits.h:
653         * bmalloc/Page.h: ditto.
654         * bmalloc/PerThread.h:
655             EFL port doesn't support __has_include definition yet.
656             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
657         * bmalloc/SmallTraits.h: ditto.
658         * bmalloc/VMAllocate.h:
659         (bmalloc::vmDeallocatePhysicalPages):
660         (bmalloc::vmAllocatePhysicalPages):
661         * bmalloc/Vector.h:
662         (bmalloc::Vector<T>::push):
663         (bmalloc::Vector<T>::reallocateBuffer):
664
665 2015-01-31  Sam Weinig  <sam@webkit.org>
666
667         Remove even more Mountain Lion support
668         https://bugs.webkit.org/show_bug.cgi?id=141124
669
670         Reviewed by Alexey Proskuryakov.
671
672         * Configurations/Base.xcconfig:
673         * Configurations/DebugRelease.xcconfig:
674
675 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
676
677         GC marking threads should clear malloc caches
678         https://bugs.webkit.org/show_bug.cgi?id=141097
679
680         Reviewed by Andreas Kling.
681
682         Split the scavenging API into per-thread vs global, so that you can
683         request to scavenge your own thread without scavenging the whole heap.
684
685         * bmalloc/Cache.cpp:
686         (bmalloc::Cache::scavenge):
687         * bmalloc/bmalloc.h:
688         (bmalloc::api::scavengeThisThread):
689         (bmalloc::api::scavenge):
690
691 2015-01-28  Dana Burkart  <dburkart@apple.com>
692
693         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
694         https://bugs.webkit.org/show_bug.cgi?id=136765
695
696         Reviewed by Alexey Proskuryakov.
697
698         * Configurations/Base.xcconfig:
699         * Configurations/DebugRelease.xcconfig:
700
701 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
702
703         bmalloc: support aligned allocation
704         https://bugs.webkit.org/show_bug.cgi?id=140732
705
706         Reviewed by Andreas Kling.
707
708         * bmalloc/Allocator.cpp:
709         (bmalloc::Allocator::allocate): New function for aligned allocation.
710
711         Small and medium requests just allocate and free until they find an
712         aligned pointer. This is slightly inefficient in the worst case, but
713         still constant-time with little-to-no space overhead.
714
715         Large requests use a new API that requires the client to specify both
716         its ideal size and alignment, and the worst-case size you would have to
717         allocate in order to produce some interior pointer of the requested size
718         and alignment. We put the burden of this calculation on the client
719         because it simplifies things if we guarantee that allocation won't fail.
720
721         XLarge requests are easy: we just forward them to vmAllocate, which
722         already supported aligned requests.
723
724         * bmalloc/BoundaryTag.h:
725         * bmalloc/BoundaryTagInlines.h:
726         (bmalloc::BoundaryTag::mergeLeft):
727         (bmalloc::BoundaryTag::mergeRight):
728         (bmalloc::BoundaryTag::merge):
729         (bmalloc::BoundaryTag::deallocate):
730         (bmalloc::BoundaryTag::split):
731         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
732         refactored the interface to remove some reference out parameters in
733         order to clarify what changes and what doesn't.
734
735         * bmalloc/Heap.cpp:
736         (bmalloc::Heap::allocateXLarge): Added an alignment API.
737
738         (bmalloc::Heap::allocateLarge):
739         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
740         a few variants, so aligned and unaligned allocation could share some code.
741
742         * bmalloc/SegregatedFreeList.cpp:
743         (bmalloc::SegregatedFreeList::take):
744         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
745         for aligned allocation. It turns out that the aligned path is pretty
746         different, since it ends up searching for two potential ways to satisfy
747         an allocation: either large enough and aligned, or large enough to split
748         into something not aligned and something large enough and aligned.
749
750         * bmalloc/VMAllocate.h:
751         (bmalloc::vmAllocate): Switched alignment to come before size because
752         that's how the memalign API specifies it.
753
754         * bmalloc/VMHeap.h:
755         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
756
757 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
758
759         bmalloc: a little bit of cleanup
760         https://bugs.webkit.org/show_bug.cgi?id=140687
761
762         Reviewed by Anders Carlsson.
763
764         * bmalloc/Algorithm.h:
765         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
766         of code.
767
768         * bmalloc/BoundaryTag.h:
769         * bmalloc/BoundaryTagInlines.h:
770         (bmalloc::BoundaryTag::mergeLeft):
771         (bmalloc::BoundaryTag::mergeRight):
772         (bmalloc::BoundaryTag::merge):
773         (bmalloc::BoundaryTag::deallocate):
774         (bmalloc::BoundaryTag::split):
775         (bmalloc::BoundaryTag::allocate):
776         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
777         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
778         (bmalloc::BoundaryTag::mergeLarge): Deleted.
779         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
780         from all these functions, since boundary tags always pertain to large
781         objects, and putting the word "Large" everywhere wasn't helping to
782         explain that.
783
784         * bmalloc/Heap.cpp:
785         (bmalloc::Heap::allocateXLarge):
786         (bmalloc::Heap::findXLarge):
787         (bmalloc::Heap::deallocateXLarge):
788         * bmalloc/Heap.h:
789         * bmalloc/VMHeap.h:
790         (bmalloc::VMHeap::allocateXLarge): Deleted.
791         (bmalloc::VMHeap::findXLarge): Deleted.
792         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
793         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
794         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
795         really make sense for the VMHeap to be involved.
796
797 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
798
799         bmalloc: refactored XLarge allocation for better alignment
800         https://bugs.webkit.org/show_bug.cgi?id=140582
801
802         Reviewed by Andreas Kling.
803
804         XLarge objects used to be Large objects with an extra bit of metadata
805         that said "actually, I'm not large -- I'm extra large".
806
807         The metadata header in an XLarge allocation made it impossible for the
808         XLarge object to honor a very large alignment request.
809
810         The solution is to stop using a metadata header for XLarge objects, and
811         instead to store explicit metadata on the side.
812
813         This is a bit less astonishing, which is also nice.
814
815         Finding XLarge metadata is now a linear search. That's probably OK, since
816         it was always so in TCMalloc, and the usual number of XLarge allocations
817         in a process is 0.
818
819         This design makes it possible for the heap to cache XLarge allocations
820         with and/or without physical pages. I haven't actually done that yet
821         because the tradeoffs are subtle, so I don't want to do anything without
822         a motivating test case.
823
824         * bmalloc.xcodeproj/project.pbxproj:
825         * bmalloc/Allocator.cpp:
826         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
827         since an XLarge allocation is now just a naked buffer without a header.
828
829         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
830         XLarge alignment, since XLargeChunk won't give this to us implicitly
831         anymore.
832
833         * bmalloc/BoundaryTag.h:
834         (bmalloc::BoundaryTag::setRange):
835         (bmalloc::BoundaryTag::isXLarge): Deleted.
836         (bmalloc::BoundaryTag::setXLarge): Deleted.
837         * bmalloc/BoundaryTagInlines.h:
838         (bmalloc::validate):
839         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
840
841         * bmalloc/Deallocator.cpp:
842         (bmalloc::Deallocator::deallocateXLarge):
843         (bmalloc::Deallocator::deallocateSlowCase):
844         * bmalloc/Heap.cpp:
845         (bmalloc::Heap::findXLarge):
846         (bmalloc::Heap::allocateXLarge):
847         (bmalloc::Heap::deallocateXLarge):
848         * bmalloc/Heap.h: Updated for interface changes.
849
850         * bmalloc/ObjectType.cpp:
851         (bmalloc::objectType):
852         * bmalloc/ObjectType.h:
853         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
854         examining its bit pattern -- just like we do for other kinds of
855         allocations -- which is nice.
856
857         * bmalloc/Sizes.h:
858         * bmalloc/VMHeap.h:
859         (bmalloc::VMHeap::allocateXLarge):
860         (bmalloc::VMHeap::findXLarge):
861         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
862         for XLarge allocations.
863
864         * bmalloc/XLargeChunk.h: Removed.
865
866 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
867
868         bmalloc: added some infrastructure for aligned allocation
869         https://bugs.webkit.org/show_bug.cgi?id=140572
870
871         Reviewed by Andreas Kling.
872
873         * bmalloc/Algorithm.h:
874         (bmalloc::isPowerOfTwo):
875         (bmalloc::roundUpToMultipleOf):
876         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
877         isPowerOfTwo helper function.
878
879         * bmalloc/Allocator.cpp:
880         (bmalloc::Allocator::allocate):
881         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
882         Doesn't do anything yet, but does correctly forward to system malloc
883         when bmalloc is disabled.
884
885         * bmalloc/Cache.cpp:
886         (bmalloc::Cache::allocateSlowCaseNullCache):
887         * bmalloc/Cache.h:
888         (bmalloc::Cache::allocate):
889         * bmalloc/bmalloc.h:
890         (bmalloc::api::memalign):
891         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
892
893 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
894
895         Consider alignment when allocating from a SegregatedFreeList
896         https://bugs.webkit.org/show_bug.cgi?id=140408
897
898         Reviewed by Sam Weinig.
899
900         In preparation for supporting aligned allocation.
901
902         No performance change.
903
904         Since this is just one extra branch in an already expensive function,
905         I decided not to duplicate the function just to avoid the branch in
906         the un-aligned case.
907
908         * bmalloc/SegregatedFreeList.cpp:
909         (bmalloc::SegregatedFreeList::take):
910         * bmalloc/SegregatedFreeList.h:
911
912 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
913
914         Renamed minimum to size in SegregatedFreeList
915         https://bugs.webkit.org/show_bug.cgi?id=140406
916
917         Reviewed by Sam Weinig.
918
919         In preparation for supporting aligned allocation.
920
921         * bmalloc/SegregatedFreeList.cpp:
922         (bmalloc::SegregatedFreeList::takeGreedy):
923         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
924         really just a minimum. Let's not imply that this value is special.
925
926 2015-01-11  Dan Bernstein  <mitz@apple.com>
927
928         Geoff is organized, but he is not an organization.
929
930         Rubber-stamped by Anders Carlsson.
931
932         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
933
934 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
935
936         Make bmalloc work with ASan
937         https://bugs.webkit.org/show_bug.cgi?id=140194
938
939         Reviewed by Mark Lam.
940
941         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
942         an OS-specific API to test for loaded runtime libraries.
943
944         * bmalloc/Environment.cpp:
945         (bmalloc::isASanEnabled):
946         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
947         ASan is enabled, since system malloc has the Asan hooks we need.
948
949         You could check for the ASan compile-time flag instead, but doing this
950         check at runtime prepares bmalloc for a world where it is a dynamic
951         library that might be loaded into projects it did not compile with.
952
953 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
954
955         Fix up bmalloc's PerThread for use on Linux
956         https://bugs.webkit.org/show_bug.cgi?id=139804
957
958         Reviewed by Anders Carlsson.
959
960         The previous implementation was a bit slow.
961
962         * bmalloc/PerThread.h:
963         (bmalloc::PerThreadStorage<Cache>::get):
964         (bmalloc::PerThreadStorage::get):
965         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
966         way to do fast per-thread access without taking a lock every time. This
967         probably works on all the platforms we care about, and it matches other
968         techniques we use elsewhere in WebKit.
969
970         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
971         this class because PerThreadStorage now encapsulates everything that
972         needs to be conditional.
973
974         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
975
976 2014-12-26  Dan Bernstein  <mitz@apple.com>
977
978         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
979         https://bugs.webkit.org/show_bug.cgi?id=139950
980
981         Reviewed by David Kilzer.
982
983         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
984         in a manner that works with Xcode 5.1.1.
985
986 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
987
988         Safari crashes when you set Malloc environment variables
989         https://bugs.webkit.org/show_bug.cgi?id=139656
990
991         Reviewed by Michael Saboff.
992
993         I forgot to cover the realloc() case. Whoops. (OoPS?)
994
995         This time around, I ran the full MallocBench test suite in Malloc=1
996         mode, and it passed.
997
998         * bmalloc/Allocator.cpp:
999         (bmalloc::Allocator::reallocate):
1000         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
1001         It needs to be down there so that we can do the short-circuiting check
1002         for whether bmalloc is enabled first.
1003
1004         Also added the check.
1005
1006         * bmalloc/Cache.cpp:
1007         (bmalloc::Cache::scavenge):
1008         (bmalloc::Cache::Cache):
1009         (bmalloc::Cache::reallocateSlowCaseNullCache):
1010         * bmalloc/Cache.h:
1011         (bmalloc::Cache::deallocator):
1012         (bmalloc::Cache::reallocate): Ditto.
1013
1014         * bmalloc/bmalloc.h:
1015         (bmalloc::api::free):
1016         (bmalloc::api::realloc): Ditto.
1017
1018         (bmalloc::api::scavenge): Pushed this down into Cache to match the
1019         surrounding functions.
1020
1021 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1022
1023         bmalloc should support system memory analysis tools (part 2)
1024         https://bugs.webkit.org/show_bug.cgi?id=139565
1025
1026         Reviewed by Mark Lam.
1027
1028         This patch actually queries the environment to see if memory analysis
1029         tools have been enabled.
1030
1031         * bmalloc/Deallocator.cpp:
1032         (bmalloc::Deallocator::scavenge): Don't process the object log if
1033         we've disabled bmalloc because it will be full of invalid nullptrs.
1034
1035         * bmalloc/Environment.cpp:
1036         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
1037         Malloc debugging flags. I also added a plain "Malloc" catch-all for
1038         when you want to disable bmalloc without enabling any kind of funny
1039         business.
1040
1041         It would be slightly nicer just to iterate the list of environment
1042         variables and strstr them, but getenv is the more portable option,
1043         and performance here doesn't really matter.
1044
1045         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
1046         environment variable.
1047
1048         (bmalloc::Environment::computeIsBmallocEnabled):
1049
1050 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1051
1052         Try to fix the iOS simulator build.
1053
1054         #include the declaration of malloc / free.
1055
1056         * bmalloc/Allocator.cpp:
1057         * bmalloc/Deallocator.cpp:
1058
1059 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1060
1061         Try to fix the build.
1062
1063         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
1064
1065 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1066
1067         bmalloc should support system memory analysis tools (part 1)
1068         https://bugs.webkit.org/show_bug.cgi?id=139559
1069
1070         Reviewed by Mark Lam.
1071
1072         This patch adds the hooks to disable bmalloc at runtime if certain
1073         environment variables are set, but doesn't actually read from the
1074         environment yet.
1075
1076         No performance change.
1077
1078         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
1079         we'll use to read environment variables and see if memory analysis tools
1080         have been enabled.
1081
1082         * bmalloc/Allocator.cpp:
1083         (bmalloc::Allocator::Allocator):
1084         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
1085         on the allocation path. We cache the setting to make the check fast.
1086
1087         * bmalloc/Allocator.h: Interface changes.
1088
1089         * bmalloc/Cache.cpp:
1090         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
1091         and deallocator. This main purpose is to enable them to query the
1092         environment for whether bmalloc is enabled; but this is also a slightly
1093         cleaner way to guarantee to them that the Heap has been pre-initialized.
1094
1095         * bmalloc/Deallocator.cpp:
1096         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
1097         fill the object log to force us to take the slow path on all deallocations.
1098
1099         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
1100
1101         * bmalloc/Deallocator.h: Interface changes.
1102
1103         * bmalloc/Environment.cpp: Added.
1104         (bmalloc::Environment::Environment):
1105         (bmalloc::Environment::computeIsBmallocEnabled):
1106         * bmalloc/Environment.h: Added.
1107         (bmalloc::Environment::isBmallocEnabled): This is the class that will
1108         encapsulate looking for environment variables that turn on heap
1109         analysis tools.
1110
1111         * bmalloc/Heap.h:
1112         (bmalloc::Heap::environment):
1113
1114         * bmalloc/Mutex.h:
1115         (bmalloc::Mutex::Mutex):
1116         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
1117         since I got super confused about them while writing this patch.
1118
1119         * bmalloc/VMHeap.cpp: Fixed an #include.
1120
1121 2014-12-09  David Kilzer  <ddkilzer@apple.com>
1122
1123         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
1124         <http://webkit.org/b/139212>
1125
1126         Reviewed by Joseph Pecoraro.
1127
1128         * Configurations/Base.xcconfig:
1129         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
1130           on OS X.
1131         * Configurations/DebugRelease.xcconfig:
1132         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
1133
1134 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
1135
1136         bmalloc uses 8X more virtual memory than necessary
1137         https://bugs.webkit.org/show_bug.cgi?id=138495
1138
1139         Reviewed by Mark Lam.
1140
1141         iOS has a per-process virtual memory cap around 1GB, so there's some
1142         value to not going totally ham with virtual memory.
1143
1144         We currently use about 8X the necessary amount:
1145             - 2X to align our VM allocation
1146             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
1147
1148         We can cut that down:
1149             - Return the unaligned portion of our VM allocation (-2X)
1150             - Use all the chunks in a superchunk, instead of allocating one
1151               chunk per superchunk (-4X)
1152
1153         * bmalloc/Algorithm.h:
1154         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
1155         function so we can call it with getpagesize() at runtime.
1156
1157         * bmalloc/Chunk.h:
1158         * bmalloc/LargeChunk.h:
1159         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
1160         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
1161
1162         * bmalloc/VMAllocate.h:
1163         (bmalloc::vmValidate):
1164         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
1165         if it does not succeed. Allocate precisely, and give back the extra.
1166
1167         * bmalloc/VMHeap.cpp:
1168         (bmalloc::VMHeap::allocateSuperChunk):
1169         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
1170         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
1171         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
1172         in a superchunk, instead of just one.
1173
1174         * bmalloc/VMHeap.h:
1175         (bmalloc::VMHeap::allocateSmallPage):
1176         (bmalloc::VMHeap::allocateMediumPage):
1177         (bmalloc::VMHeap::allocateLargeRange):
1178         * bmalloc/XLargeChunk.h:
1179         (bmalloc::XLargeChunk::create): Updated to match changes above.
1180
1181 2014-11-01  David Kilzer  <ddkilzer@apple.com>
1182
1183         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
1184         <https://webkit.org/b/138286>
1185         <rdar://problem/18847087>
1186
1187         Reviewed by Dan Bernstein.
1188
1189         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
1190         to NO for the target that produces libbmalloc.a so that the
1191         debug symbols will be linked into JavaScriptCore and end up in
1192         its dSYM file.
1193
1194 2014-10-30  Dana Burkart  <dburkart@apple.com>
1195
1196         <rdar://problem/18821260> Prepare for the mysterious future
1197
1198         Reviewed by Lucas Forschler.
1199
1200         * Configurations/Base.xcconfig:
1201         * Configurations/DebugRelease.xcconfig:
1202
1203 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
1204
1205         bmalloc: cleaned up fast path vs slow path
1206         https://bugs.webkit.org/show_bug.cgi?id=137081
1207
1208         Reviewed by Sam Weinig.
1209
1210         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
1211
1212         * bmalloc/Allocator.cpp:
1213         (bmalloc::Allocator::Allocator): Merged the small and medium range
1214         caches, just like the small and medium allocators. Ranges are abstract
1215         objects that don't really care whether they hold small or medium objects,
1216         so they don't need to be segregated.
1217
1218         (bmalloc::Allocator::scavenge): Ditto.
1219
1220         (bmalloc::Allocator::allocateBumpRangeSlowCase):
1221         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
1222         we do care a tiny bit, because we need to specify small vs medium when
1223         allocating new ranges from the heap, to ensure that the heap allocates
1224         from the right segment of VM.
1225
1226         (bmalloc::Allocator::allocateLarge):
1227         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
1228         up the fast path. Large allocation performance is dominated by allocation
1229         logic and initialization, so inlining it doesn't help.
1230
1231         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
1232         it doesn't need to distinguish small vs medium objects.
1233
1234         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
1235         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
1236
1237         * bmalloc/Allocator.h:
1238         * bmalloc/BumpRange.h:
1239
1240         * bmalloc/Cache.cpp:
1241         (bmalloc::Cache::allocateSlowCase): Deleted.
1242         (bmalloc::Cache::deallocateSlowCase): Deleted.
1243         * bmalloc/Cache.h:
1244         (bmalloc::Cache::allocate):
1245         (bmalloc::Cache::deallocate):
1246         (bmalloc::Cache::allocateFastCase): Deleted.
1247         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
1248         paths. The downside to this change is that the fast path branches to two
1249         distinct failure cases instead of one. The upside is that the slow path
1250         doesn't need to re-read the segment register, which is not as cheap as a
1251         normal register, and it doesn't need to do an extra level of function 
1252         call. Seems to be worth it.
1253
1254         * bmalloc/Deallocator.h:
1255         * bmalloc/Heap.cpp:
1256         (bmalloc::Heap::refillSmallBumpRangeCache):
1257         (bmalloc::Heap::refillMediumBumpRangeCache):
1258         * bmalloc/Heap.h: Updated for interface changes.
1259
1260         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
1261         small lines in a page / 2, since any other free lines will coalesce
1262         with their neighbors.
1263
1264 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1265
1266         Rolled out r173346.
1267
1268             bmalloc should honor the FastMalloc statistics API
1269             https://bugs.webkit.org/show_bug.cgi?id=136592
1270
1271         This didn't really work. Because we allow ranges with and without
1272         physical pages to merge, and we allow double-committing and
1273         double-decommitting, we can't rely on commit actions to track memory
1274         footprint.
1275
1276         * bmalloc/Heap.cpp:
1277         (bmalloc::Heap::size): Deleted.
1278         (bmalloc::Heap::capacity): Deleted.
1279         * bmalloc/Heap.h:
1280         * bmalloc/VMHeap.cpp:
1281         (bmalloc::VMHeap::VMHeap):
1282         (bmalloc::VMHeap::allocateSmallChunk):
1283         (bmalloc::VMHeap::allocateMediumChunk):
1284         (bmalloc::VMHeap::allocateLargeChunk):
1285         * bmalloc/VMHeap.h:
1286         (bmalloc::VMHeap::allocateSmallPage):
1287         (bmalloc::VMHeap::allocateMediumPage):
1288         (bmalloc::VMHeap::allocateLargeRange):
1289         (bmalloc::VMHeap::deallocateSmallPage):
1290         (bmalloc::VMHeap::deallocateMediumPage):
1291         (bmalloc::VMHeap::deallocateLargeRange):
1292         (bmalloc::VMHeap::size): Deleted.
1293         (bmalloc::VMHeap::capacity): Deleted.
1294         * bmalloc/bmalloc.h:
1295         (bmalloc::api::heapSize): Deleted.
1296         (bmalloc::api::heapCapacity): Deleted.
1297
1298 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1299
1300         bmalloc: Allocation should be more precise
1301         https://bugs.webkit.org/show_bug.cgi?id=136993
1302
1303         Reviewed by Gavin Barraclough.
1304
1305         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1306
1307         This patch teaches the allocator to merge adjacent free lines into a
1308         single allocatable range. This allows us to shrink the size of an
1309         individual line without increasing fragmentation or the rate of allocator
1310         slow paths.
1311
1312         We'll only take more slow paths when available memory is sparse, which
1313         is exactly when it's worth it. When available memory is dense, we'll
1314         take fewer slow paths.
1315
1316         * bmalloc.xcodeproj/project.pbxproj:
1317         * bmalloc/Algorithm.h:
1318         (bmalloc::divideRoundingUp):
1319
1320         * bmalloc/Allocator.cpp:
1321         (bmalloc::Allocator::Allocator): Updated for interface changes.
1322
1323         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
1324         Now that we merge lines, it's not convenient to scavenge by line.
1325
1326         (bmalloc::Allocator::allocateSmallBumpRange):
1327         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
1328         instead of individual lines.
1329
1330         (bmalloc::Allocator::allocateSlowCase):
1331         (bmalloc::Allocator::allocateSmallLine): Deleted.
1332         (bmalloc::Allocator::allocateMediumLine): Deleted.
1333         (bmalloc::Allocator::allocateMedium): Deleted.
1334         * bmalloc/Allocator.h:
1335         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
1336         into the standard fast path with small allocations. Since a BumpAllocator
1337         just allocates out of an arbitrary range, it doesn't need to distinguish
1338         between small and medium lines.
1339
1340         * bmalloc/BumpAllocator.h:
1341         (bmalloc::BumpAllocator::size):
1342         (bmalloc::BumpAllocator::BumpAllocator):
1343         (bmalloc::BumpAllocator::init):
1344         (bmalloc::BumpAllocator::refill):
1345         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
1346         anymore: the heap just gives us a pointer and a pre-computed number of
1347         objects, and we allocate them.
1348
1349         * bmalloc/Deallocator.cpp:
1350         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
1351
1352         * bmalloc/Heap.cpp:
1353         (bmalloc::Heap::Heap):
1354         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
1355         detailing where all objects will lie in memory. After we merge two lines,
1356         we might allocate an object that spans from one line to the next. This
1357         metadata details which bits of memory overlap in that way, and how they
1358         overlap.
1359
1360         (bmalloc::Heap::refillSmallBumpRangeCache):
1361         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
1362         and merge adjacent free lines into BumpRanges.
1363
1364         (bmalloc::Heap::allocateSmallPage):
1365         (bmalloc::Heap::allocateMediumPage):
1366         (bmalloc::Heap::deallocateSmallLine):
1367         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
1368         since we scan for free memory a page at a time.
1369
1370         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
1371         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
1372         fast path.
1373
1374         * bmalloc/Heap.h:
1375         (bmalloc::Heap::derefSmallLine):
1376         (bmalloc::Heap::derefMediumLine):
1377         (bmalloc::Heap::deallocateSmallLine): Deleted.
1378         (bmalloc::Heap::allocateSmallLine): Deleted.
1379         (bmalloc::Heap::deallocateMediumLine): Deleted.
1380         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
1381
1382         * bmalloc/Line.h:
1383         (bmalloc::Line<Traits>::ref):
1384         (bmalloc::Line<Traits>::deref):
1385         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
1386         anymore, since we only ever deref by 1 now.
1387
1388         * bmalloc/MediumAllocator.h:
1389         (bmalloc::MediumAllocator::isNull): Deleted.
1390         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
1391         (bmalloc::MediumAllocator::line): Deleted.
1392         (bmalloc::MediumAllocator::allocate): Deleted.
1393         (bmalloc::MediumAllocator::derefCount): Deleted.
1394         (bmalloc::MediumAllocator::refill): Deleted.
1395         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
1396         been dead for a while, since it doesn't build anymore with this patch.
1397
1398         * bmalloc/Page.h:
1399         (bmalloc::Page::sizeClass):
1400         (bmalloc::Page::setSizeClass):
1401         (bmalloc::Page::smallSizeClass): Deleted.
1402         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
1403         to sizeClass, since we use it for medium sizes too.
1404
1405         * bmalloc/Sizes.h:
1406         (bmalloc::Sizes::sizeClass):
1407         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
1408
1409         (bmalloc::Sizes::smallSizeClassFor): Deleted.
1410         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
1411
1412         * bmalloc/bmalloc.h:
1413         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
1414         can be a bit more precise. It also has to be, since we can't guarantee
1415         that an object ends at the end of a line anymore.
1416
1417 2014-09-19  Daniel Bates  <dabates@apple.com>
1418
1419         Always assume internal SDK when building configuration Production
1420         https://bugs.webkit.org/show_bug.cgi?id=136925
1421         <rdar://problem/18362399>
1422
1423         Reviewed by Dan Bernstein.
1424
1425         * Configurations/Base.xcconfig:
1426
1427 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
1428
1429         bmalloc: moved line caches from the deallocator to the allocator
1430         https://bugs.webkit.org/show_bug.cgi?id=136868
1431
1432         Reviewed by Gavin Barraclough.
1433
1434         I did this mostly as a simplification, to make it easier to change the
1435         allocation strategy.
1436
1437         No throughput change on MallocBench. Saves about 50kB.
1438
1439         Since the deallocator needs to lock the heap when freeing lines anyway,
1440         there isn't much benefit to giving the deallocator a local cache of
1441         deallocated lines.
1442
1443         We still give the allocator a local cache of lines because that does
1444         reduce the frequency at which it needs to lock the heap in order to
1445         acquire more lines.
1446
1447         * bmalloc/Allocator.cpp:
1448         (bmalloc::Allocator::scavenge):
1449         (bmalloc::Allocator::allocateSmallLine):
1450         (bmalloc::Allocator::allocateMediumLine):
1451         (bmalloc::Allocator::allocateMedium):
1452         (bmalloc::Allocator::allocateSlowCase):
1453         * bmalloc/Allocator.h:
1454         * bmalloc/Deallocator.cpp:
1455         (bmalloc::Deallocator::Deallocator):
1456         (bmalloc::Deallocator::scavenge):
1457         (bmalloc::Deallocator::processObjectLog):
1458         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
1459         (bmalloc::Deallocator::allocateSmallLine): Deleted.
1460         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
1461         (bmalloc::Deallocator::allocateMediumLine): Deleted.
1462         * bmalloc/Deallocator.h:
1463
1464         * bmalloc/Sizes.h:
1465         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
1466         exactly one page in size. That's about what we were shooting for anyway,
1467         and it may make it easier to switch to per-page allocation in future.
1468
1469 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
1470
1471         bmalloc: allocate small and medium objects using the same bump pointer class
1472         https://bugs.webkit.org/show_bug.cgi?id=136843
1473
1474         Reviewed by Gavin Barraclough.
1475
1476         4% speedup on MallocBench.
1477
1478         Now that medium-sized objects have dedicated per-size allocators, they
1479         don't need to use an arbitrary bump pointer allocator. This means that
1480         every allocator knows how many objects it will allocate from the start,
1481         and we don't need a post-processing step to adjust refcounts based on
1482         real allocation count.
1483
1484         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1485         since it's used for small and medium objects now.
1486
1487         * bmalloc/Allocator.cpp:
1488         (bmalloc::Allocator::Allocator): Updated to use new interface.
1489         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1490         to make sure that we finish allocating all the objects in it.
1491
1492         (bmalloc::Allocator::allocateMedium):
1493         (bmalloc::Allocator::allocateSlowCase):
1494         (bmalloc::Allocator::retire): Deleted.
1495         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1496         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1497         * bmalloc/Allocator.h:
1498         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1499         used to post-process an allocator based on how many objects it allocated.
1500
1501         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1502         (bmalloc::BumpAllocator::BumpAllocator):
1503         (bmalloc::BumpAllocator::init):
1504         (bmalloc::BumpAllocator::line):
1505         (bmalloc::BumpAllocator::validate):
1506         (bmalloc::BumpAllocator::allocate):
1507         (bmalloc::BumpAllocator::refill):
1508         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1509         about the kinds of lines they allocate into. In some cases, the line
1510         type must be provided as a template parameter by the caller.
1511
1512         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1513         (bmalloc::SmallAllocator::line): Deleted.
1514         (bmalloc::SmallAllocator::allocate): Deleted.
1515         (bmalloc::SmallAllocator::objectCount): Deleted.
1516         (bmalloc::SmallAllocator::derefCount): Deleted.
1517         (bmalloc::SmallAllocator::refill): Deleted.
1518         (bmalloc::SmallAllocator::clear): Deleted.
1519
1520         * bmalloc/ObjectType.h:
1521         (bmalloc::isMedium):
1522
1523         * bmalloc/SmallAllocator.h:
1524         (bmalloc::SmallAllocator::isNull): Deleted.
1525         (bmalloc::SmallAllocator::canAllocate): Deleted.
1526         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1527         (bmalloc::SmallAllocator::line): Deleted.
1528         (bmalloc::SmallAllocator::allocate): Deleted.
1529         (bmalloc::SmallAllocator::objectCount): Deleted.
1530         (bmalloc::SmallAllocator::derefCount): Deleted.
1531         (bmalloc::SmallAllocator::refill): Deleted.
1532         (bmalloc::SmallAllocator::clear): Deleted.
1533
1534 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1535
1536         Fixed a goof in bmalloc Vector sizing
1537         https://bugs.webkit.org/show_bug.cgi?id=136795
1538
1539         Reviewed by Gavin Barraclough and Sam Weinig.
1540
1541         We want our minimum vector to be page-sized since the OS will give us
1542         a page no matter what -- but we want that many bytes, and not enough
1543         bytes to store that many elements.
1544
1545         * bmalloc/Vector.h: Math is hard.
1546
1547 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1548
1549         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1550         https://bugs.webkit.org/show_bug.cgi?id=136693
1551
1552         Reviewed by Gavin Barraclough.
1553
1554         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1555
1556         No throughput change.
1557
1558         We keep an array of medium allocators, just like our array of small
1559         allocators.
1560
1561         In future, we can simplify the allocation fast path by merging the small
1562         and medium allocator arrays. For now, this is the simplest change that
1563         gets the win.
1564
1565         * bmalloc/Allocator.cpp:
1566         (bmalloc::Allocator::Allocator):
1567         (bmalloc::Allocator::scavenge):
1568         (bmalloc::Allocator::allocateMedium):
1569         * bmalloc/Allocator.h:
1570         * bmalloc/Sizes.h:
1571         (bmalloc::Sizes::mediumSizeClassFor):
1572
1573 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1574
1575         Reviewed by Sam Weinig.
1576
1577         Renamed log => retire for clarity.
1578
1579         * bmalloc/Allocator.cpp:
1580         (bmalloc::Allocator::scavenge):
1581         (bmalloc::Allocator::retire):
1582         (bmalloc::Allocator::allocateMedium):
1583         (bmalloc::Allocator::allocateSlowCase):
1584         (bmalloc::Allocator::log): Deleted.
1585         * bmalloc/Allocator.h:
1586
1587 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1588
1589         bmalloc: eager scavenge leaves behind a bogus allocator
1590         https://bugs.webkit.org/show_bug.cgi?id=136743
1591
1592         Reviewed by Sam Weinig.
1593
1594         Be sure to clear the allocator after logging it in the eager scavenge
1595         case, so that we don't later try to allocate out of the lines that we
1596         have thrown away.
1597
1598         We didn't need to do this previously because scavenge would only happen
1599         at thread exit time, after which no further allocation from the per-thread
1600         cache would take place.
1601
1602         * bmalloc/Allocator.cpp:
1603         (bmalloc::Allocator::scavenge):
1604         * bmalloc/MediumAllocator.h:
1605         (bmalloc::MediumAllocator::clear):
1606         * bmalloc/SmallAllocator.h:
1607         (bmalloc::SmallAllocator::clear):
1608
1609 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1610
1611         bmalloc should honor the FastMalloc statistics API
1612         https://bugs.webkit.org/show_bug.cgi?id=136592
1613
1614         Reviewed by Gavin Barraclough.
1615
1616         We do this by tracking "size" and "capacity" in the VM heap.
1617
1618         The VM heap's "capacity" is all the VM we ever allocated.
1619
1620         The VM heap's "size" the subset of VM currently held onto by the
1621         VM heap (and therefore not in use by the regular heap).
1622
1623         Somewhat ironically, reducing the process's memory footprint, increases
1624         the size of the VM heap, since the VM heap holds the pages that are
1625         purely virtual and not physical.
1626
1627         * bmalloc/Heap.cpp:
1628         (bmalloc::Heap::size):
1629         (bmalloc::Heap::capacity):
1630         * bmalloc/Heap.h:
1631         * bmalloc/VMHeap.cpp:
1632         (bmalloc::VMHeap::VMHeap):
1633         (bmalloc::VMHeap::allocateSmallChunk):
1634         (bmalloc::VMHeap::allocateMediumChunk):
1635         (bmalloc::VMHeap::allocateLargeChunk):
1636         * bmalloc/VMHeap.h:
1637         (bmalloc::VMHeap::size):
1638         (bmalloc::VMHeap::capacity):
1639         (bmalloc::VMHeap::allocateSmallPage):
1640         (bmalloc::VMHeap::allocateMediumPage):
1641         (bmalloc::VMHeap::allocateLargeRange):
1642         (bmalloc::VMHeap::deallocateSmallPage):
1643         (bmalloc::VMHeap::deallocateMediumPage):
1644         (bmalloc::VMHeap::deallocateLargeRange):
1645         * bmalloc/bmalloc.h:
1646         (bmalloc::api::heapSize):
1647         (bmalloc::api::heapCapacity):
1648
1649 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1650
1651         bmalloc crashes on the EWS bots (due to bad large object allocation)
1652         https://bugs.webkit.org/show_bug.cgi?id=136469
1653
1654         Reviewed by Andreas Kling.
1655
1656         It's possible to convince bmalloc to perform a bad large object allocation,
1657         through these steps:
1658
1659         (1) Insert object A into freelist F0.
1660
1661         (2) Split, merge and split again A's neighbors such that object B is
1662         inserted into freelist F0, with boundary tag and size equal to object A,
1663         but pointer not completely equal to object A. Put object B at the head of F0.
1664
1665         (3) Allocate some other object from F0, swapping its position in the
1666         freelist with object B, such that object A is now ahead of object B.
1667
1668         --> Now, the next allocation for size A/B will allocate object A, which
1669         has a slightly wrong idea about where the object actually begins.
1670         Immediately, you'll corrupt a little memory, and over time, you'll also
1671         corrupt boundary tag metadata.
1672
1673         The solution is to store the begin pointer in the boundary tag. Luckily,
1674         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1675         on MallocBench.
1676
1677         * bmalloc/Algorithm.h:
1678         (bmalloc::rightShift):
1679         * bmalloc/BeginTag.h:
1680         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1681         validate the start pointer when popping off the free list. Through a
1682         very uncommon set of steps, it is possible to have an item in the free
1683         list that is valid by all accounts except for its start pointer.
1684
1685         * bmalloc/BoundaryTag.h:
1686         (bmalloc::BoundaryTag::compactBegin):
1687         (bmalloc::BoundaryTag::setRange):
1688         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1689         start pointer. We don't need the whole pointer -- just the offset, in
1690         largeAlignment increments, into the relevant boundary tag bucket.
1691
1692         * bmalloc/BoundaryTagInlines.h:
1693         (bmalloc::validateNext):
1694         (bmalloc::BoundaryTag::init):
1695         (bmalloc::BoundaryTag::mergeLarge):
1696         (bmalloc::BoundaryTag::splitLarge):
1697         * bmalloc/SegregatedFreeList.cpp:
1698         (bmalloc::SegregatedFreeList::insert):
1699         (bmalloc::SegregatedFreeList::takeGreedy):
1700         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1701         the size when establishing a boundary tag, as required by the new
1702         interface.
1703
1704         * bmalloc/Sizes.h:
1705
1706 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1707
1708         Fixed a bmalloc crash seen on the EWS bot
1709         https://bugs.webkit.org/show_bug.cgi?id=135955
1710
1711         Reviewed by Andreas Kling.
1712
1713         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1714         input buffer is a malloc region, that region will get marked Copy-On-Write
1715         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1716         on older OS X's. In 10.10, they still fail, but they do not return
1717         EINVAL.
1718
1719         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1720
1721 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1722
1723         Fixed the bmalloc build
1724         https://bugs.webkit.org/show_bug.cgi?id=135953
1725
1726         Reviewed by Andreas Kling.
1727
1728         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1729         These headers are used, so they must be available outside the project.
1730
1731 2014-08-13  Daniel Bates  <dabates@apple.com>
1732
1733         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1734         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1735
1736         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1737         implementation of PerThread<T>::getFastCase().
1738
1739         * bmalloc/PerThread.h:
1740         (bmalloc::PerThread<T>::getFastCase):
1741
1742 2014-08-13  Daniel Bates  <dabates@apple.com>
1743
1744         Make bmalloc::PerThread work without C++ thread local storage
1745         https://bugs.webkit.org/show_bug.cgi?id=135895
1746
1747         Reviewed by Geoffrey Garen.
1748
1749         Implement support for building bmalloc without C++ thread local storage.
1750
1751         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1752         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1753         to determine whether the compiler supports C++ thread local storage.
1754         * bmalloc/PerThread.h:
1755         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1756         without C++ thread local storage.
1757         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1758         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1759         PerThreadStorage::initSharedKeyIfNeeded().
1760         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1761         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1762         ensure that the shared key has been initialized.
1763         (_pthread_setspecific_direct): Deleted.
1764         (_pthread_getspecific_direct): Deleted.
1765
1766 2014-08-13  Daniel Bates  <dabates@apple.com>
1767
1768         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1769         https://bugs.webkit.org/show_bug.cgi?id=135848
1770
1771         Reviewed by Geoffrey Garen.
1772
1773         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1774         when building for the iOS Simulator.
1775         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1776         using the public SDK.
1777         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1778         using the public SDK.
1779         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1780         using the public SDK.
1781
1782 2014-08-12  Daniel Bates  <dabates@apple.com>
1783
1784         BPLATFORM(IOS) always evaluates to false
1785         https://bugs.webkit.org/show_bug.cgi?id=135843
1786
1787         Reviewed by Geoffrey Garen.
1788
1789         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1790         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1791         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1792         building for iOS.
1793
1794         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1795         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1796
1797         * bmalloc/BPlatform.h:
1798
1799 2014-08-11  Andy Estes  <aestes@apple.com>
1800
1801         [iOS] Get rid of iOS.xcconfig
1802         https://bugs.webkit.org/show_bug.cgi?id=135809
1803
1804         Reviewed by Joseph Pecoraro.
1805
1806         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1807
1808         * Configurations/Base.xcconfig:
1809         * Configurations/iOS.xcconfig: Removed.
1810         * bmalloc.xcodeproj/project.pbxproj:
1811
1812 2014-05-01  Dan Bernstein  <mitz@apple.com>
1813
1814         Fixed production builds for the iOS Simulator.
1815         <rdar://problem/16792221>
1816
1817         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1818         PRIVATE_HEADERS_FOLDER_PATH when installing.
1819
1820 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1821
1822         bmalloc: Segregate pages by objects size
1823         https://bugs.webkit.org/show_bug.cgi?id=131909
1824
1825         Reviewed by Andreas Kling.
1826
1827         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1828
1829         * bmalloc/Allocator.cpp:
1830         (bmalloc::Allocator::allocateSlowCase):
1831         * bmalloc/Allocator.h:
1832         (bmalloc::Allocator::allocateFastCase):
1833         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1834         function for size class calculation.
1835
1836         * bmalloc/Deallocator.cpp:
1837         (bmalloc::Deallocator::Deallocator):
1838         (bmalloc::Deallocator::scavenge):
1839         (bmalloc::Deallocator::deallocateSmallLine):
1840         (bmalloc::Deallocator::allocateSmallLine):
1841         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1842         cache can't be shared anymore.
1843
1844         * bmalloc/Heap.cpp:
1845         (bmalloc::Heap::allocateSmallLineSlowCase):
1846         * bmalloc/Heap.h:
1847         (bmalloc::Heap::deallocateSmallLine): Ditto.
1848
1849         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1850         page refcount when allocating a line because we might have deallocated
1851         the page and the recycled it for another size class.
1852
1853         (bmalloc::Heap::deallocateMediumLine):
1854         (bmalloc::Heap::allocateMediumLine):
1855         * bmalloc/Line.h:
1856         (bmalloc::Line::refCount):
1857         * bmalloc/Page.h:
1858         (bmalloc::Page::refCount):
1859         (bmalloc::Page::smallSizeClass):
1860         (bmalloc::Page::setSmallSizeClass):
1861         (bmalloc::Page<Traits>::refCount): Deleted.
1862         * bmalloc/Sizes.h:
1863         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1864         an index into an array from a size.
1865
1866 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1867
1868         bmalloc: Improved alignment in LargeChunk
1869         https://bugs.webkit.org/show_bug.cgi?id=131895
1870
1871         Reviewed by Andreas Kling.
1872
1873         * bmalloc/Chunk.h:
1874         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1875         Technically, the previous alignment was harmless, but I would prefer,
1876         dear reader, not to have to explain the interlocking set of
1877         circumstances that made it so.
1878
1879 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1880
1881         Rolled out r167502 because it caused a crash on the facebook benchmark.
1882
1883         Unreviewed.
1884
1885             bmalloc: Added an XSmall line size
1886             https://bugs.webkit.org/show_bug.cgi?id=131851
1887
1888             Reviewed by Sam Weinig.
1889
1890 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1891
1892         bmalloc: Mutex should be harder to use wrong
1893         https://bugs.webkit.org/show_bug.cgi?id=131879
1894
1895         Reviewed by Andreas Kling.
1896
1897         Mutex now has a proper constructor, so you can't deadlock by forgetting
1898         to initialize it.
1899
1900         * bmalloc.xcodeproj/project.pbxproj:
1901         * bmalloc/Allocator.cpp:
1902         (bmalloc::Allocator::processXSmallAllocatorLog):
1903         (bmalloc::Allocator::processSmallAllocatorLog):
1904         (bmalloc::Allocator::processMediumAllocatorLog):
1905         (bmalloc::Allocator::allocateLarge):
1906         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1907         since the Heap mutex is a static.
1908
1909         * bmalloc/AsyncTask.h:
1910         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1911         need for explicit initialization anymore.
1912
1913         * bmalloc/Deallocator.cpp:
1914         (bmalloc::Deallocator::scavenge):
1915         (bmalloc::Deallocator::deallocateLarge):
1916         (bmalloc::Deallocator::deallocateXLarge):
1917         (bmalloc::Deallocator::processObjectLog):
1918         (bmalloc::Deallocator::deallocateSmallLine):
1919         (bmalloc::Deallocator::deallocateXSmallLine):
1920         (bmalloc::Deallocator::allocateSmallLine):
1921         (bmalloc::Deallocator::allocateXSmallLine):
1922         (bmalloc::Deallocator::deallocateMediumLine):
1923         (bmalloc::Deallocator::allocateMediumLine):
1924         * bmalloc/Deallocator.h:
1925         * bmalloc/Heap.cpp:
1926         (bmalloc::sleep):
1927         (bmalloc::Heap::Heap):
1928         (bmalloc::Heap::concurrentScavenge):
1929         (bmalloc::Heap::scavenge):
1930         (bmalloc::Heap::scavengeSmallPages):
1931         (bmalloc::Heap::scavengeXSmallPages):
1932         (bmalloc::Heap::scavengeMediumPages):
1933         (bmalloc::Heap::scavengeLargeRanges):
1934         (bmalloc::Heap::allocateXSmallLineSlowCase):
1935         (bmalloc::Heap::allocateSmallLineSlowCase):
1936         (bmalloc::Heap::allocateMediumLineSlowCase):
1937         (bmalloc::Heap::allocateXLarge):
1938         (bmalloc::Heap::deallocateXLarge):
1939         (bmalloc::Heap::allocateLarge):
1940         (bmalloc::Heap::deallocateLarge):
1941         * bmalloc/Heap.h:
1942         (bmalloc::Heap::deallocateXSmallLine):
1943         (bmalloc::Heap::allocateXSmallLine):
1944         (bmalloc::Heap::deallocateSmallLine):
1945         (bmalloc::Heap::allocateSmallLine):
1946         (bmalloc::Heap::deallocateMediumLine):
1947         (bmalloc::Heap::allocateMediumLine):
1948         * bmalloc/Line.h:
1949         (bmalloc::Line<Traits>::deref):
1950         * bmalloc/Mutex.cpp: Removed.
1951         * bmalloc/Mutex.h:
1952         (bmalloc::Mutex::Mutex):
1953         (bmalloc::Mutex::init): Deleted.
1954         (bmalloc::Mutex::try_lock): Deleted.
1955         (bmalloc::Mutex::lock): Deleted.
1956         (bmalloc::Mutex::unlock): Deleted.
1957         * bmalloc/Page.h:
1958         (bmalloc::Page<Traits>::ref):
1959         (bmalloc::Page<Traits>::deref):
1960         (bmalloc::Page<Traits>::refCount):
1961         * bmalloc/PerProcess.h:
1962         (bmalloc::PerProcess::mutex):
1963         (bmalloc::PerProcess<T>::getSlowCase):
1964         * bmalloc/StaticMutex.cpp: Added.
1965         (bmalloc::StaticMutex::lockSlowCase):
1966         * bmalloc/StaticMutex.h: Added.
1967         (bmalloc::StaticMutex::init):
1968         (bmalloc::StaticMutex::try_lock):
1969         (bmalloc::StaticMutex::lock):
1970         (bmalloc::StaticMutex::unlock):
1971         * bmalloc/VMHeap.h:
1972         (bmalloc::VMHeap::deallocateXSmallPage):
1973         (bmalloc::VMHeap::deallocateSmallPage):
1974         (bmalloc::VMHeap::deallocateMediumPage):
1975         (bmalloc::VMHeap::deallocateLargeRange):
1976         * bmalloc/bmalloc.h:
1977         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1978         since the Heap mutex is a static.
1979
1980 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1981
1982         bmalloc: AsyncTask should use Mutex instead of std::mutex
1983         https://bugs.webkit.org/show_bug.cgi?id=131865
1984
1985         Reviewed by Gavin Barraclough.
1986
1987         std::mutex is so slow that it makes parallelizing simple tasks through
1988         AsyncTask a net regression. Mutex fixes this.
1989
1990         * bmalloc/AsyncTask.h:
1991         (bmalloc::Function>::AsyncTask):
1992         (bmalloc::Function>::join):
1993         (bmalloc::Function>::runSlowCase):
1994         (bmalloc::Function>::entryPoint):
1995         * bmalloc/Mutex.h:
1996         (bmalloc::Mutex::init):
1997
1998 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1999
2000         bmalloc: Added an XSmall line size
2001         https://bugs.webkit.org/show_bug.cgi?id=131851
2002
2003         Reviewed by Sam Weinig.
2004
2005         Reduces malloc footprint on Membuster recordings by 10%.
2006
2007         This is a throughput regression, but we're still way ahead of TCMalloc.
2008         I have some ideas for how to recover the regression -- but I wanted to
2009         get this win in first.
2010
2011         Full set of benchmark results:
2012
2013                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2014                                                                                                 
2015                                                                nopatch                      patch                                Δ
2016                 Peak Memory:
2017                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
2018                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
2019                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
2020
2021                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
2022                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
2023                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
2024
2025                 Memory at End:
2026                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
2027                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
2028                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
2029
2030                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
2031                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
2032                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
2033
2034                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2035                                                                                                 
2036                                                    nopatch                patch                         Δ
2037                 Execution Time:
2038                     churn                            127ms                151ms            ! 1.19x slower
2039                     list_allocate                    130ms                164ms            ! 1.26x slower
2040                     tree_allocate                    109ms                127ms            ! 1.17x slower
2041                     tree_churn                       115ms                120ms            ! 1.04x slower
2042                     facebook                         240ms                259ms            ! 1.08x slower
2043                     fragment                          91ms                131ms            ! 1.44x slower
2044                     fragment_iterate                 105ms                106ms            ! 1.01x slower
2045                     message_one                      260ms                259ms             ^ 1.0x faster
2046                     message_many                     149ms                154ms            ! 1.03x slower
2047                     medium                           194ms                248ms            ! 1.28x slower
2048                     big                              157ms                160ms            ! 1.02x slower
2049
2050                     <geometric mean>                 144ms                163ms            ! 1.13x slower
2051                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
2052                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
2053
2054                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2055                                                                                                 
2056                                                                        nopatch                          patch                                     Δ
2057                 Execution Time:
2058                     churn                                                126ms                          148ms                        ! 1.17x slower
2059                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
2060                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
2061                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
2062                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
2063                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
2064                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
2065                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
2066                     facebook                                             240ms                          276ms                        ! 1.15x slower
2067                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
2068                     fragment                                              92ms                          130ms                        ! 1.41x slower
2069                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
2070                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
2071                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
2072                     message_one                                          260ms                          260ms                                      
2073                     message_many                                         170ms                          238ms                         ! 1.4x slower
2074                     medium                                               185ms                          250ms                        ! 1.35x slower
2075                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
2076                     big                                                  150ms                          169ms                        ! 1.13x slower
2077                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
2078
2079                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
2080                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
2081                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
2082
2083                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2084
2085                                                                     TC                      patch                                Δ
2086                 Peak Memory:
2087                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
2088                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
2089                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
2090
2091                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
2092                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
2093                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
2094
2095                 Memory at End:
2096                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
2097                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
2098                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
2099
2100                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
2101                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
2102                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
2103
2104                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2105                                                                                                 
2106                                                         TC                patch                         Δ
2107                 Execution Time:
2108                     churn                            416ms                148ms            ^ 2.81x faster
2109                     list_allocate                    463ms                164ms            ^ 2.82x faster
2110                     tree_allocate                    292ms                127ms             ^ 2.3x faster
2111                     tree_churn                       157ms                120ms            ^ 1.31x faster
2112                     facebook                         327ms                276ms            ^ 1.18x faster
2113                     fragment                         335ms                129ms             ^ 2.6x faster
2114                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
2115                     message_one                      386ms                258ms             ^ 1.5x faster
2116                     message_many                     410ms                154ms            ^ 2.66x faster
2117                     medium                           391ms                245ms             ^ 1.6x faster
2118                     big                              261ms                167ms            ^ 1.56x faster
2119
2120                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
2121                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
2122                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
2123
2124         * bmalloc.xcodeproj/project.pbxproj:
2125         * bmalloc/Allocator.cpp:
2126         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
2127         index corresponds with its size. Instead, use the size selection function
2128         explicitly. Now that we have XSmall, some small allocator entries are
2129         unused.
2130
2131         (bmalloc::Allocator::scavenge):
2132         (bmalloc::Allocator::log):
2133         (bmalloc::Allocator::processXSmallAllocatorLog):
2134         (bmalloc::Allocator::allocateSlowCase):
2135         * bmalloc/Allocator.h:
2136         (bmalloc::Allocator::xSmallAllocatorFor):
2137         (bmalloc::Allocator::allocateFastCase):
2138         * bmalloc/Chunk.h:
2139         * bmalloc/Deallocator.cpp:
2140         (bmalloc::Deallocator::scavenge):
2141         (bmalloc::Deallocator::processObjectLog):
2142         (bmalloc::Deallocator::deallocateSlowCase):
2143         (bmalloc::Deallocator::deallocateXSmallLine):
2144         (bmalloc::Deallocator::allocateXSmallLine):
2145         * bmalloc/Deallocator.h:
2146         (bmalloc::Deallocator::deallocateFastCase):
2147         * bmalloc/Heap.cpp:
2148         (bmalloc::Heap::scavenge):
2149         (bmalloc::Heap::scavengeXSmallPages):
2150         (bmalloc::Heap::allocateXSmallLineSlowCase):
2151         * bmalloc/Heap.h:
2152         (bmalloc::Heap::deallocateXSmallLine):
2153         (bmalloc::Heap::allocateXSmallLine):
2154         * bmalloc/LargeChunk.h:
2155         (bmalloc::LargeChunk::get):
2156         (bmalloc::LargeChunk::endTag):
2157         * bmalloc/Line.h:
2158         * bmalloc/MediumAllocator.h:
2159         (bmalloc::MediumAllocator::allocate):
2160         (bmalloc::MediumAllocator::refill):
2161         * bmalloc/ObjectType.cpp:
2162         (bmalloc::objectType):
2163         * bmalloc/ObjectType.h:
2164         (bmalloc::isXSmall):
2165         (bmalloc::isSmall):
2166         (bmalloc::isMedium):
2167         (bmalloc::isLarge):
2168         (bmalloc::isSmallOrMedium): Deleted.
2169         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
2170         handling small objects. There's probably a reasonable way to share this
2171         code in the future -- I'll look into that once it's stopped changing.
2172
2173         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
2174         is the main reason things got slower.
2175
2176         * bmalloc/SmallAllocator.h:
2177         (bmalloc::SmallAllocator::allocate):
2178         * bmalloc/SmallTraits.h:
2179         * bmalloc/VMHeap.cpp:
2180         (bmalloc::VMHeap::allocateXSmallChunk):
2181         * bmalloc/VMHeap.h:
2182         (bmalloc::VMHeap::allocateXSmallPage):
2183         (bmalloc::VMHeap::deallocateXSmallPage):
2184         * bmalloc/XSmallAllocator.h: Added.
2185         (bmalloc::XSmallAllocator::isNull):
2186         (bmalloc::XSmallAllocator::canAllocate):
2187         (bmalloc::XSmallAllocator::XSmallAllocator):
2188         (bmalloc::XSmallAllocator::line):
2189         (bmalloc::XSmallAllocator::allocate):
2190         (bmalloc::XSmallAllocator::objectCount):
2191         (bmalloc::XSmallAllocator::derefCount):
2192         (bmalloc::XSmallAllocator::refill):
2193         * bmalloc/XSmallChunk.h: Added.
2194         * bmalloc/XSmallLine.h: Added.
2195         * bmalloc/XSmallPage.h: Added.
2196         * bmalloc/XSmallTraits.h: Added.
2197         * bmalloc/bmalloc.h:
2198         (bmalloc::api::realloc): Boiler-plate copy, as above.
2199
2200 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2201
2202         MallocBench should scavenge explicitly instead of waiting
2203         https://bugs.webkit.org/show_bug.cgi?id=131661
2204
2205         Reviewed by Andreas Kling.
2206
2207         Added explicit scavenge support to bmalloc. This isn't a memory win,
2208         since bmalloc's per-thread cache is so small. But it makes testing
2209         simpler.
2210
2211         * bmalloc/Allocator.cpp:
2212         (bmalloc::Allocator::~Allocator):
2213         (bmalloc::Allocator::scavenge):
2214         * bmalloc/Allocator.h:
2215         * bmalloc/Cache.cpp:
2216         (bmalloc::Cache::operator new):
2217         (bmalloc::Cache::operator delete):
2218         (bmalloc::Cache::Cache):
2219         (bmalloc::Cache::scavenge):
2220         * bmalloc/Cache.h:
2221         * bmalloc/Deallocator.cpp:
2222         (bmalloc::Deallocator::~Deallocator):
2223         (bmalloc::Deallocator::scavenge):
2224         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
2225         functions, for reuse.
2226
2227         * bmalloc/Heap.cpp:
2228         (bmalloc::sleep):
2229         (bmalloc::Heap::concurrentScavenge):
2230         (bmalloc::Heap::scavenge):
2231         (bmalloc::Heap::scavengeSmallPages):
2232         (bmalloc::Heap::scavengeMediumPages):
2233         (bmalloc::Heap::scavengeLargeRanges):
2234         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
2235         scavenging -- in response to a benchmark or a low memory warning --
2236         wants to complete as soon as possible, so its sleep duration is 0.
2237
2238         * bmalloc/bmalloc.h:
2239         (bmalloc::api::scavenge):
2240         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
2241
2242 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2243
2244         Use 4kB pages on Mac
2245         https://bugs.webkit.org/show_bug.cgi?id=131658
2246
2247         Reviewed by Sam Weinig.
2248
2249         This reduces memory use a lot on Membuster:
2250
2251                                                                   base                      patch                                Δ
2252                 Execution Time:
2253                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
2254                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
2255                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
2256
2257                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
2258                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
2259                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
2260
2261                 Peak Memory:
2262                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
2263                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
2264                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
2265
2266                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
2267                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
2268                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
2269
2270                 Memory at End:
2271                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
2272                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
2273                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
2274
2275                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
2276                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
2277                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
2278
2279
2280         * bmalloc.xcodeproj/project.pbxproj:
2281         * bmalloc/BPlatform.h: Added.
2282         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
2283         is 4kB on Mac.
2284
2285 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
2286
2287         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
2288
2289         * bmalloc.xcodeproj: Modified property svn:ignore.
2290
2291 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
2292
2293         Fixed some mbmalloc exports
2294         https://bugs.webkit.org/show_bug.cgi?id=131599
2295
2296         Reviewed by Ryosuke Niwa.
2297
2298         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
2299         of the project, so we can call them from API.
2300
2301         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
2302         visibility, so they show up as exported in the .dylib.
2303
2304 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
2305
2306         Put bmalloc headers in the right place
2307         https://bugs.webkit.org/show_bug.cgi?id=131464
2308
2309         Reviewed by Mark Rowe.
2310
2311         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
2312         specify that we don't just want to dump all of our generically-named
2313         headers into /usr/local/include.
2314
2315 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2316
2317         Made bmalloc more #include friendly
2318         https://bugs.webkit.org/show_bug.cgi?id=131386
2319
2320         Reviewed by Andreas Kling.
2321
2322         Marked a bunch of headers private so they can be used from client code
2323         that #includes bmalloc.h.
2324
2325         Renamed ASSERT macros to BASSERT. This matches their header, which already
2326         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
2327
2328         * bmalloc.xcodeproj/project.pbxproj:
2329         * bmalloc/Allocator.cpp:
2330         (bmalloc::Allocator::allocateSlowCase):
2331         * bmalloc/AsyncTask.h:
2332         (bmalloc::Function>::runSlowCase):
2333         * bmalloc/BAssert.h:
2334         * bmalloc/BoundaryTag.h:
2335         (bmalloc::BoundaryTag::setSize):
2336         * bmalloc/BoundaryTagInlines.h:
2337         (bmalloc::validate):
2338         (bmalloc::BoundaryTag::init):
2339         (bmalloc::BoundaryTag::deallocate):
2340         (bmalloc::BoundaryTag::splitLarge):
2341         (bmalloc::BoundaryTag::allocate):
2342         * bmalloc/Chunk.h:
2343         * bmalloc/Deallocator.cpp:
2344         (bmalloc::Deallocator::processObjectLog):
2345         (bmalloc::Deallocator::deallocateSlowCase):
2346         * bmalloc/Deallocator.h:
2347         (bmalloc::Deallocator::deallocateFastCase):
2348         * bmalloc/FixedVector.h:
2349         (bmalloc::Capacity>::operator):
2350         (bmalloc::Capacity>::push):
2351         (bmalloc::Capacity>::pop):
2352         (bmalloc::Capacity>::shrink):
2353         * bmalloc/Heap.cpp:
2354         (bmalloc::Heap::allocateLarge):
2355         * bmalloc/LargeChunk.h:
2356         (bmalloc::LargeChunk::get):
2357         (bmalloc::LargeChunk::endTag):
2358         * bmalloc/Line.h:
2359         (bmalloc::Line<Traits>::concurrentRef):
2360         (bmalloc::Line<Traits>::deref):
2361         * bmalloc/MediumAllocator.h:
2362         (bmalloc::MediumAllocator::allocate):
2363         * bmalloc/ObjectType.h:
2364         (bmalloc::isSmall):
2365         * bmalloc/Page.h:
2366         (bmalloc::Page<Traits>::ref):
2367         (bmalloc::Page<Traits>::deref):
2368         * bmalloc/PerThread.h:
2369         (bmalloc::PerThread<T>::getSlowCase):
2370         * bmalloc/SegregatedFreeList.cpp:
2371         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2372         (bmalloc::SegregatedFreeList::insert):
2373         * bmalloc/SmallAllocator.h:
2374         (bmalloc::SmallAllocator::allocate):
2375         (bmalloc::SmallAllocator::refill):
2376         * bmalloc/Syscall.h:
2377         * bmalloc/VMAllocate.h:
2378         (bmalloc::vmValidate):
2379         (bmalloc::vmAllocate):
2380         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2381         * bmalloc/Vector.h:
2382         (bmalloc::Vector<T>::operator):
2383         (bmalloc::Vector<T>::pop):
2384         (bmalloc::Vector<T>::shrink):
2385         * bmalloc/XLargeChunk.h:
2386         (bmalloc::XLargeChunk::range):
2387         (bmalloc::XLargeChunk::size):
2388
2389 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2390
2391         Removed an unused file.
2392
2393         Unreviewed.
2394
2395         * bmalloc/AsyncTask.cpp: Removed.
2396
2397 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2398
2399         Build bmalloc on Mac
2400         https://bugs.webkit.org/show_bug.cgi?id=131333
2401
2402         Reviewed by Mark Rowe.
2403
2404         * Makefile: Added. For make clients.
2405
2406         These files are required for building any project in WebKit. I copied
2407         them from WTF:
2408         * Configurations: Added.
2409         * Configurations/Base.xcconfig: Added.
2410         * Configurations/DebugRelease.xcconfig: Added.
2411         * Configurations/bmalloc.xcconfig: Added.
2412         * Configurations/iOS.xcconfig: Added.
2413         * Configurations/mbmalloc.xcconfig: Added.
2414
2415         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
2416         from here because everything is in .xcconfig files now.
2417
2418         I had to fix a bunch of minor warnings, since they're enabled in our
2419         .xcconfig files:
2420
2421         * bmalloc/AsyncTask.h:
2422         (bmalloc::Function>::AsyncTask):
2423         * bmalloc/BAssert.h:
2424         * bmalloc/BoundaryTagInlines.h:
2425         (bmalloc::validate):
2426         * bmalloc/Heap.cpp:
2427         (bmalloc::Heap::Heap):
2428         (bmalloc::Heap::allocateLarge):
2429         (bmalloc::Heap::deallocateLarge):
2430         * bmalloc/Mutex.h:
2431         (bmalloc::Mutex::Mutex): Deleted.
2432         * bmalloc/VMAllocate.h:
2433         (bmalloc::vmValidate):
2434         * bmalloc/mbmalloc.cpp:
2435
2436 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2437
2438         bmalloc: Fixed a leak in the per-thread cache
2439         https://bugs.webkit.org/show_bug.cgi?id=131330
2440
2441         Reviewed by Andreas Kling.
2442
2443         Remember to deallocate our line caches upon thread exit.
2444
2445         * bmalloc/Deallocator.cpp:
2446         (bmalloc::Deallocator::~Deallocator):
2447
2448 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2449
2450         bmalloc: rolled out the tryLock experiment
2451         https://bugs.webkit.org/show_bug.cgi?id=131328
2452
2453         Reviewed by Andreas Kling.
2454
2455         It wasn't a speedup.
2456
2457         * bmalloc.xcodeproj/project.pbxproj:
2458         * bmalloc/Allocator.cpp:
2459         (bmalloc::Allocator::processSmallAllocatorLog):
2460         (bmalloc::Allocator::processMediumAllocatorLog):
2461         * bmalloc/Deallocator.cpp:
2462         (bmalloc::Deallocator::processObjectLog):
2463         (bmalloc::Deallocator::deallocateSlowCase):
2464         (bmalloc::Deallocator::deallocateSmallLine):
2465         (bmalloc::Deallocator::deallocateMediumLine):
2466         * bmalloc/Deallocator.h:
2467         (bmalloc::Deallocator::deallocateFastCase):
2468         * bmalloc/Heap.h:
2469         (bmalloc::Heap::deallocateSmallLine):
2470         (bmalloc::Heap::deallocateMediumLine):
2471         * bmalloc/Line.h:
2472         (bmalloc::Line<Traits>::deref):
2473         * bmalloc/Page.h:
2474         (bmalloc::Page<Traits>::deref):
2475
2476 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2477
2478         bmalloc
2479         https://bugs.webkit.org/show_bug.cgi?id=131170
2480
2481         Reviewed by Andreas Kling.
2482
2483         Initial commit.
2484
2485         * bmalloc: Added.
2486         * bmalloc.xcodeproj: Added.
2487         * bmalloc.xcodeproj/project.pbxproj: Added.
2488         * bmalloc/Algorithm.h: Added.
2489         (bmalloc::max):
2490         (bmalloc::min):
2491         (bmalloc::mask):
2492         (bmalloc::test):
2493         (bmalloc::roundUpToMultipleOf):
2494         (bmalloc::roundDownToMultipleOf):
2495         (bmalloc::sizeOf):
2496         (bmalloc::bitCount):
2497         (bmalloc::isPowerOfTwo):
2498         * bmalloc/Allocator.cpp: Added.
2499         (bmalloc::Allocator::Allocator):
2500         (bmalloc::Allocator::~Allocator):
2501         (bmalloc::Allocator::log):
2502         (bmalloc::Allocator::processSmallAllocatorLog):
2503         (bmalloc::Allocator::processMediumAllocatorLog):
2504         (bmalloc::Allocator::allocateLarge):
2505         (bmalloc::Allocator::allocateXLarge):
2506         (bmalloc::Allocator::allocateMedium):
2507         (bmalloc::Allocator::allocateSlowCase):
2508         * bmalloc/Allocator.h: Added.
2509         (bmalloc::Allocator::smallAllocatorFor):
2510         (bmalloc::Allocator::allocateFastCase):
2511         (bmalloc::Allocator::allocate):
2512         * bmalloc/AsyncTask.cpp: Added.
2513         (bmalloc::AsyncTask<Function>::runSlowCase):
2514         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2515         (bmalloc::AsyncTask<Function>::entryPoint):
2516         * bmalloc/AsyncTask.h: Added.
2517         (bmalloc::Function>::AsyncTask):
2518         (bmalloc::Function>::join):
2519         (bmalloc::Function>::run):
2520         (bmalloc::Function>::runSlowCase):
2521         (bmalloc::Function>::pthreadEntryPoint):
2522         (bmalloc::Function>::entryPoint):
2523         * bmalloc/BAssert.h: Added.
2524         * bmalloc/BeginTag.h: Added.
2525         (bmalloc::BeginTag::isInFreeList):
2526         * bmalloc/BoundaryTag.h: Added.
2527         (bmalloc::BoundaryTag::isXLarge):
2528         (bmalloc::BoundaryTag::setXLarge):
2529         (bmalloc::BoundaryTag::isFree):
2530         (bmalloc::BoundaryTag::setFree):
2531         (bmalloc::BoundaryTag::isEnd):
2532         (bmalloc::BoundaryTag::setEnd):
2533         (bmalloc::BoundaryTag::hasPhysicalPages):
2534         (bmalloc::BoundaryTag::setHasPhysicalPages):
2535         (bmalloc::BoundaryTag::isNull):
2536         (bmalloc::BoundaryTag::clear):
2537         (bmalloc::BoundaryTag::size):
2538         (bmalloc::BoundaryTag::setSize):
2539         (bmalloc::BoundaryTag::prev):
2540         (bmalloc::BoundaryTag::next):
2541         * bmalloc/BoundaryTagInlines.h: Added.
2542         (bmalloc::validate):
2543         (bmalloc::validatePrev):
2544         (bmalloc::validateNext):
2545         (bmalloc::BoundaryTag::init):
2546         (bmalloc::BoundaryTag::mergeLargeLeft):
2547         (bmalloc::BoundaryTag::mergeLargeRight):
2548         (bmalloc::BoundaryTag::mergeLarge):
2549         (bmalloc::BoundaryTag::deallocate):
2550         (bmalloc::BoundaryTag::splitLarge):
2551         (bmalloc::BoundaryTag::allocate):
2552         * bmalloc/Cache.cpp: Added.
2553         (bmalloc::Cache::operator new):
2554         (bmalloc::Cache::operator delete):
2555         (bmalloc::Cache::Cache):
2556         (bmalloc::Cache::allocateSlowCase):
2557         (bmalloc::Cache::allocateSlowCaseNullCache):
2558         (bmalloc::Cache::deallocateSlowCase):
2559         (bmalloc::Cache::deallocateSlowCaseNullCache):
2560         * bmalloc/Cache.h: Added.
2561         (bmalloc::Cache::allocator):
2562         (bmalloc::Cache::deallocator):
2563         (bmalloc::Cache::allocateFastCase):
2564         (bmalloc::Cache::deallocateFastCase):
2565         (bmalloc::Cache::allocate):
2566         (bmalloc::Cache::deallocate):
2567         * bmalloc/Chunk.h: Added.
2568         (bmalloc::Chunk::begin):
2569         (bmalloc::Chunk::end):
2570         (bmalloc::Chunk::lines):
2571         (bmalloc::Chunk::pages):
2572         * bmalloc/Deallocator.cpp: Added.
2573         (bmalloc::Deallocator::Deallocator):
2574         (bmalloc::Deallocator::~Deallocator):
2575         (bmalloc::Deallocator::deallocateLarge):
2576         (bmalloc::Deallocator::deallocateXLarge):
2577         (bmalloc::Deallocator::processObjectLog):
2578         (bmalloc::Deallocator::deallocateSlowCase):
2579         (bmalloc::Deallocator::deallocateSmallLine):
2580         (bmalloc::Deallocator::allocateSmallLine):
2581         (bmalloc::Deallocator::deallocateMediumLine):
2582         (bmalloc::Deallocator::allocateMediumLine):
2583         * bmalloc/Deallocator.h: Added.
2584         (bmalloc::Deallocator::deallocateFastCase):
2585         (bmalloc::Deallocator::deallocate):
2586         * bmalloc/EndTag.h: Added.
2587         (bmalloc::EndTag::operator=):
2588         * bmalloc/FixedVector.h: Added.
2589         (bmalloc::FixedVector::begin):
2590         (bmalloc::FixedVector::end):
2591         (bmalloc::FixedVector::size):
2592         (bmalloc::FixedVector::capacity):
2593         (bmalloc::FixedVector::clear):
2594         (bmalloc::FixedVector::isEmpty):
2595         (bmalloc::Capacity>::FixedVector):
2596         (bmalloc::Capacity>::operator):
2597         (bmalloc::Capacity>::push):
2598         (bmalloc::Capacity>::pop):
2599         (bmalloc::Capacity>::shrink):
2600         * bmalloc/Heap.cpp: Added.
2601         (bmalloc::sleep):
2602         (bmalloc::Heap::Heap):
2603         (bmalloc::Heap::concurrentScavenge):
2604         (bmalloc::Heap::scavengeSmallPages):
2605         (bmalloc::Heap::scavengeMediumPages):
2606         (bmalloc::Heap::scavengeLargeRanges):
2607         (bmalloc::Heap::allocateSmallLineSlowCase):
2608         (bmalloc::Heap::allocateMediumLineSlowCase):
2609         (bmalloc::Heap::allocateXLarge):
2610         (bmalloc::Heap::deallocateXLarge):
2611         (bmalloc::Heap::allocateLarge):
2612         (bmalloc::Heap::deallocateLarge):
2613         * bmalloc/Heap.h: Added.
2614         (bmalloc::Heap::deallocateSmallLine):
2615         (bmalloc::Heap::allocateSmallLine):
2616         (bmalloc::Heap::deallocateMediumLine):
2617         (bmalloc::Heap::allocateMediumLine):
2618         * bmalloc/Inline.h: Added.
2619         * bmalloc/LargeChunk.h: Added.
2620         (bmalloc::LargeChunk::begin):
2621         (bmalloc::LargeChunk::end):
2622         (bmalloc::LargeChunk::create):
2623         (bmalloc::LargeChunk::get):
2624         (bmalloc::LargeChunk::beginTag):
2625         (bmalloc::LargeChunk::endTag):
2626         * bmalloc/Line.h: Added.
2627         (bmalloc::Line<Traits>::begin):
2628         (bmalloc::Line<Traits>::end):
2629         (bmalloc::Line<Traits>::concurrentRef):
2630         (bmalloc::Line<Traits>::deref):
2631         * bmalloc/MediumAllocator.h: Added.
2632         (bmalloc::MediumAllocator::isNull):
2633         (bmalloc::MediumAllocator::MediumAllocator):
2634         (bmalloc::MediumAllocator::line):
2635         (bmalloc::MediumAllocator::allocate):
2636         (bmalloc::MediumAllocator::derefCount):
2637         (bmalloc::MediumAllocator::refill):
2638         * bmalloc/MediumChunk.h: Added.
2639         * bmalloc/MediumLine.h: Added.
2640         * bmalloc/MediumPage.h: Added.
2641         * bmalloc/MediumTraits.h: Added.
2642         * bmalloc/Mutex.cpp: Added.
2643         (bmalloc::Mutex::lockSlowCase):
2644         * bmalloc/Mutex.h: Added.
2645         (bmalloc::Mutex::Mutex):
2646         (bmalloc::Mutex::try_lock):
2647         (bmalloc::Mutex::lock):
2648         (bmalloc::Mutex::unlock):
2649         * bmalloc/ObjectType.cpp: Added.
2650         (bmalloc::objectType):
2651         * bmalloc/ObjectType.h: Added.
2652         (bmalloc::isSmallOrMedium):
2653         (bmalloc::isSmall):
2654         * bmalloc/Page.h: Added.
2655         (bmalloc::Page<Traits>::ref):
2656         (bmalloc::Page<Traits>::deref):
2657         (bmalloc::Page<Traits>::refCount):
2658         * bmalloc/PerProcess.h: Added.
2659         (bmalloc::PerProcess::mutex):
2660         (bmalloc::PerProcess<T>::getFastCase):
2661         (bmalloc::PerProcess<T>::get):
2662         (bmalloc::PerProcess<T>::getSlowCase):
2663         * bmalloc/PerThread.h: Added.
2664         (bmalloc::PerThreadStorage<Cache>::get):
2665         (bmalloc::PerThreadStorage<Cache>::init):
2666         (bmalloc::PerThreadStorage::get):
2667         (bmalloc::PerThreadStorage::init):
2668         (bmalloc::PerThread<T>::getFastCase):
2669         (bmalloc::PerThread<T>::get):
2670         (bmalloc::PerThread<T>::destructor):
2671         (bmalloc::PerThread<T>::getSlowCase):
2672         * bmalloc/Range.h: Added.
2673         (bmalloc::Range::Range):
2674         (bmalloc::Range::begin):
2675         (bmalloc::Range::end):
2676         (bmalloc::Range::size):
2677         (bmalloc::Range::operator!):
2678         (bmalloc::Range::operator<):
2679         * bmalloc/SegregatedFreeList.cpp: Added.
2680         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2681         (bmalloc::SegregatedFreeList::insert):
2682         (bmalloc::SegregatedFreeList::takeGreedy):
2683         (bmalloc::SegregatedFreeList::take):
2684         * bmalloc/SegregatedFreeList.h: Added.
2685         * bmalloc/Sizes.h: Added.
2686         * bmalloc/SmallAllocator.h: Added.
2687         (bmalloc::SmallAllocator::isNull):
2688         (bmalloc::SmallAllocator::canAllocate):
2689         (bmalloc::SmallAllocator::SmallAllocator):
2690         (bmalloc::SmallAllocator::line):
2691         (bmalloc::SmallAllocator::allocate):
2692         (bmalloc::SmallAllocator::objectCount):
2693         (bmalloc::SmallAllocator::derefCount):
2694         (bmalloc::SmallAllocator::refill):
2695         * bmalloc/SmallChunk.h: Added.
2696         * bmalloc/SmallLine.h: Added.
2697         * bmalloc/SmallPage.h: Added.
2698         * bmalloc/SmallTraits.h: Added.
2699         * bmalloc/Syscall.h: Added.
2700         * bmalloc/VMAllocate.h: Added.
2701         (bmalloc::vmSize):
2702         (bmalloc::vmValidate):
2703         (bmalloc::vmAllocate):
2704         (bmalloc::vmDeallocate):
2705         (bmalloc::vmDeallocatePhysicalPages):
2706         (bmalloc::vmAllocatePhysicalPages):
2707         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2708         (bmalloc::vmAllocatePhysicalPagesSloppy):
2709         * bmalloc/VMHeap.cpp: Added.
2710         (bmalloc::VMHeap::VMHeap):
2711         (bmalloc::VMHeap::allocateSmallChunk):
2712         (bmalloc::VMHeap::allocateMediumChunk):
2713         (bmalloc::VMHeap::allocateLargeChunk):
2714         * bmalloc/VMHeap.h: Added.
2715         (bmalloc::VMHeap::allocateSmallPage):
2716         (bmalloc::VMHeap::allocateMediumPage):
2717         (bmalloc::VMHeap::allocateLargeRange):
2718         (bmalloc::VMHeap::deallocateSmallPage):
2719         (bmalloc::VMHeap::deallocateMediumPage):
2720         (bmalloc::VMHeap::deallocateLargeRange):
2721         * bmalloc/Vector.h: Added.
2722         (bmalloc::Vector::begin):
2723         (bmalloc::Vector::end):
2724         (bmalloc::Vector::size):
2725         (bmalloc::Vector::capacity):
2726         (bmalloc::Vector::last):
2727         (bmalloc::Vector::pop):
2728         (bmalloc::Vector<T>::Vector):
2729         (bmalloc::Vector<T>::~Vector):
2730         (bmalloc::Vector<T>::operator):
2731         (bmalloc::Vector<T>::push):
2732         (bmalloc::Vector<T>::pop):
2733         (bmalloc::Vector<T>::shrink):
2734         (bmalloc::Vector<T>::reallocateBuffer):
2735         (bmalloc::Vector<T>::shrinkCapacity):
2736         (bmalloc::Vector<T>::growCapacity):
2737         * bmalloc/XLargeChunk.h: Added.
2738         (bmalloc::XLargeChunk::get):
2739         (bmalloc::XLargeChunk::begin):
2740         (bmalloc::XLargeChunk::XLargeChunk):
2741         (bmalloc::XLargeChunk::create):
2742         (bmalloc::XLargeChunk::destroy):
2743         (bmalloc::XLargeChunk::range):
2744         (bmalloc::XLargeChunk::size):
2745         * bmalloc/bmalloc.h: Added.
2746         (bmalloc::api::malloc):
2747         (bmalloc::api::free):
2748         (bmalloc::api::realloc):
2749         * bmalloc/mbmalloc.cpp: Added.
2750