fromGlobalCode has an unused Debugger* argument
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc: Don't branch when setting the owner of a large object
4         https://bugs.webkit.org/show_bug.cgi?id=142241
5
6         Reviewed by Andreas Kling.
7
8         * bmalloc/BoundaryTag.h:
9         (bmalloc::BoundaryTag::owner):
10         (bmalloc::BoundaryTag::setOwner):
11
12 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
13
14         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
15         https://bugs.webkit.org/show_bug.cgi?id=141802
16
17         Reviewed by Andreas Kling.
18
19         Re-enabled this feature on iOS, now that the iOS crash should be fixed.
20
21         * bmalloc/VMHeap.cpp:
22         (bmalloc::VMHeap::grow):
23         * bmalloc/VMHeap.h:
24
25 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
26
27         bmalloc: Added missing features to the malloc zone introspection API
28         https://bugs.webkit.org/show_bug.cgi?id=142235
29
30         Reviewed by Andreas Kling.
31
32         This should fix the crash we saw on the iOS PLT bot
33         (c.f. http://trac.webkit.org/changeset/180604).
34
35         * bmalloc/Zone.cpp:
36         (bmalloc::good_size):
37         (bmalloc::check):
38         (bmalloc::print):
39         (bmalloc::log):
40         (bmalloc::force_lock):
41         (bmalloc::force_unlock):
42         (bmalloc::statistics):
43         (bmalloc::size):
44         (bmalloc::enumerator): Provide all of these functions since they are called
45         indiscriminately on all zones.
46
47         (bmalloc::Zone::Zone):
48         (bmalloc::Zone::size): Deleted.
49         (bmalloc::Zone::enumerator): Deleted. Moved these functions out of the
50         Zone class since they can stand alone.
51
52         * bmalloc/Zone.h:
53
54 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
55
56         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
57         https://bugs.webkit.org/show_bug.cgi?id=141802
58
59         Reviewed by Andreas Kling.
60
61         Rolling back in but disabled on iOS until I can debug why the iOS PLT crashes.
62
63         * bmalloc/VMHeap.cpp:
64         (bmalloc::VMHeap::grow):
65         * bmalloc/VMHeap.h:
66         * bmalloc/Zone.cpp:
67         (bmalloc::Zone::size):
68         (bmalloc::Zone::Zone):
69         * bmalloc/Zone.h:
70
71 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
72
73         bmalloc: Miscellaneous cleanup
74         https://bugs.webkit.org/show_bug.cgi?id=142231
75
76         Reviewed by Andreas Kling.
77
78         No performance change -- maybe a tiny reduction in memory use.
79
80         * bmalloc/Heap.cpp: Moved the sleep function into StaticMutex, since
81         it's a helper for working with mutexes.
82
83         (bmalloc::Heap::scavenge): Make sure to wait before we start any
84         scavenging, since individual scavenging functions now always scavenge
85         at least one page before waiting themselves.
86
87         (bmalloc::Heap::scavengeSmallPages):
88         (bmalloc::Heap::scavengeMediumPages):
89         (bmalloc::Heap::scavengeLargeObjects): Use the new wait helper to
90         simplify this code. Also, we now require our caller to wait until at
91         least one deallocation is desirable. This simplifies our loop.
92
93         (bmalloc::Heap::allocateSmallPage):
94         (bmalloc::Heap::allocateMediumPage):
95         (bmalloc::Heap::allocateXLarge):
96         (bmalloc::Heap::allocateLarge): Don't freak out any time the heap does
97         an allocation. Only consider the heap to be growing if it actually needs
98         to allocate new VM. This allows us to shrink the heap back down from a
99         high water mark more reliably even if heap activity continues.
100
101         (bmalloc::sleep): Deleted.
102         (bmalloc::Heap::scavengeLargeRanges): Renamed to match our use of
103         "LargeObject".
104
105         * bmalloc/Heap.h:
106
107         * bmalloc/LargeObject.h:
108         (bmalloc::LargeObject::operator bool): Added to simplify a while loop.
109
110         * bmalloc/StaticMutex.h:
111         (bmalloc::sleep):
112         (bmalloc::waitUntilFalse): New helper for waiting until a condition
113         becomes reliably false.
114
115         * bmalloc/Vector.h:
116         (bmalloc::Vector<T>::~Vector): Oops! Don't deallocate the null pointer.
117         We don't actually run any Vector destructors, but an iteration of this
118         patch did, and then crashed. So, let's fix that.
119
120 2015-03-02  Geoffrey Garen  <ggaren@apple.com>
121
122         bmalloc: Eagerly remove allocated objects from the free list
123         https://bugs.webkit.org/show_bug.cgi?id=142194
124
125         Reviewed by Andreas Kling.
126
127         This reduces the pressure to garbage collect the free list.
128
129         Might be a 1% speedup on MallocBench.
130
131         * bmalloc/FreeList.cpp: Put this comment at the top of the file instead
132         of repeating it inside of each function. Tried to clarify the details.
133
134         (bmalloc::FreeList::takeGreedy): Matched the other iteration code in this
135         file for consistency -- even though either direction works fine in this
136         function.
137
138         (bmalloc::FreeList::take): Change to iterate from low to high so that we
139         can maintain an index into the vector that is not disturbed even if we
140         pop from the middle (which invalidates the last index in the vector).
141
142         Decrement i when popping from the middle to make sure that we don't
143         skip the next item after popping.
144
145         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): Ditto.
146
147 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
148
149         Fixed a typo in the previous commit.
150
151         * bmalloc/BoundaryTag.h:
152         (bmalloc::BoundaryTag::setOwner):
153
154 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
155
156         EFL build fix after r180797.
157
158         * bmalloc/BoundaryTag.h:
159         (bmalloc::BoundaryTag::owner):
160         (bmalloc::BoundaryTag::setOwner):
161
162 2015-02-27  Geoffrey Garen  <ggaren@apple.com>
163
164         bmalloc: Pathological madvise churn on the free(malloc(x)) benchmark
165         https://bugs.webkit.org/show_bug.cgi?id=142058
166
167         Reviewed by Andreas Kling.
168
169         The churn was caused by repeatedly splitting an object with physical
170         pages from an object without, and then merging them back together again.
171         The merge would conservatively forget that we had physical pages, forcing
172         a new call to madvise on the next allocation.
173
174         This patch more strictly segregates objects in the heap from objects in
175         the VM heap, with these changes:
176
177         (1) Objects in the heap are not allowed to merge with objects in the VM
178         heap, and vice versa -- since that would erase our precise knowledge of
179         which physical pages had been allocated.
180
181         (2) The VM heap is exclusively responsible for allocating and deallocating
182         physical pages.
183
184         (3) The heap free list must consider entries for objects that are in the
185         VM heap to be invalid, and vice versa. (This condition can arise
186         because the free list does not eagerly remove items.)
187
188         With these changes, we can know that any valid object in the heap's free
189         list already has physical pages, and does not need to call madvise.
190
191         Note that the VM heap -- as before -- might sometimes contain ranges
192         or pieces of ranges that have physical pages, since we allow splitting
193         of ranges at granularities smaller than the VM page size. These ranges
194         can eventually merge with ranges in the heap during scavenging.
195
196         * bmalloc.xcodeproj/project.pbxproj:
197
198         * bmalloc/BoundaryTag.h:
199         (bmalloc::BoundaryTag::owner):
200         (bmalloc::BoundaryTag::setOwner):
201         (bmalloc::BoundaryTag::initSentinel):
202         (bmalloc::BoundaryTag::hasPhysicalPages): Deleted.
203         (bmalloc::BoundaryTag::setHasPhysicalPages): Deleted. Replaced the concept
204         of "has physical pages" with a bit indicating which heap owns the large
205         object. This is a more precise concept, since the old bit was really a
206         Yes / Maybe bit.
207
208         * bmalloc/Deallocator.cpp:
209
210         * bmalloc/FreeList.cpp: Adopt
211         (bmalloc::FreeList::takeGreedy):
212         (bmalloc::FreeList::take):
213         (bmalloc::FreeList::removeInvalidAndDuplicateEntries):
214         * bmalloc/FreeList.h:
215         (bmalloc::FreeList::push): Added API for considering the owner when
216         deciding if a free list entry is valid.
217
218         * bmalloc/Heap.cpp:
219         (bmalloc::Heap::Heap): Adopt new API.
220
221         (bmalloc::Heap::scavengeLargeRanges): Scavenge all ranges with no minimum,
222         since some ranges might be able to merge with ranges in the VM heap, and
223         they won't be allowed to until we scavenge them.
224
225         (bmalloc::Heap::allocateSmallPage):
226         (bmalloc::Heap::allocateMediumPage):
227         (bmalloc::Heap::allocateLarge): New VM heap API makes this function
228         simpler, since we always get back physical pages now.
229
230         * bmalloc/Heap.h:
231         * bmalloc/LargeObject.h:
232         (bmalloc::LargeObject::end):
233         (bmalloc::LargeObject::owner):
234         (bmalloc::LargeObject::setOwner):
235         (bmalloc::LargeObject::isValidAndFree):
236         (bmalloc::LargeObject::merge): Do not merge objects across heaps since
237         that causes madvise churn.
238         (bmalloc::LargeObject::validateSelf):
239         (bmalloc::LargeObject::init):
240         (bmalloc::LargeObject::hasPhysicalPages): Deleted.
241         (bmalloc::LargeObject::setHasPhysicalPages): Deleted. Propogate the Owner API.
242
243         * bmalloc/Owner.h: Added.
244
245         * bmalloc/SegregatedFreeList.cpp:
246         (bmalloc::SegregatedFreeList::SegregatedFreeList):
247         (bmalloc::SegregatedFreeList::insert):
248         (bmalloc::SegregatedFreeList::takeGreedy):
249         (bmalloc::SegregatedFreeList::take):
250         * bmalloc/SegregatedFreeList.h: Propogate the owner API.
251
252         * bmalloc/VMAllocate.h:
253         (bmalloc::vmDeallocatePhysicalPagesSloppy):
254         (bmalloc::vmAllocatePhysicalPagesSloppy): Clarified these functions and
255         removed an edge case.
256
257         * bmalloc/VMHeap.cpp:
258         (bmalloc::VMHeap::VMHeap):
259         * bmalloc/VMHeap.h:
260         (bmalloc::VMHeap::allocateSmallPage):
261         (bmalloc::VMHeap::allocateMediumPage):
262         (bmalloc::VMHeap::allocateLargeObject):
263         (bmalloc::VMHeap::deallocateLargeObject): Be sure to give each object
264         a new chance to merge, since it might have been prohibited from merging
265         before by virtue of not being in the VM heap.
266
267         (bmalloc::VMHeap::allocateLargeRange): Deleted.
268         (bmalloc::VMHeap::deallocateLargeRange): Deleted.
269
270 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
271
272         bmalloc: Large object free list can grow infinitely
273         https://bugs.webkit.org/show_bug.cgi?id=142055
274
275         Reviewed by Andreas Kling.
276
277         By design, we don't eagerly remove large objects from the free list.
278         This creates two simple pathologies:
279
280             (1) If you free and then allocate the same object repeatedly, it will
281             duplicate itself in the free list repeatedly. Since it is never
282             invalid at the time of allocation, it will never be removed.
283
284             (2) If you split and then merge the same object repeatedly, it will
285             duplicate its split sibling in the free list repeatedly. If its
286             sibling is in a separate free list size class, it will never be
287             consulted at the time of allocation, so it will never be removed.
288
289         So, a simple "while (1) { free(malloc(x)); }" causes infinite memory
290         use in the free list.
291
292         The solution in this patch is a simple helper to remove garbage from the
293         free list if it grows too large. This pathology is not common, so the
294         cost is OK.
295
296         Long-term, perhaps we should rethink the laziness of these free lists.
297
298         * bmalloc/BoundaryTag.h:
299         (bmalloc::BoundaryTag::isMarked):
300         (bmalloc::BoundaryTag::setMarked): New bit, used by free list GC.
301
302         * bmalloc/FreeList.cpp:
303         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): The GC algorithm.
304
305         * bmalloc/FreeList.h:
306         (bmalloc::FreeList::FreeList):
307         (bmalloc::FreeList::push): Invoke the GC if we're getting huge.
308
309         * bmalloc/LargeObject.h:
310         (bmalloc::LargeObject::isMarked):
311         (bmalloc::LargeObject::setMarked):
312         (bmalloc::LargeObject::validateSelf): Expose the new bit.
313
314         * bmalloc/Sizes.h: New constant to control GC frequency.
315
316 2015-02-26  Csaba Osztrogonác  <ossy@webkit.org>
317
318         URTBF after r180693.
319
320         * CMakeLists.txt:
321
322 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
323
324         Try to fix the Mac build.
325
326         Unreviewed.
327
328         * bmalloc.xcodeproj/project.pbxproj: Make FreeList.h available.
329
330 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
331
332         bmalloc: Refactored SegregatedFreeList and BoundaryTag::init
333         https://bugs.webkit.org/show_bug.cgi?id=142049
334
335         Reviewed by Anders Carlsson.
336
337         Split out a FreeList class from SegregatedFreeList. This will make it
338         easier to add behaviors on free list insertion and removal -- and it's
339         probably how I should have designed things at the start.
340
341         Moved BoundaryTag::init into LargeObject, since all the related logic
342         lives in LargeObject now too, and this allows us to remove BoundaryTagInlines.h.
343
344         * bmalloc.xcodeproj/project.pbxproj:
345         * bmalloc/BoundaryTagInlines.h: Removed.
346         * bmalloc/FreeList.cpp: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.cpp.
347         (bmalloc::FreeList::takeGreedy):
348         (bmalloc::FreeList::take):
349         (bmalloc::SegregatedFreeList::SegregatedFreeList): Deleted.
350         (bmalloc::SegregatedFreeList::insert): Deleted.
351         (bmalloc::SegregatedFreeList::takeGreedy): Deleted.
352         (bmalloc::SegregatedFreeList::take): Deleted.
353         * bmalloc/FreeList.h: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.h.
354         (bmalloc::FreeList::push):
355         * bmalloc/LargeObject.h:
356         (bmalloc::LargeObject::init):
357         * bmalloc/SegregatedFreeList.cpp:
358         (bmalloc::SegregatedFreeList::SegregatedFreeList):
359         (bmalloc::SegregatedFreeList::insert):
360         (bmalloc::SegregatedFreeList::takeGreedy):
361         (bmalloc::SegregatedFreeList::take):
362         * bmalloc/SegregatedFreeList.h:
363         * bmalloc/Sizes.h:
364         * bmalloc/VMHeap.cpp:
365         (bmalloc::VMHeap::grow):
366
367 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
368
369         bmalloc: free up a bit in BoundaryTag
370         https://bugs.webkit.org/show_bug.cgi?id=142048
371
372         Reviewed by Brady Eidson.
373
374         We were wasting a bit by accident, and I need one now.
375
376         * bmalloc/Algorithm.h:
377         (bmalloc::rightShift): Deleted. Not needed, now that I've simplified
378         the math.
379
380         * bmalloc/BoundaryTag.h: Since each boundary tag bucket is 1024 bytes
381         long, the maximum offset into a bucket is 1023.
382
383         You need 5 bits to count up to 1024, but only 4 to count up to 1023.
384         
385         Math is hard.
386
387         (bmalloc::BoundaryTag::compactBegin): Switched to division because it
388         is simpler, and easier to match up with our ASSERT. The compiler will
389         turn division by constant power of two into a shift for us.
390
391         (bmalloc::BoundaryTag::setRange): Added an ASSERT for compactBegin
392         because we do encode it, so we should ASSERT that encoding did not
393         lose information.
394
395         * bmalloc/Sizes.h: Shifting is no longer used since we use division
396         instead.
397
398 2015-02-24  Stephanie Lewis  <slewis@apple.com>
399
400         Rolling out http://trac.webkit.org/changeset/180430 as it causes the PLT to crash.
401         <rdar://problem/19948015>
402
403         Unreviewed.
404
405         * bmalloc/VMHeap.cpp:
406         (bmalloc::VMHeap::grow):
407         * bmalloc/VMHeap.h:
408         * bmalloc/Zone.cpp:
409         (bmalloc::Zone::Zone):
410         (bmalloc::Zone::size): Deleted.
411         * bmalloc/Zone.h:
412
413 2015-02-24  Geoffrey Garen  <ggaren@apple.com>
414
415         bmalloc: Added a little more abstraction for large objects
416         https://bugs.webkit.org/show_bug.cgi?id=141978
417
418         Reviewed by Sam Weinig.
419
420         Previously, each client needed to manage the boundary tags of
421         a large object using free functions. This patch introduces a LargeObject
422         class that does things a little more automatically.
423
424         * bmalloc.xcodeproj/project.pbxproj:
425
426         * bmalloc/Allocator.cpp:
427         (bmalloc::Allocator::reallocate): Use the new LargeObject class.
428
429         * bmalloc/BeginTag.h:
430         (bmalloc::BeginTag::isInFreeList): Deleted. Moved this logic into the
431         LargeObject class.
432
433         * bmalloc/BoundaryTag.h:
434         (bmalloc::BoundaryTag::isSentinel):
435         (bmalloc::BoundaryTag::compactBegin):
436         (bmalloc::BoundaryTag::setRange):
437         (bmalloc::BoundaryTag::initSentinel): Added an explicit API for sentinels,
438         which we used to create and test for implicitly.
439
440         * bmalloc/BoundaryTagInlines.h:
441         (bmalloc::BoundaryTag::init):
442         (bmalloc::validate): Deleted.
443         (bmalloc::validatePrev): Deleted.
444         (bmalloc::validateNext): Deleted.
445         (bmalloc::BoundaryTag::mergeLeft): Deleted.
446         (bmalloc::BoundaryTag::mergeRight): Deleted.
447         (bmalloc::BoundaryTag::merge): Deleted.
448         (bmalloc::BoundaryTag::deallocate): Deleted.
449         (bmalloc::BoundaryTag::split): Deleted.
450         (bmalloc::BoundaryTag::allocate): Deleted. Moved this logic into the
451         LargeObject class.
452
453         * bmalloc/EndTag.h:
454         (bmalloc::EndTag::init):
455         (bmalloc::EndTag::operator=): Deleted. Re-reading this code, I found
456         special behavior in the assignment operator to be a surprising API.
457         So, I replaced the assignment operation with an explicit initializing
458         function.
459
460         * bmalloc/Heap.cpp:
461         (bmalloc::Heap::scavengeLargeRanges):
462         (bmalloc::Heap::allocateXLarge):
463         (bmalloc::Heap::findXLarge):
464         (bmalloc::Heap::deallocateXLarge):
465         (bmalloc::Heap::allocateLarge):
466         (bmalloc::Heap::deallocateLarge):
467         * bmalloc/Heap.h: No behavior changes here -- just adopting the
468         LargeObject interface.
469
470         * bmalloc/LargeObject.h: Added.
471         (bmalloc::LargeObject::operator!):
472         (bmalloc::LargeObject::begin):
473         (bmalloc::LargeObject::size):
474         (bmalloc::LargeObject::range):
475         (bmalloc::LargeObject::LargeObject):
476         (bmalloc::LargeObject::setFree):
477         (bmalloc::LargeObject::isFree):
478         (bmalloc::LargeObject::hasPhysicalPages):
479         (bmalloc::LargeObject::setHasPhysicalPages):
480         (bmalloc::LargeObject::isValidAndFree):
481         (bmalloc::LargeObject::merge):
482         (bmalloc::LargeObject::split):
483         (bmalloc::LargeObject::validateSelf):
484         (bmalloc::LargeObject::validate): Moved this code into a class, out of
485         BoundaryTag free functions.
486
487         New to the class are these features:
488
489             (1) Every reference to an object is validated upon creation and use.
490
491             (2) There's an explicit API for "This is a reference to an object
492             that might be stale (the DoNotValidate API)".
493
494             (3) The begin and end tags are kept in sync automatically.
495
496         * bmalloc/SegregatedFreeList.cpp:
497         (bmalloc::SegregatedFreeList::insert):
498         (bmalloc::SegregatedFreeList::takeGreedy):
499         (bmalloc::SegregatedFreeList::take):
500         * bmalloc/SegregatedFreeList.h: Adopt the LargeObject interface.
501
502         * bmalloc/VMHeap.cpp:
503         (bmalloc::VMHeap::grow):
504         * bmalloc/VMHeap.h:
505         (bmalloc::VMHeap::allocateLargeRange):
506         (bmalloc::VMHeap::deallocateLargeRange): Adopt the LargeObject interface.
507
508 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
509
510         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
511         https://bugs.webkit.org/show_bug.cgi?id=141802
512
513         Reviewed by Andreas Kling.
514
515         Rolling back in with a fix for a crash seen while using GuardMalloc.
516
517         * bmalloc/VMHeap.cpp:
518         (bmalloc::VMHeap::grow):
519         * bmalloc/VMHeap.h:
520         * bmalloc/Zone.cpp: Re-land the old patch.
521
522         (bmalloc::Zone::size): Be sure to implement the size() function since
523         it's accessible indirectly via the malloc_zone_from_ptr public API --
524         and GuardMalloc calls it all the time.
525
526         (bmalloc::Zone::Zone):
527         * bmalloc/Zone.h: Re-land the old patch.
528
529 2015-02-19  Commit Queue  <commit-queue@webkit.org>
530
531         Unreviewed, rolling out r180363.
532         https://bugs.webkit.org/show_bug.cgi?id=141814
533
534         Caused >50 crashes when running LayoutTests in GuardMalloc or
535         ASAN modes. (Requested by jernoble on #webkit).
536
537         Reverted changeset:
538
539         "bmalloc should implement malloc introspection (to stop false-
540         positive leaks when MallocStackLogging is off)"
541         https://bugs.webkit.org/show_bug.cgi?id=141802
542         http://trac.webkit.org/changeset/180363
543
544 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
545
546         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
547         https://bugs.webkit.org/show_bug.cgi?id=141802
548
549         Reviewed by Andreas Kling.
550
551         Fixed a last-minute type.
552
553         The macro is OS, not PLATFORM.
554
555         * bmalloc/VMHeap.cpp:
556         (bmalloc::VMHeap::grow):
557         * bmalloc/VMHeap.h:
558         * bmalloc/Zone.h:
559
560 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
561
562         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
563         https://bugs.webkit.org/show_bug.cgi?id=141802
564
565         Reviewed by Andreas Kling.
566
567         This patch does the bare minimum to stop false positive leaks from
568         being reported by the Darwin leaks tool. We register each super chunk
569         as a single object, and then request that the leaks tool scan it.
570
571         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
572         zone introspection API.
573
574         * bmalloc/Algorithm.h: Missing #include.
575
576         * bmalloc/VMHeap.cpp:
577         (bmalloc::VMHeap::grow):
578         * bmalloc/VMHeap.h: Adopt the new abstraction.
579
580         * bmalloc/Zone.cpp: Added.
581         (bmalloc::remoteRead): Helper for reading an object out of another process.
582         (bmalloc::Zone::enumerator):
583         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
584         in introspection.
585
586         * bmalloc/Zone.h: Added.
587         (bmalloc::Zone::superChunks):
588         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
589         since our dynamic allocations will not be scanned by leaks since they
590         will have the malloc VM tag.
591
592 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
593
594         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
595         https://bugs.webkit.org/show_bug.cgi?id=141759
596
597         Reviewed by Andreas Kling.
598
599         * bmalloc.xcodeproj/project.pbxproj:
600         * bmalloc/SuperChunk.h: Added.
601         (bmalloc::SuperChunk::create):
602         (bmalloc::SuperChunk::SuperChunk):
603         (bmalloc::SuperChunk::smallChunk):
604         (bmalloc::SuperChunk::mediumChunk):
605         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
606         into a separate class, for clarity and type safety.
607
608         * bmalloc/VMHeap.cpp:
609         (bmalloc::VMHeap::grow):
610         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
611         "grow" because Andreas found "allocateSuperChunk" to be unclear.
612
613         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
614         for malloc introspection.
615
616         (bmalloc::VMHeap::allocateSmallPage):
617         (bmalloc::VMHeap::allocateMediumPage):
618         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
619
620 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
621
622         Build bmalloc through CMake as a static library. It's then linked either
623         into the WTF library (if built as a shared library) or into the JSC and
624         WebKit2 libraries. There's no need to build it as a standalone shared library.
625
626         Rubber-stamped by Carlos Garcia Campos.
627
628         * CMakeLists.txt:
629
630 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
631
632         [BMalloc] Add a FIXME comment for memory alignas
633         https://bugs.webkit.org/show_bug.cgi?id=141556
634
635         Reviewed by Csaba Osztrogonác.
636
637         * bmalloc/Chunk.h: Add a FIXME comment.
638         * bmalloc/LargeChunk.h: ditto.
639
640 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
641
642         bmalloc buildfix on 32 bit Linux (x86/ARM)
643         https://bugs.webkit.org/show_bug.cgi?id=141472
644
645         Reviewed by Gyuyoung Kim.
646
647         * bmalloc/Algorithm.h:
648         (bmalloc::roundUpToMultipleOf):
649         * bmalloc/FixedVector.h:
650         (bmalloc::FixedVector::clear):
651         * bmalloc/Sizes.h:
652         (bmalloc::Sizes::sizeClass):
653
654 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
655
656         [EFL][GTK] Use bmalloc instead of tcmalloc
657         https://bugs.webkit.org/show_bug.cgi?id=140162
658
659         Reviewed by Carlos Garcia Campos.
660
661         Support to use bmalloc on EFL and GTK ports.
662
663         * CMakeLists.txt: Added.
664         * bmalloc/Allocator.cpp:
665         (bmalloc::Allocator::allocate):
666             Fix unused return value caused by posix_memalign().
667         * bmalloc/AsyncTask.h:
668         * bmalloc/BoundaryTag.h:
669         (bmalloc::BoundaryTag::clear):
670         * bmalloc/Chunk.h:
671             Change Traits::Page with Traits::PageType in order to fix
672             -fpermitive build error on EFL and GTK port. 
673         * bmalloc/EndTag.h:
674         (bmalloc::EndTag::operator=):
675         * bmalloc/Line.h: ditto.
676         * bmalloc/MediumTraits.h:
677         * bmalloc/Page.h: ditto.
678         * bmalloc/PerThread.h:
679             EFL port doesn't support __has_include definition yet.
680             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
681         * bmalloc/SmallTraits.h: ditto.
682         * bmalloc/VMAllocate.h:
683         (bmalloc::vmDeallocatePhysicalPages):
684         (bmalloc::vmAllocatePhysicalPages):
685         * bmalloc/Vector.h:
686         (bmalloc::Vector<T>::push):
687         (bmalloc::Vector<T>::reallocateBuffer):
688
689 2015-01-31  Sam Weinig  <sam@webkit.org>
690
691         Remove even more Mountain Lion support
692         https://bugs.webkit.org/show_bug.cgi?id=141124
693
694         Reviewed by Alexey Proskuryakov.
695
696         * Configurations/Base.xcconfig:
697         * Configurations/DebugRelease.xcconfig:
698
699 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
700
701         GC marking threads should clear malloc caches
702         https://bugs.webkit.org/show_bug.cgi?id=141097
703
704         Reviewed by Andreas Kling.
705
706         Split the scavenging API into per-thread vs global, so that you can
707         request to scavenge your own thread without scavenging the whole heap.
708
709         * bmalloc/Cache.cpp:
710         (bmalloc::Cache::scavenge):
711         * bmalloc/bmalloc.h:
712         (bmalloc::api::scavengeThisThread):
713         (bmalloc::api::scavenge):
714
715 2015-01-28  Dana Burkart  <dburkart@apple.com>
716
717         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
718         https://bugs.webkit.org/show_bug.cgi?id=136765
719
720         Reviewed by Alexey Proskuryakov.
721
722         * Configurations/Base.xcconfig:
723         * Configurations/DebugRelease.xcconfig:
724
725 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
726
727         bmalloc: support aligned allocation
728         https://bugs.webkit.org/show_bug.cgi?id=140732
729
730         Reviewed by Andreas Kling.
731
732         * bmalloc/Allocator.cpp:
733         (bmalloc::Allocator::allocate): New function for aligned allocation.
734
735         Small and medium requests just allocate and free until they find an
736         aligned pointer. This is slightly inefficient in the worst case, but
737         still constant-time with little-to-no space overhead.
738
739         Large requests use a new API that requires the client to specify both
740         its ideal size and alignment, and the worst-case size you would have to
741         allocate in order to produce some interior pointer of the requested size
742         and alignment. We put the burden of this calculation on the client
743         because it simplifies things if we guarantee that allocation won't fail.
744
745         XLarge requests are easy: we just forward them to vmAllocate, which
746         already supported aligned requests.
747
748         * bmalloc/BoundaryTag.h:
749         * bmalloc/BoundaryTagInlines.h:
750         (bmalloc::BoundaryTag::mergeLeft):
751         (bmalloc::BoundaryTag::mergeRight):
752         (bmalloc::BoundaryTag::merge):
753         (bmalloc::BoundaryTag::deallocate):
754         (bmalloc::BoundaryTag::split):
755         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
756         refactored the interface to remove some reference out parameters in
757         order to clarify what changes and what doesn't.
758
759         * bmalloc/Heap.cpp:
760         (bmalloc::Heap::allocateXLarge): Added an alignment API.
761
762         (bmalloc::Heap::allocateLarge):
763         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
764         a few variants, so aligned and unaligned allocation could share some code.
765
766         * bmalloc/SegregatedFreeList.cpp:
767         (bmalloc::SegregatedFreeList::take):
768         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
769         for aligned allocation. It turns out that the aligned path is pretty
770         different, since it ends up searching for two potential ways to satisfy
771         an allocation: either large enough and aligned, or large enough to split
772         into something not aligned and something large enough and aligned.
773
774         * bmalloc/VMAllocate.h:
775         (bmalloc::vmAllocate): Switched alignment to come before size because
776         that's how the memalign API specifies it.
777
778         * bmalloc/VMHeap.h:
779         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
780
781 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
782
783         bmalloc: a little bit of cleanup
784         https://bugs.webkit.org/show_bug.cgi?id=140687
785
786         Reviewed by Anders Carlsson.
787
788         * bmalloc/Algorithm.h:
789         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
790         of code.
791
792         * bmalloc/BoundaryTag.h:
793         * bmalloc/BoundaryTagInlines.h:
794         (bmalloc::BoundaryTag::mergeLeft):
795         (bmalloc::BoundaryTag::mergeRight):
796         (bmalloc::BoundaryTag::merge):
797         (bmalloc::BoundaryTag::deallocate):
798         (bmalloc::BoundaryTag::split):
799         (bmalloc::BoundaryTag::allocate):
800         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
801         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
802         (bmalloc::BoundaryTag::mergeLarge): Deleted.
803         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
804         from all these functions, since boundary tags always pertain to large
805         objects, and putting the word "Large" everywhere wasn't helping to
806         explain that.
807
808         * bmalloc/Heap.cpp:
809         (bmalloc::Heap::allocateXLarge):
810         (bmalloc::Heap::findXLarge):
811         (bmalloc::Heap::deallocateXLarge):
812         * bmalloc/Heap.h:
813         * bmalloc/VMHeap.h:
814         (bmalloc::VMHeap::allocateXLarge): Deleted.
815         (bmalloc::VMHeap::findXLarge): Deleted.
816         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
817         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
818         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
819         really make sense for the VMHeap to be involved.
820
821 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
822
823         bmalloc: refactored XLarge allocation for better alignment
824         https://bugs.webkit.org/show_bug.cgi?id=140582
825
826         Reviewed by Andreas Kling.
827
828         XLarge objects used to be Large objects with an extra bit of metadata
829         that said "actually, I'm not large -- I'm extra large".
830
831         The metadata header in an XLarge allocation made it impossible for the
832         XLarge object to honor a very large alignment request.
833
834         The solution is to stop using a metadata header for XLarge objects, and
835         instead to store explicit metadata on the side.
836
837         This is a bit less astonishing, which is also nice.
838
839         Finding XLarge metadata is now a linear search. That's probably OK, since
840         it was always so in TCMalloc, and the usual number of XLarge allocations
841         in a process is 0.
842
843         This design makes it possible for the heap to cache XLarge allocations
844         with and/or without physical pages. I haven't actually done that yet
845         because the tradeoffs are subtle, so I don't want to do anything without
846         a motivating test case.
847
848         * bmalloc.xcodeproj/project.pbxproj:
849         * bmalloc/Allocator.cpp:
850         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
851         since an XLarge allocation is now just a naked buffer without a header.
852
853         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
854         XLarge alignment, since XLargeChunk won't give this to us implicitly
855         anymore.
856
857         * bmalloc/BoundaryTag.h:
858         (bmalloc::BoundaryTag::setRange):
859         (bmalloc::BoundaryTag::isXLarge): Deleted.
860         (bmalloc::BoundaryTag::setXLarge): Deleted.
861         * bmalloc/BoundaryTagInlines.h:
862         (bmalloc::validate):
863         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
864
865         * bmalloc/Deallocator.cpp:
866         (bmalloc::Deallocator::deallocateXLarge):
867         (bmalloc::Deallocator::deallocateSlowCase):
868         * bmalloc/Heap.cpp:
869         (bmalloc::Heap::findXLarge):
870         (bmalloc::Heap::allocateXLarge):
871         (bmalloc::Heap::deallocateXLarge):
872         * bmalloc/Heap.h: Updated for interface changes.
873
874         * bmalloc/ObjectType.cpp:
875         (bmalloc::objectType):
876         * bmalloc/ObjectType.h:
877         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
878         examining its bit pattern -- just like we do for other kinds of
879         allocations -- which is nice.
880
881         * bmalloc/Sizes.h:
882         * bmalloc/VMHeap.h:
883         (bmalloc::VMHeap::allocateXLarge):
884         (bmalloc::VMHeap::findXLarge):
885         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
886         for XLarge allocations.
887
888         * bmalloc/XLargeChunk.h: Removed.
889
890 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
891
892         bmalloc: added some infrastructure for aligned allocation
893         https://bugs.webkit.org/show_bug.cgi?id=140572
894
895         Reviewed by Andreas Kling.
896
897         * bmalloc/Algorithm.h:
898         (bmalloc::isPowerOfTwo):
899         (bmalloc::roundUpToMultipleOf):
900         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
901         isPowerOfTwo helper function.
902
903         * bmalloc/Allocator.cpp:
904         (bmalloc::Allocator::allocate):
905         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
906         Doesn't do anything yet, but does correctly forward to system malloc
907         when bmalloc is disabled.
908
909         * bmalloc/Cache.cpp:
910         (bmalloc::Cache::allocateSlowCaseNullCache):
911         * bmalloc/Cache.h:
912         (bmalloc::Cache::allocate):
913         * bmalloc/bmalloc.h:
914         (bmalloc::api::memalign):
915         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
916
917 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
918
919         Consider alignment when allocating from a SegregatedFreeList
920         https://bugs.webkit.org/show_bug.cgi?id=140408
921
922         Reviewed by Sam Weinig.
923
924         In preparation for supporting aligned allocation.
925
926         No performance change.
927
928         Since this is just one extra branch in an already expensive function,
929         I decided not to duplicate the function just to avoid the branch in
930         the un-aligned case.
931
932         * bmalloc/SegregatedFreeList.cpp:
933         (bmalloc::SegregatedFreeList::take):
934         * bmalloc/SegregatedFreeList.h:
935
936 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
937
938         Renamed minimum to size in SegregatedFreeList
939         https://bugs.webkit.org/show_bug.cgi?id=140406
940
941         Reviewed by Sam Weinig.
942
943         In preparation for supporting aligned allocation.
944
945         * bmalloc/SegregatedFreeList.cpp:
946         (bmalloc::SegregatedFreeList::takeGreedy):
947         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
948         really just a minimum. Let's not imply that this value is special.
949
950 2015-01-11  Dan Bernstein  <mitz@apple.com>
951
952         Geoff is organized, but he is not an organization.
953
954         Rubber-stamped by Anders Carlsson.
955
956         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
957
958 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
959
960         Make bmalloc work with ASan
961         https://bugs.webkit.org/show_bug.cgi?id=140194
962
963         Reviewed by Mark Lam.
964
965         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
966         an OS-specific API to test for loaded runtime libraries.
967
968         * bmalloc/Environment.cpp:
969         (bmalloc::isASanEnabled):
970         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
971         ASan is enabled, since system malloc has the Asan hooks we need.
972
973         You could check for the ASan compile-time flag instead, but doing this
974         check at runtime prepares bmalloc for a world where it is a dynamic
975         library that might be loaded into projects it did not compile with.
976
977 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
978
979         Fix up bmalloc's PerThread for use on Linux
980         https://bugs.webkit.org/show_bug.cgi?id=139804
981
982         Reviewed by Anders Carlsson.
983
984         The previous implementation was a bit slow.
985
986         * bmalloc/PerThread.h:
987         (bmalloc::PerThreadStorage<Cache>::get):
988         (bmalloc::PerThreadStorage::get):
989         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
990         way to do fast per-thread access without taking a lock every time. This
991         probably works on all the platforms we care about, and it matches other
992         techniques we use elsewhere in WebKit.
993
994         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
995         this class because PerThreadStorage now encapsulates everything that
996         needs to be conditional.
997
998         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
999
1000 2014-12-26  Dan Bernstein  <mitz@apple.com>
1001
1002         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
1003         https://bugs.webkit.org/show_bug.cgi?id=139950
1004
1005         Reviewed by David Kilzer.
1006
1007         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
1008         in a manner that works with Xcode 5.1.1.
1009
1010 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
1011
1012         Safari crashes when you set Malloc environment variables
1013         https://bugs.webkit.org/show_bug.cgi?id=139656
1014
1015         Reviewed by Michael Saboff.
1016
1017         I forgot to cover the realloc() case. Whoops. (OoPS?)
1018
1019         This time around, I ran the full MallocBench test suite in Malloc=1
1020         mode, and it passed.
1021
1022         * bmalloc/Allocator.cpp:
1023         (bmalloc::Allocator::reallocate):
1024         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
1025         It needs to be down there so that we can do the short-circuiting check
1026         for whether bmalloc is enabled first.
1027
1028         Also added the check.
1029
1030         * bmalloc/Cache.cpp:
1031         (bmalloc::Cache::scavenge):
1032         (bmalloc::Cache::Cache):
1033         (bmalloc::Cache::reallocateSlowCaseNullCache):
1034         * bmalloc/Cache.h:
1035         (bmalloc::Cache::deallocator):
1036         (bmalloc::Cache::reallocate): Ditto.
1037
1038         * bmalloc/bmalloc.h:
1039         (bmalloc::api::free):
1040         (bmalloc::api::realloc): Ditto.
1041
1042         (bmalloc::api::scavenge): Pushed this down into Cache to match the
1043         surrounding functions.
1044
1045 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1046
1047         bmalloc should support system memory analysis tools (part 2)
1048         https://bugs.webkit.org/show_bug.cgi?id=139565
1049
1050         Reviewed by Mark Lam.
1051
1052         This patch actually queries the environment to see if memory analysis
1053         tools have been enabled.
1054
1055         * bmalloc/Deallocator.cpp:
1056         (bmalloc::Deallocator::scavenge): Don't process the object log if
1057         we've disabled bmalloc because it will be full of invalid nullptrs.
1058
1059         * bmalloc/Environment.cpp:
1060         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
1061         Malloc debugging flags. I also added a plain "Malloc" catch-all for
1062         when you want to disable bmalloc without enabling any kind of funny
1063         business.
1064
1065         It would be slightly nicer just to iterate the list of environment
1066         variables and strstr them, but getenv is the more portable option,
1067         and performance here doesn't really matter.
1068
1069         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
1070         environment variable.
1071
1072         (bmalloc::Environment::computeIsBmallocEnabled):
1073
1074 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1075
1076         Try to fix the iOS simulator build.
1077
1078         #include the declaration of malloc / free.
1079
1080         * bmalloc/Allocator.cpp:
1081         * bmalloc/Deallocator.cpp:
1082
1083 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1084
1085         Try to fix the build.
1086
1087         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
1088
1089 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1090
1091         bmalloc should support system memory analysis tools (part 1)
1092         https://bugs.webkit.org/show_bug.cgi?id=139559
1093
1094         Reviewed by Mark Lam.
1095
1096         This patch adds the hooks to disable bmalloc at runtime if certain
1097         environment variables are set, but doesn't actually read from the
1098         environment yet.
1099
1100         No performance change.
1101
1102         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
1103         we'll use to read environment variables and see if memory analysis tools
1104         have been enabled.
1105
1106         * bmalloc/Allocator.cpp:
1107         (bmalloc::Allocator::Allocator):
1108         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
1109         on the allocation path. We cache the setting to make the check fast.
1110
1111         * bmalloc/Allocator.h: Interface changes.
1112
1113         * bmalloc/Cache.cpp:
1114         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
1115         and deallocator. This main purpose is to enable them to query the
1116         environment for whether bmalloc is enabled; but this is also a slightly
1117         cleaner way to guarantee to them that the Heap has been pre-initialized.
1118
1119         * bmalloc/Deallocator.cpp:
1120         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
1121         fill the object log to force us to take the slow path on all deallocations.
1122
1123         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
1124
1125         * bmalloc/Deallocator.h: Interface changes.
1126
1127         * bmalloc/Environment.cpp: Added.
1128         (bmalloc::Environment::Environment):
1129         (bmalloc::Environment::computeIsBmallocEnabled):
1130         * bmalloc/Environment.h: Added.
1131         (bmalloc::Environment::isBmallocEnabled): This is the class that will
1132         encapsulate looking for environment variables that turn on heap
1133         analysis tools.
1134
1135         * bmalloc/Heap.h:
1136         (bmalloc::Heap::environment):
1137
1138         * bmalloc/Mutex.h:
1139         (bmalloc::Mutex::Mutex):
1140         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
1141         since I got super confused about them while writing this patch.
1142
1143         * bmalloc/VMHeap.cpp: Fixed an #include.
1144
1145 2014-12-09  David Kilzer  <ddkilzer@apple.com>
1146
1147         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
1148         <http://webkit.org/b/139212>
1149
1150         Reviewed by Joseph Pecoraro.
1151
1152         * Configurations/Base.xcconfig:
1153         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
1154           on OS X.
1155         * Configurations/DebugRelease.xcconfig:
1156         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
1157
1158 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
1159
1160         bmalloc uses 8X more virtual memory than necessary
1161         https://bugs.webkit.org/show_bug.cgi?id=138495
1162
1163         Reviewed by Mark Lam.
1164
1165         iOS has a per-process virtual memory cap around 1GB, so there's some
1166         value to not going totally ham with virtual memory.
1167
1168         We currently use about 8X the necessary amount:
1169             - 2X to align our VM allocation
1170             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
1171
1172         We can cut that down:
1173             - Return the unaligned portion of our VM allocation (-2X)
1174             - Use all the chunks in a superchunk, instead of allocating one
1175               chunk per superchunk (-4X)
1176
1177         * bmalloc/Algorithm.h:
1178         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
1179         function so we can call it with getpagesize() at runtime.
1180
1181         * bmalloc/Chunk.h:
1182         * bmalloc/LargeChunk.h:
1183         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
1184         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
1185
1186         * bmalloc/VMAllocate.h:
1187         (bmalloc::vmValidate):
1188         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
1189         if it does not succeed. Allocate precisely, and give back the extra.
1190
1191         * bmalloc/VMHeap.cpp:
1192         (bmalloc::VMHeap::allocateSuperChunk):
1193         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
1194         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
1195         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
1196         in a superchunk, instead of just one.
1197
1198         * bmalloc/VMHeap.h:
1199         (bmalloc::VMHeap::allocateSmallPage):
1200         (bmalloc::VMHeap::allocateMediumPage):
1201         (bmalloc::VMHeap::allocateLargeRange):
1202         * bmalloc/XLargeChunk.h:
1203         (bmalloc::XLargeChunk::create): Updated to match changes above.
1204
1205 2014-11-01  David Kilzer  <ddkilzer@apple.com>
1206
1207         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
1208         <https://webkit.org/b/138286>
1209         <rdar://problem/18847087>
1210
1211         Reviewed by Dan Bernstein.
1212
1213         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
1214         to NO for the target that produces libbmalloc.a so that the
1215         debug symbols will be linked into JavaScriptCore and end up in
1216         its dSYM file.
1217
1218 2014-10-30  Dana Burkart  <dburkart@apple.com>
1219
1220         <rdar://problem/18821260> Prepare for the mysterious future
1221
1222         Reviewed by Lucas Forschler.
1223
1224         * Configurations/Base.xcconfig:
1225         * Configurations/DebugRelease.xcconfig:
1226
1227 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
1228
1229         bmalloc: cleaned up fast path vs slow path
1230         https://bugs.webkit.org/show_bug.cgi?id=137081
1231
1232         Reviewed by Sam Weinig.
1233
1234         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
1235
1236         * bmalloc/Allocator.cpp:
1237         (bmalloc::Allocator::Allocator): Merged the small and medium range
1238         caches, just like the small and medium allocators. Ranges are abstract
1239         objects that don't really care whether they hold small or medium objects,
1240         so they don't need to be segregated.
1241
1242         (bmalloc::Allocator::scavenge): Ditto.
1243
1244         (bmalloc::Allocator::allocateBumpRangeSlowCase):
1245         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
1246         we do care a tiny bit, because we need to specify small vs medium when
1247         allocating new ranges from the heap, to ensure that the heap allocates
1248         from the right segment of VM.
1249
1250         (bmalloc::Allocator::allocateLarge):
1251         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
1252         up the fast path. Large allocation performance is dominated by allocation
1253         logic and initialization, so inlining it doesn't help.
1254
1255         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
1256         it doesn't need to distinguish small vs medium objects.
1257
1258         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
1259         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
1260
1261         * bmalloc/Allocator.h:
1262         * bmalloc/BumpRange.h:
1263
1264         * bmalloc/Cache.cpp:
1265         (bmalloc::Cache::allocateSlowCase): Deleted.
1266         (bmalloc::Cache::deallocateSlowCase): Deleted.
1267         * bmalloc/Cache.h:
1268         (bmalloc::Cache::allocate):
1269         (bmalloc::Cache::deallocate):
1270         (bmalloc::Cache::allocateFastCase): Deleted.
1271         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
1272         paths. The downside to this change is that the fast path branches to two
1273         distinct failure cases instead of one. The upside is that the slow path
1274         doesn't need to re-read the segment register, which is not as cheap as a
1275         normal register, and it doesn't need to do an extra level of function 
1276         call. Seems to be worth it.
1277
1278         * bmalloc/Deallocator.h:
1279         * bmalloc/Heap.cpp:
1280         (bmalloc::Heap::refillSmallBumpRangeCache):
1281         (bmalloc::Heap::refillMediumBumpRangeCache):
1282         * bmalloc/Heap.h: Updated for interface changes.
1283
1284         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
1285         small lines in a page / 2, since any other free lines will coalesce
1286         with their neighbors.
1287
1288 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1289
1290         Rolled out r173346.
1291
1292             bmalloc should honor the FastMalloc statistics API
1293             https://bugs.webkit.org/show_bug.cgi?id=136592
1294
1295         This didn't really work. Because we allow ranges with and without
1296         physical pages to merge, and we allow double-committing and
1297         double-decommitting, we can't rely on commit actions to track memory
1298         footprint.
1299
1300         * bmalloc/Heap.cpp:
1301         (bmalloc::Heap::size): Deleted.
1302         (bmalloc::Heap::capacity): Deleted.
1303         * bmalloc/Heap.h:
1304         * bmalloc/VMHeap.cpp:
1305         (bmalloc::VMHeap::VMHeap):
1306         (bmalloc::VMHeap::allocateSmallChunk):
1307         (bmalloc::VMHeap::allocateMediumChunk):
1308         (bmalloc::VMHeap::allocateLargeChunk):
1309         * bmalloc/VMHeap.h:
1310         (bmalloc::VMHeap::allocateSmallPage):
1311         (bmalloc::VMHeap::allocateMediumPage):
1312         (bmalloc::VMHeap::allocateLargeRange):
1313         (bmalloc::VMHeap::deallocateSmallPage):
1314         (bmalloc::VMHeap::deallocateMediumPage):
1315         (bmalloc::VMHeap::deallocateLargeRange):
1316         (bmalloc::VMHeap::size): Deleted.
1317         (bmalloc::VMHeap::capacity): Deleted.
1318         * bmalloc/bmalloc.h:
1319         (bmalloc::api::heapSize): Deleted.
1320         (bmalloc::api::heapCapacity): Deleted.
1321
1322 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1323
1324         bmalloc: Allocation should be more precise
1325         https://bugs.webkit.org/show_bug.cgi?id=136993
1326
1327         Reviewed by Gavin Barraclough.
1328
1329         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1330
1331         This patch teaches the allocator to merge adjacent free lines into a
1332         single allocatable range. This allows us to shrink the size of an
1333         individual line without increasing fragmentation or the rate of allocator
1334         slow paths.
1335
1336         We'll only take more slow paths when available memory is sparse, which
1337         is exactly when it's worth it. When available memory is dense, we'll
1338         take fewer slow paths.
1339
1340         * bmalloc.xcodeproj/project.pbxproj:
1341         * bmalloc/Algorithm.h:
1342         (bmalloc::divideRoundingUp):
1343
1344         * bmalloc/Allocator.cpp:
1345         (bmalloc::Allocator::Allocator): Updated for interface changes.
1346
1347         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
1348         Now that we merge lines, it's not convenient to scavenge by line.
1349
1350         (bmalloc::Allocator::allocateSmallBumpRange):
1351         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
1352         instead of individual lines.
1353
1354         (bmalloc::Allocator::allocateSlowCase):
1355         (bmalloc::Allocator::allocateSmallLine): Deleted.
1356         (bmalloc::Allocator::allocateMediumLine): Deleted.
1357         (bmalloc::Allocator::allocateMedium): Deleted.
1358         * bmalloc/Allocator.h:
1359         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
1360         into the standard fast path with small allocations. Since a BumpAllocator
1361         just allocates out of an arbitrary range, it doesn't need to distinguish
1362         between small and medium lines.
1363
1364         * bmalloc/BumpAllocator.h:
1365         (bmalloc::BumpAllocator::size):
1366         (bmalloc::BumpAllocator::BumpAllocator):
1367         (bmalloc::BumpAllocator::init):
1368         (bmalloc::BumpAllocator::refill):
1369         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
1370         anymore: the heap just gives us a pointer and a pre-computed number of
1371         objects, and we allocate them.
1372
1373         * bmalloc/Deallocator.cpp:
1374         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
1375
1376         * bmalloc/Heap.cpp:
1377         (bmalloc::Heap::Heap):
1378         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
1379         detailing where all objects will lie in memory. After we merge two lines,
1380         we might allocate an object that spans from one line to the next. This
1381         metadata details which bits of memory overlap in that way, and how they
1382         overlap.
1383
1384         (bmalloc::Heap::refillSmallBumpRangeCache):
1385         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
1386         and merge adjacent free lines into BumpRanges.
1387
1388         (bmalloc::Heap::allocateSmallPage):
1389         (bmalloc::Heap::allocateMediumPage):
1390         (bmalloc::Heap::deallocateSmallLine):
1391         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
1392         since we scan for free memory a page at a time.
1393
1394         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
1395         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
1396         fast path.
1397
1398         * bmalloc/Heap.h:
1399         (bmalloc::Heap::derefSmallLine):
1400         (bmalloc::Heap::derefMediumLine):
1401         (bmalloc::Heap::deallocateSmallLine): Deleted.
1402         (bmalloc::Heap::allocateSmallLine): Deleted.
1403         (bmalloc::Heap::deallocateMediumLine): Deleted.
1404         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
1405
1406         * bmalloc/Line.h:
1407         (bmalloc::Line<Traits>::ref):
1408         (bmalloc::Line<Traits>::deref):
1409         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
1410         anymore, since we only ever deref by 1 now.
1411
1412         * bmalloc/MediumAllocator.h:
1413         (bmalloc::MediumAllocator::isNull): Deleted.
1414         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
1415         (bmalloc::MediumAllocator::line): Deleted.
1416         (bmalloc::MediumAllocator::allocate): Deleted.
1417         (bmalloc::MediumAllocator::derefCount): Deleted.
1418         (bmalloc::MediumAllocator::refill): Deleted.
1419         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
1420         been dead for a while, since it doesn't build anymore with this patch.
1421
1422         * bmalloc/Page.h:
1423         (bmalloc::Page::sizeClass):
1424         (bmalloc::Page::setSizeClass):
1425         (bmalloc::Page::smallSizeClass): Deleted.
1426         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
1427         to sizeClass, since we use it for medium sizes too.
1428
1429         * bmalloc/Sizes.h:
1430         (bmalloc::Sizes::sizeClass):
1431         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
1432
1433         (bmalloc::Sizes::smallSizeClassFor): Deleted.
1434         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
1435
1436         * bmalloc/bmalloc.h:
1437         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
1438         can be a bit more precise. It also has to be, since we can't guarantee
1439         that an object ends at the end of a line anymore.
1440
1441 2014-09-19  Daniel Bates  <dabates@apple.com>
1442
1443         Always assume internal SDK when building configuration Production
1444         https://bugs.webkit.org/show_bug.cgi?id=136925
1445         <rdar://problem/18362399>
1446
1447         Reviewed by Dan Bernstein.
1448
1449         * Configurations/Base.xcconfig:
1450
1451 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
1452
1453         bmalloc: moved line caches from the deallocator to the allocator
1454         https://bugs.webkit.org/show_bug.cgi?id=136868
1455
1456         Reviewed by Gavin Barraclough.
1457
1458         I did this mostly as a simplification, to make it easier to change the
1459         allocation strategy.
1460
1461         No throughput change on MallocBench. Saves about 50kB.
1462
1463         Since the deallocator needs to lock the heap when freeing lines anyway,
1464         there isn't much benefit to giving the deallocator a local cache of
1465         deallocated lines.
1466
1467         We still give the allocator a local cache of lines because that does
1468         reduce the frequency at which it needs to lock the heap in order to
1469         acquire more lines.
1470
1471         * bmalloc/Allocator.cpp:
1472         (bmalloc::Allocator::scavenge):
1473         (bmalloc::Allocator::allocateSmallLine):
1474         (bmalloc::Allocator::allocateMediumLine):
1475         (bmalloc::Allocator::allocateMedium):
1476         (bmalloc::Allocator::allocateSlowCase):
1477         * bmalloc/Allocator.h:
1478         * bmalloc/Deallocator.cpp:
1479         (bmalloc::Deallocator::Deallocator):
1480         (bmalloc::Deallocator::scavenge):
1481         (bmalloc::Deallocator::processObjectLog):
1482         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
1483         (bmalloc::Deallocator::allocateSmallLine): Deleted.
1484         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
1485         (bmalloc::Deallocator::allocateMediumLine): Deleted.
1486         * bmalloc/Deallocator.h:
1487
1488         * bmalloc/Sizes.h:
1489         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
1490         exactly one page in size. That's about what we were shooting for anyway,
1491         and it may make it easier to switch to per-page allocation in future.
1492
1493 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
1494
1495         bmalloc: allocate small and medium objects using the same bump pointer class
1496         https://bugs.webkit.org/show_bug.cgi?id=136843
1497
1498         Reviewed by Gavin Barraclough.
1499
1500         4% speedup on MallocBench.
1501
1502         Now that medium-sized objects have dedicated per-size allocators, they
1503         don't need to use an arbitrary bump pointer allocator. This means that
1504         every allocator knows how many objects it will allocate from the start,
1505         and we don't need a post-processing step to adjust refcounts based on
1506         real allocation count.
1507
1508         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1509         since it's used for small and medium objects now.
1510
1511         * bmalloc/Allocator.cpp:
1512         (bmalloc::Allocator::Allocator): Updated to use new interface.
1513         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1514         to make sure that we finish allocating all the objects in it.
1515
1516         (bmalloc::Allocator::allocateMedium):
1517         (bmalloc::Allocator::allocateSlowCase):
1518         (bmalloc::Allocator::retire): Deleted.
1519         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1520         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1521         * bmalloc/Allocator.h:
1522         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1523         used to post-process an allocator based on how many objects it allocated.
1524
1525         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1526         (bmalloc::BumpAllocator::BumpAllocator):
1527         (bmalloc::BumpAllocator::init):
1528         (bmalloc::BumpAllocator::line):
1529         (bmalloc::BumpAllocator::validate):
1530         (bmalloc::BumpAllocator::allocate):
1531         (bmalloc::BumpAllocator::refill):
1532         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1533         about the kinds of lines they allocate into. In some cases, the line
1534         type must be provided as a template parameter by the caller.
1535
1536         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1537         (bmalloc::SmallAllocator::line): Deleted.
1538         (bmalloc::SmallAllocator::allocate): Deleted.
1539         (bmalloc::SmallAllocator::objectCount): Deleted.
1540         (bmalloc::SmallAllocator::derefCount): Deleted.
1541         (bmalloc::SmallAllocator::refill): Deleted.
1542         (bmalloc::SmallAllocator::clear): Deleted.
1543
1544         * bmalloc/ObjectType.h:
1545         (bmalloc::isMedium):
1546
1547         * bmalloc/SmallAllocator.h:
1548         (bmalloc::SmallAllocator::isNull): Deleted.
1549         (bmalloc::SmallAllocator::canAllocate): Deleted.
1550         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1551         (bmalloc::SmallAllocator::line): Deleted.
1552         (bmalloc::SmallAllocator::allocate): Deleted.
1553         (bmalloc::SmallAllocator::objectCount): Deleted.
1554         (bmalloc::SmallAllocator::derefCount): Deleted.
1555         (bmalloc::SmallAllocator::refill): Deleted.
1556         (bmalloc::SmallAllocator::clear): Deleted.
1557
1558 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1559
1560         Fixed a goof in bmalloc Vector sizing
1561         https://bugs.webkit.org/show_bug.cgi?id=136795
1562
1563         Reviewed by Gavin Barraclough and Sam Weinig.
1564
1565         We want our minimum vector to be page-sized since the OS will give us
1566         a page no matter what -- but we want that many bytes, and not enough
1567         bytes to store that many elements.
1568
1569         * bmalloc/Vector.h: Math is hard.
1570
1571 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1572
1573         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1574         https://bugs.webkit.org/show_bug.cgi?id=136693
1575
1576         Reviewed by Gavin Barraclough.
1577
1578         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1579
1580         No throughput change.
1581
1582         We keep an array of medium allocators, just like our array of small
1583         allocators.
1584
1585         In future, we can simplify the allocation fast path by merging the small
1586         and medium allocator arrays. For now, this is the simplest change that
1587         gets the win.
1588
1589         * bmalloc/Allocator.cpp:
1590         (bmalloc::Allocator::Allocator):
1591         (bmalloc::Allocator::scavenge):
1592         (bmalloc::Allocator::allocateMedium):
1593         * bmalloc/Allocator.h:
1594         * bmalloc/Sizes.h:
1595         (bmalloc::Sizes::mediumSizeClassFor):
1596
1597 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1598
1599         Reviewed by Sam Weinig.
1600
1601         Renamed log => retire for clarity.
1602
1603         * bmalloc/Allocator.cpp:
1604         (bmalloc::Allocator::scavenge):
1605         (bmalloc::Allocator::retire):
1606         (bmalloc::Allocator::allocateMedium):
1607         (bmalloc::Allocator::allocateSlowCase):
1608         (bmalloc::Allocator::log): Deleted.
1609         * bmalloc/Allocator.h:
1610
1611 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1612
1613         bmalloc: eager scavenge leaves behind a bogus allocator
1614         https://bugs.webkit.org/show_bug.cgi?id=136743
1615
1616         Reviewed by Sam Weinig.
1617
1618         Be sure to clear the allocator after logging it in the eager scavenge
1619         case, so that we don't later try to allocate out of the lines that we
1620         have thrown away.
1621
1622         We didn't need to do this previously because scavenge would only happen
1623         at thread exit time, after which no further allocation from the per-thread
1624         cache would take place.
1625
1626         * bmalloc/Allocator.cpp:
1627         (bmalloc::Allocator::scavenge):
1628         * bmalloc/MediumAllocator.h:
1629         (bmalloc::MediumAllocator::clear):
1630         * bmalloc/SmallAllocator.h:
1631         (bmalloc::SmallAllocator::clear):
1632
1633 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1634
1635         bmalloc should honor the FastMalloc statistics API
1636         https://bugs.webkit.org/show_bug.cgi?id=136592
1637
1638         Reviewed by Gavin Barraclough.
1639
1640         We do this by tracking "size" and "capacity" in the VM heap.
1641
1642         The VM heap's "capacity" is all the VM we ever allocated.
1643
1644         The VM heap's "size" the subset of VM currently held onto by the
1645         VM heap (and therefore not in use by the regular heap).
1646
1647         Somewhat ironically, reducing the process's memory footprint, increases
1648         the size of the VM heap, since the VM heap holds the pages that are
1649         purely virtual and not physical.
1650
1651         * bmalloc/Heap.cpp:
1652         (bmalloc::Heap::size):
1653         (bmalloc::Heap::capacity):
1654         * bmalloc/Heap.h:
1655         * bmalloc/VMHeap.cpp:
1656         (bmalloc::VMHeap::VMHeap):
1657         (bmalloc::VMHeap::allocateSmallChunk):
1658         (bmalloc::VMHeap::allocateMediumChunk):
1659         (bmalloc::VMHeap::allocateLargeChunk):
1660         * bmalloc/VMHeap.h:
1661         (bmalloc::VMHeap::size):
1662         (bmalloc::VMHeap::capacity):
1663         (bmalloc::VMHeap::allocateSmallPage):
1664         (bmalloc::VMHeap::allocateMediumPage):
1665         (bmalloc::VMHeap::allocateLargeRange):
1666         (bmalloc::VMHeap::deallocateSmallPage):
1667         (bmalloc::VMHeap::deallocateMediumPage):
1668         (bmalloc::VMHeap::deallocateLargeRange):
1669         * bmalloc/bmalloc.h:
1670         (bmalloc::api::heapSize):
1671         (bmalloc::api::heapCapacity):
1672
1673 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1674
1675         bmalloc crashes on the EWS bots (due to bad large object allocation)
1676         https://bugs.webkit.org/show_bug.cgi?id=136469
1677
1678         Reviewed by Andreas Kling.
1679
1680         It's possible to convince bmalloc to perform a bad large object allocation,
1681         through these steps:
1682
1683         (1) Insert object A into freelist F0.
1684
1685         (2) Split, merge and split again A's neighbors such that object B is
1686         inserted into freelist F0, with boundary tag and size equal to object A,
1687         but pointer not completely equal to object A. Put object B at the head of F0.
1688
1689         (3) Allocate some other object from F0, swapping its position in the
1690         freelist with object B, such that object A is now ahead of object B.
1691
1692         --> Now, the next allocation for size A/B will allocate object A, which
1693         has a slightly wrong idea about where the object actually begins.
1694         Immediately, you'll corrupt a little memory, and over time, you'll also
1695         corrupt boundary tag metadata.
1696
1697         The solution is to store the begin pointer in the boundary tag. Luckily,
1698         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1699         on MallocBench.
1700
1701         * bmalloc/Algorithm.h:
1702         (bmalloc::rightShift):
1703         * bmalloc/BeginTag.h:
1704         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1705         validate the start pointer when popping off the free list. Through a
1706         very uncommon set of steps, it is possible to have an item in the free
1707         list that is valid by all accounts except for its start pointer.
1708
1709         * bmalloc/BoundaryTag.h:
1710         (bmalloc::BoundaryTag::compactBegin):
1711         (bmalloc::BoundaryTag::setRange):
1712         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1713         start pointer. We don't need the whole pointer -- just the offset, in
1714         largeAlignment increments, into the relevant boundary tag bucket.
1715
1716         * bmalloc/BoundaryTagInlines.h:
1717         (bmalloc::validateNext):
1718         (bmalloc::BoundaryTag::init):
1719         (bmalloc::BoundaryTag::mergeLarge):
1720         (bmalloc::BoundaryTag::splitLarge):
1721         * bmalloc/SegregatedFreeList.cpp:
1722         (bmalloc::SegregatedFreeList::insert):
1723         (bmalloc::SegregatedFreeList::takeGreedy):
1724         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1725         the size when establishing a boundary tag, as required by the new
1726         interface.
1727
1728         * bmalloc/Sizes.h:
1729
1730 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1731
1732         Fixed a bmalloc crash seen on the EWS bot
1733         https://bugs.webkit.org/show_bug.cgi?id=135955
1734
1735         Reviewed by Andreas Kling.
1736
1737         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1738         input buffer is a malloc region, that region will get marked Copy-On-Write
1739         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1740         on older OS X's. In 10.10, they still fail, but they do not return
1741         EINVAL.
1742
1743         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1744
1745 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1746
1747         Fixed the bmalloc build
1748         https://bugs.webkit.org/show_bug.cgi?id=135953
1749
1750         Reviewed by Andreas Kling.
1751
1752         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1753         These headers are used, so they must be available outside the project.
1754
1755 2014-08-13  Daniel Bates  <dabates@apple.com>
1756
1757         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1758         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1759
1760         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1761         implementation of PerThread<T>::getFastCase().
1762
1763         * bmalloc/PerThread.h:
1764         (bmalloc::PerThread<T>::getFastCase):
1765
1766 2014-08-13  Daniel Bates  <dabates@apple.com>
1767
1768         Make bmalloc::PerThread work without C++ thread local storage
1769         https://bugs.webkit.org/show_bug.cgi?id=135895
1770
1771         Reviewed by Geoffrey Garen.
1772
1773         Implement support for building bmalloc without C++ thread local storage.
1774
1775         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1776         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1777         to determine whether the compiler supports C++ thread local storage.
1778         * bmalloc/PerThread.h:
1779         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1780         without C++ thread local storage.
1781         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1782         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1783         PerThreadStorage::initSharedKeyIfNeeded().
1784         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1785         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1786         ensure that the shared key has been initialized.
1787         (_pthread_setspecific_direct): Deleted.
1788         (_pthread_getspecific_direct): Deleted.
1789
1790 2014-08-13  Daniel Bates  <dabates@apple.com>
1791
1792         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1793         https://bugs.webkit.org/show_bug.cgi?id=135848
1794
1795         Reviewed by Geoffrey Garen.
1796
1797         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1798         when building for the iOS Simulator.
1799         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1800         using the public SDK.
1801         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1802         using the public SDK.
1803         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1804         using the public SDK.
1805
1806 2014-08-12  Daniel Bates  <dabates@apple.com>
1807
1808         BPLATFORM(IOS) always evaluates to false
1809         https://bugs.webkit.org/show_bug.cgi?id=135843
1810
1811         Reviewed by Geoffrey Garen.
1812
1813         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1814         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1815         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1816         building for iOS.
1817
1818         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1819         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1820
1821         * bmalloc/BPlatform.h:
1822
1823 2014-08-11  Andy Estes  <aestes@apple.com>
1824
1825         [iOS] Get rid of iOS.xcconfig
1826         https://bugs.webkit.org/show_bug.cgi?id=135809
1827
1828         Reviewed by Joseph Pecoraro.
1829
1830         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1831
1832         * Configurations/Base.xcconfig:
1833         * Configurations/iOS.xcconfig: Removed.
1834         * bmalloc.xcodeproj/project.pbxproj:
1835
1836 2014-05-01  Dan Bernstein  <mitz@apple.com>
1837
1838         Fixed production builds for the iOS Simulator.
1839         <rdar://problem/16792221>
1840
1841         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1842         PRIVATE_HEADERS_FOLDER_PATH when installing.
1843
1844 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1845
1846         bmalloc: Segregate pages by objects size
1847         https://bugs.webkit.org/show_bug.cgi?id=131909
1848
1849         Reviewed by Andreas Kling.
1850
1851         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1852
1853         * bmalloc/Allocator.cpp:
1854         (bmalloc::Allocator::allocateSlowCase):
1855         * bmalloc/Allocator.h:
1856         (bmalloc::Allocator::allocateFastCase):
1857         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1858         function for size class calculation.
1859
1860         * bmalloc/Deallocator.cpp:
1861         (bmalloc::Deallocator::Deallocator):
1862         (bmalloc::Deallocator::scavenge):
1863         (bmalloc::Deallocator::deallocateSmallLine):
1864         (bmalloc::Deallocator::allocateSmallLine):
1865         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1866         cache can't be shared anymore.
1867
1868         * bmalloc/Heap.cpp:
1869         (bmalloc::Heap::allocateSmallLineSlowCase):
1870         * bmalloc/Heap.h:
1871         (bmalloc::Heap::deallocateSmallLine): Ditto.
1872
1873         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1874         page refcount when allocating a line because we might have deallocated
1875         the page and the recycled it for another size class.
1876
1877         (bmalloc::Heap::deallocateMediumLine):
1878         (bmalloc::Heap::allocateMediumLine):
1879         * bmalloc/Line.h:
1880         (bmalloc::Line::refCount):
1881         * bmalloc/Page.h:
1882         (bmalloc::Page::refCount):
1883         (bmalloc::Page::smallSizeClass):
1884         (bmalloc::Page::setSmallSizeClass):
1885         (bmalloc::Page<Traits>::refCount): Deleted.
1886         * bmalloc/Sizes.h:
1887         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1888         an index into an array from a size.
1889
1890 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1891
1892         bmalloc: Improved alignment in LargeChunk
1893         https://bugs.webkit.org/show_bug.cgi?id=131895
1894
1895         Reviewed by Andreas Kling.
1896
1897         * bmalloc/Chunk.h:
1898         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1899         Technically, the previous alignment was harmless, but I would prefer,
1900         dear reader, not to have to explain the interlocking set of
1901         circumstances that made it so.
1902
1903 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1904
1905         Rolled out r167502 because it caused a crash on the facebook benchmark.
1906
1907         Unreviewed.
1908
1909             bmalloc: Added an XSmall line size
1910             https://bugs.webkit.org/show_bug.cgi?id=131851
1911
1912             Reviewed by Sam Weinig.
1913
1914 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1915
1916         bmalloc: Mutex should be harder to use wrong
1917         https://bugs.webkit.org/show_bug.cgi?id=131879
1918
1919         Reviewed by Andreas Kling.
1920
1921         Mutex now has a proper constructor, so you can't deadlock by forgetting
1922         to initialize it.
1923
1924         * bmalloc.xcodeproj/project.pbxproj:
1925         * bmalloc/Allocator.cpp:
1926         (bmalloc::Allocator::processXSmallAllocatorLog):
1927         (bmalloc::Allocator::processSmallAllocatorLog):
1928         (bmalloc::Allocator::processMediumAllocatorLog):
1929         (bmalloc::Allocator::allocateLarge):
1930         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1931         since the Heap mutex is a static.
1932
1933         * bmalloc/AsyncTask.h:
1934         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1935         need for explicit initialization anymore.
1936
1937         * bmalloc/Deallocator.cpp:
1938         (bmalloc::Deallocator::scavenge):
1939         (bmalloc::Deallocator::deallocateLarge):
1940         (bmalloc::Deallocator::deallocateXLarge):
1941         (bmalloc::Deallocator::processObjectLog):
1942         (bmalloc::Deallocator::deallocateSmallLine):
1943         (bmalloc::Deallocator::deallocateXSmallLine):
1944         (bmalloc::Deallocator::allocateSmallLine):
1945         (bmalloc::Deallocator::allocateXSmallLine):
1946         (bmalloc::Deallocator::deallocateMediumLine):
1947         (bmalloc::Deallocator::allocateMediumLine):
1948         * bmalloc/Deallocator.h:
1949         * bmalloc/Heap.cpp:
1950         (bmalloc::sleep):
1951         (bmalloc::Heap::Heap):
1952         (bmalloc::Heap::concurrentScavenge):
1953         (bmalloc::Heap::scavenge):
1954         (bmalloc::Heap::scavengeSmallPages):
1955         (bmalloc::Heap::scavengeXSmallPages):
1956         (bmalloc::Heap::scavengeMediumPages):
1957         (bmalloc::Heap::scavengeLargeRanges):
1958         (bmalloc::Heap::allocateXSmallLineSlowCase):
1959         (bmalloc::Heap::allocateSmallLineSlowCase):
1960         (bmalloc::Heap::allocateMediumLineSlowCase):
1961         (bmalloc::Heap::allocateXLarge):
1962         (bmalloc::Heap::deallocateXLarge):
1963         (bmalloc::Heap::allocateLarge):
1964         (bmalloc::Heap::deallocateLarge):
1965         * bmalloc/Heap.h:
1966         (bmalloc::Heap::deallocateXSmallLine):
1967         (bmalloc::Heap::allocateXSmallLine):
1968         (bmalloc::Heap::deallocateSmallLine):
1969         (bmalloc::Heap::allocateSmallLine):
1970         (bmalloc::Heap::deallocateMediumLine):
1971         (bmalloc::Heap::allocateMediumLine):
1972         * bmalloc/Line.h:
1973         (bmalloc::Line<Traits>::deref):
1974         * bmalloc/Mutex.cpp: Removed.
1975         * bmalloc/Mutex.h:
1976         (bmalloc::Mutex::Mutex):
1977         (bmalloc::Mutex::init): Deleted.
1978         (bmalloc::Mutex::try_lock): Deleted.
1979         (bmalloc::Mutex::lock): Deleted.
1980         (bmalloc::Mutex::unlock): Deleted.
1981         * bmalloc/Page.h:
1982         (bmalloc::Page<Traits>::ref):
1983         (bmalloc::Page<Traits>::deref):
1984         (bmalloc::Page<Traits>::refCount):
1985         * bmalloc/PerProcess.h:
1986         (bmalloc::PerProcess::mutex):
1987         (bmalloc::PerProcess<T>::getSlowCase):
1988         * bmalloc/StaticMutex.cpp: Added.
1989         (bmalloc::StaticMutex::lockSlowCase):
1990         * bmalloc/StaticMutex.h: Added.
1991         (bmalloc::StaticMutex::init):
1992         (bmalloc::StaticMutex::try_lock):
1993         (bmalloc::StaticMutex::lock):
1994         (bmalloc::StaticMutex::unlock):
1995         * bmalloc/VMHeap.h:
1996         (bmalloc::VMHeap::deallocateXSmallPage):
1997         (bmalloc::VMHeap::deallocateSmallPage):
1998         (bmalloc::VMHeap::deallocateMediumPage):
1999         (bmalloc::VMHeap::deallocateLargeRange):
2000         * bmalloc/bmalloc.h:
2001         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
2002         since the Heap mutex is a static.
2003
2004 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
2005
2006         bmalloc: AsyncTask should use Mutex instead of std::mutex
2007         https://bugs.webkit.org/show_bug.cgi?id=131865
2008
2009         Reviewed by Gavin Barraclough.
2010
2011         std::mutex is so slow that it makes parallelizing simple tasks through
2012         AsyncTask a net regression. Mutex fixes this.
2013
2014         * bmalloc/AsyncTask.h:
2015         (bmalloc::Function>::AsyncTask):
2016         (bmalloc::Function>::join):
2017         (bmalloc::Function>::runSlowCase):
2018         (bmalloc::Function>::entryPoint):
2019         * bmalloc/Mutex.h:
2020         (bmalloc::Mutex::init):
2021
2022 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
2023
2024         bmalloc: Added an XSmall line size
2025         https://bugs.webkit.org/show_bug.cgi?id=131851
2026
2027         Reviewed by Sam Weinig.
2028
2029         Reduces malloc footprint on Membuster recordings by 10%.
2030
2031         This is a throughput regression, but we're still way ahead of TCMalloc.
2032         I have some ideas for how to recover the regression -- but I wanted to
2033         get this win in first.
2034
2035         Full set of benchmark results:
2036
2037                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2038                                                                                                 
2039                                                                nopatch                      patch                                Δ
2040                 Peak Memory:
2041                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
2042                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
2043                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
2044
2045                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
2046                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
2047                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
2048
2049                 Memory at End:
2050                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
2051                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
2052                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
2053
2054                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
2055                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
2056                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
2057
2058                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2059                                                                                                 
2060                                                    nopatch                patch                         Δ
2061                 Execution Time:
2062                     churn                            127ms                151ms            ! 1.19x slower
2063                     list_allocate                    130ms                164ms            ! 1.26x slower
2064                     tree_allocate                    109ms                127ms            ! 1.17x slower
2065                     tree_churn                       115ms                120ms            ! 1.04x slower
2066                     facebook                         240ms                259ms            ! 1.08x slower
2067                     fragment                          91ms                131ms            ! 1.44x slower
2068                     fragment_iterate                 105ms                106ms            ! 1.01x slower
2069                     message_one                      260ms                259ms             ^ 1.0x faster
2070                     message_many                     149ms                154ms            ! 1.03x slower
2071                     medium                           194ms                248ms            ! 1.28x slower
2072                     big                              157ms                160ms            ! 1.02x slower
2073
2074                     <geometric mean>                 144ms                163ms            ! 1.13x slower
2075                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
2076                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
2077
2078                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2079                                                                                                 
2080                                                                        nopatch                          patch                                     Δ
2081                 Execution Time:
2082                     churn                                                126ms                          148ms                        ! 1.17x slower
2083                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
2084                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
2085                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
2086                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
2087                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
2088                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
2089                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
2090                     facebook                                             240ms                          276ms                        ! 1.15x slower
2091                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
2092                     fragment                                              92ms                          130ms                        ! 1.41x slower
2093                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
2094                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
2095                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
2096                     message_one                                          260ms                          260ms                                      
2097                     message_many                                         170ms                          238ms                         ! 1.4x slower
2098                     medium                                               185ms                          250ms                        ! 1.35x slower
2099                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
2100                     big                                                  150ms                          169ms                        ! 1.13x slower
2101                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
2102
2103                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
2104                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
2105                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
2106
2107                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2108
2109                                                                     TC                      patch                                Δ
2110                 Peak Memory:
2111                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
2112                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
2113                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
2114
2115                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
2116                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
2117                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
2118
2119                 Memory at End:
2120                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
2121                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
2122                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
2123
2124                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
2125                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
2126                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
2127
2128                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2129                                                                                                 
2130                                                         TC                patch                         Δ
2131                 Execution Time:
2132                     churn                            416ms                148ms            ^ 2.81x faster
2133                     list_allocate                    463ms                164ms            ^ 2.82x faster
2134                     tree_allocate                    292ms                127ms             ^ 2.3x faster
2135                     tree_churn                       157ms                120ms            ^ 1.31x faster
2136                     facebook                         327ms                276ms            ^ 1.18x faster
2137                     fragment                         335ms                129ms             ^ 2.6x faster
2138                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
2139                     message_one                      386ms                258ms             ^ 1.5x faster
2140                     message_many                     410ms                154ms            ^ 2.66x faster
2141                     medium                           391ms                245ms             ^ 1.6x faster
2142                     big                              261ms                167ms            ^ 1.56x faster
2143
2144                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
2145                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
2146                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
2147
2148         * bmalloc.xcodeproj/project.pbxproj:
2149         * bmalloc/Allocator.cpp:
2150         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
2151         index corresponds with its size. Instead, use the size selection function
2152         explicitly. Now that we have XSmall, some small allocator entries are
2153         unused.
2154
2155         (bmalloc::Allocator::scavenge):
2156         (bmalloc::Allocator::log):
2157         (bmalloc::Allocator::processXSmallAllocatorLog):
2158         (bmalloc::Allocator::allocateSlowCase):
2159         * bmalloc/Allocator.h:
2160         (bmalloc::Allocator::xSmallAllocatorFor):
2161         (bmalloc::Allocator::allocateFastCase):
2162         * bmalloc/Chunk.h:
2163         * bmalloc/Deallocator.cpp:
2164         (bmalloc::Deallocator::scavenge):
2165         (bmalloc::Deallocator::processObjectLog):
2166         (bmalloc::Deallocator::deallocateSlowCase):
2167         (bmalloc::Deallocator::deallocateXSmallLine):
2168         (bmalloc::Deallocator::allocateXSmallLine):
2169         * bmalloc/Deallocator.h:
2170         (bmalloc::Deallocator::deallocateFastCase):
2171         * bmalloc/Heap.cpp:
2172         (bmalloc::Heap::scavenge):
2173         (bmalloc::Heap::scavengeXSmallPages):
2174         (bmalloc::Heap::allocateXSmallLineSlowCase):
2175         * bmalloc/Heap.h:
2176         (bmalloc::Heap::deallocateXSmallLine):
2177         (bmalloc::Heap::allocateXSmallLine):
2178         * bmalloc/LargeChunk.h:
2179         (bmalloc::LargeChunk::get):
2180         (bmalloc::LargeChunk::endTag):
2181         * bmalloc/Line.h:
2182         * bmalloc/MediumAllocator.h:
2183         (bmalloc::MediumAllocator::allocate):
2184         (bmalloc::MediumAllocator::refill):
2185         * bmalloc/ObjectType.cpp:
2186         (bmalloc::objectType):
2187         * bmalloc/ObjectType.h:
2188         (bmalloc::isXSmall):
2189         (bmalloc::isSmall):
2190         (bmalloc::isMedium):
2191         (bmalloc::isLarge):
2192         (bmalloc::isSmallOrMedium): Deleted.
2193         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
2194         handling small objects. There's probably a reasonable way to share this
2195         code in the future -- I'll look into that once it's stopped changing.
2196
2197         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
2198         is the main reason things got slower.
2199
2200         * bmalloc/SmallAllocator.h:
2201         (bmalloc::SmallAllocator::allocate):
2202         * bmalloc/SmallTraits.h:
2203         * bmalloc/VMHeap.cpp:
2204         (bmalloc::VMHeap::allocateXSmallChunk):
2205         * bmalloc/VMHeap.h:
2206         (bmalloc::VMHeap::allocateXSmallPage):
2207         (bmalloc::VMHeap::deallocateXSmallPage):
2208         * bmalloc/XSmallAllocator.h: Added.
2209         (bmalloc::XSmallAllocator::isNull):
2210         (bmalloc::XSmallAllocator::canAllocate):
2211         (bmalloc::XSmallAllocator::XSmallAllocator):
2212         (bmalloc::XSmallAllocator::line):
2213         (bmalloc::XSmallAllocator::allocate):
2214         (bmalloc::XSmallAllocator::objectCount):
2215         (bmalloc::XSmallAllocator::derefCount):
2216         (bmalloc::XSmallAllocator::refill):
2217         * bmalloc/XSmallChunk.h: Added.
2218         * bmalloc/XSmallLine.h: Added.
2219         * bmalloc/XSmallPage.h: Added.
2220         * bmalloc/XSmallTraits.h: Added.
2221         * bmalloc/bmalloc.h:
2222         (bmalloc::api::realloc): Boiler-plate copy, as above.
2223
2224 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2225
2226         MallocBench should scavenge explicitly instead of waiting
2227         https://bugs.webkit.org/show_bug.cgi?id=131661
2228
2229         Reviewed by Andreas Kling.
2230
2231         Added explicit scavenge support to bmalloc. This isn't a memory win,
2232         since bmalloc's per-thread cache is so small. But it makes testing
2233         simpler.
2234
2235         * bmalloc/Allocator.cpp:
2236         (bmalloc::Allocator::~Allocator):
2237         (bmalloc::Allocator::scavenge):
2238         * bmalloc/Allocator.h:
2239         * bmalloc/Cache.cpp:
2240         (bmalloc::Cache::operator new):
2241         (bmalloc::Cache::operator delete):
2242         (bmalloc::Cache::Cache):
2243         (bmalloc::Cache::scavenge):
2244         * bmalloc/Cache.h:
2245         * bmalloc/Deallocator.cpp:
2246         (bmalloc::Deallocator::~Deallocator):
2247         (bmalloc::Deallocator::scavenge):
2248         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
2249         functions, for reuse.
2250
2251         * bmalloc/Heap.cpp:
2252         (bmalloc::sleep):
2253         (bmalloc::Heap::concurrentScavenge):
2254         (bmalloc::Heap::scavenge):
2255         (bmalloc::Heap::scavengeSmallPages):
2256         (bmalloc::Heap::scavengeMediumPages):
2257         (bmalloc::Heap::scavengeLargeRanges):
2258         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
2259         scavenging -- in response to a benchmark or a low memory warning --
2260         wants to complete as soon as possible, so its sleep duration is 0.
2261
2262         * bmalloc/bmalloc.h:
2263         (bmalloc::api::scavenge):
2264         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
2265
2266 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2267
2268         Use 4kB pages on Mac
2269         https://bugs.webkit.org/show_bug.cgi?id=131658
2270
2271         Reviewed by Sam Weinig.
2272
2273         This reduces memory use a lot on Membuster:
2274
2275                                                                   base                      patch                                Δ
2276                 Execution Time:
2277                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
2278                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
2279                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
2280
2281                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
2282                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
2283                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
2284
2285                 Peak Memory:
2286                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
2287                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
2288                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
2289
2290                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
2291                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
2292                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
2293
2294                 Memory at End:
2295                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
2296                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
2297                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
2298
2299                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
2300                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
2301                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
2302
2303
2304         * bmalloc.xcodeproj/project.pbxproj:
2305         * bmalloc/BPlatform.h: Added.
2306         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
2307         is 4kB on Mac.
2308
2309 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
2310
2311         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
2312
2313         * bmalloc.xcodeproj: Modified property svn:ignore.
2314
2315 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
2316
2317         Fixed some mbmalloc exports
2318         https://bugs.webkit.org/show_bug.cgi?id=131599
2319
2320         Reviewed by Ryosuke Niwa.
2321
2322         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
2323         of the project, so we can call them from API.
2324
2325         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
2326         visibility, so they show up as exported in the .dylib.
2327
2328 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
2329
2330         Put bmalloc headers in the right place
2331         https://bugs.webkit.org/show_bug.cgi?id=131464
2332
2333         Reviewed by Mark Rowe.
2334
2335         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
2336         specify that we don't just want to dump all of our generically-named
2337         headers into /usr/local/include.
2338
2339 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2340
2341         Made bmalloc more #include friendly
2342         https://bugs.webkit.org/show_bug.cgi?id=131386
2343
2344         Reviewed by Andreas Kling.
2345
2346         Marked a bunch of headers private so they can be used from client code
2347         that #includes bmalloc.h.
2348
2349         Renamed ASSERT macros to BASSERT. This matches their header, which already
2350         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
2351
2352         * bmalloc.xcodeproj/project.pbxproj:
2353         * bmalloc/Allocator.cpp:
2354         (bmalloc::Allocator::allocateSlowCase):
2355         * bmalloc/AsyncTask.h:
2356         (bmalloc::Function>::runSlowCase):
2357         * bmalloc/BAssert.h:
2358         * bmalloc/BoundaryTag.h:
2359         (bmalloc::BoundaryTag::setSize):
2360         * bmalloc/BoundaryTagInlines.h:
2361         (bmalloc::validate):
2362         (bmalloc::BoundaryTag::init):
2363         (bmalloc::BoundaryTag::deallocate):
2364         (bmalloc::BoundaryTag::splitLarge):
2365         (bmalloc::BoundaryTag::allocate):
2366         * bmalloc/Chunk.h:
2367         * bmalloc/Deallocator.cpp:
2368         (bmalloc::Deallocator::processObjectLog):
2369         (bmalloc::Deallocator::deallocateSlowCase):
2370         * bmalloc/Deallocator.h:
2371         (bmalloc::Deallocator::deallocateFastCase):
2372         * bmalloc/FixedVector.h:
2373         (bmalloc::Capacity>::operator):
2374         (bmalloc::Capacity>::push):
2375         (bmalloc::Capacity>::pop):
2376         (bmalloc::Capacity>::shrink):
2377         * bmalloc/Heap.cpp:
2378         (bmalloc::Heap::allocateLarge):
2379         * bmalloc/LargeChunk.h:
2380         (bmalloc::LargeChunk::get):
2381         (bmalloc::LargeChunk::endTag):
2382         * bmalloc/Line.h:
2383         (bmalloc::Line<Traits>::concurrentRef):
2384         (bmalloc::Line<Traits>::deref):
2385         * bmalloc/MediumAllocator.h:
2386         (bmalloc::MediumAllocator::allocate):
2387         * bmalloc/ObjectType.h:
2388         (bmalloc::isSmall):
2389         * bmalloc/Page.h:
2390         (bmalloc::Page<Traits>::ref):
2391         (bmalloc::Page<Traits>::deref):
2392         * bmalloc/PerThread.h:
2393         (bmalloc::PerThread<T>::getSlowCase):
2394         * bmalloc/SegregatedFreeList.cpp:
2395         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2396         (bmalloc::SegregatedFreeList::insert):
2397         * bmalloc/SmallAllocator.h:
2398         (bmalloc::SmallAllocator::allocate):
2399         (bmalloc::SmallAllocator::refill):
2400         * bmalloc/Syscall.h:
2401         * bmalloc/VMAllocate.h:
2402         (bmalloc::vmValidate):
2403         (bmalloc::vmAllocate):
2404         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2405         * bmalloc/Vector.h:
2406         (bmalloc::Vector<T>::operator):
2407         (bmalloc::Vector<T>::pop):
2408         (bmalloc::Vector<T>::shrink):
2409         * bmalloc/XLargeChunk.h:
2410         (bmalloc::XLargeChunk::range):
2411         (bmalloc::XLargeChunk::size):
2412
2413 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2414
2415         Removed an unused file.
2416
2417         Unreviewed.
2418
2419         * bmalloc/AsyncTask.cpp: Removed.
2420
2421 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2422
2423         Build bmalloc on Mac
2424         https://bugs.webkit.org/show_bug.cgi?id=131333
2425
2426         Reviewed by Mark Rowe.
2427
2428         * Makefile: Added. For make clients.
2429
2430         These files are required for building any project in WebKit. I copied
2431         them from WTF:
2432         * Configurations: Added.
2433         * Configurations/Base.xcconfig: Added.
2434         * Configurations/DebugRelease.xcconfig: Added.
2435         * Configurations/bmalloc.xcconfig: Added.
2436         * Configurations/iOS.xcconfig: Added.
2437         * Configurations/mbmalloc.xcconfig: Added.
2438
2439         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
2440         from here because everything is in .xcconfig files now.
2441
2442         I had to fix a bunch of minor warnings, since they're enabled in our
2443         .xcconfig files:
2444
2445         * bmalloc/AsyncTask.h:
2446         (bmalloc::Function>::AsyncTask):
2447         * bmalloc/BAssert.h:
2448         * bmalloc/BoundaryTagInlines.h:
2449         (bmalloc::validate):
2450         * bmalloc/Heap.cpp:
2451         (bmalloc::Heap::Heap):
2452         (bmalloc::Heap::allocateLarge):
2453         (bmalloc::Heap::deallocateLarge):
2454         * bmalloc/Mutex.h:
2455         (bmalloc::Mutex::Mutex): Deleted.
2456         * bmalloc/VMAllocate.h:
2457         (bmalloc::vmValidate):
2458         * bmalloc/mbmalloc.cpp:
2459
2460 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2461
2462         bmalloc: Fixed a leak in the per-thread cache
2463         https://bugs.webkit.org/show_bug.cgi?id=131330
2464
2465         Reviewed by Andreas Kling.
2466
2467         Remember to deallocate our line caches upon thread exit.
2468
2469         * bmalloc/Deallocator.cpp:
2470         (bmalloc::Deallocator::~Deallocator):
2471
2472 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2473
2474         bmalloc: rolled out the tryLock experiment
2475         https://bugs.webkit.org/show_bug.cgi?id=131328
2476
2477         Reviewed by Andreas Kling.
2478
2479         It wasn't a speedup.
2480
2481         * bmalloc.xcodeproj/project.pbxproj:
2482         * bmalloc/Allocator.cpp:
2483         (bmalloc::Allocator::processSmallAllocatorLog):
2484         (bmalloc::Allocator::processMediumAllocatorLog):
2485         * bmalloc/Deallocator.cpp:
2486         (bmalloc::Deallocator::processObjectLog):
2487         (bmalloc::Deallocator::deallocateSlowCase):
2488         (bmalloc::Deallocator::deallocateSmallLine):
2489         (bmalloc::Deallocator::deallocateMediumLine):
2490         * bmalloc/Deallocator.h:
2491         (bmalloc::Deallocator::deallocateFastCase):
2492         * bmalloc/Heap.h:
2493         (bmalloc::Heap::deallocateSmallLine):
2494         (bmalloc::Heap::deallocateMediumLine):
2495         * bmalloc/Line.h:
2496         (bmalloc::Line<Traits>::deref):
2497         * bmalloc/Page.h:
2498         (bmalloc::Page<Traits>::deref):
2499
2500 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2501
2502         bmalloc
2503         https://bugs.webkit.org/show_bug.cgi?id=131170
2504
2505         Reviewed by Andreas Kling.
2506
2507         Initial commit.
2508
2509         * bmalloc: Added.
2510         * bmalloc.xcodeproj: Added.
2511         * bmalloc.xcodeproj/project.pbxproj: Added.
2512         * bmalloc/Algorithm.h: Added.
2513         (bmalloc::max):
2514         (bmalloc::min):
2515         (bmalloc::mask):
2516         (bmalloc::test):
2517         (bmalloc::roundUpToMultipleOf):
2518         (bmalloc::roundDownToMultipleOf):
2519         (bmalloc::sizeOf):
2520         (bmalloc::bitCount):
2521         (bmalloc::isPowerOfTwo):
2522         * bmalloc/Allocator.cpp: Added.
2523         (bmalloc::Allocator::Allocator):
2524         (bmalloc::Allocator::~Allocator):
2525         (bmalloc::Allocator::log):
2526         (bmalloc::Allocator::processSmallAllocatorLog):
2527         (bmalloc::Allocator::processMediumAllocatorLog):
2528         (bmalloc::Allocator::allocateLarge):
2529         (bmalloc::Allocator::allocateXLarge):
2530         (bmalloc::Allocator::allocateMedium):
2531         (bmalloc::Allocator::allocateSlowCase):
2532         * bmalloc/Allocator.h: Added.
2533         (bmalloc::Allocator::smallAllocatorFor):
2534         (bmalloc::Allocator::allocateFastCase):
2535         (bmalloc::Allocator::allocate):
2536         * bmalloc/AsyncTask.cpp: Added.
2537         (bmalloc::AsyncTask<Function>::runSlowCase):
2538         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2539         (bmalloc::AsyncTask<Function>::entryPoint):
2540         * bmalloc/AsyncTask.h: Added.
2541         (bmalloc::Function>::AsyncTask):
2542         (bmalloc::Function>::join):
2543         (bmalloc::Function>::run):
2544         (bmalloc::Function>::runSlowCase):
2545         (bmalloc::Function>::pthreadEntryPoint):
2546         (bmalloc::Function>::entryPoint):
2547         * bmalloc/BAssert.h: Added.
2548         * bmalloc/BeginTag.h: Added.
2549         (bmalloc::BeginTag::isInFreeList):
2550         * bmalloc/BoundaryTag.h: Added.
2551         (bmalloc::BoundaryTag::isXLarge):
2552         (bmalloc::BoundaryTag::setXLarge):
2553         (bmalloc::BoundaryTag::isFree):
2554         (bmalloc::BoundaryTag::setFree):
2555         (bmalloc::BoundaryTag::isEnd):
2556         (bmalloc::BoundaryTag::setEnd):
2557         (bmalloc::BoundaryTag::hasPhysicalPages):
2558         (bmalloc::BoundaryTag::setHasPhysicalPages):
2559         (bmalloc::BoundaryTag::isNull):
2560         (bmalloc::BoundaryTag::clear):
2561         (bmalloc::BoundaryTag::size):
2562         (bmalloc::BoundaryTag::setSize):
2563         (bmalloc::BoundaryTag::prev):
2564         (bmalloc::BoundaryTag::next):
2565         * bmalloc/BoundaryTagInlines.h: Added.
2566         (bmalloc::validate):
2567         (bmalloc::validatePrev):
2568         (bmalloc::validateNext):
2569         (bmalloc::BoundaryTag::init):
2570         (bmalloc::BoundaryTag::mergeLargeLeft):
2571         (bmalloc::BoundaryTag::mergeLargeRight):
2572         (bmalloc::BoundaryTag::mergeLarge):
2573         (bmalloc::BoundaryTag::deallocate):
2574         (bmalloc::BoundaryTag::splitLarge):
2575         (bmalloc::BoundaryTag::allocate):
2576         * bmalloc/Cache.cpp: Added.
2577         (bmalloc::Cache::operator new):
2578         (bmalloc::Cache::operator delete):
2579         (bmalloc::Cache::Cache):
2580         (bmalloc::Cache::allocateSlowCase):
2581         (bmalloc::Cache::allocateSlowCaseNullCache):
2582         (bmalloc::Cache::deallocateSlowCase):
2583         (bmalloc::Cache::deallocateSlowCaseNullCache):
2584         * bmalloc/Cache.h: Added.
2585         (bmalloc::Cache::allocator):
2586         (bmalloc::Cache::deallocator):
2587         (bmalloc::Cache::allocateFastCase):
2588         (bmalloc::Cache::deallocateFastCase):
2589         (bmalloc::Cache::allocate):
2590         (bmalloc::Cache::deallocate):
2591         * bmalloc/Chunk.h: Added.
2592         (bmalloc::Chunk::begin):
2593         (bmalloc::Chunk::end):
2594         (bmalloc::Chunk::lines):
2595         (bmalloc::Chunk::pages):
2596         * bmalloc/Deallocator.cpp: Added.
2597         (bmalloc::Deallocator::Deallocator):
2598         (bmalloc::Deallocator::~Deallocator):
2599         (bmalloc::Deallocator::deallocateLarge):
2600         (bmalloc::Deallocator::deallocateXLarge):
2601         (bmalloc::Deallocator::processObjectLog):
2602         (bmalloc::Deallocator::deallocateSlowCase):
2603         (bmalloc::Deallocator::deallocateSmallLine):
2604         (bmalloc::Deallocator::allocateSmallLine):
2605         (bmalloc::Deallocator::deallocateMediumLine):
2606         (bmalloc::Deallocator::allocateMediumLine):
2607         * bmalloc/Deallocator.h: Added.
2608         (bmalloc::Deallocator::deallocateFastCase):
2609         (bmalloc::Deallocator::deallocate):
2610         * bmalloc/EndTag.h: Added.
2611         (bmalloc::EndTag::operator=):
2612         * bmalloc/FixedVector.h: Added.
2613         (bmalloc::FixedVector::begin):
2614         (bmalloc::FixedVector::end):
2615         (bmalloc::FixedVector::size):
2616         (bmalloc::FixedVector::capacity):
2617         (bmalloc::FixedVector::clear):
2618         (bmalloc::FixedVector::isEmpty):
2619         (bmalloc::Capacity>::FixedVector):
2620         (bmalloc::Capacity>::operator):
2621         (bmalloc::Capacity>::push):
2622         (bmalloc::Capacity>::pop):
2623         (bmalloc::Capacity>::shrink):
2624         * bmalloc/Heap.cpp: Added.
2625         (bmalloc::sleep):
2626         (bmalloc::Heap::Heap):
2627         (bmalloc::Heap::concurrentScavenge):
2628         (bmalloc::Heap::scavengeSmallPages):
2629         (bmalloc::Heap::scavengeMediumPages):
2630         (bmalloc::Heap::scavengeLargeRanges):
2631         (bmalloc::Heap::allocateSmallLineSlowCase):
2632         (bmalloc::Heap::allocateMediumLineSlowCase):
2633         (bmalloc::Heap::allocateXLarge):
2634         (bmalloc::Heap::deallocateXLarge):
2635         (bmalloc::Heap::allocateLarge):
2636         (bmalloc::Heap::deallocateLarge):
2637         * bmalloc/Heap.h: Added.
2638         (bmalloc::Heap::deallocateSmallLine):
2639         (bmalloc::Heap::allocateSmallLine):
2640         (bmalloc::Heap::deallocateMediumLine):
2641         (bmalloc::Heap::allocateMediumLine):
2642         * bmalloc/Inline.h: Added.
2643         * bmalloc/LargeChunk.h: Added.
2644         (bmalloc::LargeChunk::begin):
2645         (bmalloc::LargeChunk::end):
2646         (bmalloc::LargeChunk::create):
2647         (bmalloc::LargeChunk::get):
2648         (bmalloc::LargeChunk::beginTag):
2649         (bmalloc::LargeChunk::endTag):
2650         * bmalloc/Line.h: Added.
2651         (bmalloc::Line<Traits>::begin):
2652         (bmalloc::Line<Traits>::end):
2653         (bmalloc::Line<Traits>::concurrentRef):
2654         (bmalloc::Line<Traits>::deref):
2655         * bmalloc/MediumAllocator.h: Added.
2656         (bmalloc::MediumAllocator::isNull):
2657         (bmalloc::MediumAllocator::MediumAllocator):
2658         (bmalloc::MediumAllocator::line):
2659         (bmalloc::MediumAllocator::allocate):
2660         (bmalloc::MediumAllocator::derefCount):
2661         (bmalloc::MediumAllocator::refill):
2662         * bmalloc/MediumChunk.h: Added.
2663         * bmalloc/MediumLine.h: Added.
2664         * bmalloc/MediumPage.h: Added.
2665         * bmalloc/MediumTraits.h: Added.
2666         * bmalloc/Mutex.cpp: Added.
2667         (bmalloc::Mutex::lockSlowCase):
2668         * bmalloc/Mutex.h: Added.
2669         (bmalloc::Mutex::Mutex):
2670         (bmalloc::Mutex::try_lock):
2671         (bmalloc::Mutex::lock):
2672         (bmalloc::Mutex::unlock):
2673         * bmalloc/ObjectType.cpp: Added.
2674         (bmalloc::objectType):
2675         * bmalloc/ObjectType.h: Added.
2676         (bmalloc::isSmallOrMedium):
2677         (bmalloc::isSmall):
2678         * bmalloc/Page.h: Added.
2679         (bmalloc::Page<Traits>::ref):
2680         (bmalloc::Page<Traits>::deref):
2681         (bmalloc::Page<Traits>::refCount):
2682         * bmalloc/PerProcess.h: Added.
2683         (bmalloc::PerProcess::mutex):
2684         (bmalloc::PerProcess<T>::getFastCase):
2685         (bmalloc::PerProcess<T>::get):
2686         (bmalloc::PerProcess<T>::getSlowCase):
2687         * bmalloc/PerThread.h: Added.
2688         (bmalloc::PerThreadStorage<Cache>::get):
2689         (bmalloc::PerThreadStorage<Cache>::init):
2690         (bmalloc::PerThreadStorage::get):
2691         (bmalloc::PerThreadStorage::init):
2692         (bmalloc::PerThread<T>::getFastCase):
2693         (bmalloc::PerThread<T>::get):
2694         (bmalloc::PerThread<T>::destructor):
2695         (bmalloc::PerThread<T>::getSlowCase):
2696         * bmalloc/Range.h: Added.
2697         (bmalloc::Range::Range):
2698         (bmalloc::Range::begin):
2699         (bmalloc::Range::end):
2700         (bmalloc::Range::size):
2701         (bmalloc::Range::operator!):
2702         (bmalloc::Range::operator<):
2703         * bmalloc/SegregatedFreeList.cpp: Added.
2704         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2705         (bmalloc::SegregatedFreeList::insert):
2706         (bmalloc::SegregatedFreeList::takeGreedy):
2707         (bmalloc::SegregatedFreeList::take):
2708         * bmalloc/SegregatedFreeList.h: Added.
2709         * bmalloc/Sizes.h: Added.
2710         * bmalloc/SmallAllocator.h: Added.
2711         (bmalloc::SmallAllocator::isNull):
2712         (bmalloc::SmallAllocator::canAllocate):
2713         (bmalloc::SmallAllocator::SmallAllocator):
2714         (bmalloc::SmallAllocator::line):
2715         (bmalloc::SmallAllocator::allocate):
2716         (bmalloc::SmallAllocator::objectCount):
2717         (bmalloc::SmallAllocator::derefCount):
2718         (bmalloc::SmallAllocator::refill):
2719         * bmalloc/SmallChunk.h: Added.
2720         * bmalloc/SmallLine.h: Added.
2721         * bmalloc/SmallPage.h: Added.
2722         * bmalloc/SmallTraits.h: Added.
2723         * bmalloc/Syscall.h: Added.
2724         * bmalloc/VMAllocate.h: Added.
2725         (bmalloc::vmSize):
2726         (bmalloc::vmValidate):
2727         (bmalloc::vmAllocate):
2728         (bmalloc::vmDeallocate):
2729         (bmalloc::vmDeallocatePhysicalPages):
2730         (bmalloc::vmAllocatePhysicalPages):
2731         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2732         (bmalloc::vmAllocatePhysicalPagesSloppy):
2733         * bmalloc/VMHeap.cpp: Added.
2734         (bmalloc::VMHeap::VMHeap):
2735         (bmalloc::VMHeap::allocateSmallChunk):
2736         (bmalloc::VMHeap::allocateMediumChunk):
2737         (bmalloc::VMHeap::allocateLargeChunk):
2738         * bmalloc/VMHeap.h: Added.
2739         (bmalloc::VMHeap::allocateSmallPage):
2740         (bmalloc::VMHeap::allocateMediumPage):
2741         (bmalloc::VMHeap::allocateLargeRange):
2742         (bmalloc::VMHeap::deallocateSmallPage):
2743         (bmalloc::VMHeap::deallocateMediumPage):
2744         (bmalloc::VMHeap::deallocateLargeRange):
2745         * bmalloc/Vector.h: Added.
2746         (bmalloc::Vector::begin):
2747         (bmalloc::Vector::end):
2748         (bmalloc::Vector::size):
2749         (bmalloc::Vector::capacity):
2750         (bmalloc::Vector::last):
2751         (bmalloc::Vector::pop):
2752         (bmalloc::Vector<T>::Vector):
2753         (bmalloc::Vector<T>::~Vector):
2754         (bmalloc::Vector<T>::operator):
2755         (bmalloc::Vector<T>::push):
2756         (bmalloc::Vector<T>::pop):
2757         (bmalloc::Vector<T>::shrink):
2758         (bmalloc::Vector<T>::reallocateBuffer):
2759         (bmalloc::Vector<T>::shrinkCapacity):
2760         (bmalloc::Vector<T>::growCapacity):
2761         * bmalloc/XLargeChunk.h: Added.
2762         (bmalloc::XLargeChunk::get):
2763         (bmalloc::XLargeChunk::begin):
2764         (bmalloc::XLargeChunk::XLargeChunk):
2765         (bmalloc::XLargeChunk::create):
2766         (bmalloc::XLargeChunk::destroy):
2767         (bmalloc::XLargeChunk::range):
2768         (bmalloc::XLargeChunk::size):
2769         * bmalloc/bmalloc.h: Added.
2770         (bmalloc::api::malloc):
2771         (bmalloc::api::free):
2772         (bmalloc::api::realloc):
2773         * bmalloc/mbmalloc.cpp: Added.
2774