Progress towards CMake on Mac.
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-03-13  Alex Christensen  <achristensen@webkit.org>
2
3         Progress towards CMake on Mac.
4         https://bugs.webkit.org/show_bug.cgi?id=142680
5
6         Reviewed by Gyuyoung Kim.
7
8         * CMakeLists.txt:
9         * PlatformMac.cmake:
10         Added Zone.cpp to Mac CMake builds.
11
12 2015-03-12  Geoffrey Garen  <ggaren@apple.com>
13
14         Assertion failure in bmalloc::LargeObject::validateSelf on Mavericks Debug layout test bot
15         https://bugs.webkit.org/show_bug.cgi?id=142642
16
17         Reviewed by Michael Saboff.
18
19         The typical backtrace to this crash shows the main thread trying to
20         realloc a large string while a DFG compiler thread tries to
21         free a large vector buffer.
22
23         I believe that this is a race condition -- at least in debug builds --
24         since the main thread will try to validate its object's neighbors
25         without holding a lock, even though those neighbors might be in the
26         midst of changing.
27
28         In general, there may be sneaky times when it is valid to look at an
29         object's metadata without holding the heap lock, but it is best not to
30         do so unless we have a really really good reason to.
31
32         * bmalloc/Allocator.cpp:
33         (bmalloc::Allocator::reallocate): Take a lock before reading the metadata
34         for this object, since we generally require any access to shared heap
35         metadata to take a lock.
36
37 2015-03-10  Geoffrey Garen  <ggaren@apple.com>
38
39         bmalloc: tryFastMalloc shouldn't crash
40         https://bugs.webkit.org/show_bug.cgi?id=142443
41
42         Reviewed by Sam Weinig.
43
44         Rolling back in r181307 with a check for whether bmalloc is enabled, to
45         avoid crashes when running with ASan and GuardMalloc.
46
47         * bmalloc/Allocator.cpp:
48         (bmalloc::Allocator::tryAllocate):
49         * bmalloc/Allocator.h:
50         * bmalloc/Cache.cpp:
51         (bmalloc::Cache::tryAllocateSlowCaseNullCache):
52         * bmalloc/Cache.h:
53         (bmalloc::Cache::tryAllocate):
54         * bmalloc/Heap.cpp:
55         (bmalloc::Heap::allocateXLarge):
56         (bmalloc::Heap::tryAllocateXLarge):
57         * bmalloc/Heap.h:
58         * bmalloc/VMAllocate.h:
59         (bmalloc::tryVMAllocate):
60         (bmalloc::vmAllocate):
61         * bmalloc/bmalloc.h:
62         (bmalloc::api::tryMalloc):
63         (bmalloc::api::realloc):
64         (bmalloc::api::free):
65
66 2015-03-09  Commit Queue  <commit-queue@webkit.org>
67
68         Unreviewed, rolling out r181307.
69         https://bugs.webkit.org/show_bug.cgi?id=142525
70
71         Broke ASan tests (Requested by ap on #webkit).
72
73         Reverted changeset:
74
75         "bmalloc: tryFastMalloc shouldn't crash"
76         https://bugs.webkit.org/show_bug.cgi?id=142443
77         http://trac.webkit.org/changeset/181307
78
79 2015-03-09  Geoffrey Garen  <ggaren@apple.com>
80
81         bmalloc: tryFastMalloc shouldn't crash
82         https://bugs.webkit.org/show_bug.cgi?id=142443
83
84         Reviewed by Darin Adler.
85
86         Added support for tryMalloc.
87
88         We assume that non-x-large allocations always succeed, and we crash
89         otherwise, since normal allocation failure will just cause the next
90         non-try allocation or internal metadata allocation to fail, and it's
91         hard and not really useful to keep limping along after that. But
92         extra-large allocations can meaningfully fail, and we can recover.
93
94         * bmalloc/Heap.cpp:
95         (bmalloc::Heap::allocateXLarge):
96         (bmalloc::Heap::tryAllocateXLarge):
97         * bmalloc/Heap.h: Added support for non-crashy x-large allocation.
98
99         * bmalloc/VMAllocate.h:
100         (bmalloc::tryVMAllocate):
101         (bmalloc::vmAllocate): Added support for non-crashy VM allocation.
102
103         * bmalloc/bmalloc.h:
104         (bmalloc::api::tryMalloc):
105         (bmalloc::api::realloc):
106         (bmalloc::api::free): Tried to clarify our behavior with some comments.
107         Unfortunately, calling what we do "malloc" is still not quite right, since
108         malloc returns null on failure and we don't.
109
110 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
111
112         bmalloc: Don't branch when setting the owner of a large object
113         https://bugs.webkit.org/show_bug.cgi?id=142241
114
115         Reviewed by Andreas Kling.
116
117         * bmalloc/BoundaryTag.h:
118         (bmalloc::BoundaryTag::owner):
119         (bmalloc::BoundaryTag::setOwner):
120
121 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
122
123         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
124         https://bugs.webkit.org/show_bug.cgi?id=141802
125
126         Reviewed by Andreas Kling.
127
128         Re-enabled this feature on iOS, now that the iOS crash should be fixed.
129
130         * bmalloc/VMHeap.cpp:
131         (bmalloc::VMHeap::grow):
132         * bmalloc/VMHeap.h:
133
134 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
135
136         bmalloc: Added missing features to the malloc zone introspection API
137         https://bugs.webkit.org/show_bug.cgi?id=142235
138
139         Reviewed by Andreas Kling.
140
141         This should fix the crash we saw on the iOS PLT bot
142         (c.f. http://trac.webkit.org/changeset/180604).
143
144         * bmalloc/Zone.cpp:
145         (bmalloc::good_size):
146         (bmalloc::check):
147         (bmalloc::print):
148         (bmalloc::log):
149         (bmalloc::force_lock):
150         (bmalloc::force_unlock):
151         (bmalloc::statistics):
152         (bmalloc::size):
153         (bmalloc::enumerator): Provide all of these functions since they are called
154         indiscriminately on all zones.
155
156         (bmalloc::Zone::Zone):
157         (bmalloc::Zone::size): Deleted.
158         (bmalloc::Zone::enumerator): Deleted. Moved these functions out of the
159         Zone class since they can stand alone.
160
161         * bmalloc/Zone.h:
162
163 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
164
165         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
166         https://bugs.webkit.org/show_bug.cgi?id=141802
167
168         Reviewed by Andreas Kling.
169
170         Rolling back in but disabled on iOS until I can debug why the iOS PLT crashes.
171
172         * bmalloc/VMHeap.cpp:
173         (bmalloc::VMHeap::grow):
174         * bmalloc/VMHeap.h:
175         * bmalloc/Zone.cpp:
176         (bmalloc::Zone::size):
177         (bmalloc::Zone::Zone):
178         * bmalloc/Zone.h:
179
180 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
181
182         bmalloc: Miscellaneous cleanup
183         https://bugs.webkit.org/show_bug.cgi?id=142231
184
185         Reviewed by Andreas Kling.
186
187         No performance change -- maybe a tiny reduction in memory use.
188
189         * bmalloc/Heap.cpp: Moved the sleep function into StaticMutex, since
190         it's a helper for working with mutexes.
191
192         (bmalloc::Heap::scavenge): Make sure to wait before we start any
193         scavenging, since individual scavenging functions now always scavenge
194         at least one page before waiting themselves.
195
196         (bmalloc::Heap::scavengeSmallPages):
197         (bmalloc::Heap::scavengeMediumPages):
198         (bmalloc::Heap::scavengeLargeObjects): Use the new wait helper to
199         simplify this code. Also, we now require our caller to wait until at
200         least one deallocation is desirable. This simplifies our loop.
201
202         (bmalloc::Heap::allocateSmallPage):
203         (bmalloc::Heap::allocateMediumPage):
204         (bmalloc::Heap::allocateXLarge):
205         (bmalloc::Heap::allocateLarge): Don't freak out any time the heap does
206         an allocation. Only consider the heap to be growing if it actually needs
207         to allocate new VM. This allows us to shrink the heap back down from a
208         high water mark more reliably even if heap activity continues.
209
210         (bmalloc::sleep): Deleted.
211         (bmalloc::Heap::scavengeLargeRanges): Renamed to match our use of
212         "LargeObject".
213
214         * bmalloc/Heap.h:
215
216         * bmalloc/LargeObject.h:
217         (bmalloc::LargeObject::operator bool): Added to simplify a while loop.
218
219         * bmalloc/StaticMutex.h:
220         (bmalloc::sleep):
221         (bmalloc::waitUntilFalse): New helper for waiting until a condition
222         becomes reliably false.
223
224         * bmalloc/Vector.h:
225         (bmalloc::Vector<T>::~Vector): Oops! Don't deallocate the null pointer.
226         We don't actually run any Vector destructors, but an iteration of this
227         patch did, and then crashed. So, let's fix that.
228
229 2015-03-02  Geoffrey Garen  <ggaren@apple.com>
230
231         bmalloc: Eagerly remove allocated objects from the free list
232         https://bugs.webkit.org/show_bug.cgi?id=142194
233
234         Reviewed by Andreas Kling.
235
236         This reduces the pressure to garbage collect the free list.
237
238         Might be a 1% speedup on MallocBench.
239
240         * bmalloc/FreeList.cpp: Put this comment at the top of the file instead
241         of repeating it inside of each function. Tried to clarify the details.
242
243         (bmalloc::FreeList::takeGreedy): Matched the other iteration code in this
244         file for consistency -- even though either direction works fine in this
245         function.
246
247         (bmalloc::FreeList::take): Change to iterate from low to high so that we
248         can maintain an index into the vector that is not disturbed even if we
249         pop from the middle (which invalidates the last index in the vector).
250
251         Decrement i when popping from the middle to make sure that we don't
252         skip the next item after popping.
253
254         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): Ditto.
255
256 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
257
258         Fixed a typo in the previous commit.
259
260         * bmalloc/BoundaryTag.h:
261         (bmalloc::BoundaryTag::setOwner):
262
263 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
264
265         EFL build fix after r180797.
266
267         * bmalloc/BoundaryTag.h:
268         (bmalloc::BoundaryTag::owner):
269         (bmalloc::BoundaryTag::setOwner):
270
271 2015-02-27  Geoffrey Garen  <ggaren@apple.com>
272
273         bmalloc: Pathological madvise churn on the free(malloc(x)) benchmark
274         https://bugs.webkit.org/show_bug.cgi?id=142058
275
276         Reviewed by Andreas Kling.
277
278         The churn was caused by repeatedly splitting an object with physical
279         pages from an object without, and then merging them back together again.
280         The merge would conservatively forget that we had physical pages, forcing
281         a new call to madvise on the next allocation.
282
283         This patch more strictly segregates objects in the heap from objects in
284         the VM heap, with these changes:
285
286         (1) Objects in the heap are not allowed to merge with objects in the VM
287         heap, and vice versa -- since that would erase our precise knowledge of
288         which physical pages had been allocated.
289
290         (2) The VM heap is exclusively responsible for allocating and deallocating
291         physical pages.
292
293         (3) The heap free list must consider entries for objects that are in the
294         VM heap to be invalid, and vice versa. (This condition can arise
295         because the free list does not eagerly remove items.)
296
297         With these changes, we can know that any valid object in the heap's free
298         list already has physical pages, and does not need to call madvise.
299
300         Note that the VM heap -- as before -- might sometimes contain ranges
301         or pieces of ranges that have physical pages, since we allow splitting
302         of ranges at granularities smaller than the VM page size. These ranges
303         can eventually merge with ranges in the heap during scavenging.
304
305         * bmalloc.xcodeproj/project.pbxproj:
306
307         * bmalloc/BoundaryTag.h:
308         (bmalloc::BoundaryTag::owner):
309         (bmalloc::BoundaryTag::setOwner):
310         (bmalloc::BoundaryTag::initSentinel):
311         (bmalloc::BoundaryTag::hasPhysicalPages): Deleted.
312         (bmalloc::BoundaryTag::setHasPhysicalPages): Deleted. Replaced the concept
313         of "has physical pages" with a bit indicating which heap owns the large
314         object. This is a more precise concept, since the old bit was really a
315         Yes / Maybe bit.
316
317         * bmalloc/Deallocator.cpp:
318
319         * bmalloc/FreeList.cpp: Adopt
320         (bmalloc::FreeList::takeGreedy):
321         (bmalloc::FreeList::take):
322         (bmalloc::FreeList::removeInvalidAndDuplicateEntries):
323         * bmalloc/FreeList.h:
324         (bmalloc::FreeList::push): Added API for considering the owner when
325         deciding if a free list entry is valid.
326
327         * bmalloc/Heap.cpp:
328         (bmalloc::Heap::Heap): Adopt new API.
329
330         (bmalloc::Heap::scavengeLargeRanges): Scavenge all ranges with no minimum,
331         since some ranges might be able to merge with ranges in the VM heap, and
332         they won't be allowed to until we scavenge them.
333
334         (bmalloc::Heap::allocateSmallPage):
335         (bmalloc::Heap::allocateMediumPage):
336         (bmalloc::Heap::allocateLarge): New VM heap API makes this function
337         simpler, since we always get back physical pages now.
338
339         * bmalloc/Heap.h:
340         * bmalloc/LargeObject.h:
341         (bmalloc::LargeObject::end):
342         (bmalloc::LargeObject::owner):
343         (bmalloc::LargeObject::setOwner):
344         (bmalloc::LargeObject::isValidAndFree):
345         (bmalloc::LargeObject::merge): Do not merge objects across heaps since
346         that causes madvise churn.
347         (bmalloc::LargeObject::validateSelf):
348         (bmalloc::LargeObject::init):
349         (bmalloc::LargeObject::hasPhysicalPages): Deleted.
350         (bmalloc::LargeObject::setHasPhysicalPages): Deleted. Propogate the Owner API.
351
352         * bmalloc/Owner.h: Added.
353
354         * bmalloc/SegregatedFreeList.cpp:
355         (bmalloc::SegregatedFreeList::SegregatedFreeList):
356         (bmalloc::SegregatedFreeList::insert):
357         (bmalloc::SegregatedFreeList::takeGreedy):
358         (bmalloc::SegregatedFreeList::take):
359         * bmalloc/SegregatedFreeList.h: Propogate the owner API.
360
361         * bmalloc/VMAllocate.h:
362         (bmalloc::vmDeallocatePhysicalPagesSloppy):
363         (bmalloc::vmAllocatePhysicalPagesSloppy): Clarified these functions and
364         removed an edge case.
365
366         * bmalloc/VMHeap.cpp:
367         (bmalloc::VMHeap::VMHeap):
368         * bmalloc/VMHeap.h:
369         (bmalloc::VMHeap::allocateSmallPage):
370         (bmalloc::VMHeap::allocateMediumPage):
371         (bmalloc::VMHeap::allocateLargeObject):
372         (bmalloc::VMHeap::deallocateLargeObject): Be sure to give each object
373         a new chance to merge, since it might have been prohibited from merging
374         before by virtue of not being in the VM heap.
375
376         (bmalloc::VMHeap::allocateLargeRange): Deleted.
377         (bmalloc::VMHeap::deallocateLargeRange): Deleted.
378
379 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
380
381         bmalloc: Large object free list can grow infinitely
382         https://bugs.webkit.org/show_bug.cgi?id=142055
383
384         Reviewed by Andreas Kling.
385
386         By design, we don't eagerly remove large objects from the free list.
387         This creates two simple pathologies:
388
389             (1) If you free and then allocate the same object repeatedly, it will
390             duplicate itself in the free list repeatedly. Since it is never
391             invalid at the time of allocation, it will never be removed.
392
393             (2) If you split and then merge the same object repeatedly, it will
394             duplicate its split sibling in the free list repeatedly. If its
395             sibling is in a separate free list size class, it will never be
396             consulted at the time of allocation, so it will never be removed.
397
398         So, a simple "while (1) { free(malloc(x)); }" causes infinite memory
399         use in the free list.
400
401         The solution in this patch is a simple helper to remove garbage from the
402         free list if it grows too large. This pathology is not common, so the
403         cost is OK.
404
405         Long-term, perhaps we should rethink the laziness of these free lists.
406
407         * bmalloc/BoundaryTag.h:
408         (bmalloc::BoundaryTag::isMarked):
409         (bmalloc::BoundaryTag::setMarked): New bit, used by free list GC.
410
411         * bmalloc/FreeList.cpp:
412         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): The GC algorithm.
413
414         * bmalloc/FreeList.h:
415         (bmalloc::FreeList::FreeList):
416         (bmalloc::FreeList::push): Invoke the GC if we're getting huge.
417
418         * bmalloc/LargeObject.h:
419         (bmalloc::LargeObject::isMarked):
420         (bmalloc::LargeObject::setMarked):
421         (bmalloc::LargeObject::validateSelf): Expose the new bit.
422
423         * bmalloc/Sizes.h: New constant to control GC frequency.
424
425 2015-02-26  Csaba Osztrogonác  <ossy@webkit.org>
426
427         URTBF after r180693.
428
429         * CMakeLists.txt:
430
431 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
432
433         Try to fix the Mac build.
434
435         Unreviewed.
436
437         * bmalloc.xcodeproj/project.pbxproj: Make FreeList.h available.
438
439 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
440
441         bmalloc: Refactored SegregatedFreeList and BoundaryTag::init
442         https://bugs.webkit.org/show_bug.cgi?id=142049
443
444         Reviewed by Anders Carlsson.
445
446         Split out a FreeList class from SegregatedFreeList. This will make it
447         easier to add behaviors on free list insertion and removal -- and it's
448         probably how I should have designed things at the start.
449
450         Moved BoundaryTag::init into LargeObject, since all the related logic
451         lives in LargeObject now too, and this allows us to remove BoundaryTagInlines.h.
452
453         * bmalloc.xcodeproj/project.pbxproj:
454         * bmalloc/BoundaryTagInlines.h: Removed.
455         * bmalloc/FreeList.cpp: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.cpp.
456         (bmalloc::FreeList::takeGreedy):
457         (bmalloc::FreeList::take):
458         (bmalloc::SegregatedFreeList::SegregatedFreeList): Deleted.
459         (bmalloc::SegregatedFreeList::insert): Deleted.
460         (bmalloc::SegregatedFreeList::takeGreedy): Deleted.
461         (bmalloc::SegregatedFreeList::take): Deleted.
462         * bmalloc/FreeList.h: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.h.
463         (bmalloc::FreeList::push):
464         * bmalloc/LargeObject.h:
465         (bmalloc::LargeObject::init):
466         * bmalloc/SegregatedFreeList.cpp:
467         (bmalloc::SegregatedFreeList::SegregatedFreeList):
468         (bmalloc::SegregatedFreeList::insert):
469         (bmalloc::SegregatedFreeList::takeGreedy):
470         (bmalloc::SegregatedFreeList::take):
471         * bmalloc/SegregatedFreeList.h:
472         * bmalloc/Sizes.h:
473         * bmalloc/VMHeap.cpp:
474         (bmalloc::VMHeap::grow):
475
476 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
477
478         bmalloc: free up a bit in BoundaryTag
479         https://bugs.webkit.org/show_bug.cgi?id=142048
480
481         Reviewed by Brady Eidson.
482
483         We were wasting a bit by accident, and I need one now.
484
485         * bmalloc/Algorithm.h:
486         (bmalloc::rightShift): Deleted. Not needed, now that I've simplified
487         the math.
488
489         * bmalloc/BoundaryTag.h: Since each boundary tag bucket is 1024 bytes
490         long, the maximum offset into a bucket is 1023.
491
492         You need 5 bits to count up to 1024, but only 4 to count up to 1023.
493         
494         Math is hard.
495
496         (bmalloc::BoundaryTag::compactBegin): Switched to division because it
497         is simpler, and easier to match up with our ASSERT. The compiler will
498         turn division by constant power of two into a shift for us.
499
500         (bmalloc::BoundaryTag::setRange): Added an ASSERT for compactBegin
501         because we do encode it, so we should ASSERT that encoding did not
502         lose information.
503
504         * bmalloc/Sizes.h: Shifting is no longer used since we use division
505         instead.
506
507 2015-02-24  Stephanie Lewis  <slewis@apple.com>
508
509         Rolling out http://trac.webkit.org/changeset/180430 as it causes the PLT to crash.
510         <rdar://problem/19948015>
511
512         Unreviewed.
513
514         * bmalloc/VMHeap.cpp:
515         (bmalloc::VMHeap::grow):
516         * bmalloc/VMHeap.h:
517         * bmalloc/Zone.cpp:
518         (bmalloc::Zone::Zone):
519         (bmalloc::Zone::size): Deleted.
520         * bmalloc/Zone.h:
521
522 2015-02-24  Geoffrey Garen  <ggaren@apple.com>
523
524         bmalloc: Added a little more abstraction for large objects
525         https://bugs.webkit.org/show_bug.cgi?id=141978
526
527         Reviewed by Sam Weinig.
528
529         Previously, each client needed to manage the boundary tags of
530         a large object using free functions. This patch introduces a LargeObject
531         class that does things a little more automatically.
532
533         * bmalloc.xcodeproj/project.pbxproj:
534
535         * bmalloc/Allocator.cpp:
536         (bmalloc::Allocator::reallocate): Use the new LargeObject class.
537
538         * bmalloc/BeginTag.h:
539         (bmalloc::BeginTag::isInFreeList): Deleted. Moved this logic into the
540         LargeObject class.
541
542         * bmalloc/BoundaryTag.h:
543         (bmalloc::BoundaryTag::isSentinel):
544         (bmalloc::BoundaryTag::compactBegin):
545         (bmalloc::BoundaryTag::setRange):
546         (bmalloc::BoundaryTag::initSentinel): Added an explicit API for sentinels,
547         which we used to create and test for implicitly.
548
549         * bmalloc/BoundaryTagInlines.h:
550         (bmalloc::BoundaryTag::init):
551         (bmalloc::validate): Deleted.
552         (bmalloc::validatePrev): Deleted.
553         (bmalloc::validateNext): Deleted.
554         (bmalloc::BoundaryTag::mergeLeft): Deleted.
555         (bmalloc::BoundaryTag::mergeRight): Deleted.
556         (bmalloc::BoundaryTag::merge): Deleted.
557         (bmalloc::BoundaryTag::deallocate): Deleted.
558         (bmalloc::BoundaryTag::split): Deleted.
559         (bmalloc::BoundaryTag::allocate): Deleted. Moved this logic into the
560         LargeObject class.
561
562         * bmalloc/EndTag.h:
563         (bmalloc::EndTag::init):
564         (bmalloc::EndTag::operator=): Deleted. Re-reading this code, I found
565         special behavior in the assignment operator to be a surprising API.
566         So, I replaced the assignment operation with an explicit initializing
567         function.
568
569         * bmalloc/Heap.cpp:
570         (bmalloc::Heap::scavengeLargeRanges):
571         (bmalloc::Heap::allocateXLarge):
572         (bmalloc::Heap::findXLarge):
573         (bmalloc::Heap::deallocateXLarge):
574         (bmalloc::Heap::allocateLarge):
575         (bmalloc::Heap::deallocateLarge):
576         * bmalloc/Heap.h: No behavior changes here -- just adopting the
577         LargeObject interface.
578
579         * bmalloc/LargeObject.h: Added.
580         (bmalloc::LargeObject::operator!):
581         (bmalloc::LargeObject::begin):
582         (bmalloc::LargeObject::size):
583         (bmalloc::LargeObject::range):
584         (bmalloc::LargeObject::LargeObject):
585         (bmalloc::LargeObject::setFree):
586         (bmalloc::LargeObject::isFree):
587         (bmalloc::LargeObject::hasPhysicalPages):
588         (bmalloc::LargeObject::setHasPhysicalPages):
589         (bmalloc::LargeObject::isValidAndFree):
590         (bmalloc::LargeObject::merge):
591         (bmalloc::LargeObject::split):
592         (bmalloc::LargeObject::validateSelf):
593         (bmalloc::LargeObject::validate): Moved this code into a class, out of
594         BoundaryTag free functions.
595
596         New to the class are these features:
597
598             (1) Every reference to an object is validated upon creation and use.
599
600             (2) There's an explicit API for "This is a reference to an object
601             that might be stale (the DoNotValidate API)".
602
603             (3) The begin and end tags are kept in sync automatically.
604
605         * bmalloc/SegregatedFreeList.cpp:
606         (bmalloc::SegregatedFreeList::insert):
607         (bmalloc::SegregatedFreeList::takeGreedy):
608         (bmalloc::SegregatedFreeList::take):
609         * bmalloc/SegregatedFreeList.h: Adopt the LargeObject interface.
610
611         * bmalloc/VMHeap.cpp:
612         (bmalloc::VMHeap::grow):
613         * bmalloc/VMHeap.h:
614         (bmalloc::VMHeap::allocateLargeRange):
615         (bmalloc::VMHeap::deallocateLargeRange): Adopt the LargeObject interface.
616
617 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
618
619         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
620         https://bugs.webkit.org/show_bug.cgi?id=141802
621
622         Reviewed by Andreas Kling.
623
624         Rolling back in with a fix for a crash seen while using GuardMalloc.
625
626         * bmalloc/VMHeap.cpp:
627         (bmalloc::VMHeap::grow):
628         * bmalloc/VMHeap.h:
629         * bmalloc/Zone.cpp: Re-land the old patch.
630
631         (bmalloc::Zone::size): Be sure to implement the size() function since
632         it's accessible indirectly via the malloc_zone_from_ptr public API --
633         and GuardMalloc calls it all the time.
634
635         (bmalloc::Zone::Zone):
636         * bmalloc/Zone.h: Re-land the old patch.
637
638 2015-02-19  Commit Queue  <commit-queue@webkit.org>
639
640         Unreviewed, rolling out r180363.
641         https://bugs.webkit.org/show_bug.cgi?id=141814
642
643         Caused >50 crashes when running LayoutTests in GuardMalloc or
644         ASAN modes. (Requested by jernoble on #webkit).
645
646         Reverted changeset:
647
648         "bmalloc should implement malloc introspection (to stop false-
649         positive leaks when MallocStackLogging is off)"
650         https://bugs.webkit.org/show_bug.cgi?id=141802
651         http://trac.webkit.org/changeset/180363
652
653 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
654
655         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
656         https://bugs.webkit.org/show_bug.cgi?id=141802
657
658         Reviewed by Andreas Kling.
659
660         Fixed a last-minute type.
661
662         The macro is OS, not PLATFORM.
663
664         * bmalloc/VMHeap.cpp:
665         (bmalloc::VMHeap::grow):
666         * bmalloc/VMHeap.h:
667         * bmalloc/Zone.h:
668
669 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
670
671         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
672         https://bugs.webkit.org/show_bug.cgi?id=141802
673
674         Reviewed by Andreas Kling.
675
676         This patch does the bare minimum to stop false positive leaks from
677         being reported by the Darwin leaks tool. We register each super chunk
678         as a single object, and then request that the leaks tool scan it.
679
680         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
681         zone introspection API.
682
683         * bmalloc/Algorithm.h: Missing #include.
684
685         * bmalloc/VMHeap.cpp:
686         (bmalloc::VMHeap::grow):
687         * bmalloc/VMHeap.h: Adopt the new abstraction.
688
689         * bmalloc/Zone.cpp: Added.
690         (bmalloc::remoteRead): Helper for reading an object out of another process.
691         (bmalloc::Zone::enumerator):
692         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
693         in introspection.
694
695         * bmalloc/Zone.h: Added.
696         (bmalloc::Zone::superChunks):
697         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
698         since our dynamic allocations will not be scanned by leaks since they
699         will have the malloc VM tag.
700
701 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
702
703         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
704         https://bugs.webkit.org/show_bug.cgi?id=141759
705
706         Reviewed by Andreas Kling.
707
708         * bmalloc.xcodeproj/project.pbxproj:
709         * bmalloc/SuperChunk.h: Added.
710         (bmalloc::SuperChunk::create):
711         (bmalloc::SuperChunk::SuperChunk):
712         (bmalloc::SuperChunk::smallChunk):
713         (bmalloc::SuperChunk::mediumChunk):
714         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
715         into a separate class, for clarity and type safety.
716
717         * bmalloc/VMHeap.cpp:
718         (bmalloc::VMHeap::grow):
719         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
720         "grow" because Andreas found "allocateSuperChunk" to be unclear.
721
722         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
723         for malloc introspection.
724
725         (bmalloc::VMHeap::allocateSmallPage):
726         (bmalloc::VMHeap::allocateMediumPage):
727         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
728
729 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
730
731         Build bmalloc through CMake as a static library. It's then linked either
732         into the WTF library (if built as a shared library) or into the JSC and
733         WebKit2 libraries. There's no need to build it as a standalone shared library.
734
735         Rubber-stamped by Carlos Garcia Campos.
736
737         * CMakeLists.txt:
738
739 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
740
741         [BMalloc] Add a FIXME comment for memory alignas
742         https://bugs.webkit.org/show_bug.cgi?id=141556
743
744         Reviewed by Csaba Osztrogonác.
745
746         * bmalloc/Chunk.h: Add a FIXME comment.
747         * bmalloc/LargeChunk.h: ditto.
748
749 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
750
751         bmalloc buildfix on 32 bit Linux (x86/ARM)
752         https://bugs.webkit.org/show_bug.cgi?id=141472
753
754         Reviewed by Gyuyoung Kim.
755
756         * bmalloc/Algorithm.h:
757         (bmalloc::roundUpToMultipleOf):
758         * bmalloc/FixedVector.h:
759         (bmalloc::FixedVector::clear):
760         * bmalloc/Sizes.h:
761         (bmalloc::Sizes::sizeClass):
762
763 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
764
765         [EFL][GTK] Use bmalloc instead of tcmalloc
766         https://bugs.webkit.org/show_bug.cgi?id=140162
767
768         Reviewed by Carlos Garcia Campos.
769
770         Support to use bmalloc on EFL and GTK ports.
771
772         * CMakeLists.txt: Added.
773         * bmalloc/Allocator.cpp:
774         (bmalloc::Allocator::allocate):
775             Fix unused return value caused by posix_memalign().
776         * bmalloc/AsyncTask.h:
777         * bmalloc/BoundaryTag.h:
778         (bmalloc::BoundaryTag::clear):
779         * bmalloc/Chunk.h:
780             Change Traits::Page with Traits::PageType in order to fix
781             -fpermitive build error on EFL and GTK port. 
782         * bmalloc/EndTag.h:
783         (bmalloc::EndTag::operator=):
784         * bmalloc/Line.h: ditto.
785         * bmalloc/MediumTraits.h:
786         * bmalloc/Page.h: ditto.
787         * bmalloc/PerThread.h:
788             EFL port doesn't support __has_include definition yet.
789             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
790         * bmalloc/SmallTraits.h: ditto.
791         * bmalloc/VMAllocate.h:
792         (bmalloc::vmDeallocatePhysicalPages):
793         (bmalloc::vmAllocatePhysicalPages):
794         * bmalloc/Vector.h:
795         (bmalloc::Vector<T>::push):
796         (bmalloc::Vector<T>::reallocateBuffer):
797
798 2015-01-31  Sam Weinig  <sam@webkit.org>
799
800         Remove even more Mountain Lion support
801         https://bugs.webkit.org/show_bug.cgi?id=141124
802
803         Reviewed by Alexey Proskuryakov.
804
805         * Configurations/Base.xcconfig:
806         * Configurations/DebugRelease.xcconfig:
807
808 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
809
810         GC marking threads should clear malloc caches
811         https://bugs.webkit.org/show_bug.cgi?id=141097
812
813         Reviewed by Andreas Kling.
814
815         Split the scavenging API into per-thread vs global, so that you can
816         request to scavenge your own thread without scavenging the whole heap.
817
818         * bmalloc/Cache.cpp:
819         (bmalloc::Cache::scavenge):
820         * bmalloc/bmalloc.h:
821         (bmalloc::api::scavengeThisThread):
822         (bmalloc::api::scavenge):
823
824 2015-01-28  Dana Burkart  <dburkart@apple.com>
825
826         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
827         https://bugs.webkit.org/show_bug.cgi?id=136765
828
829         Reviewed by Alexey Proskuryakov.
830
831         * Configurations/Base.xcconfig:
832         * Configurations/DebugRelease.xcconfig:
833
834 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
835
836         bmalloc: support aligned allocation
837         https://bugs.webkit.org/show_bug.cgi?id=140732
838
839         Reviewed by Andreas Kling.
840
841         * bmalloc/Allocator.cpp:
842         (bmalloc::Allocator::allocate): New function for aligned allocation.
843
844         Small and medium requests just allocate and free until they find an
845         aligned pointer. This is slightly inefficient in the worst case, but
846         still constant-time with little-to-no space overhead.
847
848         Large requests use a new API that requires the client to specify both
849         its ideal size and alignment, and the worst-case size you would have to
850         allocate in order to produce some interior pointer of the requested size
851         and alignment. We put the burden of this calculation on the client
852         because it simplifies things if we guarantee that allocation won't fail.
853
854         XLarge requests are easy: we just forward them to vmAllocate, which
855         already supported aligned requests.
856
857         * bmalloc/BoundaryTag.h:
858         * bmalloc/BoundaryTagInlines.h:
859         (bmalloc::BoundaryTag::mergeLeft):
860         (bmalloc::BoundaryTag::mergeRight):
861         (bmalloc::BoundaryTag::merge):
862         (bmalloc::BoundaryTag::deallocate):
863         (bmalloc::BoundaryTag::split):
864         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
865         refactored the interface to remove some reference out parameters in
866         order to clarify what changes and what doesn't.
867
868         * bmalloc/Heap.cpp:
869         (bmalloc::Heap::allocateXLarge): Added an alignment API.
870
871         (bmalloc::Heap::allocateLarge):
872         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
873         a few variants, so aligned and unaligned allocation could share some code.
874
875         * bmalloc/SegregatedFreeList.cpp:
876         (bmalloc::SegregatedFreeList::take):
877         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
878         for aligned allocation. It turns out that the aligned path is pretty
879         different, since it ends up searching for two potential ways to satisfy
880         an allocation: either large enough and aligned, or large enough to split
881         into something not aligned and something large enough and aligned.
882
883         * bmalloc/VMAllocate.h:
884         (bmalloc::vmAllocate): Switched alignment to come before size because
885         that's how the memalign API specifies it.
886
887         * bmalloc/VMHeap.h:
888         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
889
890 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
891
892         bmalloc: a little bit of cleanup
893         https://bugs.webkit.org/show_bug.cgi?id=140687
894
895         Reviewed by Anders Carlsson.
896
897         * bmalloc/Algorithm.h:
898         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
899         of code.
900
901         * bmalloc/BoundaryTag.h:
902         * bmalloc/BoundaryTagInlines.h:
903         (bmalloc::BoundaryTag::mergeLeft):
904         (bmalloc::BoundaryTag::mergeRight):
905         (bmalloc::BoundaryTag::merge):
906         (bmalloc::BoundaryTag::deallocate):
907         (bmalloc::BoundaryTag::split):
908         (bmalloc::BoundaryTag::allocate):
909         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
910         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
911         (bmalloc::BoundaryTag::mergeLarge): Deleted.
912         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
913         from all these functions, since boundary tags always pertain to large
914         objects, and putting the word "Large" everywhere wasn't helping to
915         explain that.
916
917         * bmalloc/Heap.cpp:
918         (bmalloc::Heap::allocateXLarge):
919         (bmalloc::Heap::findXLarge):
920         (bmalloc::Heap::deallocateXLarge):
921         * bmalloc/Heap.h:
922         * bmalloc/VMHeap.h:
923         (bmalloc::VMHeap::allocateXLarge): Deleted.
924         (bmalloc::VMHeap::findXLarge): Deleted.
925         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
926         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
927         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
928         really make sense for the VMHeap to be involved.
929
930 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
931
932         bmalloc: refactored XLarge allocation for better alignment
933         https://bugs.webkit.org/show_bug.cgi?id=140582
934
935         Reviewed by Andreas Kling.
936
937         XLarge objects used to be Large objects with an extra bit of metadata
938         that said "actually, I'm not large -- I'm extra large".
939
940         The metadata header in an XLarge allocation made it impossible for the
941         XLarge object to honor a very large alignment request.
942
943         The solution is to stop using a metadata header for XLarge objects, and
944         instead to store explicit metadata on the side.
945
946         This is a bit less astonishing, which is also nice.
947
948         Finding XLarge metadata is now a linear search. That's probably OK, since
949         it was always so in TCMalloc, and the usual number of XLarge allocations
950         in a process is 0.
951
952         This design makes it possible for the heap to cache XLarge allocations
953         with and/or without physical pages. I haven't actually done that yet
954         because the tradeoffs are subtle, so I don't want to do anything without
955         a motivating test case.
956
957         * bmalloc.xcodeproj/project.pbxproj:
958         * bmalloc/Allocator.cpp:
959         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
960         since an XLarge allocation is now just a naked buffer without a header.
961
962         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
963         XLarge alignment, since XLargeChunk won't give this to us implicitly
964         anymore.
965
966         * bmalloc/BoundaryTag.h:
967         (bmalloc::BoundaryTag::setRange):
968         (bmalloc::BoundaryTag::isXLarge): Deleted.
969         (bmalloc::BoundaryTag::setXLarge): Deleted.
970         * bmalloc/BoundaryTagInlines.h:
971         (bmalloc::validate):
972         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
973
974         * bmalloc/Deallocator.cpp:
975         (bmalloc::Deallocator::deallocateXLarge):
976         (bmalloc::Deallocator::deallocateSlowCase):
977         * bmalloc/Heap.cpp:
978         (bmalloc::Heap::findXLarge):
979         (bmalloc::Heap::allocateXLarge):
980         (bmalloc::Heap::deallocateXLarge):
981         * bmalloc/Heap.h: Updated for interface changes.
982
983         * bmalloc/ObjectType.cpp:
984         (bmalloc::objectType):
985         * bmalloc/ObjectType.h:
986         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
987         examining its bit pattern -- just like we do for other kinds of
988         allocations -- which is nice.
989
990         * bmalloc/Sizes.h:
991         * bmalloc/VMHeap.h:
992         (bmalloc::VMHeap::allocateXLarge):
993         (bmalloc::VMHeap::findXLarge):
994         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
995         for XLarge allocations.
996
997         * bmalloc/XLargeChunk.h: Removed.
998
999 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
1000
1001         bmalloc: added some infrastructure for aligned allocation
1002         https://bugs.webkit.org/show_bug.cgi?id=140572
1003
1004         Reviewed by Andreas Kling.
1005
1006         * bmalloc/Algorithm.h:
1007         (bmalloc::isPowerOfTwo):
1008         (bmalloc::roundUpToMultipleOf):
1009         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
1010         isPowerOfTwo helper function.
1011
1012         * bmalloc/Allocator.cpp:
1013         (bmalloc::Allocator::allocate):
1014         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
1015         Doesn't do anything yet, but does correctly forward to system malloc
1016         when bmalloc is disabled.
1017
1018         * bmalloc/Cache.cpp:
1019         (bmalloc::Cache::allocateSlowCaseNullCache):
1020         * bmalloc/Cache.h:
1021         (bmalloc::Cache::allocate):
1022         * bmalloc/bmalloc.h:
1023         (bmalloc::api::memalign):
1024         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
1025
1026 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
1027
1028         Consider alignment when allocating from a SegregatedFreeList
1029         https://bugs.webkit.org/show_bug.cgi?id=140408
1030
1031         Reviewed by Sam Weinig.
1032
1033         In preparation for supporting aligned allocation.
1034
1035         No performance change.
1036
1037         Since this is just one extra branch in an already expensive function,
1038         I decided not to duplicate the function just to avoid the branch in
1039         the un-aligned case.
1040
1041         * bmalloc/SegregatedFreeList.cpp:
1042         (bmalloc::SegregatedFreeList::take):
1043         * bmalloc/SegregatedFreeList.h:
1044
1045 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
1046
1047         Renamed minimum to size in SegregatedFreeList
1048         https://bugs.webkit.org/show_bug.cgi?id=140406
1049
1050         Reviewed by Sam Weinig.
1051
1052         In preparation for supporting aligned allocation.
1053
1054         * bmalloc/SegregatedFreeList.cpp:
1055         (bmalloc::SegregatedFreeList::takeGreedy):
1056         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
1057         really just a minimum. Let's not imply that this value is special.
1058
1059 2015-01-11  Dan Bernstein  <mitz@apple.com>
1060
1061         Geoff is organized, but he is not an organization.
1062
1063         Rubber-stamped by Anders Carlsson.
1064
1065         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
1066
1067 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
1068
1069         Make bmalloc work with ASan
1070         https://bugs.webkit.org/show_bug.cgi?id=140194
1071
1072         Reviewed by Mark Lam.
1073
1074         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
1075         an OS-specific API to test for loaded runtime libraries.
1076
1077         * bmalloc/Environment.cpp:
1078         (bmalloc::isASanEnabled):
1079         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
1080         ASan is enabled, since system malloc has the Asan hooks we need.
1081
1082         You could check for the ASan compile-time flag instead, but doing this
1083         check at runtime prepares bmalloc for a world where it is a dynamic
1084         library that might be loaded into projects it did not compile with.
1085
1086 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
1087
1088         Fix up bmalloc's PerThread for use on Linux
1089         https://bugs.webkit.org/show_bug.cgi?id=139804
1090
1091         Reviewed by Anders Carlsson.
1092
1093         The previous implementation was a bit slow.
1094
1095         * bmalloc/PerThread.h:
1096         (bmalloc::PerThreadStorage<Cache>::get):
1097         (bmalloc::PerThreadStorage::get):
1098         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
1099         way to do fast per-thread access without taking a lock every time. This
1100         probably works on all the platforms we care about, and it matches other
1101         techniques we use elsewhere in WebKit.
1102
1103         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
1104         this class because PerThreadStorage now encapsulates everything that
1105         needs to be conditional.
1106
1107         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
1108
1109 2014-12-26  Dan Bernstein  <mitz@apple.com>
1110
1111         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
1112         https://bugs.webkit.org/show_bug.cgi?id=139950
1113
1114         Reviewed by David Kilzer.
1115
1116         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
1117         in a manner that works with Xcode 5.1.1.
1118
1119 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
1120
1121         Safari crashes when you set Malloc environment variables
1122         https://bugs.webkit.org/show_bug.cgi?id=139656
1123
1124         Reviewed by Michael Saboff.
1125
1126         I forgot to cover the realloc() case. Whoops. (OoPS?)
1127
1128         This time around, I ran the full MallocBench test suite in Malloc=1
1129         mode, and it passed.
1130
1131         * bmalloc/Allocator.cpp:
1132         (bmalloc::Allocator::reallocate):
1133         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
1134         It needs to be down there so that we can do the short-circuiting check
1135         for whether bmalloc is enabled first.
1136
1137         Also added the check.
1138
1139         * bmalloc/Cache.cpp:
1140         (bmalloc::Cache::scavenge):
1141         (bmalloc::Cache::Cache):
1142         (bmalloc::Cache::reallocateSlowCaseNullCache):
1143         * bmalloc/Cache.h:
1144         (bmalloc::Cache::deallocator):
1145         (bmalloc::Cache::reallocate): Ditto.
1146
1147         * bmalloc/bmalloc.h:
1148         (bmalloc::api::free):
1149         (bmalloc::api::realloc): Ditto.
1150
1151         (bmalloc::api::scavenge): Pushed this down into Cache to match the
1152         surrounding functions.
1153
1154 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1155
1156         bmalloc should support system memory analysis tools (part 2)
1157         https://bugs.webkit.org/show_bug.cgi?id=139565
1158
1159         Reviewed by Mark Lam.
1160
1161         This patch actually queries the environment to see if memory analysis
1162         tools have been enabled.
1163
1164         * bmalloc/Deallocator.cpp:
1165         (bmalloc::Deallocator::scavenge): Don't process the object log if
1166         we've disabled bmalloc because it will be full of invalid nullptrs.
1167
1168         * bmalloc/Environment.cpp:
1169         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
1170         Malloc debugging flags. I also added a plain "Malloc" catch-all for
1171         when you want to disable bmalloc without enabling any kind of funny
1172         business.
1173
1174         It would be slightly nicer just to iterate the list of environment
1175         variables and strstr them, but getenv is the more portable option,
1176         and performance here doesn't really matter.
1177
1178         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
1179         environment variable.
1180
1181         (bmalloc::Environment::computeIsBmallocEnabled):
1182
1183 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1184
1185         Try to fix the iOS simulator build.
1186
1187         #include the declaration of malloc / free.
1188
1189         * bmalloc/Allocator.cpp:
1190         * bmalloc/Deallocator.cpp:
1191
1192 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1193
1194         Try to fix the build.
1195
1196         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
1197
1198 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1199
1200         bmalloc should support system memory analysis tools (part 1)
1201         https://bugs.webkit.org/show_bug.cgi?id=139559
1202
1203         Reviewed by Mark Lam.
1204
1205         This patch adds the hooks to disable bmalloc at runtime if certain
1206         environment variables are set, but doesn't actually read from the
1207         environment yet.
1208
1209         No performance change.
1210
1211         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
1212         we'll use to read environment variables and see if memory analysis tools
1213         have been enabled.
1214
1215         * bmalloc/Allocator.cpp:
1216         (bmalloc::Allocator::Allocator):
1217         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
1218         on the allocation path. We cache the setting to make the check fast.
1219
1220         * bmalloc/Allocator.h: Interface changes.
1221
1222         * bmalloc/Cache.cpp:
1223         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
1224         and deallocator. This main purpose is to enable them to query the
1225         environment for whether bmalloc is enabled; but this is also a slightly
1226         cleaner way to guarantee to them that the Heap has been pre-initialized.
1227
1228         * bmalloc/Deallocator.cpp:
1229         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
1230         fill the object log to force us to take the slow path on all deallocations.
1231
1232         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
1233
1234         * bmalloc/Deallocator.h: Interface changes.
1235
1236         * bmalloc/Environment.cpp: Added.
1237         (bmalloc::Environment::Environment):
1238         (bmalloc::Environment::computeIsBmallocEnabled):
1239         * bmalloc/Environment.h: Added.
1240         (bmalloc::Environment::isBmallocEnabled): This is the class that will
1241         encapsulate looking for environment variables that turn on heap
1242         analysis tools.
1243
1244         * bmalloc/Heap.h:
1245         (bmalloc::Heap::environment):
1246
1247         * bmalloc/Mutex.h:
1248         (bmalloc::Mutex::Mutex):
1249         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
1250         since I got super confused about them while writing this patch.
1251
1252         * bmalloc/VMHeap.cpp: Fixed an #include.
1253
1254 2014-12-09  David Kilzer  <ddkilzer@apple.com>
1255
1256         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
1257         <http://webkit.org/b/139212>
1258
1259         Reviewed by Joseph Pecoraro.
1260
1261         * Configurations/Base.xcconfig:
1262         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
1263           on OS X.
1264         * Configurations/DebugRelease.xcconfig:
1265         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
1266
1267 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
1268
1269         bmalloc uses 8X more virtual memory than necessary
1270         https://bugs.webkit.org/show_bug.cgi?id=138495
1271
1272         Reviewed by Mark Lam.
1273
1274         iOS has a per-process virtual memory cap around 1GB, so there's some
1275         value to not going totally ham with virtual memory.
1276
1277         We currently use about 8X the necessary amount:
1278             - 2X to align our VM allocation
1279             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
1280
1281         We can cut that down:
1282             - Return the unaligned portion of our VM allocation (-2X)
1283             - Use all the chunks in a superchunk, instead of allocating one
1284               chunk per superchunk (-4X)
1285
1286         * bmalloc/Algorithm.h:
1287         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
1288         function so we can call it with getpagesize() at runtime.
1289
1290         * bmalloc/Chunk.h:
1291         * bmalloc/LargeChunk.h:
1292         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
1293         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
1294
1295         * bmalloc/VMAllocate.h:
1296         (bmalloc::vmValidate):
1297         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
1298         if it does not succeed. Allocate precisely, and give back the extra.
1299
1300         * bmalloc/VMHeap.cpp:
1301         (bmalloc::VMHeap::allocateSuperChunk):
1302         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
1303         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
1304         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
1305         in a superchunk, instead of just one.
1306
1307         * bmalloc/VMHeap.h:
1308         (bmalloc::VMHeap::allocateSmallPage):
1309         (bmalloc::VMHeap::allocateMediumPage):
1310         (bmalloc::VMHeap::allocateLargeRange):
1311         * bmalloc/XLargeChunk.h:
1312         (bmalloc::XLargeChunk::create): Updated to match changes above.
1313
1314 2014-11-01  David Kilzer  <ddkilzer@apple.com>
1315
1316         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
1317         <https://webkit.org/b/138286>
1318         <rdar://problem/18847087>
1319
1320         Reviewed by Dan Bernstein.
1321
1322         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
1323         to NO for the target that produces libbmalloc.a so that the
1324         debug symbols will be linked into JavaScriptCore and end up in
1325         its dSYM file.
1326
1327 2014-10-30  Dana Burkart  <dburkart@apple.com>
1328
1329         <rdar://problem/18821260> Prepare for the mysterious future
1330
1331         Reviewed by Lucas Forschler.
1332
1333         * Configurations/Base.xcconfig:
1334         * Configurations/DebugRelease.xcconfig:
1335
1336 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
1337
1338         bmalloc: cleaned up fast path vs slow path
1339         https://bugs.webkit.org/show_bug.cgi?id=137081
1340
1341         Reviewed by Sam Weinig.
1342
1343         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
1344
1345         * bmalloc/Allocator.cpp:
1346         (bmalloc::Allocator::Allocator): Merged the small and medium range
1347         caches, just like the small and medium allocators. Ranges are abstract
1348         objects that don't really care whether they hold small or medium objects,
1349         so they don't need to be segregated.
1350
1351         (bmalloc::Allocator::scavenge): Ditto.
1352
1353         (bmalloc::Allocator::allocateBumpRangeSlowCase):
1354         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
1355         we do care a tiny bit, because we need to specify small vs medium when
1356         allocating new ranges from the heap, to ensure that the heap allocates
1357         from the right segment of VM.
1358
1359         (bmalloc::Allocator::allocateLarge):
1360         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
1361         up the fast path. Large allocation performance is dominated by allocation
1362         logic and initialization, so inlining it doesn't help.
1363
1364         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
1365         it doesn't need to distinguish small vs medium objects.
1366
1367         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
1368         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
1369
1370         * bmalloc/Allocator.h:
1371         * bmalloc/BumpRange.h:
1372
1373         * bmalloc/Cache.cpp:
1374         (bmalloc::Cache::allocateSlowCase): Deleted.
1375         (bmalloc::Cache::deallocateSlowCase): Deleted.
1376         * bmalloc/Cache.h:
1377         (bmalloc::Cache::allocate):
1378         (bmalloc::Cache::deallocate):
1379         (bmalloc::Cache::allocateFastCase): Deleted.
1380         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
1381         paths. The downside to this change is that the fast path branches to two
1382         distinct failure cases instead of one. The upside is that the slow path
1383         doesn't need to re-read the segment register, which is not as cheap as a
1384         normal register, and it doesn't need to do an extra level of function 
1385         call. Seems to be worth it.
1386
1387         * bmalloc/Deallocator.h:
1388         * bmalloc/Heap.cpp:
1389         (bmalloc::Heap::refillSmallBumpRangeCache):
1390         (bmalloc::Heap::refillMediumBumpRangeCache):
1391         * bmalloc/Heap.h: Updated for interface changes.
1392
1393         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
1394         small lines in a page / 2, since any other free lines will coalesce
1395         with their neighbors.
1396
1397 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1398
1399         Rolled out r173346.
1400
1401             bmalloc should honor the FastMalloc statistics API
1402             https://bugs.webkit.org/show_bug.cgi?id=136592
1403
1404         This didn't really work. Because we allow ranges with and without
1405         physical pages to merge, and we allow double-committing and
1406         double-decommitting, we can't rely on commit actions to track memory
1407         footprint.
1408
1409         * bmalloc/Heap.cpp:
1410         (bmalloc::Heap::size): Deleted.
1411         (bmalloc::Heap::capacity): Deleted.
1412         * bmalloc/Heap.h:
1413         * bmalloc/VMHeap.cpp:
1414         (bmalloc::VMHeap::VMHeap):
1415         (bmalloc::VMHeap::allocateSmallChunk):
1416         (bmalloc::VMHeap::allocateMediumChunk):
1417         (bmalloc::VMHeap::allocateLargeChunk):
1418         * bmalloc/VMHeap.h:
1419         (bmalloc::VMHeap::allocateSmallPage):
1420         (bmalloc::VMHeap::allocateMediumPage):
1421         (bmalloc::VMHeap::allocateLargeRange):
1422         (bmalloc::VMHeap::deallocateSmallPage):
1423         (bmalloc::VMHeap::deallocateMediumPage):
1424         (bmalloc::VMHeap::deallocateLargeRange):
1425         (bmalloc::VMHeap::size): Deleted.
1426         (bmalloc::VMHeap::capacity): Deleted.
1427         * bmalloc/bmalloc.h:
1428         (bmalloc::api::heapSize): Deleted.
1429         (bmalloc::api::heapCapacity): Deleted.
1430
1431 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1432
1433         bmalloc: Allocation should be more precise
1434         https://bugs.webkit.org/show_bug.cgi?id=136993
1435
1436         Reviewed by Gavin Barraclough.
1437
1438         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1439
1440         This patch teaches the allocator to merge adjacent free lines into a
1441         single allocatable range. This allows us to shrink the size of an
1442         individual line without increasing fragmentation or the rate of allocator
1443         slow paths.
1444
1445         We'll only take more slow paths when available memory is sparse, which
1446         is exactly when it's worth it. When available memory is dense, we'll
1447         take fewer slow paths.
1448
1449         * bmalloc.xcodeproj/project.pbxproj:
1450         * bmalloc/Algorithm.h:
1451         (bmalloc::divideRoundingUp):
1452
1453         * bmalloc/Allocator.cpp:
1454         (bmalloc::Allocator::Allocator): Updated for interface changes.
1455
1456         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
1457         Now that we merge lines, it's not convenient to scavenge by line.
1458
1459         (bmalloc::Allocator::allocateSmallBumpRange):
1460         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
1461         instead of individual lines.
1462
1463         (bmalloc::Allocator::allocateSlowCase):
1464         (bmalloc::Allocator::allocateSmallLine): Deleted.
1465         (bmalloc::Allocator::allocateMediumLine): Deleted.
1466         (bmalloc::Allocator::allocateMedium): Deleted.
1467         * bmalloc/Allocator.h:
1468         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
1469         into the standard fast path with small allocations. Since a BumpAllocator
1470         just allocates out of an arbitrary range, it doesn't need to distinguish
1471         between small and medium lines.
1472
1473         * bmalloc/BumpAllocator.h:
1474         (bmalloc::BumpAllocator::size):
1475         (bmalloc::BumpAllocator::BumpAllocator):
1476         (bmalloc::BumpAllocator::init):
1477         (bmalloc::BumpAllocator::refill):
1478         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
1479         anymore: the heap just gives us a pointer and a pre-computed number of
1480         objects, and we allocate them.
1481
1482         * bmalloc/Deallocator.cpp:
1483         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
1484
1485         * bmalloc/Heap.cpp:
1486         (bmalloc::Heap::Heap):
1487         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
1488         detailing where all objects will lie in memory. After we merge two lines,
1489         we might allocate an object that spans from one line to the next. This
1490         metadata details which bits of memory overlap in that way, and how they
1491         overlap.
1492
1493         (bmalloc::Heap::refillSmallBumpRangeCache):
1494         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
1495         and merge adjacent free lines into BumpRanges.
1496
1497         (bmalloc::Heap::allocateSmallPage):
1498         (bmalloc::Heap::allocateMediumPage):
1499         (bmalloc::Heap::deallocateSmallLine):
1500         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
1501         since we scan for free memory a page at a time.
1502
1503         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
1504         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
1505         fast path.
1506
1507         * bmalloc/Heap.h:
1508         (bmalloc::Heap::derefSmallLine):
1509         (bmalloc::Heap::derefMediumLine):
1510         (bmalloc::Heap::deallocateSmallLine): Deleted.
1511         (bmalloc::Heap::allocateSmallLine): Deleted.
1512         (bmalloc::Heap::deallocateMediumLine): Deleted.
1513         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
1514
1515         * bmalloc/Line.h:
1516         (bmalloc::Line<Traits>::ref):
1517         (bmalloc::Line<Traits>::deref):
1518         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
1519         anymore, since we only ever deref by 1 now.
1520
1521         * bmalloc/MediumAllocator.h:
1522         (bmalloc::MediumAllocator::isNull): Deleted.
1523         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
1524         (bmalloc::MediumAllocator::line): Deleted.
1525         (bmalloc::MediumAllocator::allocate): Deleted.
1526         (bmalloc::MediumAllocator::derefCount): Deleted.
1527         (bmalloc::MediumAllocator::refill): Deleted.
1528         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
1529         been dead for a while, since it doesn't build anymore with this patch.
1530
1531         * bmalloc/Page.h:
1532         (bmalloc::Page::sizeClass):
1533         (bmalloc::Page::setSizeClass):
1534         (bmalloc::Page::smallSizeClass): Deleted.
1535         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
1536         to sizeClass, since we use it for medium sizes too.
1537
1538         * bmalloc/Sizes.h:
1539         (bmalloc::Sizes::sizeClass):
1540         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
1541
1542         (bmalloc::Sizes::smallSizeClassFor): Deleted.
1543         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
1544
1545         * bmalloc/bmalloc.h:
1546         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
1547         can be a bit more precise. It also has to be, since we can't guarantee
1548         that an object ends at the end of a line anymore.
1549
1550 2014-09-19  Daniel Bates  <dabates@apple.com>
1551
1552         Always assume internal SDK when building configuration Production
1553         https://bugs.webkit.org/show_bug.cgi?id=136925
1554         <rdar://problem/18362399>
1555
1556         Reviewed by Dan Bernstein.
1557
1558         * Configurations/Base.xcconfig:
1559
1560 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
1561
1562         bmalloc: moved line caches from the deallocator to the allocator
1563         https://bugs.webkit.org/show_bug.cgi?id=136868
1564
1565         Reviewed by Gavin Barraclough.
1566
1567         I did this mostly as a simplification, to make it easier to change the
1568         allocation strategy.
1569
1570         No throughput change on MallocBench. Saves about 50kB.
1571
1572         Since the deallocator needs to lock the heap when freeing lines anyway,
1573         there isn't much benefit to giving the deallocator a local cache of
1574         deallocated lines.
1575
1576         We still give the allocator a local cache of lines because that does
1577         reduce the frequency at which it needs to lock the heap in order to
1578         acquire more lines.
1579
1580         * bmalloc/Allocator.cpp:
1581         (bmalloc::Allocator::scavenge):
1582         (bmalloc::Allocator::allocateSmallLine):
1583         (bmalloc::Allocator::allocateMediumLine):
1584         (bmalloc::Allocator::allocateMedium):
1585         (bmalloc::Allocator::allocateSlowCase):
1586         * bmalloc/Allocator.h:
1587         * bmalloc/Deallocator.cpp:
1588         (bmalloc::Deallocator::Deallocator):
1589         (bmalloc::Deallocator::scavenge):
1590         (bmalloc::Deallocator::processObjectLog):
1591         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
1592         (bmalloc::Deallocator::allocateSmallLine): Deleted.
1593         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
1594         (bmalloc::Deallocator::allocateMediumLine): Deleted.
1595         * bmalloc/Deallocator.h:
1596
1597         * bmalloc/Sizes.h:
1598         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
1599         exactly one page in size. That's about what we were shooting for anyway,
1600         and it may make it easier to switch to per-page allocation in future.
1601
1602 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
1603
1604         bmalloc: allocate small and medium objects using the same bump pointer class
1605         https://bugs.webkit.org/show_bug.cgi?id=136843
1606
1607         Reviewed by Gavin Barraclough.
1608
1609         4% speedup on MallocBench.
1610
1611         Now that medium-sized objects have dedicated per-size allocators, they
1612         don't need to use an arbitrary bump pointer allocator. This means that
1613         every allocator knows how many objects it will allocate from the start,
1614         and we don't need a post-processing step to adjust refcounts based on
1615         real allocation count.
1616
1617         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1618         since it's used for small and medium objects now.
1619
1620         * bmalloc/Allocator.cpp:
1621         (bmalloc::Allocator::Allocator): Updated to use new interface.
1622         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1623         to make sure that we finish allocating all the objects in it.
1624
1625         (bmalloc::Allocator::allocateMedium):
1626         (bmalloc::Allocator::allocateSlowCase):
1627         (bmalloc::Allocator::retire): Deleted.
1628         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1629         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1630         * bmalloc/Allocator.h:
1631         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1632         used to post-process an allocator based on how many objects it allocated.
1633
1634         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1635         (bmalloc::BumpAllocator::BumpAllocator):
1636         (bmalloc::BumpAllocator::init):
1637         (bmalloc::BumpAllocator::line):
1638         (bmalloc::BumpAllocator::validate):
1639         (bmalloc::BumpAllocator::allocate):
1640         (bmalloc::BumpAllocator::refill):
1641         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1642         about the kinds of lines they allocate into. In some cases, the line
1643         type must be provided as a template parameter by the caller.
1644
1645         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1646         (bmalloc::SmallAllocator::line): Deleted.
1647         (bmalloc::SmallAllocator::allocate): Deleted.
1648         (bmalloc::SmallAllocator::objectCount): Deleted.
1649         (bmalloc::SmallAllocator::derefCount): Deleted.
1650         (bmalloc::SmallAllocator::refill): Deleted.
1651         (bmalloc::SmallAllocator::clear): Deleted.
1652
1653         * bmalloc/ObjectType.h:
1654         (bmalloc::isMedium):
1655
1656         * bmalloc/SmallAllocator.h:
1657         (bmalloc::SmallAllocator::isNull): Deleted.
1658         (bmalloc::SmallAllocator::canAllocate): Deleted.
1659         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1660         (bmalloc::SmallAllocator::line): Deleted.
1661         (bmalloc::SmallAllocator::allocate): Deleted.
1662         (bmalloc::SmallAllocator::objectCount): Deleted.
1663         (bmalloc::SmallAllocator::derefCount): Deleted.
1664         (bmalloc::SmallAllocator::refill): Deleted.
1665         (bmalloc::SmallAllocator::clear): Deleted.
1666
1667 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1668
1669         Fixed a goof in bmalloc Vector sizing
1670         https://bugs.webkit.org/show_bug.cgi?id=136795
1671
1672         Reviewed by Gavin Barraclough and Sam Weinig.
1673
1674         We want our minimum vector to be page-sized since the OS will give us
1675         a page no matter what -- but we want that many bytes, and not enough
1676         bytes to store that many elements.
1677
1678         * bmalloc/Vector.h: Math is hard.
1679
1680 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1681
1682         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1683         https://bugs.webkit.org/show_bug.cgi?id=136693
1684
1685         Reviewed by Gavin Barraclough.
1686
1687         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1688
1689         No throughput change.
1690
1691         We keep an array of medium allocators, just like our array of small
1692         allocators.
1693
1694         In future, we can simplify the allocation fast path by merging the small
1695         and medium allocator arrays. For now, this is the simplest change that
1696         gets the win.
1697
1698         * bmalloc/Allocator.cpp:
1699         (bmalloc::Allocator::Allocator):
1700         (bmalloc::Allocator::scavenge):
1701         (bmalloc::Allocator::allocateMedium):
1702         * bmalloc/Allocator.h:
1703         * bmalloc/Sizes.h:
1704         (bmalloc::Sizes::mediumSizeClassFor):
1705
1706 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1707
1708         Reviewed by Sam Weinig.
1709
1710         Renamed log => retire for clarity.
1711
1712         * bmalloc/Allocator.cpp:
1713         (bmalloc::Allocator::scavenge):
1714         (bmalloc::Allocator::retire):
1715         (bmalloc::Allocator::allocateMedium):
1716         (bmalloc::Allocator::allocateSlowCase):
1717         (bmalloc::Allocator::log): Deleted.
1718         * bmalloc/Allocator.h:
1719
1720 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1721
1722         bmalloc: eager scavenge leaves behind a bogus allocator
1723         https://bugs.webkit.org/show_bug.cgi?id=136743
1724
1725         Reviewed by Sam Weinig.
1726
1727         Be sure to clear the allocator after logging it in the eager scavenge
1728         case, so that we don't later try to allocate out of the lines that we
1729         have thrown away.
1730
1731         We didn't need to do this previously because scavenge would only happen
1732         at thread exit time, after which no further allocation from the per-thread
1733         cache would take place.
1734
1735         * bmalloc/Allocator.cpp:
1736         (bmalloc::Allocator::scavenge):
1737         * bmalloc/MediumAllocator.h:
1738         (bmalloc::MediumAllocator::clear):
1739         * bmalloc/SmallAllocator.h:
1740         (bmalloc::SmallAllocator::clear):
1741
1742 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1743
1744         bmalloc should honor the FastMalloc statistics API
1745         https://bugs.webkit.org/show_bug.cgi?id=136592
1746
1747         Reviewed by Gavin Barraclough.
1748
1749         We do this by tracking "size" and "capacity" in the VM heap.
1750
1751         The VM heap's "capacity" is all the VM we ever allocated.
1752
1753         The VM heap's "size" the subset of VM currently held onto by the
1754         VM heap (and therefore not in use by the regular heap).
1755
1756         Somewhat ironically, reducing the process's memory footprint, increases
1757         the size of the VM heap, since the VM heap holds the pages that are
1758         purely virtual and not physical.
1759
1760         * bmalloc/Heap.cpp:
1761         (bmalloc::Heap::size):
1762         (bmalloc::Heap::capacity):
1763         * bmalloc/Heap.h:
1764         * bmalloc/VMHeap.cpp:
1765         (bmalloc::VMHeap::VMHeap):
1766         (bmalloc::VMHeap::allocateSmallChunk):
1767         (bmalloc::VMHeap::allocateMediumChunk):
1768         (bmalloc::VMHeap::allocateLargeChunk):
1769         * bmalloc/VMHeap.h:
1770         (bmalloc::VMHeap::size):
1771         (bmalloc::VMHeap::capacity):
1772         (bmalloc::VMHeap::allocateSmallPage):
1773         (bmalloc::VMHeap::allocateMediumPage):
1774         (bmalloc::VMHeap::allocateLargeRange):
1775         (bmalloc::VMHeap::deallocateSmallPage):
1776         (bmalloc::VMHeap::deallocateMediumPage):
1777         (bmalloc::VMHeap::deallocateLargeRange):
1778         * bmalloc/bmalloc.h:
1779         (bmalloc::api::heapSize):
1780         (bmalloc::api::heapCapacity):
1781
1782 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1783
1784         bmalloc crashes on the EWS bots (due to bad large object allocation)
1785         https://bugs.webkit.org/show_bug.cgi?id=136469
1786
1787         Reviewed by Andreas Kling.
1788
1789         It's possible to convince bmalloc to perform a bad large object allocation,
1790         through these steps:
1791
1792         (1) Insert object A into freelist F0.
1793
1794         (2) Split, merge and split again A's neighbors such that object B is
1795         inserted into freelist F0, with boundary tag and size equal to object A,
1796         but pointer not completely equal to object A. Put object B at the head of F0.
1797
1798         (3) Allocate some other object from F0, swapping its position in the
1799         freelist with object B, such that object A is now ahead of object B.
1800
1801         --> Now, the next allocation for size A/B will allocate object A, which
1802         has a slightly wrong idea about where the object actually begins.
1803         Immediately, you'll corrupt a little memory, and over time, you'll also
1804         corrupt boundary tag metadata.
1805
1806         The solution is to store the begin pointer in the boundary tag. Luckily,
1807         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1808         on MallocBench.
1809
1810         * bmalloc/Algorithm.h:
1811         (bmalloc::rightShift):
1812         * bmalloc/BeginTag.h:
1813         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1814         validate the start pointer when popping off the free list. Through a
1815         very uncommon set of steps, it is possible to have an item in the free
1816         list that is valid by all accounts except for its start pointer.
1817
1818         * bmalloc/BoundaryTag.h:
1819         (bmalloc::BoundaryTag::compactBegin):
1820         (bmalloc::BoundaryTag::setRange):
1821         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1822         start pointer. We don't need the whole pointer -- just the offset, in
1823         largeAlignment increments, into the relevant boundary tag bucket.
1824
1825         * bmalloc/BoundaryTagInlines.h:
1826         (bmalloc::validateNext):
1827         (bmalloc::BoundaryTag::init):
1828         (bmalloc::BoundaryTag::mergeLarge):
1829         (bmalloc::BoundaryTag::splitLarge):
1830         * bmalloc/SegregatedFreeList.cpp:
1831         (bmalloc::SegregatedFreeList::insert):
1832         (bmalloc::SegregatedFreeList::takeGreedy):
1833         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1834         the size when establishing a boundary tag, as required by the new
1835         interface.
1836
1837         * bmalloc/Sizes.h:
1838
1839 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1840
1841         Fixed a bmalloc crash seen on the EWS bot
1842         https://bugs.webkit.org/show_bug.cgi?id=135955
1843
1844         Reviewed by Andreas Kling.
1845
1846         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1847         input buffer is a malloc region, that region will get marked Copy-On-Write
1848         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1849         on older OS X's. In 10.10, they still fail, but they do not return
1850         EINVAL.
1851
1852         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1853
1854 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1855
1856         Fixed the bmalloc build
1857         https://bugs.webkit.org/show_bug.cgi?id=135953
1858
1859         Reviewed by Andreas Kling.
1860
1861         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1862         These headers are used, so they must be available outside the project.
1863
1864 2014-08-13  Daniel Bates  <dabates@apple.com>
1865
1866         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1867         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1868
1869         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1870         implementation of PerThread<T>::getFastCase().
1871
1872         * bmalloc/PerThread.h:
1873         (bmalloc::PerThread<T>::getFastCase):
1874
1875 2014-08-13  Daniel Bates  <dabates@apple.com>
1876
1877         Make bmalloc::PerThread work without C++ thread local storage
1878         https://bugs.webkit.org/show_bug.cgi?id=135895
1879
1880         Reviewed by Geoffrey Garen.
1881
1882         Implement support for building bmalloc without C++ thread local storage.
1883
1884         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1885         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1886         to determine whether the compiler supports C++ thread local storage.
1887         * bmalloc/PerThread.h:
1888         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1889         without C++ thread local storage.
1890         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1891         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1892         PerThreadStorage::initSharedKeyIfNeeded().
1893         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1894         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1895         ensure that the shared key has been initialized.
1896         (_pthread_setspecific_direct): Deleted.
1897         (_pthread_getspecific_direct): Deleted.
1898
1899 2014-08-13  Daniel Bates  <dabates@apple.com>
1900
1901         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1902         https://bugs.webkit.org/show_bug.cgi?id=135848
1903
1904         Reviewed by Geoffrey Garen.
1905
1906         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1907         when building for the iOS Simulator.
1908         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1909         using the public SDK.
1910         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1911         using the public SDK.
1912         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1913         using the public SDK.
1914
1915 2014-08-12  Daniel Bates  <dabates@apple.com>
1916
1917         BPLATFORM(IOS) always evaluates to false
1918         https://bugs.webkit.org/show_bug.cgi?id=135843
1919
1920         Reviewed by Geoffrey Garen.
1921
1922         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1923         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1924         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1925         building for iOS.
1926
1927         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1928         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1929
1930         * bmalloc/BPlatform.h:
1931
1932 2014-08-11  Andy Estes  <aestes@apple.com>
1933
1934         [iOS] Get rid of iOS.xcconfig
1935         https://bugs.webkit.org/show_bug.cgi?id=135809
1936
1937         Reviewed by Joseph Pecoraro.
1938
1939         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1940
1941         * Configurations/Base.xcconfig:
1942         * Configurations/iOS.xcconfig: Removed.
1943         * bmalloc.xcodeproj/project.pbxproj:
1944
1945 2014-05-01  Dan Bernstein  <mitz@apple.com>
1946
1947         Fixed production builds for the iOS Simulator.
1948         <rdar://problem/16792221>
1949
1950         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1951         PRIVATE_HEADERS_FOLDER_PATH when installing.
1952
1953 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1954
1955         bmalloc: Segregate pages by objects size
1956         https://bugs.webkit.org/show_bug.cgi?id=131909
1957
1958         Reviewed by Andreas Kling.
1959
1960         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1961
1962         * bmalloc/Allocator.cpp:
1963         (bmalloc::Allocator::allocateSlowCase):
1964         * bmalloc/Allocator.h:
1965         (bmalloc::Allocator::allocateFastCase):
1966         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1967         function for size class calculation.
1968
1969         * bmalloc/Deallocator.cpp:
1970         (bmalloc::Deallocator::Deallocator):
1971         (bmalloc::Deallocator::scavenge):
1972         (bmalloc::Deallocator::deallocateSmallLine):
1973         (bmalloc::Deallocator::allocateSmallLine):
1974         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1975         cache can't be shared anymore.
1976
1977         * bmalloc/Heap.cpp:
1978         (bmalloc::Heap::allocateSmallLineSlowCase):
1979         * bmalloc/Heap.h:
1980         (bmalloc::Heap::deallocateSmallLine): Ditto.
1981
1982         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1983         page refcount when allocating a line because we might have deallocated
1984         the page and the recycled it for another size class.
1985
1986         (bmalloc::Heap::deallocateMediumLine):
1987         (bmalloc::Heap::allocateMediumLine):
1988         * bmalloc/Line.h:
1989         (bmalloc::Line::refCount):
1990         * bmalloc/Page.h:
1991         (bmalloc::Page::refCount):
1992         (bmalloc::Page::smallSizeClass):
1993         (bmalloc::Page::setSmallSizeClass):
1994         (bmalloc::Page<Traits>::refCount): Deleted.
1995         * bmalloc/Sizes.h:
1996         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1997         an index into an array from a size.
1998
1999 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
2000
2001         bmalloc: Improved alignment in LargeChunk
2002         https://bugs.webkit.org/show_bug.cgi?id=131895
2003
2004         Reviewed by Andreas Kling.
2005
2006         * bmalloc/Chunk.h:
2007         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
2008         Technically, the previous alignment was harmless, but I would prefer,
2009         dear reader, not to have to explain the interlocking set of
2010         circumstances that made it so.
2011
2012 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
2013
2014         Rolled out r167502 because it caused a crash on the facebook benchmark.
2015
2016         Unreviewed.
2017
2018             bmalloc: Added an XSmall line size
2019             https://bugs.webkit.org/show_bug.cgi?id=131851
2020
2021             Reviewed by Sam Weinig.
2022
2023 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
2024
2025         bmalloc: Mutex should be harder to use wrong
2026         https://bugs.webkit.org/show_bug.cgi?id=131879
2027
2028         Reviewed by Andreas Kling.
2029
2030         Mutex now has a proper constructor, so you can't deadlock by forgetting
2031         to initialize it.
2032
2033         * bmalloc.xcodeproj/project.pbxproj:
2034         * bmalloc/Allocator.cpp:
2035         (bmalloc::Allocator::processXSmallAllocatorLog):
2036         (bmalloc::Allocator::processSmallAllocatorLog):
2037         (bmalloc::Allocator::processMediumAllocatorLog):
2038         (bmalloc::Allocator::allocateLarge):
2039         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
2040         since the Heap mutex is a static.
2041
2042         * bmalloc/AsyncTask.h:
2043         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
2044         need for explicit initialization anymore.
2045
2046         * bmalloc/Deallocator.cpp:
2047         (bmalloc::Deallocator::scavenge):
2048         (bmalloc::Deallocator::deallocateLarge):
2049         (bmalloc::Deallocator::deallocateXLarge):
2050         (bmalloc::Deallocator::processObjectLog):
2051         (bmalloc::Deallocator::deallocateSmallLine):
2052         (bmalloc::Deallocator::deallocateXSmallLine):
2053         (bmalloc::Deallocator::allocateSmallLine):
2054         (bmalloc::Deallocator::allocateXSmallLine):
2055         (bmalloc::Deallocator::deallocateMediumLine):
2056         (bmalloc::Deallocator::allocateMediumLine):
2057         * bmalloc/Deallocator.h:
2058         * bmalloc/Heap.cpp:
2059         (bmalloc::sleep):
2060         (bmalloc::Heap::Heap):
2061         (bmalloc::Heap::concurrentScavenge):
2062         (bmalloc::Heap::scavenge):
2063         (bmalloc::Heap::scavengeSmallPages):
2064         (bmalloc::Heap::scavengeXSmallPages):
2065         (bmalloc::Heap::scavengeMediumPages):
2066         (bmalloc::Heap::scavengeLargeRanges):
2067         (bmalloc::Heap::allocateXSmallLineSlowCase):
2068         (bmalloc::Heap::allocateSmallLineSlowCase):
2069         (bmalloc::Heap::allocateMediumLineSlowCase):
2070         (bmalloc::Heap::allocateXLarge):
2071         (bmalloc::Heap::deallocateXLarge):
2072         (bmalloc::Heap::allocateLarge):
2073         (bmalloc::Heap::deallocateLarge):
2074         * bmalloc/Heap.h:
2075         (bmalloc::Heap::deallocateXSmallLine):
2076         (bmalloc::Heap::allocateXSmallLine):
2077         (bmalloc::Heap::deallocateSmallLine):
2078         (bmalloc::Heap::allocateSmallLine):
2079         (bmalloc::Heap::deallocateMediumLine):
2080         (bmalloc::Heap::allocateMediumLine):
2081         * bmalloc/Line.h:
2082         (bmalloc::Line<Traits>::deref):
2083         * bmalloc/Mutex.cpp: Removed.
2084         * bmalloc/Mutex.h:
2085         (bmalloc::Mutex::Mutex):
2086         (bmalloc::Mutex::init): Deleted.
2087         (bmalloc::Mutex::try_lock): Deleted.
2088         (bmalloc::Mutex::lock): Deleted.
2089         (bmalloc::Mutex::unlock): Deleted.
2090         * bmalloc/Page.h:
2091         (bmalloc::Page<Traits>::ref):
2092         (bmalloc::Page<Traits>::deref):
2093         (bmalloc::Page<Traits>::refCount):
2094         * bmalloc/PerProcess.h:
2095         (bmalloc::PerProcess::mutex):
2096         (bmalloc::PerProcess<T>::getSlowCase):
2097         * bmalloc/StaticMutex.cpp: Added.
2098         (bmalloc::StaticMutex::lockSlowCase):
2099         * bmalloc/StaticMutex.h: Added.
2100         (bmalloc::StaticMutex::init):
2101         (bmalloc::StaticMutex::try_lock):
2102         (bmalloc::StaticMutex::lock):
2103         (bmalloc::StaticMutex::unlock):
2104         * bmalloc/VMHeap.h:
2105         (bmalloc::VMHeap::deallocateXSmallPage):
2106         (bmalloc::VMHeap::deallocateSmallPage):
2107         (bmalloc::VMHeap::deallocateMediumPage):
2108         (bmalloc::VMHeap::deallocateLargeRange):
2109         * bmalloc/bmalloc.h:
2110         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
2111         since the Heap mutex is a static.
2112
2113 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
2114
2115         bmalloc: AsyncTask should use Mutex instead of std::mutex
2116         https://bugs.webkit.org/show_bug.cgi?id=131865
2117
2118         Reviewed by Gavin Barraclough.
2119
2120         std::mutex is so slow that it makes parallelizing simple tasks through
2121         AsyncTask a net regression. Mutex fixes this.
2122
2123         * bmalloc/AsyncTask.h:
2124         (bmalloc::Function>::AsyncTask):
2125         (bmalloc::Function>::join):
2126         (bmalloc::Function>::runSlowCase):
2127         (bmalloc::Function>::entryPoint):
2128         * bmalloc/Mutex.h:
2129         (bmalloc::Mutex::init):
2130
2131 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
2132
2133         bmalloc: Added an XSmall line size
2134         https://bugs.webkit.org/show_bug.cgi?id=131851
2135
2136         Reviewed by Sam Weinig.
2137
2138         Reduces malloc footprint on Membuster recordings by 10%.
2139
2140         This is a throughput regression, but we're still way ahead of TCMalloc.
2141         I have some ideas for how to recover the regression -- but I wanted to
2142         get this win in first.
2143
2144         Full set of benchmark results:
2145
2146                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2147                                                                                                 
2148                                                                nopatch                      patch                                Δ
2149                 Peak Memory:
2150                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
2151                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
2152                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
2153
2154                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
2155                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
2156                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
2157
2158                 Memory at End:
2159                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
2160                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
2161                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
2162
2163                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
2164                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
2165                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
2166
2167                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2168                                                                                                 
2169                                                    nopatch                patch                         Δ
2170                 Execution Time:
2171                     churn                            127ms                151ms            ! 1.19x slower
2172                     list_allocate                    130ms                164ms            ! 1.26x slower
2173                     tree_allocate                    109ms                127ms            ! 1.17x slower
2174                     tree_churn                       115ms                120ms            ! 1.04x slower
2175                     facebook                         240ms                259ms            ! 1.08x slower
2176                     fragment                          91ms                131ms            ! 1.44x slower
2177                     fragment_iterate                 105ms                106ms            ! 1.01x slower
2178                     message_one                      260ms                259ms             ^ 1.0x faster
2179                     message_many                     149ms                154ms            ! 1.03x slower
2180                     medium                           194ms                248ms            ! 1.28x slower
2181                     big                              157ms                160ms            ! 1.02x slower
2182
2183                     <geometric mean>                 144ms                163ms            ! 1.13x slower
2184                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
2185                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
2186
2187                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2188                                                                                                 
2189                                                                        nopatch                          patch                                     Δ
2190                 Execution Time:
2191                     churn                                                126ms                          148ms                        ! 1.17x slower
2192                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
2193                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
2194                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
2195                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
2196                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
2197                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
2198                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
2199                     facebook                                             240ms                          276ms                        ! 1.15x slower
2200                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
2201                     fragment                                              92ms                          130ms                        ! 1.41x slower
2202                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
2203                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
2204                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
2205                     message_one                                          260ms                          260ms                                      
2206                     message_many                                         170ms                          238ms                         ! 1.4x slower
2207                     medium                                               185ms                          250ms                        ! 1.35x slower
2208                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
2209                     big                                                  150ms                          169ms                        ! 1.13x slower
2210                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
2211
2212                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
2213                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
2214                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
2215
2216                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2217
2218                                                                     TC                      patch                                Δ
2219                 Peak Memory:
2220                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
2221                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
2222                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
2223
2224                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
2225                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
2226                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
2227
2228                 Memory at End:
2229                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
2230                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
2231                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
2232
2233                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
2234                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
2235                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
2236
2237                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2238                                                                                                 
2239                                                         TC                patch                         Δ
2240                 Execution Time:
2241                     churn                            416ms                148ms            ^ 2.81x faster
2242                     list_allocate                    463ms                164ms            ^ 2.82x faster
2243                     tree_allocate                    292ms                127ms             ^ 2.3x faster
2244                     tree_churn                       157ms                120ms            ^ 1.31x faster
2245                     facebook                         327ms                276ms            ^ 1.18x faster
2246                     fragment                         335ms                129ms             ^ 2.6x faster
2247                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
2248                     message_one                      386ms                258ms             ^ 1.5x faster
2249                     message_many                     410ms                154ms            ^ 2.66x faster
2250                     medium                           391ms                245ms             ^ 1.6x faster
2251                     big                              261ms                167ms            ^ 1.56x faster
2252
2253                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
2254                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
2255                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
2256
2257         * bmalloc.xcodeproj/project.pbxproj:
2258         * bmalloc/Allocator.cpp:
2259         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
2260         index corresponds with its size. Instead, use the size selection function
2261         explicitly. Now that we have XSmall, some small allocator entries are
2262         unused.
2263
2264         (bmalloc::Allocator::scavenge):
2265         (bmalloc::Allocator::log):
2266         (bmalloc::Allocator::processXSmallAllocatorLog):
2267         (bmalloc::Allocator::allocateSlowCase):
2268         * bmalloc/Allocator.h:
2269         (bmalloc::Allocator::xSmallAllocatorFor):
2270         (bmalloc::Allocator::allocateFastCase):
2271         * bmalloc/Chunk.h:
2272         * bmalloc/Deallocator.cpp:
2273         (bmalloc::Deallocator::scavenge):
2274         (bmalloc::Deallocator::processObjectLog):
2275         (bmalloc::Deallocator::deallocateSlowCase):
2276         (bmalloc::Deallocator::deallocateXSmallLine):
2277         (bmalloc::Deallocator::allocateXSmallLine):
2278         * bmalloc/Deallocator.h:
2279         (bmalloc::Deallocator::deallocateFastCase):
2280         * bmalloc/Heap.cpp:
2281         (bmalloc::Heap::scavenge):
2282         (bmalloc::Heap::scavengeXSmallPages):
2283         (bmalloc::Heap::allocateXSmallLineSlowCase):
2284         * bmalloc/Heap.h:
2285         (bmalloc::Heap::deallocateXSmallLine):
2286         (bmalloc::Heap::allocateXSmallLine):
2287         * bmalloc/LargeChunk.h:
2288         (bmalloc::LargeChunk::get):
2289         (bmalloc::LargeChunk::endTag):
2290         * bmalloc/Line.h:
2291         * bmalloc/MediumAllocator.h:
2292         (bmalloc::MediumAllocator::allocate):
2293         (bmalloc::MediumAllocator::refill):
2294         * bmalloc/ObjectType.cpp:
2295         (bmalloc::objectType):
2296         * bmalloc/ObjectType.h:
2297         (bmalloc::isXSmall):
2298         (bmalloc::isSmall):
2299         (bmalloc::isMedium):
2300         (bmalloc::isLarge):
2301         (bmalloc::isSmallOrMedium): Deleted.
2302         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
2303         handling small objects. There's probably a reasonable way to share this
2304         code in the future -- I'll look into that once it's stopped changing.
2305
2306         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
2307         is the main reason things got slower.
2308
2309         * bmalloc/SmallAllocator.h:
2310         (bmalloc::SmallAllocator::allocate):
2311         * bmalloc/SmallTraits.h:
2312         * bmalloc/VMHeap.cpp:
2313         (bmalloc::VMHeap::allocateXSmallChunk):
2314         * bmalloc/VMHeap.h:
2315         (bmalloc::VMHeap::allocateXSmallPage):
2316         (bmalloc::VMHeap::deallocateXSmallPage):
2317         * bmalloc/XSmallAllocator.h: Added.
2318         (bmalloc::XSmallAllocator::isNull):
2319         (bmalloc::XSmallAllocator::canAllocate):
2320         (bmalloc::XSmallAllocator::XSmallAllocator):
2321         (bmalloc::XSmallAllocator::line):
2322         (bmalloc::XSmallAllocator::allocate):
2323         (bmalloc::XSmallAllocator::objectCount):
2324         (bmalloc::XSmallAllocator::derefCount):
2325         (bmalloc::XSmallAllocator::refill):
2326         * bmalloc/XSmallChunk.h: Added.
2327         * bmalloc/XSmallLine.h: Added.
2328         * bmalloc/XSmallPage.h: Added.
2329         * bmalloc/XSmallTraits.h: Added.
2330         * bmalloc/bmalloc.h:
2331         (bmalloc::api::realloc): Boiler-plate copy, as above.
2332
2333 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2334
2335         MallocBench should scavenge explicitly instead of waiting
2336         https://bugs.webkit.org/show_bug.cgi?id=131661
2337
2338         Reviewed by Andreas Kling.
2339
2340         Added explicit scavenge support to bmalloc. This isn't a memory win,
2341         since bmalloc's per-thread cache is so small. But it makes testing
2342         simpler.
2343
2344         * bmalloc/Allocator.cpp:
2345         (bmalloc::Allocator::~Allocator):
2346         (bmalloc::Allocator::scavenge):
2347         * bmalloc/Allocator.h:
2348         * bmalloc/Cache.cpp:
2349         (bmalloc::Cache::operator new):
2350         (bmalloc::Cache::operator delete):
2351         (bmalloc::Cache::Cache):
2352         (bmalloc::Cache::scavenge):
2353         * bmalloc/Cache.h:
2354         * bmalloc/Deallocator.cpp:
2355         (bmalloc::Deallocator::~Deallocator):
2356         (bmalloc::Deallocator::scavenge):
2357         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
2358         functions, for reuse.
2359
2360         * bmalloc/Heap.cpp:
2361         (bmalloc::sleep):
2362         (bmalloc::Heap::concurrentScavenge):
2363         (bmalloc::Heap::scavenge):
2364         (bmalloc::Heap::scavengeSmallPages):
2365         (bmalloc::Heap::scavengeMediumPages):
2366         (bmalloc::Heap::scavengeLargeRanges):
2367         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
2368         scavenging -- in response to a benchmark or a low memory warning --
2369         wants to complete as soon as possible, so its sleep duration is 0.
2370
2371         * bmalloc/bmalloc.h:
2372         (bmalloc::api::scavenge):
2373         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
2374
2375 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2376
2377         Use 4kB pages on Mac
2378         https://bugs.webkit.org/show_bug.cgi?id=131658
2379
2380         Reviewed by Sam Weinig.
2381
2382         This reduces memory use a lot on Membuster:
2383
2384                                                                   base                      patch                                Δ
2385                 Execution Time:
2386                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
2387                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
2388                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
2389
2390                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
2391                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
2392                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
2393
2394                 Peak Memory:
2395                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
2396                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
2397                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
2398
2399                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
2400                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
2401                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
2402
2403                 Memory at End:
2404                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
2405                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
2406                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
2407
2408                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
2409                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
2410                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
2411
2412
2413         * bmalloc.xcodeproj/project.pbxproj:
2414         * bmalloc/BPlatform.h: Added.
2415         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
2416         is 4kB on Mac.
2417
2418 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
2419
2420         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
2421
2422         * bmalloc.xcodeproj: Modified property svn:ignore.
2423
2424 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
2425
2426         Fixed some mbmalloc exports
2427         https://bugs.webkit.org/show_bug.cgi?id=131599
2428
2429         Reviewed by Ryosuke Niwa.
2430
2431         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
2432         of the project, so we can call them from API.
2433
2434         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
2435         visibility, so they show up as exported in the .dylib.
2436
2437 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
2438
2439         Put bmalloc headers in the right place
2440         https://bugs.webkit.org/show_bug.cgi?id=131464
2441
2442         Reviewed by Mark Rowe.
2443
2444         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
2445         specify that we don't just want to dump all of our generically-named
2446         headers into /usr/local/include.
2447
2448 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2449
2450         Made bmalloc more #include friendly
2451         https://bugs.webkit.org/show_bug.cgi?id=131386
2452
2453         Reviewed by Andreas Kling.
2454
2455         Marked a bunch of headers private so they can be used from client code
2456         that #includes bmalloc.h.
2457
2458         Renamed ASSERT macros to BASSERT. This matches their header, which already
2459         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
2460
2461         * bmalloc.xcodeproj/project.pbxproj:
2462         * bmalloc/Allocator.cpp:
2463         (bmalloc::Allocator::allocateSlowCase):
2464         * bmalloc/AsyncTask.h:
2465         (bmalloc::Function>::runSlowCase):
2466         * bmalloc/BAssert.h:
2467         * bmalloc/BoundaryTag.h:
2468         (bmalloc::BoundaryTag::setSize):
2469         * bmalloc/BoundaryTagInlines.h:
2470         (bmalloc::validate):
2471         (bmalloc::BoundaryTag::init):
2472         (bmalloc::BoundaryTag::deallocate):
2473         (bmalloc::BoundaryTag::splitLarge):
2474         (bmalloc::BoundaryTag::allocate):
2475         * bmalloc/Chunk.h:
2476         * bmalloc/Deallocator.cpp:
2477         (bmalloc::Deallocator::processObjectLog):
2478         (bmalloc::Deallocator::deallocateSlowCase):
2479         * bmalloc/Deallocator.h:
2480         (bmalloc::Deallocator::deallocateFastCase):
2481         * bmalloc/FixedVector.h:
2482         (bmalloc::Capacity>::operator):
2483         (bmalloc::Capacity>::push):
2484         (bmalloc::Capacity>::pop):
2485         (bmalloc::Capacity>::shrink):
2486         * bmalloc/Heap.cpp:
2487         (bmalloc::Heap::allocateLarge):
2488         * bmalloc/LargeChunk.h:
2489         (bmalloc::LargeChunk::get):
2490         (bmalloc::LargeChunk::endTag):
2491         * bmalloc/Line.h:
2492         (bmalloc::Line<Traits>::concurrentRef):
2493         (bmalloc::Line<Traits>::deref):
2494         * bmalloc/MediumAllocator.h:
2495         (bmalloc::MediumAllocator::allocate):
2496         * bmalloc/ObjectType.h:
2497         (bmalloc::isSmall):
2498         * bmalloc/Page.h:
2499         (bmalloc::Page<Traits>::ref):
2500         (bmalloc::Page<Traits>::deref):
2501         * bmalloc/PerThread.h:
2502         (bmalloc::PerThread<T>::getSlowCase):
2503         * bmalloc/SegregatedFreeList.cpp:
2504         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2505         (bmalloc::SegregatedFreeList::insert):
2506         * bmalloc/SmallAllocator.h:
2507         (bmalloc::SmallAllocator::allocate):
2508         (bmalloc::SmallAllocator::refill):
2509         * bmalloc/Syscall.h:
2510         * bmalloc/VMAllocate.h:
2511         (bmalloc::vmValidate):
2512         (bmalloc::vmAllocate):
2513         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2514         * bmalloc/Vector.h:
2515         (bmalloc::Vector<T>::operator):
2516         (bmalloc::Vector<T>::pop):
2517         (bmalloc::Vector<T>::shrink):
2518         * bmalloc/XLargeChunk.h:
2519         (bmalloc::XLargeChunk::range):
2520         (bmalloc::XLargeChunk::size):
2521
2522 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2523
2524         Removed an unused file.
2525
2526         Unreviewed.
2527
2528         * bmalloc/AsyncTask.cpp: Removed.
2529
2530 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2531
2532         Build bmalloc on Mac
2533         https://bugs.webkit.org/show_bug.cgi?id=131333
2534
2535         Reviewed by Mark Rowe.
2536
2537         * Makefile: Added. For make clients.
2538
2539         These files are required for building any project in WebKit. I copied
2540         them from WTF:
2541         * Configurations: Added.
2542         * Configurations/Base.xcconfig: Added.
2543         * Configurations/DebugRelease.xcconfig: Added.
2544         * Configurations/bmalloc.xcconfig: Added.
2545         * Configurations/iOS.xcconfig: Added.
2546         * Configurations/mbmalloc.xcconfig: Added.
2547
2548         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
2549         from here because everything is in .xcconfig files now.
2550
2551         I had to fix a bunch of minor warnings, since they're enabled in our
2552         .xcconfig files:
2553
2554         * bmalloc/AsyncTask.h:
2555         (bmalloc::Function>::AsyncTask):
2556         * bmalloc/BAssert.h:
2557         * bmalloc/BoundaryTagInlines.h:
2558         (bmalloc::validate):
2559         * bmalloc/Heap.cpp:
2560         (bmalloc::Heap::Heap):
2561         (bmalloc::Heap::allocateLarge):
2562         (bmalloc::Heap::deallocateLarge):
2563         * bmalloc/Mutex.h:
2564         (bmalloc::Mutex::Mutex): Deleted.
2565         * bmalloc/VMAllocate.h:
2566         (bmalloc::vmValidate):
2567         * bmalloc/mbmalloc.cpp:
2568
2569 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2570
2571         bmalloc: Fixed a leak in the per-thread cache
2572         https://bugs.webkit.org/show_bug.cgi?id=131330
2573
2574         Reviewed by Andreas Kling.
2575
2576         Remember to deallocate our line caches upon thread exit.
2577
2578         * bmalloc/Deallocator.cpp:
2579         (bmalloc::Deallocator::~Deallocator):
2580
2581 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2582
2583         bmalloc: rolled out the tryLock experiment
2584         https://bugs.webkit.org/show_bug.cgi?id=131328
2585
2586         Reviewed by Andreas Kling.
2587
2588         It wasn't a speedup.
2589
2590         * bmalloc.xcodeproj/project.pbxproj:
2591         * bmalloc/Allocator.cpp:
2592         (bmalloc::Allocator::processSmallAllocatorLog):
2593         (bmalloc::Allocator::processMediumAllocatorLog):
2594         * bmalloc/Deallocator.cpp:
2595         (bmalloc::Deallocator::processObjectLog):
2596         (bmalloc::Deallocator::deallocateSlowCase):
2597         (bmalloc::Deallocator::deallocateSmallLine):
2598         (bmalloc::Deallocator::deallocateMediumLine):
2599         * bmalloc/Deallocator.h:
2600         (bmalloc::Deallocator::deallocateFastCase):
2601         * bmalloc/Heap.h:
2602         (bmalloc::Heap::deallocateSmallLine):
2603         (bmalloc::Heap::deallocateMediumLine):
2604         * bmalloc/Line.h:
2605         (bmalloc::Line<Traits>::deref):
2606         * bmalloc/Page.h:
2607         (bmalloc::Page<Traits>::deref):
2608
2609 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2610
2611         bmalloc
2612         https://bugs.webkit.org/show_bug.cgi?id=131170
2613
2614         Reviewed by Andreas Kling.
2615
2616         Initial commit.
2617
2618         * bmalloc: Added.
2619         * bmalloc.xcodeproj: Added.
2620         * bmalloc.xcodeproj/project.pbxproj: Added.
2621         * bmalloc/Algorithm.h: Added.
2622         (bmalloc::max):
2623         (bmalloc::min):
2624         (bmalloc::mask):
2625         (bmalloc::test):
2626         (bmalloc::roundUpToMultipleOf):
2627         (bmalloc::roundDownToMultipleOf):
2628         (bmalloc::sizeOf):
2629         (bmalloc::bitCount):
2630         (bmalloc::isPowerOfTwo):
2631         * bmalloc/Allocator.cpp: Added.
2632         (bmalloc::Allocator::Allocator):
2633         (bmalloc::Allocator::~Allocator):
2634         (bmalloc::Allocator::log):
2635         (bmalloc::Allocator::processSmallAllocatorLog):
2636         (bmalloc::Allocator::processMediumAllocatorLog):
2637         (bmalloc::Allocator::allocateLarge):
2638         (bmalloc::Allocator::allocateXLarge):
2639         (bmalloc::Allocator::allocateMedium):
2640         (bmalloc::Allocator::allocateSlowCase):
2641         * bmalloc/Allocator.h: Added.
2642         (bmalloc::Allocator::smallAllocatorFor):
2643         (bmalloc::Allocator::allocateFastCase):
2644         (bmalloc::Allocator::allocate):
2645         * bmalloc/AsyncTask.cpp: Added.
2646         (bmalloc::AsyncTask<Function>::runSlowCase):
2647         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2648         (bmalloc::AsyncTask<Function>::entryPoint):
2649         * bmalloc/AsyncTask.h: Added.
2650         (bmalloc::Function>::AsyncTask):
2651         (bmalloc::Function>::join):
2652         (bmalloc::Function>::run):
2653         (bmalloc::Function>::runSlowCase):
2654         (bmalloc::Function>::pthreadEntryPoint):
2655         (bmalloc::Function>::entryPoint):
2656         * bmalloc/BAssert.h: Added.
2657         * bmalloc/BeginTag.h: Added.
2658         (bmalloc::BeginTag::isInFreeList):
2659         * bmalloc/BoundaryTag.h: Added.
2660         (bmalloc::BoundaryTag::isXLarge):
2661         (bmalloc::BoundaryTag::setXLarge):
2662         (bmalloc::BoundaryTag::isFree):
2663         (bmalloc::BoundaryTag::setFree):
2664         (bmalloc::BoundaryTag::isEnd):
2665         (bmalloc::BoundaryTag::setEnd):
2666         (bmalloc::BoundaryTag::hasPhysicalPages):
2667         (bmalloc::BoundaryTag::setHasPhysicalPages):
2668         (bmalloc::BoundaryTag::isNull):
2669         (bmalloc::BoundaryTag::clear):
2670         (bmalloc::BoundaryTag::size):
2671         (bmalloc::BoundaryTag::setSize):
2672         (bmalloc::BoundaryTag::prev):
2673         (bmalloc::BoundaryTag::next):
2674         * bmalloc/BoundaryTagInlines.h: Added.
2675         (bmalloc::validate):
2676         (bmalloc::validatePrev):
2677         (bmalloc::validateNext):
2678         (bmalloc::BoundaryTag::init):
2679         (bmalloc::BoundaryTag::mergeLargeLeft):
2680         (bmalloc::BoundaryTag::mergeLargeRight):
2681         (bmalloc::BoundaryTag::mergeLarge):
2682         (bmalloc::BoundaryTag::deallocate):
2683         (bmalloc::BoundaryTag::splitLarge):
2684         (bmalloc::BoundaryTag::allocate):
2685         * bmalloc/Cache.cpp: Added.
2686         (bmalloc::Cache::operator new):
2687         (bmalloc::Cache::operator delete):
2688         (bmalloc::Cache::Cache):
2689         (bmalloc::Cache::allocateSlowCase):
2690         (bmalloc::Cache::allocateSlowCaseNullCache):
2691         (bmalloc::Cache::deallocateSlowCase):
2692         (bmalloc::Cache::deallocateSlowCaseNullCache):
2693         * bmalloc/Cache.h: Added.
2694         (bmalloc::Cache::allocator):
2695         (bmalloc::Cache::deallocator):
2696         (bmalloc::Cache::allocateFastCase):
2697         (bmalloc::Cache::deallocateFastCase):
2698         (bmalloc::Cache::allocate):
2699         (bmalloc::Cache::deallocate):
2700         * bmalloc/Chunk.h: Added.
2701         (bmalloc::Chunk::begin):
2702         (bmalloc::Chunk::end):
2703         (bmalloc::Chunk::lines):
2704         (bmalloc::Chunk::pages):
2705         * bmalloc/Deallocator.cpp: Added.
2706         (bmalloc::Deallocator::Deallocator):
2707         (bmalloc::Deallocator::~Deallocator):
2708         (bmalloc::Deallocator::deallocateLarge):
2709         (bmalloc::Deallocator::deallocateXLarge):
2710         (bmalloc::Deallocator::processObjectLog):
2711         (bmalloc::Deallocator::deallocateSlowCase):
2712         (bmalloc::Deallocator::deallocateSmallLine):
2713         (bmalloc::Deallocator::allocateSmallLine):
2714         (bmalloc::Deallocator::deallocateMediumLine):
2715         (bmalloc::Deallocator::allocateMediumLine):
2716         * bmalloc/Deallocator.h: Added.
2717         (bmalloc::Deallocator::deallocateFastCase):
2718         (bmalloc::Deallocator::deallocate):
2719         * bmalloc/EndTag.h: Added.
2720         (bmalloc::EndTag::operator=):
2721         * bmalloc/FixedVector.h: Added.
2722         (bmalloc::FixedVector::begin):
2723         (bmalloc::FixedVector::end):
2724         (bmalloc::FixedVector::size):
2725         (bmalloc::FixedVector::capacity):
2726         (bmalloc::FixedVector::clear):
2727         (bmalloc::FixedVector::isEmpty):
2728         (bmalloc::Capacity>::FixedVector):
2729         (bmalloc::Capacity>::operator):
2730         (bmalloc::Capacity>::push):
2731         (bmalloc::Capacity>::pop):
2732         (bmalloc::Capacity>::shrink):
2733         * bmalloc/Heap.cpp: Added.
2734         (bmalloc::sleep):
2735         (bmalloc::Heap::Heap):
2736         (bmalloc::Heap::concurrentScavenge):
2737         (bmalloc::Heap::scavengeSmallPages):
2738         (bmalloc::Heap::scavengeMediumPages):
2739         (bmalloc::Heap::scavengeLargeRanges):
2740         (bmalloc::Heap::allocateSmallLineSlowCase):
2741         (bmalloc::Heap::allocateMediumLineSlowCase):
2742         (bmalloc::Heap::allocateXLarge):
2743         (bmalloc::Heap::deallocateXLarge):
2744         (bmalloc::Heap::allocateLarge):
2745         (bmalloc::Heap::deallocateLarge):
2746         * bmalloc/Heap.h: Added.
2747         (bmalloc::Heap::deallocateSmallLine):
2748         (bmalloc::Heap::allocateSmallLine):
2749         (bmalloc::Heap::deallocateMediumLine):
2750         (bmalloc::Heap::allocateMediumLine):
2751         * bmalloc/Inline.h: Added.
2752         * bmalloc/LargeChunk.h: Added.
2753         (bmalloc::LargeChunk::begin):
2754         (bmalloc::LargeChunk::end):
2755         (bmalloc::LargeChunk::create):
2756         (bmalloc::LargeChunk::get):
2757         (bmalloc::LargeChunk::beginTag):
2758         (bmalloc::LargeChunk::endTag):
2759         * bmalloc/Line.h: Added.
2760         (bmalloc::Line<Traits>::begin):
2761         (bmalloc::Line<Traits>::end):
2762         (bmalloc::Line<Traits>::concurrentRef):
2763         (bmalloc::Line<Traits>::deref):
2764         * bmalloc/MediumAllocator.h: Added.
2765         (bmalloc::MediumAllocator::isNull):
2766         (bmalloc::MediumAllocator::MediumAllocator):
2767         (bmalloc::MediumAllocator::line):
2768         (bmalloc::MediumAllocator::allocate):
2769         (bmalloc::MediumAllocator::derefCount):
2770         (bmalloc::MediumAllocator::refill):
2771         * bmalloc/MediumChunk.h: Added.
2772         * bmalloc/MediumLine.h: Added.
2773         * bmalloc/MediumPage.h: Added.
2774         * bmalloc/MediumTraits.h: Added.
2775         * bmalloc/Mutex.cpp: Added.
2776         (bmalloc::Mutex::lockSlowCase):
2777         * bmalloc/Mutex.h: Added.
2778         (bmalloc::Mutex::Mutex):
2779         (bmalloc::Mutex::try_lock):
2780         (bmalloc::Mutex::lock):
2781         (bmalloc::Mutex::unlock):
2782         * bmalloc/ObjectType.cpp: Added.
2783         (bmalloc::objectType):
2784         * bmalloc/ObjectType.h: Added.
2785         (bmalloc::isSmallOrMedium):
2786         (bmalloc::isSmall):
2787         * bmalloc/Page.h: Added.
2788         (bmalloc::Page<Traits>::ref):
2789         (bmalloc::Page<Traits>::deref):
2790         (bmalloc::Page<Traits>::refCount):
2791         * bmalloc/PerProcess.h: Added.
2792         (bmalloc::PerProcess::mutex):
2793         (bmalloc::PerProcess<T>::getFastCase):
2794         (bmalloc::PerProcess<T>::get):
2795         (bmalloc::PerProcess<T>::getSlowCase):
2796         * bmalloc/PerThread.h: Added.
2797         (bmalloc::PerThreadStorage<Cache>::get):
2798         (bmalloc::PerThreadStorage<Cache>::init):
2799         (bmalloc::PerThreadStorage::get):
2800         (bmalloc::PerThreadStorage::init):
2801         (bmalloc::PerThread<T>::getFastCase):
2802         (bmalloc::PerThread<T>::get):
2803         (bmalloc::PerThread<T>::destructor):
2804         (bmalloc::PerThread<T>::getSlowCase):
2805         * bmalloc/Range.h: Added.
2806         (bmalloc::Range::Range):
2807         (bmalloc::Range::begin):
2808         (bmalloc::Range::end):
2809         (bmalloc::Range::size):
2810         (bmalloc::Range::operator!):
2811         (bmalloc::Range::operator<):
2812         * bmalloc/SegregatedFreeList.cpp: Added.
2813         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2814         (bmalloc::SegregatedFreeList::insert):
2815         (bmalloc::SegregatedFreeList::takeGreedy):
2816         (bmalloc::SegregatedFreeList::take):
2817         * bmalloc/SegregatedFreeList.h: Added.
2818         * bmalloc/Sizes.h: Added.
2819         * bmalloc/SmallAllocator.h: Added.
2820         (bmalloc::SmallAllocator::isNull):
2821         (bmalloc::SmallAllocator::canAllocate):
2822         (bmalloc::SmallAllocator::SmallAllocator):
2823         (bmalloc::SmallAllocator::line):
2824         (bmalloc::SmallAllocator::allocate):
2825         (bmalloc::SmallAllocator::objectCount):
2826         (bmalloc::SmallAllocator::derefCount):
2827         (bmalloc::SmallAllocator::refill):
2828         * bmalloc/SmallChunk.h: Added.
2829         * bmalloc/SmallLine.h: Added.
2830         * bmalloc/SmallPage.h: Added.
2831         * bmalloc/SmallTraits.h: Added.
2832         * bmalloc/Syscall.h: Added.
2833         * bmalloc/VMAllocate.h: Added.
2834         (bmalloc::vmSize):
2835         (bmalloc::vmValidate):
2836         (bmalloc::vmAllocate):
2837         (bmalloc::vmDeallocate):
2838         (bmalloc::vmDeallocatePhysicalPages):
2839         (bmalloc::vmAllocatePhysicalPages):
2840         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2841         (bmalloc::vmAllocatePhysicalPagesSloppy):
2842         * bmalloc/VMHeap.cpp: Added.
2843         (bmalloc::VMHeap::VMHeap):
2844         (bmalloc::VMHeap::allocateSmallChunk):
2845         (bmalloc::VMHeap::allocateMediumChunk):
2846         (bmalloc::VMHeap::allocateLargeChunk):
2847         * bmalloc/VMHeap.h: Added.
2848         (bmalloc::VMHeap::allocateSmallPage):
2849         (bmalloc::VMHeap::allocateMediumPage):
2850         (bmalloc::VMHeap::allocateLargeRange):
2851         (bmalloc::VMHeap::deallocateSmallPage):
2852         (bmalloc::VMHeap::deallocateMediumPage):
2853         (bmalloc::VMHeap::deallocateLargeRange):
2854         * bmalloc/Vector.h: Added.
2855         (bmalloc::Vector::begin):
2856         (bmalloc::Vector::end):
2857         (bmalloc::Vector::size):
2858         (bmalloc::Vector::capacity):
2859         (bmalloc::Vector::last):
2860         (bmalloc::Vector::pop):
2861         (bmalloc::Vector<T>::Vector):
2862         (bmalloc::Vector<T>::~Vector):
2863         (bmalloc::Vector<T>::operator):
2864         (bmalloc::Vector<T>::push):
2865         (bmalloc::Vector<T>::pop):
2866         (bmalloc::Vector<T>::shrink):
2867         (bmalloc::Vector<T>::reallocateBuffer):
2868         (bmalloc::Vector<T>::shrinkCapacity):
2869         (bmalloc::Vector<T>::growCapacity):
2870         * bmalloc/XLargeChunk.h: Added.
2871         (bmalloc::XLargeChunk::get):
2872         (bmalloc::XLargeChunk::begin):
2873         (bmalloc::XLargeChunk::XLargeChunk):
2874         (bmalloc::XLargeChunk::create):
2875         (bmalloc::XLargeChunk::destroy):
2876         (bmalloc::XLargeChunk::range):
2877         (bmalloc::XLargeChunk::size):
2878         * bmalloc/bmalloc.h: Added.
2879         (bmalloc::api::malloc):
2880         (bmalloc::api::free):
2881         (bmalloc::api::realloc):
2882         * bmalloc/mbmalloc.cpp: Added.
2883