7e1948af9118608d31c90cc9c10d5946b586bc7a
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-05-07  Geoffrey Garen  <ggaren@apple.com>
2
3         Release assert in com.apple.WebKit.WebContent under JavaScriptCore: JSC::JSONProtoFuncStringify
4         https://bugs.webkit.org/show_bug.cgi?id=144758
5
6         Reviewed by Andreas Kling.
7
8         This was an out-of-memory error when trying to shrink a string builder.
9         bmalloc was missing the optimization that allowed realloc() to shrink
10         without copying. So, let's add it.
11
12         * bmalloc/Allocator.cpp:
13         (bmalloc::Allocator::reallocate): Added Large and XLarge cases for
14         shrinking without copying. This isn't possible for small and medium
15         objects, and probably not very profitable, either.
16
17         * bmalloc/Heap.cpp:
18         (bmalloc::Heap::findXLarge):
19         (bmalloc::Heap::deallocateXLarge):
20         * bmalloc/Heap.h: Refactored this code to return a reference to an
21         XLarge range. This makes the code reusable, and also makes it easier
22         for realloc() to update metadata.
23
24         * bmalloc/LargeObject.h:
25         (bmalloc::LargeObject::split): Allow allocated objects to split because
26         that's what realloc() wants to do, and there's nothing intrinsically
27         wrong with it.
28
29 2015-05-07  Dan Bernstein  <mitz@apple.com>
30
31         <rdar://problem/19317140> [Xcode] Remove usage of AspenFamily.xcconfig in Source/
32         https://bugs.webkit.org/show_bug.cgi?id=144727
33
34         Reviewed by Darin Adler.
35
36         * Configurations/Base.xcconfig: Dont’s include AspenFamily.xcconfig, and define
37         INSTALL_PATH_PREFIX for the iOS 8.x Simulator.
38
39 2015-04-01  Alex Christensen  <achristensen@webkit.org>
40
41         Progress towards CMake on Windows and Mac.
42         https://bugs.webkit.org/show_bug.cgi?id=143293
43
44         Reviewed by Filip Pizlo.
45
46         * bmalloc/BAssert.h:
47         Removed ellipses from macros to appease Visual Studio.
48
49 2015-03-13  Alex Christensen  <achristensen@webkit.org>
50
51         Progress towards CMake on Mac.
52         https://bugs.webkit.org/show_bug.cgi?id=142680
53
54         Reviewed by Gyuyoung Kim.
55
56         * CMakeLists.txt:
57         * PlatformMac.cmake:
58         Added Zone.cpp to Mac CMake builds.
59
60 2015-03-12  Geoffrey Garen  <ggaren@apple.com>
61
62         Assertion failure in bmalloc::LargeObject::validateSelf on Mavericks Debug layout test bot
63         https://bugs.webkit.org/show_bug.cgi?id=142642
64
65         Reviewed by Michael Saboff.
66
67         The typical backtrace to this crash shows the main thread trying to
68         realloc a large string while a DFG compiler thread tries to
69         free a large vector buffer.
70
71         I believe that this is a race condition -- at least in debug builds --
72         since the main thread will try to validate its object's neighbors
73         without holding a lock, even though those neighbors might be in the
74         midst of changing.
75
76         In general, there may be sneaky times when it is valid to look at an
77         object's metadata without holding the heap lock, but it is best not to
78         do so unless we have a really really good reason to.
79
80         * bmalloc/Allocator.cpp:
81         (bmalloc::Allocator::reallocate): Take a lock before reading the metadata
82         for this object, since we generally require any access to shared heap
83         metadata to take a lock.
84
85 2015-03-10  Geoffrey Garen  <ggaren@apple.com>
86
87         bmalloc: tryFastMalloc shouldn't crash
88         https://bugs.webkit.org/show_bug.cgi?id=142443
89
90         Reviewed by Sam Weinig.
91
92         Rolling back in r181307 with a check for whether bmalloc is enabled, to
93         avoid crashes when running with ASan and GuardMalloc.
94
95         * bmalloc/Allocator.cpp:
96         (bmalloc::Allocator::tryAllocate):
97         * bmalloc/Allocator.h:
98         * bmalloc/Cache.cpp:
99         (bmalloc::Cache::tryAllocateSlowCaseNullCache):
100         * bmalloc/Cache.h:
101         (bmalloc::Cache::tryAllocate):
102         * bmalloc/Heap.cpp:
103         (bmalloc::Heap::allocateXLarge):
104         (bmalloc::Heap::tryAllocateXLarge):
105         * bmalloc/Heap.h:
106         * bmalloc/VMAllocate.h:
107         (bmalloc::tryVMAllocate):
108         (bmalloc::vmAllocate):
109         * bmalloc/bmalloc.h:
110         (bmalloc::api::tryMalloc):
111         (bmalloc::api::realloc):
112         (bmalloc::api::free):
113
114 2015-03-09  Commit Queue  <commit-queue@webkit.org>
115
116         Unreviewed, rolling out r181307.
117         https://bugs.webkit.org/show_bug.cgi?id=142525
118
119         Broke ASan tests (Requested by ap on #webkit).
120
121         Reverted changeset:
122
123         "bmalloc: tryFastMalloc shouldn't crash"
124         https://bugs.webkit.org/show_bug.cgi?id=142443
125         http://trac.webkit.org/changeset/181307
126
127 2015-03-09  Geoffrey Garen  <ggaren@apple.com>
128
129         bmalloc: tryFastMalloc shouldn't crash
130         https://bugs.webkit.org/show_bug.cgi?id=142443
131
132         Reviewed by Darin Adler.
133
134         Added support for tryMalloc.
135
136         We assume that non-x-large allocations always succeed, and we crash
137         otherwise, since normal allocation failure will just cause the next
138         non-try allocation or internal metadata allocation to fail, and it's
139         hard and not really useful to keep limping along after that. But
140         extra-large allocations can meaningfully fail, and we can recover.
141
142         * bmalloc/Heap.cpp:
143         (bmalloc::Heap::allocateXLarge):
144         (bmalloc::Heap::tryAllocateXLarge):
145         * bmalloc/Heap.h: Added support for non-crashy x-large allocation.
146
147         * bmalloc/VMAllocate.h:
148         (bmalloc::tryVMAllocate):
149         (bmalloc::vmAllocate): Added support for non-crashy VM allocation.
150
151         * bmalloc/bmalloc.h:
152         (bmalloc::api::tryMalloc):
153         (bmalloc::api::realloc):
154         (bmalloc::api::free): Tried to clarify our behavior with some comments.
155         Unfortunately, calling what we do "malloc" is still not quite right, since
156         malloc returns null on failure and we don't.
157
158 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
159
160         bmalloc: Don't branch when setting the owner of a large object
161         https://bugs.webkit.org/show_bug.cgi?id=142241
162
163         Reviewed by Andreas Kling.
164
165         * bmalloc/BoundaryTag.h:
166         (bmalloc::BoundaryTag::owner):
167         (bmalloc::BoundaryTag::setOwner):
168
169 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
170
171         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
172         https://bugs.webkit.org/show_bug.cgi?id=141802
173
174         Reviewed by Andreas Kling.
175
176         Re-enabled this feature on iOS, now that the iOS crash should be fixed.
177
178         * bmalloc/VMHeap.cpp:
179         (bmalloc::VMHeap::grow):
180         * bmalloc/VMHeap.h:
181
182 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
183
184         bmalloc: Added missing features to the malloc zone introspection API
185         https://bugs.webkit.org/show_bug.cgi?id=142235
186
187         Reviewed by Andreas Kling.
188
189         This should fix the crash we saw on the iOS PLT bot
190         (c.f. http://trac.webkit.org/changeset/180604).
191
192         * bmalloc/Zone.cpp:
193         (bmalloc::good_size):
194         (bmalloc::check):
195         (bmalloc::print):
196         (bmalloc::log):
197         (bmalloc::force_lock):
198         (bmalloc::force_unlock):
199         (bmalloc::statistics):
200         (bmalloc::size):
201         (bmalloc::enumerator): Provide all of these functions since they are called
202         indiscriminately on all zones.
203
204         (bmalloc::Zone::Zone):
205         (bmalloc::Zone::size): Deleted.
206         (bmalloc::Zone::enumerator): Deleted. Moved these functions out of the
207         Zone class since they can stand alone.
208
209         * bmalloc/Zone.h:
210
211 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
212
213         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
214         https://bugs.webkit.org/show_bug.cgi?id=141802
215
216         Reviewed by Andreas Kling.
217
218         Rolling back in but disabled on iOS until I can debug why the iOS PLT crashes.
219
220         * bmalloc/VMHeap.cpp:
221         (bmalloc::VMHeap::grow):
222         * bmalloc/VMHeap.h:
223         * bmalloc/Zone.cpp:
224         (bmalloc::Zone::size):
225         (bmalloc::Zone::Zone):
226         * bmalloc/Zone.h:
227
228 2015-03-03  Geoffrey Garen  <ggaren@apple.com>
229
230         bmalloc: Miscellaneous cleanup
231         https://bugs.webkit.org/show_bug.cgi?id=142231
232
233         Reviewed by Andreas Kling.
234
235         No performance change -- maybe a tiny reduction in memory use.
236
237         * bmalloc/Heap.cpp: Moved the sleep function into StaticMutex, since
238         it's a helper for working with mutexes.
239
240         (bmalloc::Heap::scavenge): Make sure to wait before we start any
241         scavenging, since individual scavenging functions now always scavenge
242         at least one page before waiting themselves.
243
244         (bmalloc::Heap::scavengeSmallPages):
245         (bmalloc::Heap::scavengeMediumPages):
246         (bmalloc::Heap::scavengeLargeObjects): Use the new wait helper to
247         simplify this code. Also, we now require our caller to wait until at
248         least one deallocation is desirable. This simplifies our loop.
249
250         (bmalloc::Heap::allocateSmallPage):
251         (bmalloc::Heap::allocateMediumPage):
252         (bmalloc::Heap::allocateXLarge):
253         (bmalloc::Heap::allocateLarge): Don't freak out any time the heap does
254         an allocation. Only consider the heap to be growing if it actually needs
255         to allocate new VM. This allows us to shrink the heap back down from a
256         high water mark more reliably even if heap activity continues.
257
258         (bmalloc::sleep): Deleted.
259         (bmalloc::Heap::scavengeLargeRanges): Renamed to match our use of
260         "LargeObject".
261
262         * bmalloc/Heap.h:
263
264         * bmalloc/LargeObject.h:
265         (bmalloc::LargeObject::operator bool): Added to simplify a while loop.
266
267         * bmalloc/StaticMutex.h:
268         (bmalloc::sleep):
269         (bmalloc::waitUntilFalse): New helper for waiting until a condition
270         becomes reliably false.
271
272         * bmalloc/Vector.h:
273         (bmalloc::Vector<T>::~Vector): Oops! Don't deallocate the null pointer.
274         We don't actually run any Vector destructors, but an iteration of this
275         patch did, and then crashed. So, let's fix that.
276
277 2015-03-02  Geoffrey Garen  <ggaren@apple.com>
278
279         bmalloc: Eagerly remove allocated objects from the free list
280         https://bugs.webkit.org/show_bug.cgi?id=142194
281
282         Reviewed by Andreas Kling.
283
284         This reduces the pressure to garbage collect the free list.
285
286         Might be a 1% speedup on MallocBench.
287
288         * bmalloc/FreeList.cpp: Put this comment at the top of the file instead
289         of repeating it inside of each function. Tried to clarify the details.
290
291         (bmalloc::FreeList::takeGreedy): Matched the other iteration code in this
292         file for consistency -- even though either direction works fine in this
293         function.
294
295         (bmalloc::FreeList::take): Change to iterate from low to high so that we
296         can maintain an index into the vector that is not disturbed even if we
297         pop from the middle (which invalidates the last index in the vector).
298
299         Decrement i when popping from the middle to make sure that we don't
300         skip the next item after popping.
301
302         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): Ditto.
303
304 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
305
306         Fixed a typo in the previous commit.
307
308         * bmalloc/BoundaryTag.h:
309         (bmalloc::BoundaryTag::setOwner):
310
311 2015-02-27  Ryosuke Niwa  <rniwa@webkit.org>
312
313         EFL build fix after r180797.
314
315         * bmalloc/BoundaryTag.h:
316         (bmalloc::BoundaryTag::owner):
317         (bmalloc::BoundaryTag::setOwner):
318
319 2015-02-27  Geoffrey Garen  <ggaren@apple.com>
320
321         bmalloc: Pathological madvise churn on the free(malloc(x)) benchmark
322         https://bugs.webkit.org/show_bug.cgi?id=142058
323
324         Reviewed by Andreas Kling.
325
326         The churn was caused by repeatedly splitting an object with physical
327         pages from an object without, and then merging them back together again.
328         The merge would conservatively forget that we had physical pages, forcing
329         a new call to madvise on the next allocation.
330
331         This patch more strictly segregates objects in the heap from objects in
332         the VM heap, with these changes:
333
334         (1) Objects in the heap are not allowed to merge with objects in the VM
335         heap, and vice versa -- since that would erase our precise knowledge of
336         which physical pages had been allocated.
337
338         (2) The VM heap is exclusively responsible for allocating and deallocating
339         physical pages.
340
341         (3) The heap free list must consider entries for objects that are in the
342         VM heap to be invalid, and vice versa. (This condition can arise
343         because the free list does not eagerly remove items.)
344
345         With these changes, we can know that any valid object in the heap's free
346         list already has physical pages, and does not need to call madvise.
347
348         Note that the VM heap -- as before -- might sometimes contain ranges
349         or pieces of ranges that have physical pages, since we allow splitting
350         of ranges at granularities smaller than the VM page size. These ranges
351         can eventually merge with ranges in the heap during scavenging.
352
353         * bmalloc.xcodeproj/project.pbxproj:
354
355         * bmalloc/BoundaryTag.h:
356         (bmalloc::BoundaryTag::owner):
357         (bmalloc::BoundaryTag::setOwner):
358         (bmalloc::BoundaryTag::initSentinel):
359         (bmalloc::BoundaryTag::hasPhysicalPages): Deleted.
360         (bmalloc::BoundaryTag::setHasPhysicalPages): Deleted. Replaced the concept
361         of "has physical pages" with a bit indicating which heap owns the large
362         object. This is a more precise concept, since the old bit was really a
363         Yes / Maybe bit.
364
365         * bmalloc/Deallocator.cpp:
366
367         * bmalloc/FreeList.cpp: Adopt
368         (bmalloc::FreeList::takeGreedy):
369         (bmalloc::FreeList::take):
370         (bmalloc::FreeList::removeInvalidAndDuplicateEntries):
371         * bmalloc/FreeList.h:
372         (bmalloc::FreeList::push): Added API for considering the owner when
373         deciding if a free list entry is valid.
374
375         * bmalloc/Heap.cpp:
376         (bmalloc::Heap::Heap): Adopt new API.
377
378         (bmalloc::Heap::scavengeLargeRanges): Scavenge all ranges with no minimum,
379         since some ranges might be able to merge with ranges in the VM heap, and
380         they won't be allowed to until we scavenge them.
381
382         (bmalloc::Heap::allocateSmallPage):
383         (bmalloc::Heap::allocateMediumPage):
384         (bmalloc::Heap::allocateLarge): New VM heap API makes this function
385         simpler, since we always get back physical pages now.
386
387         * bmalloc/Heap.h:
388         * bmalloc/LargeObject.h:
389         (bmalloc::LargeObject::end):
390         (bmalloc::LargeObject::owner):
391         (bmalloc::LargeObject::setOwner):
392         (bmalloc::LargeObject::isValidAndFree):
393         (bmalloc::LargeObject::merge): Do not merge objects across heaps since
394         that causes madvise churn.
395         (bmalloc::LargeObject::validateSelf):
396         (bmalloc::LargeObject::init):
397         (bmalloc::LargeObject::hasPhysicalPages): Deleted.
398         (bmalloc::LargeObject::setHasPhysicalPages): Deleted. Propogate the Owner API.
399
400         * bmalloc/Owner.h: Added.
401
402         * bmalloc/SegregatedFreeList.cpp:
403         (bmalloc::SegregatedFreeList::SegregatedFreeList):
404         (bmalloc::SegregatedFreeList::insert):
405         (bmalloc::SegregatedFreeList::takeGreedy):
406         (bmalloc::SegregatedFreeList::take):
407         * bmalloc/SegregatedFreeList.h: Propogate the owner API.
408
409         * bmalloc/VMAllocate.h:
410         (bmalloc::vmDeallocatePhysicalPagesSloppy):
411         (bmalloc::vmAllocatePhysicalPagesSloppy): Clarified these functions and
412         removed an edge case.
413
414         * bmalloc/VMHeap.cpp:
415         (bmalloc::VMHeap::VMHeap):
416         * bmalloc/VMHeap.h:
417         (bmalloc::VMHeap::allocateSmallPage):
418         (bmalloc::VMHeap::allocateMediumPage):
419         (bmalloc::VMHeap::allocateLargeObject):
420         (bmalloc::VMHeap::deallocateLargeObject): Be sure to give each object
421         a new chance to merge, since it might have been prohibited from merging
422         before by virtue of not being in the VM heap.
423
424         (bmalloc::VMHeap::allocateLargeRange): Deleted.
425         (bmalloc::VMHeap::deallocateLargeRange): Deleted.
426
427 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
428
429         bmalloc: Large object free list can grow infinitely
430         https://bugs.webkit.org/show_bug.cgi?id=142055
431
432         Reviewed by Andreas Kling.
433
434         By design, we don't eagerly remove large objects from the free list.
435         This creates two simple pathologies:
436
437             (1) If you free and then allocate the same object repeatedly, it will
438             duplicate itself in the free list repeatedly. Since it is never
439             invalid at the time of allocation, it will never be removed.
440
441             (2) If you split and then merge the same object repeatedly, it will
442             duplicate its split sibling in the free list repeatedly. If its
443             sibling is in a separate free list size class, it will never be
444             consulted at the time of allocation, so it will never be removed.
445
446         So, a simple "while (1) { free(malloc(x)); }" causes infinite memory
447         use in the free list.
448
449         The solution in this patch is a simple helper to remove garbage from the
450         free list if it grows too large. This pathology is not common, so the
451         cost is OK.
452
453         Long-term, perhaps we should rethink the laziness of these free lists.
454
455         * bmalloc/BoundaryTag.h:
456         (bmalloc::BoundaryTag::isMarked):
457         (bmalloc::BoundaryTag::setMarked): New bit, used by free list GC.
458
459         * bmalloc/FreeList.cpp:
460         (bmalloc::FreeList::removeInvalidAndDuplicateEntries): The GC algorithm.
461
462         * bmalloc/FreeList.h:
463         (bmalloc::FreeList::FreeList):
464         (bmalloc::FreeList::push): Invoke the GC if we're getting huge.
465
466         * bmalloc/LargeObject.h:
467         (bmalloc::LargeObject::isMarked):
468         (bmalloc::LargeObject::setMarked):
469         (bmalloc::LargeObject::validateSelf): Expose the new bit.
470
471         * bmalloc/Sizes.h: New constant to control GC frequency.
472
473 2015-02-26  Csaba Osztrogonác  <ossy@webkit.org>
474
475         URTBF after r180693.
476
477         * CMakeLists.txt:
478
479 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
480
481         Try to fix the Mac build.
482
483         Unreviewed.
484
485         * bmalloc.xcodeproj/project.pbxproj: Make FreeList.h available.
486
487 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
488
489         bmalloc: Refactored SegregatedFreeList and BoundaryTag::init
490         https://bugs.webkit.org/show_bug.cgi?id=142049
491
492         Reviewed by Anders Carlsson.
493
494         Split out a FreeList class from SegregatedFreeList. This will make it
495         easier to add behaviors on free list insertion and removal -- and it's
496         probably how I should have designed things at the start.
497
498         Moved BoundaryTag::init into LargeObject, since all the related logic
499         lives in LargeObject now too, and this allows us to remove BoundaryTagInlines.h.
500
501         * bmalloc.xcodeproj/project.pbxproj:
502         * bmalloc/BoundaryTagInlines.h: Removed.
503         * bmalloc/FreeList.cpp: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.cpp.
504         (bmalloc::FreeList::takeGreedy):
505         (bmalloc::FreeList::take):
506         (bmalloc::SegregatedFreeList::SegregatedFreeList): Deleted.
507         (bmalloc::SegregatedFreeList::insert): Deleted.
508         (bmalloc::SegregatedFreeList::takeGreedy): Deleted.
509         (bmalloc::SegregatedFreeList::take): Deleted.
510         * bmalloc/FreeList.h: Copied from Source/bmalloc/bmalloc/SegregatedFreeList.h.
511         (bmalloc::FreeList::push):
512         * bmalloc/LargeObject.h:
513         (bmalloc::LargeObject::init):
514         * bmalloc/SegregatedFreeList.cpp:
515         (bmalloc::SegregatedFreeList::SegregatedFreeList):
516         (bmalloc::SegregatedFreeList::insert):
517         (bmalloc::SegregatedFreeList::takeGreedy):
518         (bmalloc::SegregatedFreeList::take):
519         * bmalloc/SegregatedFreeList.h:
520         * bmalloc/Sizes.h:
521         * bmalloc/VMHeap.cpp:
522         (bmalloc::VMHeap::grow):
523
524 2015-02-26  Geoffrey Garen  <ggaren@apple.com>
525
526         bmalloc: free up a bit in BoundaryTag
527         https://bugs.webkit.org/show_bug.cgi?id=142048
528
529         Reviewed by Brady Eidson.
530
531         We were wasting a bit by accident, and I need one now.
532
533         * bmalloc/Algorithm.h:
534         (bmalloc::rightShift): Deleted. Not needed, now that I've simplified
535         the math.
536
537         * bmalloc/BoundaryTag.h: Since each boundary tag bucket is 1024 bytes
538         long, the maximum offset into a bucket is 1023.
539
540         You need 5 bits to count up to 1024, but only 4 to count up to 1023.
541         
542         Math is hard.
543
544         (bmalloc::BoundaryTag::compactBegin): Switched to division because it
545         is simpler, and easier to match up with our ASSERT. The compiler will
546         turn division by constant power of two into a shift for us.
547
548         (bmalloc::BoundaryTag::setRange): Added an ASSERT for compactBegin
549         because we do encode it, so we should ASSERT that encoding did not
550         lose information.
551
552         * bmalloc/Sizes.h: Shifting is no longer used since we use division
553         instead.
554
555 2015-02-24  Stephanie Lewis  <slewis@apple.com>
556
557         Rolling out http://trac.webkit.org/changeset/180430 as it causes the PLT to crash.
558         <rdar://problem/19948015>
559
560         Unreviewed.
561
562         * bmalloc/VMHeap.cpp:
563         (bmalloc::VMHeap::grow):
564         * bmalloc/VMHeap.h:
565         * bmalloc/Zone.cpp:
566         (bmalloc::Zone::Zone):
567         (bmalloc::Zone::size): Deleted.
568         * bmalloc/Zone.h:
569
570 2015-02-24  Geoffrey Garen  <ggaren@apple.com>
571
572         bmalloc: Added a little more abstraction for large objects
573         https://bugs.webkit.org/show_bug.cgi?id=141978
574
575         Reviewed by Sam Weinig.
576
577         Previously, each client needed to manage the boundary tags of
578         a large object using free functions. This patch introduces a LargeObject
579         class that does things a little more automatically.
580
581         * bmalloc.xcodeproj/project.pbxproj:
582
583         * bmalloc/Allocator.cpp:
584         (bmalloc::Allocator::reallocate): Use the new LargeObject class.
585
586         * bmalloc/BeginTag.h:
587         (bmalloc::BeginTag::isInFreeList): Deleted. Moved this logic into the
588         LargeObject class.
589
590         * bmalloc/BoundaryTag.h:
591         (bmalloc::BoundaryTag::isSentinel):
592         (bmalloc::BoundaryTag::compactBegin):
593         (bmalloc::BoundaryTag::setRange):
594         (bmalloc::BoundaryTag::initSentinel): Added an explicit API for sentinels,
595         which we used to create and test for implicitly.
596
597         * bmalloc/BoundaryTagInlines.h:
598         (bmalloc::BoundaryTag::init):
599         (bmalloc::validate): Deleted.
600         (bmalloc::validatePrev): Deleted.
601         (bmalloc::validateNext): Deleted.
602         (bmalloc::BoundaryTag::mergeLeft): Deleted.
603         (bmalloc::BoundaryTag::mergeRight): Deleted.
604         (bmalloc::BoundaryTag::merge): Deleted.
605         (bmalloc::BoundaryTag::deallocate): Deleted.
606         (bmalloc::BoundaryTag::split): Deleted.
607         (bmalloc::BoundaryTag::allocate): Deleted. Moved this logic into the
608         LargeObject class.
609
610         * bmalloc/EndTag.h:
611         (bmalloc::EndTag::init):
612         (bmalloc::EndTag::operator=): Deleted. Re-reading this code, I found
613         special behavior in the assignment operator to be a surprising API.
614         So, I replaced the assignment operation with an explicit initializing
615         function.
616
617         * bmalloc/Heap.cpp:
618         (bmalloc::Heap::scavengeLargeRanges):
619         (bmalloc::Heap::allocateXLarge):
620         (bmalloc::Heap::findXLarge):
621         (bmalloc::Heap::deallocateXLarge):
622         (bmalloc::Heap::allocateLarge):
623         (bmalloc::Heap::deallocateLarge):
624         * bmalloc/Heap.h: No behavior changes here -- just adopting the
625         LargeObject interface.
626
627         * bmalloc/LargeObject.h: Added.
628         (bmalloc::LargeObject::operator!):
629         (bmalloc::LargeObject::begin):
630         (bmalloc::LargeObject::size):
631         (bmalloc::LargeObject::range):
632         (bmalloc::LargeObject::LargeObject):
633         (bmalloc::LargeObject::setFree):
634         (bmalloc::LargeObject::isFree):
635         (bmalloc::LargeObject::hasPhysicalPages):
636         (bmalloc::LargeObject::setHasPhysicalPages):
637         (bmalloc::LargeObject::isValidAndFree):
638         (bmalloc::LargeObject::merge):
639         (bmalloc::LargeObject::split):
640         (bmalloc::LargeObject::validateSelf):
641         (bmalloc::LargeObject::validate): Moved this code into a class, out of
642         BoundaryTag free functions.
643
644         New to the class are these features:
645
646             (1) Every reference to an object is validated upon creation and use.
647
648             (2) There's an explicit API for "This is a reference to an object
649             that might be stale (the DoNotValidate API)".
650
651             (3) The begin and end tags are kept in sync automatically.
652
653         * bmalloc/SegregatedFreeList.cpp:
654         (bmalloc::SegregatedFreeList::insert):
655         (bmalloc::SegregatedFreeList::takeGreedy):
656         (bmalloc::SegregatedFreeList::take):
657         * bmalloc/SegregatedFreeList.h: Adopt the LargeObject interface.
658
659         * bmalloc/VMHeap.cpp:
660         (bmalloc::VMHeap::grow):
661         * bmalloc/VMHeap.h:
662         (bmalloc::VMHeap::allocateLargeRange):
663         (bmalloc::VMHeap::deallocateLargeRange): Adopt the LargeObject interface.
664
665 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
666
667         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
668         https://bugs.webkit.org/show_bug.cgi?id=141802
669
670         Reviewed by Andreas Kling.
671
672         Rolling back in with a fix for a crash seen while using GuardMalloc.
673
674         * bmalloc/VMHeap.cpp:
675         (bmalloc::VMHeap::grow):
676         * bmalloc/VMHeap.h:
677         * bmalloc/Zone.cpp: Re-land the old patch.
678
679         (bmalloc::Zone::size): Be sure to implement the size() function since
680         it's accessible indirectly via the malloc_zone_from_ptr public API --
681         and GuardMalloc calls it all the time.
682
683         (bmalloc::Zone::Zone):
684         * bmalloc/Zone.h: Re-land the old patch.
685
686 2015-02-19  Commit Queue  <commit-queue@webkit.org>
687
688         Unreviewed, rolling out r180363.
689         https://bugs.webkit.org/show_bug.cgi?id=141814
690
691         Caused >50 crashes when running LayoutTests in GuardMalloc or
692         ASAN modes. (Requested by jernoble on #webkit).
693
694         Reverted changeset:
695
696         "bmalloc should implement malloc introspection (to stop false-
697         positive leaks when MallocStackLogging is off)"
698         https://bugs.webkit.org/show_bug.cgi?id=141802
699         http://trac.webkit.org/changeset/180363
700
701 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
702
703         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
704         https://bugs.webkit.org/show_bug.cgi?id=141802
705
706         Reviewed by Andreas Kling.
707
708         Fixed a last-minute type.
709
710         The macro is OS, not PLATFORM.
711
712         * bmalloc/VMHeap.cpp:
713         (bmalloc::VMHeap::grow):
714         * bmalloc/VMHeap.h:
715         * bmalloc/Zone.h:
716
717 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
718
719         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
720         https://bugs.webkit.org/show_bug.cgi?id=141802
721
722         Reviewed by Andreas Kling.
723
724         This patch does the bare minimum to stop false positive leaks from
725         being reported by the Darwin leaks tool. We register each super chunk
726         as a single object, and then request that the leaks tool scan it.
727
728         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
729         zone introspection API.
730
731         * bmalloc/Algorithm.h: Missing #include.
732
733         * bmalloc/VMHeap.cpp:
734         (bmalloc::VMHeap::grow):
735         * bmalloc/VMHeap.h: Adopt the new abstraction.
736
737         * bmalloc/Zone.cpp: Added.
738         (bmalloc::remoteRead): Helper for reading an object out of another process.
739         (bmalloc::Zone::enumerator):
740         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
741         in introspection.
742
743         * bmalloc/Zone.h: Added.
744         (bmalloc::Zone::superChunks):
745         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
746         since our dynamic allocations will not be scanned by leaks since they
747         will have the malloc VM tag.
748
749 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
750
751         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
752         https://bugs.webkit.org/show_bug.cgi?id=141759
753
754         Reviewed by Andreas Kling.
755
756         * bmalloc.xcodeproj/project.pbxproj:
757         * bmalloc/SuperChunk.h: Added.
758         (bmalloc::SuperChunk::create):
759         (bmalloc::SuperChunk::SuperChunk):
760         (bmalloc::SuperChunk::smallChunk):
761         (bmalloc::SuperChunk::mediumChunk):
762         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
763         into a separate class, for clarity and type safety.
764
765         * bmalloc/VMHeap.cpp:
766         (bmalloc::VMHeap::grow):
767         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
768         "grow" because Andreas found "allocateSuperChunk" to be unclear.
769
770         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
771         for malloc introspection.
772
773         (bmalloc::VMHeap::allocateSmallPage):
774         (bmalloc::VMHeap::allocateMediumPage):
775         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
776
777 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
778
779         Build bmalloc through CMake as a static library. It's then linked either
780         into the WTF library (if built as a shared library) or into the JSC and
781         WebKit2 libraries. There's no need to build it as a standalone shared library.
782
783         Rubber-stamped by Carlos Garcia Campos.
784
785         * CMakeLists.txt:
786
787 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
788
789         [BMalloc] Add a FIXME comment for memory alignas
790         https://bugs.webkit.org/show_bug.cgi?id=141556
791
792         Reviewed by Csaba Osztrogonác.
793
794         * bmalloc/Chunk.h: Add a FIXME comment.
795         * bmalloc/LargeChunk.h: ditto.
796
797 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
798
799         bmalloc buildfix on 32 bit Linux (x86/ARM)
800         https://bugs.webkit.org/show_bug.cgi?id=141472
801
802         Reviewed by Gyuyoung Kim.
803
804         * bmalloc/Algorithm.h:
805         (bmalloc::roundUpToMultipleOf):
806         * bmalloc/FixedVector.h:
807         (bmalloc::FixedVector::clear):
808         * bmalloc/Sizes.h:
809         (bmalloc::Sizes::sizeClass):
810
811 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
812
813         [EFL][GTK] Use bmalloc instead of tcmalloc
814         https://bugs.webkit.org/show_bug.cgi?id=140162
815
816         Reviewed by Carlos Garcia Campos.
817
818         Support to use bmalloc on EFL and GTK ports.
819
820         * CMakeLists.txt: Added.
821         * bmalloc/Allocator.cpp:
822         (bmalloc::Allocator::allocate):
823             Fix unused return value caused by posix_memalign().
824         * bmalloc/AsyncTask.h:
825         * bmalloc/BoundaryTag.h:
826         (bmalloc::BoundaryTag::clear):
827         * bmalloc/Chunk.h:
828             Change Traits::Page with Traits::PageType in order to fix
829             -fpermitive build error on EFL and GTK port. 
830         * bmalloc/EndTag.h:
831         (bmalloc::EndTag::operator=):
832         * bmalloc/Line.h: ditto.
833         * bmalloc/MediumTraits.h:
834         * bmalloc/Page.h: ditto.
835         * bmalloc/PerThread.h:
836             EFL port doesn't support __has_include definition yet.
837             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
838         * bmalloc/SmallTraits.h: ditto.
839         * bmalloc/VMAllocate.h:
840         (bmalloc::vmDeallocatePhysicalPages):
841         (bmalloc::vmAllocatePhysicalPages):
842         * bmalloc/Vector.h:
843         (bmalloc::Vector<T>::push):
844         (bmalloc::Vector<T>::reallocateBuffer):
845
846 2015-01-31  Sam Weinig  <sam@webkit.org>
847
848         Remove even more Mountain Lion support
849         https://bugs.webkit.org/show_bug.cgi?id=141124
850
851         Reviewed by Alexey Proskuryakov.
852
853         * Configurations/Base.xcconfig:
854         * Configurations/DebugRelease.xcconfig:
855
856 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
857
858         GC marking threads should clear malloc caches
859         https://bugs.webkit.org/show_bug.cgi?id=141097
860
861         Reviewed by Andreas Kling.
862
863         Split the scavenging API into per-thread vs global, so that you can
864         request to scavenge your own thread without scavenging the whole heap.
865
866         * bmalloc/Cache.cpp:
867         (bmalloc::Cache::scavenge):
868         * bmalloc/bmalloc.h:
869         (bmalloc::api::scavengeThisThread):
870         (bmalloc::api::scavenge):
871
872 2015-01-28  Dana Burkart  <dburkart@apple.com>
873
874         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
875         https://bugs.webkit.org/show_bug.cgi?id=136765
876
877         Reviewed by Alexey Proskuryakov.
878
879         * Configurations/Base.xcconfig:
880         * Configurations/DebugRelease.xcconfig:
881
882 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
883
884         bmalloc: support aligned allocation
885         https://bugs.webkit.org/show_bug.cgi?id=140732
886
887         Reviewed by Andreas Kling.
888
889         * bmalloc/Allocator.cpp:
890         (bmalloc::Allocator::allocate): New function for aligned allocation.
891
892         Small and medium requests just allocate and free until they find an
893         aligned pointer. This is slightly inefficient in the worst case, but
894         still constant-time with little-to-no space overhead.
895
896         Large requests use a new API that requires the client to specify both
897         its ideal size and alignment, and the worst-case size you would have to
898         allocate in order to produce some interior pointer of the requested size
899         and alignment. We put the burden of this calculation on the client
900         because it simplifies things if we guarantee that allocation won't fail.
901
902         XLarge requests are easy: we just forward them to vmAllocate, which
903         already supported aligned requests.
904
905         * bmalloc/BoundaryTag.h:
906         * bmalloc/BoundaryTagInlines.h:
907         (bmalloc::BoundaryTag::mergeLeft):
908         (bmalloc::BoundaryTag::mergeRight):
909         (bmalloc::BoundaryTag::merge):
910         (bmalloc::BoundaryTag::deallocate):
911         (bmalloc::BoundaryTag::split):
912         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
913         refactored the interface to remove some reference out parameters in
914         order to clarify what changes and what doesn't.
915
916         * bmalloc/Heap.cpp:
917         (bmalloc::Heap::allocateXLarge): Added an alignment API.
918
919         (bmalloc::Heap::allocateLarge):
920         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
921         a few variants, so aligned and unaligned allocation could share some code.
922
923         * bmalloc/SegregatedFreeList.cpp:
924         (bmalloc::SegregatedFreeList::take):
925         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
926         for aligned allocation. It turns out that the aligned path is pretty
927         different, since it ends up searching for two potential ways to satisfy
928         an allocation: either large enough and aligned, or large enough to split
929         into something not aligned and something large enough and aligned.
930
931         * bmalloc/VMAllocate.h:
932         (bmalloc::vmAllocate): Switched alignment to come before size because
933         that's how the memalign API specifies it.
934
935         * bmalloc/VMHeap.h:
936         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
937
938 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
939
940         bmalloc: a little bit of cleanup
941         https://bugs.webkit.org/show_bug.cgi?id=140687
942
943         Reviewed by Anders Carlsson.
944
945         * bmalloc/Algorithm.h:
946         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
947         of code.
948
949         * bmalloc/BoundaryTag.h:
950         * bmalloc/BoundaryTagInlines.h:
951         (bmalloc::BoundaryTag::mergeLeft):
952         (bmalloc::BoundaryTag::mergeRight):
953         (bmalloc::BoundaryTag::merge):
954         (bmalloc::BoundaryTag::deallocate):
955         (bmalloc::BoundaryTag::split):
956         (bmalloc::BoundaryTag::allocate):
957         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
958         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
959         (bmalloc::BoundaryTag::mergeLarge): Deleted.
960         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
961         from all these functions, since boundary tags always pertain to large
962         objects, and putting the word "Large" everywhere wasn't helping to
963         explain that.
964
965         * bmalloc/Heap.cpp:
966         (bmalloc::Heap::allocateXLarge):
967         (bmalloc::Heap::findXLarge):
968         (bmalloc::Heap::deallocateXLarge):
969         * bmalloc/Heap.h:
970         * bmalloc/VMHeap.h:
971         (bmalloc::VMHeap::allocateXLarge): Deleted.
972         (bmalloc::VMHeap::findXLarge): Deleted.
973         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
974         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
975         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
976         really make sense for the VMHeap to be involved.
977
978 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
979
980         bmalloc: refactored XLarge allocation for better alignment
981         https://bugs.webkit.org/show_bug.cgi?id=140582
982
983         Reviewed by Andreas Kling.
984
985         XLarge objects used to be Large objects with an extra bit of metadata
986         that said "actually, I'm not large -- I'm extra large".
987
988         The metadata header in an XLarge allocation made it impossible for the
989         XLarge object to honor a very large alignment request.
990
991         The solution is to stop using a metadata header for XLarge objects, and
992         instead to store explicit metadata on the side.
993
994         This is a bit less astonishing, which is also nice.
995
996         Finding XLarge metadata is now a linear search. That's probably OK, since
997         it was always so in TCMalloc, and the usual number of XLarge allocations
998         in a process is 0.
999
1000         This design makes it possible for the heap to cache XLarge allocations
1001         with and/or without physical pages. I haven't actually done that yet
1002         because the tradeoffs are subtle, so I don't want to do anything without
1003         a motivating test case.
1004
1005         * bmalloc.xcodeproj/project.pbxproj:
1006         * bmalloc/Allocator.cpp:
1007         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
1008         since an XLarge allocation is now just a naked buffer without a header.
1009
1010         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
1011         XLarge alignment, since XLargeChunk won't give this to us implicitly
1012         anymore.
1013
1014         * bmalloc/BoundaryTag.h:
1015         (bmalloc::BoundaryTag::setRange):
1016         (bmalloc::BoundaryTag::isXLarge): Deleted.
1017         (bmalloc::BoundaryTag::setXLarge): Deleted.
1018         * bmalloc/BoundaryTagInlines.h:
1019         (bmalloc::validate):
1020         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
1021
1022         * bmalloc/Deallocator.cpp:
1023         (bmalloc::Deallocator::deallocateXLarge):
1024         (bmalloc::Deallocator::deallocateSlowCase):
1025         * bmalloc/Heap.cpp:
1026         (bmalloc::Heap::findXLarge):
1027         (bmalloc::Heap::allocateXLarge):
1028         (bmalloc::Heap::deallocateXLarge):
1029         * bmalloc/Heap.h: Updated for interface changes.
1030
1031         * bmalloc/ObjectType.cpp:
1032         (bmalloc::objectType):
1033         * bmalloc/ObjectType.h:
1034         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
1035         examining its bit pattern -- just like we do for other kinds of
1036         allocations -- which is nice.
1037
1038         * bmalloc/Sizes.h:
1039         * bmalloc/VMHeap.h:
1040         (bmalloc::VMHeap::allocateXLarge):
1041         (bmalloc::VMHeap::findXLarge):
1042         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
1043         for XLarge allocations.
1044
1045         * bmalloc/XLargeChunk.h: Removed.
1046
1047 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
1048
1049         bmalloc: added some infrastructure for aligned allocation
1050         https://bugs.webkit.org/show_bug.cgi?id=140572
1051
1052         Reviewed by Andreas Kling.
1053
1054         * bmalloc/Algorithm.h:
1055         (bmalloc::isPowerOfTwo):
1056         (bmalloc::roundUpToMultipleOf):
1057         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
1058         isPowerOfTwo helper function.
1059
1060         * bmalloc/Allocator.cpp:
1061         (bmalloc::Allocator::allocate):
1062         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
1063         Doesn't do anything yet, but does correctly forward to system malloc
1064         when bmalloc is disabled.
1065
1066         * bmalloc/Cache.cpp:
1067         (bmalloc::Cache::allocateSlowCaseNullCache):
1068         * bmalloc/Cache.h:
1069         (bmalloc::Cache::allocate):
1070         * bmalloc/bmalloc.h:
1071         (bmalloc::api::memalign):
1072         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
1073
1074 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
1075
1076         Consider alignment when allocating from a SegregatedFreeList
1077         https://bugs.webkit.org/show_bug.cgi?id=140408
1078
1079         Reviewed by Sam Weinig.
1080
1081         In preparation for supporting aligned allocation.
1082
1083         No performance change.
1084
1085         Since this is just one extra branch in an already expensive function,
1086         I decided not to duplicate the function just to avoid the branch in
1087         the un-aligned case.
1088
1089         * bmalloc/SegregatedFreeList.cpp:
1090         (bmalloc::SegregatedFreeList::take):
1091         * bmalloc/SegregatedFreeList.h:
1092
1093 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
1094
1095         Renamed minimum to size in SegregatedFreeList
1096         https://bugs.webkit.org/show_bug.cgi?id=140406
1097
1098         Reviewed by Sam Weinig.
1099
1100         In preparation for supporting aligned allocation.
1101
1102         * bmalloc/SegregatedFreeList.cpp:
1103         (bmalloc::SegregatedFreeList::takeGreedy):
1104         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
1105         really just a minimum. Let's not imply that this value is special.
1106
1107 2015-01-11  Dan Bernstein  <mitz@apple.com>
1108
1109         Geoff is organized, but he is not an organization.
1110
1111         Rubber-stamped by Anders Carlsson.
1112
1113         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
1114
1115 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
1116
1117         Make bmalloc work with ASan
1118         https://bugs.webkit.org/show_bug.cgi?id=140194
1119
1120         Reviewed by Mark Lam.
1121
1122         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
1123         an OS-specific API to test for loaded runtime libraries.
1124
1125         * bmalloc/Environment.cpp:
1126         (bmalloc::isASanEnabled):
1127         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
1128         ASan is enabled, since system malloc has the Asan hooks we need.
1129
1130         You could check for the ASan compile-time flag instead, but doing this
1131         check at runtime prepares bmalloc for a world where it is a dynamic
1132         library that might be loaded into projects it did not compile with.
1133
1134 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
1135
1136         Fix up bmalloc's PerThread for use on Linux
1137         https://bugs.webkit.org/show_bug.cgi?id=139804
1138
1139         Reviewed by Anders Carlsson.
1140
1141         The previous implementation was a bit slow.
1142
1143         * bmalloc/PerThread.h:
1144         (bmalloc::PerThreadStorage<Cache>::get):
1145         (bmalloc::PerThreadStorage::get):
1146         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
1147         way to do fast per-thread access without taking a lock every time. This
1148         probably works on all the platforms we care about, and it matches other
1149         techniques we use elsewhere in WebKit.
1150
1151         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
1152         this class because PerThreadStorage now encapsulates everything that
1153         needs to be conditional.
1154
1155         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
1156
1157 2014-12-26  Dan Bernstein  <mitz@apple.com>
1158
1159         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
1160         https://bugs.webkit.org/show_bug.cgi?id=139950
1161
1162         Reviewed by David Kilzer.
1163
1164         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
1165         in a manner that works with Xcode 5.1.1.
1166
1167 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
1168
1169         Safari crashes when you set Malloc environment variables
1170         https://bugs.webkit.org/show_bug.cgi?id=139656
1171
1172         Reviewed by Michael Saboff.
1173
1174         I forgot to cover the realloc() case. Whoops. (OoPS?)
1175
1176         This time around, I ran the full MallocBench test suite in Malloc=1
1177         mode, and it passed.
1178
1179         * bmalloc/Allocator.cpp:
1180         (bmalloc::Allocator::reallocate):
1181         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
1182         It needs to be down there so that we can do the short-circuiting check
1183         for whether bmalloc is enabled first.
1184
1185         Also added the check.
1186
1187         * bmalloc/Cache.cpp:
1188         (bmalloc::Cache::scavenge):
1189         (bmalloc::Cache::Cache):
1190         (bmalloc::Cache::reallocateSlowCaseNullCache):
1191         * bmalloc/Cache.h:
1192         (bmalloc::Cache::deallocator):
1193         (bmalloc::Cache::reallocate): Ditto.
1194
1195         * bmalloc/bmalloc.h:
1196         (bmalloc::api::free):
1197         (bmalloc::api::realloc): Ditto.
1198
1199         (bmalloc::api::scavenge): Pushed this down into Cache to match the
1200         surrounding functions.
1201
1202 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1203
1204         bmalloc should support system memory analysis tools (part 2)
1205         https://bugs.webkit.org/show_bug.cgi?id=139565
1206
1207         Reviewed by Mark Lam.
1208
1209         This patch actually queries the environment to see if memory analysis
1210         tools have been enabled.
1211
1212         * bmalloc/Deallocator.cpp:
1213         (bmalloc::Deallocator::scavenge): Don't process the object log if
1214         we've disabled bmalloc because it will be full of invalid nullptrs.
1215
1216         * bmalloc/Environment.cpp:
1217         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
1218         Malloc debugging flags. I also added a plain "Malloc" catch-all for
1219         when you want to disable bmalloc without enabling any kind of funny
1220         business.
1221
1222         It would be slightly nicer just to iterate the list of environment
1223         variables and strstr them, but getenv is the more portable option,
1224         and performance here doesn't really matter.
1225
1226         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
1227         environment variable.
1228
1229         (bmalloc::Environment::computeIsBmallocEnabled):
1230
1231 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1232
1233         Try to fix the iOS simulator build.
1234
1235         #include the declaration of malloc / free.
1236
1237         * bmalloc/Allocator.cpp:
1238         * bmalloc/Deallocator.cpp:
1239
1240 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1241
1242         Try to fix the build.
1243
1244         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
1245
1246 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
1247
1248         bmalloc should support system memory analysis tools (part 1)
1249         https://bugs.webkit.org/show_bug.cgi?id=139559
1250
1251         Reviewed by Mark Lam.
1252
1253         This patch adds the hooks to disable bmalloc at runtime if certain
1254         environment variables are set, but doesn't actually read from the
1255         environment yet.
1256
1257         No performance change.
1258
1259         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
1260         we'll use to read environment variables and see if memory analysis tools
1261         have been enabled.
1262
1263         * bmalloc/Allocator.cpp:
1264         (bmalloc::Allocator::Allocator):
1265         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
1266         on the allocation path. We cache the setting to make the check fast.
1267
1268         * bmalloc/Allocator.h: Interface changes.
1269
1270         * bmalloc/Cache.cpp:
1271         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
1272         and deallocator. This main purpose is to enable them to query the
1273         environment for whether bmalloc is enabled; but this is also a slightly
1274         cleaner way to guarantee to them that the Heap has been pre-initialized.
1275
1276         * bmalloc/Deallocator.cpp:
1277         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
1278         fill the object log to force us to take the slow path on all deallocations.
1279
1280         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
1281
1282         * bmalloc/Deallocator.h: Interface changes.
1283
1284         * bmalloc/Environment.cpp: Added.
1285         (bmalloc::Environment::Environment):
1286         (bmalloc::Environment::computeIsBmallocEnabled):
1287         * bmalloc/Environment.h: Added.
1288         (bmalloc::Environment::isBmallocEnabled): This is the class that will
1289         encapsulate looking for environment variables that turn on heap
1290         analysis tools.
1291
1292         * bmalloc/Heap.h:
1293         (bmalloc::Heap::environment):
1294
1295         * bmalloc/Mutex.h:
1296         (bmalloc::Mutex::Mutex):
1297         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
1298         since I got super confused about them while writing this patch.
1299
1300         * bmalloc/VMHeap.cpp: Fixed an #include.
1301
1302 2014-12-09  David Kilzer  <ddkilzer@apple.com>
1303
1304         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
1305         <http://webkit.org/b/139212>
1306
1307         Reviewed by Joseph Pecoraro.
1308
1309         * Configurations/Base.xcconfig:
1310         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
1311           on OS X.
1312         * Configurations/DebugRelease.xcconfig:
1313         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
1314
1315 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
1316
1317         bmalloc uses 8X more virtual memory than necessary
1318         https://bugs.webkit.org/show_bug.cgi?id=138495
1319
1320         Reviewed by Mark Lam.
1321
1322         iOS has a per-process virtual memory cap around 1GB, so there's some
1323         value to not going totally ham with virtual memory.
1324
1325         We currently use about 8X the necessary amount:
1326             - 2X to align our VM allocation
1327             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
1328
1329         We can cut that down:
1330             - Return the unaligned portion of our VM allocation (-2X)
1331             - Use all the chunks in a superchunk, instead of allocating one
1332               chunk per superchunk (-4X)
1333
1334         * bmalloc/Algorithm.h:
1335         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
1336         function so we can call it with getpagesize() at runtime.
1337
1338         * bmalloc/Chunk.h:
1339         * bmalloc/LargeChunk.h:
1340         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
1341         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
1342
1343         * bmalloc/VMAllocate.h:
1344         (bmalloc::vmValidate):
1345         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
1346         if it does not succeed. Allocate precisely, and give back the extra.
1347
1348         * bmalloc/VMHeap.cpp:
1349         (bmalloc::VMHeap::allocateSuperChunk):
1350         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
1351         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
1352         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
1353         in a superchunk, instead of just one.
1354
1355         * bmalloc/VMHeap.h:
1356         (bmalloc::VMHeap::allocateSmallPage):
1357         (bmalloc::VMHeap::allocateMediumPage):
1358         (bmalloc::VMHeap::allocateLargeRange):
1359         * bmalloc/XLargeChunk.h:
1360         (bmalloc::XLargeChunk::create): Updated to match changes above.
1361
1362 2014-11-01  David Kilzer  <ddkilzer@apple.com>
1363
1364         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
1365         <https://webkit.org/b/138286>
1366         <rdar://problem/18847087>
1367
1368         Reviewed by Dan Bernstein.
1369
1370         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
1371         to NO for the target that produces libbmalloc.a so that the
1372         debug symbols will be linked into JavaScriptCore and end up in
1373         its dSYM file.
1374
1375 2014-10-30  Dana Burkart  <dburkart@apple.com>
1376
1377         <rdar://problem/18821260> Prepare for the mysterious future
1378
1379         Reviewed by Lucas Forschler.
1380
1381         * Configurations/Base.xcconfig:
1382         * Configurations/DebugRelease.xcconfig:
1383
1384 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
1385
1386         bmalloc: cleaned up fast path vs slow path
1387         https://bugs.webkit.org/show_bug.cgi?id=137081
1388
1389         Reviewed by Sam Weinig.
1390
1391         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
1392
1393         * bmalloc/Allocator.cpp:
1394         (bmalloc::Allocator::Allocator): Merged the small and medium range
1395         caches, just like the small and medium allocators. Ranges are abstract
1396         objects that don't really care whether they hold small or medium objects,
1397         so they don't need to be segregated.
1398
1399         (bmalloc::Allocator::scavenge): Ditto.
1400
1401         (bmalloc::Allocator::allocateBumpRangeSlowCase):
1402         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
1403         we do care a tiny bit, because we need to specify small vs medium when
1404         allocating new ranges from the heap, to ensure that the heap allocates
1405         from the right segment of VM.
1406
1407         (bmalloc::Allocator::allocateLarge):
1408         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
1409         up the fast path. Large allocation performance is dominated by allocation
1410         logic and initialization, so inlining it doesn't help.
1411
1412         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
1413         it doesn't need to distinguish small vs medium objects.
1414
1415         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
1416         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
1417
1418         * bmalloc/Allocator.h:
1419         * bmalloc/BumpRange.h:
1420
1421         * bmalloc/Cache.cpp:
1422         (bmalloc::Cache::allocateSlowCase): Deleted.
1423         (bmalloc::Cache::deallocateSlowCase): Deleted.
1424         * bmalloc/Cache.h:
1425         (bmalloc::Cache::allocate):
1426         (bmalloc::Cache::deallocate):
1427         (bmalloc::Cache::allocateFastCase): Deleted.
1428         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
1429         paths. The downside to this change is that the fast path branches to two
1430         distinct failure cases instead of one. The upside is that the slow path
1431         doesn't need to re-read the segment register, which is not as cheap as a
1432         normal register, and it doesn't need to do an extra level of function 
1433         call. Seems to be worth it.
1434
1435         * bmalloc/Deallocator.h:
1436         * bmalloc/Heap.cpp:
1437         (bmalloc::Heap::refillSmallBumpRangeCache):
1438         (bmalloc::Heap::refillMediumBumpRangeCache):
1439         * bmalloc/Heap.h: Updated for interface changes.
1440
1441         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
1442         small lines in a page / 2, since any other free lines will coalesce
1443         with their neighbors.
1444
1445 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1446
1447         Rolled out r173346.
1448
1449             bmalloc should honor the FastMalloc statistics API
1450             https://bugs.webkit.org/show_bug.cgi?id=136592
1451
1452         This didn't really work. Because we allow ranges with and without
1453         physical pages to merge, and we allow double-committing and
1454         double-decommitting, we can't rely on commit actions to track memory
1455         footprint.
1456
1457         * bmalloc/Heap.cpp:
1458         (bmalloc::Heap::size): Deleted.
1459         (bmalloc::Heap::capacity): Deleted.
1460         * bmalloc/Heap.h:
1461         * bmalloc/VMHeap.cpp:
1462         (bmalloc::VMHeap::VMHeap):
1463         (bmalloc::VMHeap::allocateSmallChunk):
1464         (bmalloc::VMHeap::allocateMediumChunk):
1465         (bmalloc::VMHeap::allocateLargeChunk):
1466         * bmalloc/VMHeap.h:
1467         (bmalloc::VMHeap::allocateSmallPage):
1468         (bmalloc::VMHeap::allocateMediumPage):
1469         (bmalloc::VMHeap::allocateLargeRange):
1470         (bmalloc::VMHeap::deallocateSmallPage):
1471         (bmalloc::VMHeap::deallocateMediumPage):
1472         (bmalloc::VMHeap::deallocateLargeRange):
1473         (bmalloc::VMHeap::size): Deleted.
1474         (bmalloc::VMHeap::capacity): Deleted.
1475         * bmalloc/bmalloc.h:
1476         (bmalloc::api::heapSize): Deleted.
1477         (bmalloc::api::heapCapacity): Deleted.
1478
1479 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
1480
1481         bmalloc: Allocation should be more precise
1482         https://bugs.webkit.org/show_bug.cgi?id=136993
1483
1484         Reviewed by Gavin Barraclough.
1485
1486         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1487
1488         This patch teaches the allocator to merge adjacent free lines into a
1489         single allocatable range. This allows us to shrink the size of an
1490         individual line without increasing fragmentation or the rate of allocator
1491         slow paths.
1492
1493         We'll only take more slow paths when available memory is sparse, which
1494         is exactly when it's worth it. When available memory is dense, we'll
1495         take fewer slow paths.
1496
1497         * bmalloc.xcodeproj/project.pbxproj:
1498         * bmalloc/Algorithm.h:
1499         (bmalloc::divideRoundingUp):
1500
1501         * bmalloc/Allocator.cpp:
1502         (bmalloc::Allocator::Allocator): Updated for interface changes.
1503
1504         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
1505         Now that we merge lines, it's not convenient to scavenge by line.
1506
1507         (bmalloc::Allocator::allocateSmallBumpRange):
1508         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
1509         instead of individual lines.
1510
1511         (bmalloc::Allocator::allocateSlowCase):
1512         (bmalloc::Allocator::allocateSmallLine): Deleted.
1513         (bmalloc::Allocator::allocateMediumLine): Deleted.
1514         (bmalloc::Allocator::allocateMedium): Deleted.
1515         * bmalloc/Allocator.h:
1516         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
1517         into the standard fast path with small allocations. Since a BumpAllocator
1518         just allocates out of an arbitrary range, it doesn't need to distinguish
1519         between small and medium lines.
1520
1521         * bmalloc/BumpAllocator.h:
1522         (bmalloc::BumpAllocator::size):
1523         (bmalloc::BumpAllocator::BumpAllocator):
1524         (bmalloc::BumpAllocator::init):
1525         (bmalloc::BumpAllocator::refill):
1526         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
1527         anymore: the heap just gives us a pointer and a pre-computed number of
1528         objects, and we allocate them.
1529
1530         * bmalloc/Deallocator.cpp:
1531         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
1532
1533         * bmalloc/Heap.cpp:
1534         (bmalloc::Heap::Heap):
1535         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
1536         detailing where all objects will lie in memory. After we merge two lines,
1537         we might allocate an object that spans from one line to the next. This
1538         metadata details which bits of memory overlap in that way, and how they
1539         overlap.
1540
1541         (bmalloc::Heap::refillSmallBumpRangeCache):
1542         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
1543         and merge adjacent free lines into BumpRanges.
1544
1545         (bmalloc::Heap::allocateSmallPage):
1546         (bmalloc::Heap::allocateMediumPage):
1547         (bmalloc::Heap::deallocateSmallLine):
1548         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
1549         since we scan for free memory a page at a time.
1550
1551         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
1552         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
1553         fast path.
1554
1555         * bmalloc/Heap.h:
1556         (bmalloc::Heap::derefSmallLine):
1557         (bmalloc::Heap::derefMediumLine):
1558         (bmalloc::Heap::deallocateSmallLine): Deleted.
1559         (bmalloc::Heap::allocateSmallLine): Deleted.
1560         (bmalloc::Heap::deallocateMediumLine): Deleted.
1561         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
1562
1563         * bmalloc/Line.h:
1564         (bmalloc::Line<Traits>::ref):
1565         (bmalloc::Line<Traits>::deref):
1566         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
1567         anymore, since we only ever deref by 1 now.
1568
1569         * bmalloc/MediumAllocator.h:
1570         (bmalloc::MediumAllocator::isNull): Deleted.
1571         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
1572         (bmalloc::MediumAllocator::line): Deleted.
1573         (bmalloc::MediumAllocator::allocate): Deleted.
1574         (bmalloc::MediumAllocator::derefCount): Deleted.
1575         (bmalloc::MediumAllocator::refill): Deleted.
1576         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
1577         been dead for a while, since it doesn't build anymore with this patch.
1578
1579         * bmalloc/Page.h:
1580         (bmalloc::Page::sizeClass):
1581         (bmalloc::Page::setSizeClass):
1582         (bmalloc::Page::smallSizeClass): Deleted.
1583         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
1584         to sizeClass, since we use it for medium sizes too.
1585
1586         * bmalloc/Sizes.h:
1587         (bmalloc::Sizes::sizeClass):
1588         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
1589
1590         (bmalloc::Sizes::smallSizeClassFor): Deleted.
1591         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
1592
1593         * bmalloc/bmalloc.h:
1594         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
1595         can be a bit more precise. It also has to be, since we can't guarantee
1596         that an object ends at the end of a line anymore.
1597
1598 2014-09-19  Daniel Bates  <dabates@apple.com>
1599
1600         Always assume internal SDK when building configuration Production
1601         https://bugs.webkit.org/show_bug.cgi?id=136925
1602         <rdar://problem/18362399>
1603
1604         Reviewed by Dan Bernstein.
1605
1606         * Configurations/Base.xcconfig:
1607
1608 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
1609
1610         bmalloc: moved line caches from the deallocator to the allocator
1611         https://bugs.webkit.org/show_bug.cgi?id=136868
1612
1613         Reviewed by Gavin Barraclough.
1614
1615         I did this mostly as a simplification, to make it easier to change the
1616         allocation strategy.
1617
1618         No throughput change on MallocBench. Saves about 50kB.
1619
1620         Since the deallocator needs to lock the heap when freeing lines anyway,
1621         there isn't much benefit to giving the deallocator a local cache of
1622         deallocated lines.
1623
1624         We still give the allocator a local cache of lines because that does
1625         reduce the frequency at which it needs to lock the heap in order to
1626         acquire more lines.
1627
1628         * bmalloc/Allocator.cpp:
1629         (bmalloc::Allocator::scavenge):
1630         (bmalloc::Allocator::allocateSmallLine):
1631         (bmalloc::Allocator::allocateMediumLine):
1632         (bmalloc::Allocator::allocateMedium):
1633         (bmalloc::Allocator::allocateSlowCase):
1634         * bmalloc/Allocator.h:
1635         * bmalloc/Deallocator.cpp:
1636         (bmalloc::Deallocator::Deallocator):
1637         (bmalloc::Deallocator::scavenge):
1638         (bmalloc::Deallocator::processObjectLog):
1639         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
1640         (bmalloc::Deallocator::allocateSmallLine): Deleted.
1641         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
1642         (bmalloc::Deallocator::allocateMediumLine): Deleted.
1643         * bmalloc/Deallocator.h:
1644
1645         * bmalloc/Sizes.h:
1646         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
1647         exactly one page in size. That's about what we were shooting for anyway,
1648         and it may make it easier to switch to per-page allocation in future.
1649
1650 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
1651
1652         bmalloc: allocate small and medium objects using the same bump pointer class
1653         https://bugs.webkit.org/show_bug.cgi?id=136843
1654
1655         Reviewed by Gavin Barraclough.
1656
1657         4% speedup on MallocBench.
1658
1659         Now that medium-sized objects have dedicated per-size allocators, they
1660         don't need to use an arbitrary bump pointer allocator. This means that
1661         every allocator knows how many objects it will allocate from the start,
1662         and we don't need a post-processing step to adjust refcounts based on
1663         real allocation count.
1664
1665         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1666         since it's used for small and medium objects now.
1667
1668         * bmalloc/Allocator.cpp:
1669         (bmalloc::Allocator::Allocator): Updated to use new interface.
1670         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1671         to make sure that we finish allocating all the objects in it.
1672
1673         (bmalloc::Allocator::allocateMedium):
1674         (bmalloc::Allocator::allocateSlowCase):
1675         (bmalloc::Allocator::retire): Deleted.
1676         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1677         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1678         * bmalloc/Allocator.h:
1679         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1680         used to post-process an allocator based on how many objects it allocated.
1681
1682         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1683         (bmalloc::BumpAllocator::BumpAllocator):
1684         (bmalloc::BumpAllocator::init):
1685         (bmalloc::BumpAllocator::line):
1686         (bmalloc::BumpAllocator::validate):
1687         (bmalloc::BumpAllocator::allocate):
1688         (bmalloc::BumpAllocator::refill):
1689         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1690         about the kinds of lines they allocate into. In some cases, the line
1691         type must be provided as a template parameter by the caller.
1692
1693         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1694         (bmalloc::SmallAllocator::line): Deleted.
1695         (bmalloc::SmallAllocator::allocate): Deleted.
1696         (bmalloc::SmallAllocator::objectCount): Deleted.
1697         (bmalloc::SmallAllocator::derefCount): Deleted.
1698         (bmalloc::SmallAllocator::refill): Deleted.
1699         (bmalloc::SmallAllocator::clear): Deleted.
1700
1701         * bmalloc/ObjectType.h:
1702         (bmalloc::isMedium):
1703
1704         * bmalloc/SmallAllocator.h:
1705         (bmalloc::SmallAllocator::isNull): Deleted.
1706         (bmalloc::SmallAllocator::canAllocate): Deleted.
1707         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1708         (bmalloc::SmallAllocator::line): Deleted.
1709         (bmalloc::SmallAllocator::allocate): Deleted.
1710         (bmalloc::SmallAllocator::objectCount): Deleted.
1711         (bmalloc::SmallAllocator::derefCount): Deleted.
1712         (bmalloc::SmallAllocator::refill): Deleted.
1713         (bmalloc::SmallAllocator::clear): Deleted.
1714
1715 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1716
1717         Fixed a goof in bmalloc Vector sizing
1718         https://bugs.webkit.org/show_bug.cgi?id=136795
1719
1720         Reviewed by Gavin Barraclough and Sam Weinig.
1721
1722         We want our minimum vector to be page-sized since the OS will give us
1723         a page no matter what -- but we want that many bytes, and not enough
1724         bytes to store that many elements.
1725
1726         * bmalloc/Vector.h: Math is hard.
1727
1728 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1729
1730         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1731         https://bugs.webkit.org/show_bug.cgi?id=136693
1732
1733         Reviewed by Gavin Barraclough.
1734
1735         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1736
1737         No throughput change.
1738
1739         We keep an array of medium allocators, just like our array of small
1740         allocators.
1741
1742         In future, we can simplify the allocation fast path by merging the small
1743         and medium allocator arrays. For now, this is the simplest change that
1744         gets the win.
1745
1746         * bmalloc/Allocator.cpp:
1747         (bmalloc::Allocator::Allocator):
1748         (bmalloc::Allocator::scavenge):
1749         (bmalloc::Allocator::allocateMedium):
1750         * bmalloc/Allocator.h:
1751         * bmalloc/Sizes.h:
1752         (bmalloc::Sizes::mediumSizeClassFor):
1753
1754 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1755
1756         Reviewed by Sam Weinig.
1757
1758         Renamed log => retire for clarity.
1759
1760         * bmalloc/Allocator.cpp:
1761         (bmalloc::Allocator::scavenge):
1762         (bmalloc::Allocator::retire):
1763         (bmalloc::Allocator::allocateMedium):
1764         (bmalloc::Allocator::allocateSlowCase):
1765         (bmalloc::Allocator::log): Deleted.
1766         * bmalloc/Allocator.h:
1767
1768 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1769
1770         bmalloc: eager scavenge leaves behind a bogus allocator
1771         https://bugs.webkit.org/show_bug.cgi?id=136743
1772
1773         Reviewed by Sam Weinig.
1774
1775         Be sure to clear the allocator after logging it in the eager scavenge
1776         case, so that we don't later try to allocate out of the lines that we
1777         have thrown away.
1778
1779         We didn't need to do this previously because scavenge would only happen
1780         at thread exit time, after which no further allocation from the per-thread
1781         cache would take place.
1782
1783         * bmalloc/Allocator.cpp:
1784         (bmalloc::Allocator::scavenge):
1785         * bmalloc/MediumAllocator.h:
1786         (bmalloc::MediumAllocator::clear):
1787         * bmalloc/SmallAllocator.h:
1788         (bmalloc::SmallAllocator::clear):
1789
1790 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1791
1792         bmalloc should honor the FastMalloc statistics API
1793         https://bugs.webkit.org/show_bug.cgi?id=136592
1794
1795         Reviewed by Gavin Barraclough.
1796
1797         We do this by tracking "size" and "capacity" in the VM heap.
1798
1799         The VM heap's "capacity" is all the VM we ever allocated.
1800
1801         The VM heap's "size" the subset of VM currently held onto by the
1802         VM heap (and therefore not in use by the regular heap).
1803
1804         Somewhat ironically, reducing the process's memory footprint, increases
1805         the size of the VM heap, since the VM heap holds the pages that are
1806         purely virtual and not physical.
1807
1808         * bmalloc/Heap.cpp:
1809         (bmalloc::Heap::size):
1810         (bmalloc::Heap::capacity):
1811         * bmalloc/Heap.h:
1812         * bmalloc/VMHeap.cpp:
1813         (bmalloc::VMHeap::VMHeap):
1814         (bmalloc::VMHeap::allocateSmallChunk):
1815         (bmalloc::VMHeap::allocateMediumChunk):
1816         (bmalloc::VMHeap::allocateLargeChunk):
1817         * bmalloc/VMHeap.h:
1818         (bmalloc::VMHeap::size):
1819         (bmalloc::VMHeap::capacity):
1820         (bmalloc::VMHeap::allocateSmallPage):
1821         (bmalloc::VMHeap::allocateMediumPage):
1822         (bmalloc::VMHeap::allocateLargeRange):
1823         (bmalloc::VMHeap::deallocateSmallPage):
1824         (bmalloc::VMHeap::deallocateMediumPage):
1825         (bmalloc::VMHeap::deallocateLargeRange):
1826         * bmalloc/bmalloc.h:
1827         (bmalloc::api::heapSize):
1828         (bmalloc::api::heapCapacity):
1829
1830 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1831
1832         bmalloc crashes on the EWS bots (due to bad large object allocation)
1833         https://bugs.webkit.org/show_bug.cgi?id=136469
1834
1835         Reviewed by Andreas Kling.
1836
1837         It's possible to convince bmalloc to perform a bad large object allocation,
1838         through these steps:
1839
1840         (1) Insert object A into freelist F0.
1841
1842         (2) Split, merge and split again A's neighbors such that object B is
1843         inserted into freelist F0, with boundary tag and size equal to object A,
1844         but pointer not completely equal to object A. Put object B at the head of F0.
1845
1846         (3) Allocate some other object from F0, swapping its position in the
1847         freelist with object B, such that object A is now ahead of object B.
1848
1849         --> Now, the next allocation for size A/B will allocate object A, which
1850         has a slightly wrong idea about where the object actually begins.
1851         Immediately, you'll corrupt a little memory, and over time, you'll also
1852         corrupt boundary tag metadata.
1853
1854         The solution is to store the begin pointer in the boundary tag. Luckily,
1855         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1856         on MallocBench.
1857
1858         * bmalloc/Algorithm.h:
1859         (bmalloc::rightShift):
1860         * bmalloc/BeginTag.h:
1861         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1862         validate the start pointer when popping off the free list. Through a
1863         very uncommon set of steps, it is possible to have an item in the free
1864         list that is valid by all accounts except for its start pointer.
1865
1866         * bmalloc/BoundaryTag.h:
1867         (bmalloc::BoundaryTag::compactBegin):
1868         (bmalloc::BoundaryTag::setRange):
1869         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1870         start pointer. We don't need the whole pointer -- just the offset, in
1871         largeAlignment increments, into the relevant boundary tag bucket.
1872
1873         * bmalloc/BoundaryTagInlines.h:
1874         (bmalloc::validateNext):
1875         (bmalloc::BoundaryTag::init):
1876         (bmalloc::BoundaryTag::mergeLarge):
1877         (bmalloc::BoundaryTag::splitLarge):
1878         * bmalloc/SegregatedFreeList.cpp:
1879         (bmalloc::SegregatedFreeList::insert):
1880         (bmalloc::SegregatedFreeList::takeGreedy):
1881         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1882         the size when establishing a boundary tag, as required by the new
1883         interface.
1884
1885         * bmalloc/Sizes.h:
1886
1887 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1888
1889         Fixed a bmalloc crash seen on the EWS bot
1890         https://bugs.webkit.org/show_bug.cgi?id=135955
1891
1892         Reviewed by Andreas Kling.
1893
1894         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1895         input buffer is a malloc region, that region will get marked Copy-On-Write
1896         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1897         on older OS X's. In 10.10, they still fail, but they do not return
1898         EINVAL.
1899
1900         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1901
1902 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1903
1904         Fixed the bmalloc build
1905         https://bugs.webkit.org/show_bug.cgi?id=135953
1906
1907         Reviewed by Andreas Kling.
1908
1909         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1910         These headers are used, so they must be available outside the project.
1911
1912 2014-08-13  Daniel Bates  <dabates@apple.com>
1913
1914         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1915         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1916
1917         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1918         implementation of PerThread<T>::getFastCase().
1919
1920         * bmalloc/PerThread.h:
1921         (bmalloc::PerThread<T>::getFastCase):
1922
1923 2014-08-13  Daniel Bates  <dabates@apple.com>
1924
1925         Make bmalloc::PerThread work without C++ thread local storage
1926         https://bugs.webkit.org/show_bug.cgi?id=135895
1927
1928         Reviewed by Geoffrey Garen.
1929
1930         Implement support for building bmalloc without C++ thread local storage.
1931
1932         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1933         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1934         to determine whether the compiler supports C++ thread local storage.
1935         * bmalloc/PerThread.h:
1936         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1937         without C++ thread local storage.
1938         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1939         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1940         PerThreadStorage::initSharedKeyIfNeeded().
1941         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1942         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1943         ensure that the shared key has been initialized.
1944         (_pthread_setspecific_direct): Deleted.
1945         (_pthread_getspecific_direct): Deleted.
1946
1947 2014-08-13  Daniel Bates  <dabates@apple.com>
1948
1949         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1950         https://bugs.webkit.org/show_bug.cgi?id=135848
1951
1952         Reviewed by Geoffrey Garen.
1953
1954         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1955         when building for the iOS Simulator.
1956         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1957         using the public SDK.
1958         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1959         using the public SDK.
1960         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1961         using the public SDK.
1962
1963 2014-08-12  Daniel Bates  <dabates@apple.com>
1964
1965         BPLATFORM(IOS) always evaluates to false
1966         https://bugs.webkit.org/show_bug.cgi?id=135843
1967
1968         Reviewed by Geoffrey Garen.
1969
1970         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1971         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1972         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1973         building for iOS.
1974
1975         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1976         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1977
1978         * bmalloc/BPlatform.h:
1979
1980 2014-08-11  Andy Estes  <aestes@apple.com>
1981
1982         [iOS] Get rid of iOS.xcconfig
1983         https://bugs.webkit.org/show_bug.cgi?id=135809
1984
1985         Reviewed by Joseph Pecoraro.
1986
1987         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1988
1989         * Configurations/Base.xcconfig:
1990         * Configurations/iOS.xcconfig: Removed.
1991         * bmalloc.xcodeproj/project.pbxproj:
1992
1993 2014-05-01  Dan Bernstein  <mitz@apple.com>
1994
1995         Fixed production builds for the iOS Simulator.
1996         <rdar://problem/16792221>
1997
1998         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1999         PRIVATE_HEADERS_FOLDER_PATH when installing.
2000
2001 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
2002
2003         bmalloc: Segregate pages by objects size
2004         https://bugs.webkit.org/show_bug.cgi?id=131909
2005
2006         Reviewed by Andreas Kling.
2007
2008         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
2009
2010         * bmalloc/Allocator.cpp:
2011         (bmalloc::Allocator::allocateSlowCase):
2012         * bmalloc/Allocator.h:
2013         (bmalloc::Allocator::allocateFastCase):
2014         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
2015         function for size class calculation.
2016
2017         * bmalloc/Deallocator.cpp:
2018         (bmalloc::Deallocator::Deallocator):
2019         (bmalloc::Deallocator::scavenge):
2020         (bmalloc::Deallocator::deallocateSmallLine):
2021         (bmalloc::Deallocator::allocateSmallLine):
2022         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
2023         cache can't be shared anymore.
2024
2025         * bmalloc/Heap.cpp:
2026         (bmalloc::Heap::allocateSmallLineSlowCase):
2027         * bmalloc/Heap.h:
2028         (bmalloc::Heap::deallocateSmallLine): Ditto.
2029
2030         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
2031         page refcount when allocating a line because we might have deallocated
2032         the page and the recycled it for another size class.
2033
2034         (bmalloc::Heap::deallocateMediumLine):
2035         (bmalloc::Heap::allocateMediumLine):
2036         * bmalloc/Line.h:
2037         (bmalloc::Line::refCount):
2038         * bmalloc/Page.h:
2039         (bmalloc::Page::refCount):
2040         (bmalloc::Page::smallSizeClass):
2041         (bmalloc::Page::setSmallSizeClass):
2042         (bmalloc::Page<Traits>::refCount): Deleted.
2043         * bmalloc/Sizes.h:
2044         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
2045         an index into an array from a size.
2046
2047 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
2048
2049         bmalloc: Improved alignment in LargeChunk
2050         https://bugs.webkit.org/show_bug.cgi?id=131895
2051
2052         Reviewed by Andreas Kling.
2053
2054         * bmalloc/Chunk.h:
2055         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
2056         Technically, the previous alignment was harmless, but I would prefer,
2057         dear reader, not to have to explain the interlocking set of
2058         circumstances that made it so.
2059
2060 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
2061
2062         Rolled out r167502 because it caused a crash on the facebook benchmark.
2063
2064         Unreviewed.
2065
2066             bmalloc: Added an XSmall line size
2067             https://bugs.webkit.org/show_bug.cgi?id=131851
2068
2069             Reviewed by Sam Weinig.
2070
2071 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
2072
2073         bmalloc: Mutex should be harder to use wrong
2074         https://bugs.webkit.org/show_bug.cgi?id=131879
2075
2076         Reviewed by Andreas Kling.
2077
2078         Mutex now has a proper constructor, so you can't deadlock by forgetting
2079         to initialize it.
2080
2081         * bmalloc.xcodeproj/project.pbxproj:
2082         * bmalloc/Allocator.cpp:
2083         (bmalloc::Allocator::processXSmallAllocatorLog):
2084         (bmalloc::Allocator::processSmallAllocatorLog):
2085         (bmalloc::Allocator::processMediumAllocatorLog):
2086         (bmalloc::Allocator::allocateLarge):
2087         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
2088         since the Heap mutex is a static.
2089
2090         * bmalloc/AsyncTask.h:
2091         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
2092         need for explicit initialization anymore.
2093
2094         * bmalloc/Deallocator.cpp:
2095         (bmalloc::Deallocator::scavenge):
2096         (bmalloc::Deallocator::deallocateLarge):
2097         (bmalloc::Deallocator::deallocateXLarge):
2098         (bmalloc::Deallocator::processObjectLog):
2099         (bmalloc::Deallocator::deallocateSmallLine):
2100         (bmalloc::Deallocator::deallocateXSmallLine):
2101         (bmalloc::Deallocator::allocateSmallLine):
2102         (bmalloc::Deallocator::allocateXSmallLine):
2103         (bmalloc::Deallocator::deallocateMediumLine):
2104         (bmalloc::Deallocator::allocateMediumLine):
2105         * bmalloc/Deallocator.h:
2106         * bmalloc/Heap.cpp:
2107         (bmalloc::sleep):
2108         (bmalloc::Heap::Heap):
2109         (bmalloc::Heap::concurrentScavenge):
2110         (bmalloc::Heap::scavenge):
2111         (bmalloc::Heap::scavengeSmallPages):
2112         (bmalloc::Heap::scavengeXSmallPages):
2113         (bmalloc::Heap::scavengeMediumPages):
2114         (bmalloc::Heap::scavengeLargeRanges):
2115         (bmalloc::Heap::allocateXSmallLineSlowCase):
2116         (bmalloc::Heap::allocateSmallLineSlowCase):
2117         (bmalloc::Heap::allocateMediumLineSlowCase):
2118         (bmalloc::Heap::allocateXLarge):
2119         (bmalloc::Heap::deallocateXLarge):
2120         (bmalloc::Heap::allocateLarge):
2121         (bmalloc::Heap::deallocateLarge):
2122         * bmalloc/Heap.h:
2123         (bmalloc::Heap::deallocateXSmallLine):
2124         (bmalloc::Heap::allocateXSmallLine):
2125         (bmalloc::Heap::deallocateSmallLine):
2126         (bmalloc::Heap::allocateSmallLine):
2127         (bmalloc::Heap::deallocateMediumLine):
2128         (bmalloc::Heap::allocateMediumLine):
2129         * bmalloc/Line.h:
2130         (bmalloc::Line<Traits>::deref):
2131         * bmalloc/Mutex.cpp: Removed.
2132         * bmalloc/Mutex.h:
2133         (bmalloc::Mutex::Mutex):
2134         (bmalloc::Mutex::init): Deleted.
2135         (bmalloc::Mutex::try_lock): Deleted.
2136         (bmalloc::Mutex::lock): Deleted.
2137         (bmalloc::Mutex::unlock): Deleted.
2138         * bmalloc/Page.h:
2139         (bmalloc::Page<Traits>::ref):
2140         (bmalloc::Page<Traits>::deref):
2141         (bmalloc::Page<Traits>::refCount):
2142         * bmalloc/PerProcess.h:
2143         (bmalloc::PerProcess::mutex):
2144         (bmalloc::PerProcess<T>::getSlowCase):
2145         * bmalloc/StaticMutex.cpp: Added.
2146         (bmalloc::StaticMutex::lockSlowCase):
2147         * bmalloc/StaticMutex.h: Added.
2148         (bmalloc::StaticMutex::init):
2149         (bmalloc::StaticMutex::try_lock):
2150         (bmalloc::StaticMutex::lock):
2151         (bmalloc::StaticMutex::unlock):
2152         * bmalloc/VMHeap.h:
2153         (bmalloc::VMHeap::deallocateXSmallPage):
2154         (bmalloc::VMHeap::deallocateSmallPage):
2155         (bmalloc::VMHeap::deallocateMediumPage):
2156         (bmalloc::VMHeap::deallocateLargeRange):
2157         * bmalloc/bmalloc.h:
2158         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
2159         since the Heap mutex is a static.
2160
2161 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
2162
2163         bmalloc: AsyncTask should use Mutex instead of std::mutex
2164         https://bugs.webkit.org/show_bug.cgi?id=131865
2165
2166         Reviewed by Gavin Barraclough.
2167
2168         std::mutex is so slow that it makes parallelizing simple tasks through
2169         AsyncTask a net regression. Mutex fixes this.
2170
2171         * bmalloc/AsyncTask.h:
2172         (bmalloc::Function>::AsyncTask):
2173         (bmalloc::Function>::join):
2174         (bmalloc::Function>::runSlowCase):
2175         (bmalloc::Function>::entryPoint):
2176         * bmalloc/Mutex.h:
2177         (bmalloc::Mutex::init):
2178
2179 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
2180
2181         bmalloc: Added an XSmall line size
2182         https://bugs.webkit.org/show_bug.cgi?id=131851
2183
2184         Reviewed by Sam Weinig.
2185
2186         Reduces malloc footprint on Membuster recordings by 10%.
2187
2188         This is a throughput regression, but we're still way ahead of TCMalloc.
2189         I have some ideas for how to recover the regression -- but I wanted to
2190         get this win in first.
2191
2192         Full set of benchmark results:
2193
2194                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2195                                                                                                 
2196                                                                nopatch                      patch                                Δ
2197                 Peak Memory:
2198                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
2199                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
2200                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
2201
2202                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
2203                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
2204                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
2205
2206                 Memory at End:
2207                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
2208                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
2209                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
2210
2211                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
2212                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
2213                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
2214
2215                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2216                                                                                                 
2217                                                    nopatch                patch                         Δ
2218                 Execution Time:
2219                     churn                            127ms                151ms            ! 1.19x slower
2220                     list_allocate                    130ms                164ms            ! 1.26x slower
2221                     tree_allocate                    109ms                127ms            ! 1.17x slower
2222                     tree_churn                       115ms                120ms            ! 1.04x slower
2223                     facebook                         240ms                259ms            ! 1.08x slower
2224                     fragment                          91ms                131ms            ! 1.44x slower
2225                     fragment_iterate                 105ms                106ms            ! 1.01x slower
2226                     message_one                      260ms                259ms             ^ 1.0x faster
2227                     message_many                     149ms                154ms            ! 1.03x slower
2228                     medium                           194ms                248ms            ! 1.28x slower
2229                     big                              157ms                160ms            ! 1.02x slower
2230
2231                     <geometric mean>                 144ms                163ms            ! 1.13x slower
2232                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
2233                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
2234
2235                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
2236                                                                                                 
2237                                                                        nopatch                          patch                                     Δ
2238                 Execution Time:
2239                     churn                                                126ms                          148ms                        ! 1.17x slower
2240                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
2241                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
2242                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
2243                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
2244                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
2245                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
2246                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
2247                     facebook                                             240ms                          276ms                        ! 1.15x slower
2248                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
2249                     fragment                                              92ms                          130ms                        ! 1.41x slower
2250                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
2251                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
2252                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
2253                     message_one                                          260ms                          260ms                                      
2254                     message_many                                         170ms                          238ms                         ! 1.4x slower
2255                     medium                                               185ms                          250ms                        ! 1.35x slower
2256                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
2257                     big                                                  150ms                          169ms                        ! 1.13x slower
2258                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
2259
2260                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
2261                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
2262                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
2263
2264                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2265
2266                                                                     TC                      patch                                Δ
2267                 Peak Memory:
2268                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
2269                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
2270                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
2271
2272                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
2273                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
2274                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
2275
2276                 Memory at End:
2277                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
2278                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
2279                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
2280
2281                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
2282                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
2283                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
2284
2285                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
2286                                                                                                 
2287                                                         TC                patch                         Δ
2288                 Execution Time:
2289                     churn                            416ms                148ms            ^ 2.81x faster
2290                     list_allocate                    463ms                164ms            ^ 2.82x faster
2291                     tree_allocate                    292ms                127ms             ^ 2.3x faster
2292                     tree_churn                       157ms                120ms            ^ 1.31x faster
2293                     facebook                         327ms                276ms            ^ 1.18x faster
2294                     fragment                         335ms                129ms             ^ 2.6x faster
2295                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
2296                     message_one                      386ms                258ms             ^ 1.5x faster
2297                     message_many                     410ms                154ms            ^ 2.66x faster
2298                     medium                           391ms                245ms             ^ 1.6x faster
2299                     big                              261ms                167ms            ^ 1.56x faster
2300
2301                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
2302                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
2303                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
2304
2305         * bmalloc.xcodeproj/project.pbxproj:
2306         * bmalloc/Allocator.cpp:
2307         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
2308         index corresponds with its size. Instead, use the size selection function
2309         explicitly. Now that we have XSmall, some small allocator entries are
2310         unused.
2311
2312         (bmalloc::Allocator::scavenge):
2313         (bmalloc::Allocator::log):
2314         (bmalloc::Allocator::processXSmallAllocatorLog):
2315         (bmalloc::Allocator::allocateSlowCase):
2316         * bmalloc/Allocator.h:
2317         (bmalloc::Allocator::xSmallAllocatorFor):
2318         (bmalloc::Allocator::allocateFastCase):
2319         * bmalloc/Chunk.h:
2320         * bmalloc/Deallocator.cpp:
2321         (bmalloc::Deallocator::scavenge):
2322         (bmalloc::Deallocator::processObjectLog):
2323         (bmalloc::Deallocator::deallocateSlowCase):
2324         (bmalloc::Deallocator::deallocateXSmallLine):
2325         (bmalloc::Deallocator::allocateXSmallLine):
2326         * bmalloc/Deallocator.h:
2327         (bmalloc::Deallocator::deallocateFastCase):
2328         * bmalloc/Heap.cpp:
2329         (bmalloc::Heap::scavenge):
2330         (bmalloc::Heap::scavengeXSmallPages):
2331         (bmalloc::Heap::allocateXSmallLineSlowCase):
2332         * bmalloc/Heap.h:
2333         (bmalloc::Heap::deallocateXSmallLine):
2334         (bmalloc::Heap::allocateXSmallLine):
2335         * bmalloc/LargeChunk.h:
2336         (bmalloc::LargeChunk::get):
2337         (bmalloc::LargeChunk::endTag):
2338         * bmalloc/Line.h:
2339         * bmalloc/MediumAllocator.h:
2340         (bmalloc::MediumAllocator::allocate):
2341         (bmalloc::MediumAllocator::refill):
2342         * bmalloc/ObjectType.cpp:
2343         (bmalloc::objectType):
2344         * bmalloc/ObjectType.h:
2345         (bmalloc::isXSmall):
2346         (bmalloc::isSmall):
2347         (bmalloc::isMedium):
2348         (bmalloc::isLarge):
2349         (bmalloc::isSmallOrMedium): Deleted.
2350         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
2351         handling small objects. There's probably a reasonable way to share this
2352         code in the future -- I'll look into that once it's stopped changing.
2353
2354         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
2355         is the main reason things got slower.
2356
2357         * bmalloc/SmallAllocator.h:
2358         (bmalloc::SmallAllocator::allocate):
2359         * bmalloc/SmallTraits.h:
2360         * bmalloc/VMHeap.cpp:
2361         (bmalloc::VMHeap::allocateXSmallChunk):
2362         * bmalloc/VMHeap.h:
2363         (bmalloc::VMHeap::allocateXSmallPage):
2364         (bmalloc::VMHeap::deallocateXSmallPage):
2365         * bmalloc/XSmallAllocator.h: Added.
2366         (bmalloc::XSmallAllocator::isNull):
2367         (bmalloc::XSmallAllocator::canAllocate):
2368         (bmalloc::XSmallAllocator::XSmallAllocator):
2369         (bmalloc::XSmallAllocator::line):
2370         (bmalloc::XSmallAllocator::allocate):
2371         (bmalloc::XSmallAllocator::objectCount):
2372         (bmalloc::XSmallAllocator::derefCount):
2373         (bmalloc::XSmallAllocator::refill):
2374         * bmalloc/XSmallChunk.h: Added.
2375         * bmalloc/XSmallLine.h: Added.
2376         * bmalloc/XSmallPage.h: Added.
2377         * bmalloc/XSmallTraits.h: Added.
2378         * bmalloc/bmalloc.h:
2379         (bmalloc::api::realloc): Boiler-plate copy, as above.
2380
2381 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2382
2383         MallocBench should scavenge explicitly instead of waiting
2384         https://bugs.webkit.org/show_bug.cgi?id=131661
2385
2386         Reviewed by Andreas Kling.
2387
2388         Added explicit scavenge support to bmalloc. This isn't a memory win,
2389         since bmalloc's per-thread cache is so small. But it makes testing
2390         simpler.
2391
2392         * bmalloc/Allocator.cpp:
2393         (bmalloc::Allocator::~Allocator):
2394         (bmalloc::Allocator::scavenge):
2395         * bmalloc/Allocator.h:
2396         * bmalloc/Cache.cpp:
2397         (bmalloc::Cache::operator new):
2398         (bmalloc::Cache::operator delete):
2399         (bmalloc::Cache::Cache):
2400         (bmalloc::Cache::scavenge):
2401         * bmalloc/Cache.h:
2402         * bmalloc/Deallocator.cpp:
2403         (bmalloc::Deallocator::~Deallocator):
2404         (bmalloc::Deallocator::scavenge):
2405         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
2406         functions, for reuse.
2407
2408         * bmalloc/Heap.cpp:
2409         (bmalloc::sleep):
2410         (bmalloc::Heap::concurrentScavenge):
2411         (bmalloc::Heap::scavenge):
2412         (bmalloc::Heap::scavengeSmallPages):
2413         (bmalloc::Heap::scavengeMediumPages):
2414         (bmalloc::Heap::scavengeLargeRanges):
2415         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
2416         scavenging -- in response to a benchmark or a low memory warning --
2417         wants to complete as soon as possible, so its sleep duration is 0.
2418
2419         * bmalloc/bmalloc.h:
2420         (bmalloc::api::scavenge):
2421         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
2422
2423 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
2424
2425         Use 4kB pages on Mac
2426         https://bugs.webkit.org/show_bug.cgi?id=131658
2427
2428         Reviewed by Sam Weinig.
2429
2430         This reduces memory use a lot on Membuster:
2431
2432                                                                   base                      patch                                Δ
2433                 Execution Time:
2434                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
2435                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
2436                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
2437
2438                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
2439                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
2440                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
2441
2442                 Peak Memory:
2443                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
2444                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
2445                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
2446
2447                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
2448                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
2449                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
2450
2451                 Memory at End:
2452                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
2453                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
2454                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
2455
2456                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
2457                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
2458                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
2459
2460
2461         * bmalloc.xcodeproj/project.pbxproj:
2462         * bmalloc/BPlatform.h: Added.
2463         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
2464         is 4kB on Mac.
2465
2466 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
2467
2468         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
2469
2470         * bmalloc.xcodeproj: Modified property svn:ignore.
2471
2472 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
2473
2474         Fixed some mbmalloc exports
2475         https://bugs.webkit.org/show_bug.cgi?id=131599
2476
2477         Reviewed by Ryosuke Niwa.
2478
2479         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
2480         of the project, so we can call them from API.
2481
2482         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
2483         visibility, so they show up as exported in the .dylib.
2484
2485 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
2486
2487         Put bmalloc headers in the right place
2488         https://bugs.webkit.org/show_bug.cgi?id=131464
2489
2490         Reviewed by Mark Rowe.
2491
2492         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
2493         specify that we don't just want to dump all of our generically-named
2494         headers into /usr/local/include.
2495
2496 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2497
2498         Made bmalloc more #include friendly
2499         https://bugs.webkit.org/show_bug.cgi?id=131386
2500
2501         Reviewed by Andreas Kling.
2502
2503         Marked a bunch of headers private so they can be used from client code
2504         that #includes bmalloc.h.
2505
2506         Renamed ASSERT macros to BASSERT. This matches their header, which already
2507         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
2508
2509         * bmalloc.xcodeproj/project.pbxproj:
2510         * bmalloc/Allocator.cpp:
2511         (bmalloc::Allocator::allocateSlowCase):
2512         * bmalloc/AsyncTask.h:
2513         (bmalloc::Function>::runSlowCase):
2514         * bmalloc/BAssert.h:
2515         * bmalloc/BoundaryTag.h:
2516         (bmalloc::BoundaryTag::setSize):
2517         * bmalloc/BoundaryTagInlines.h:
2518         (bmalloc::validate):
2519         (bmalloc::BoundaryTag::init):
2520         (bmalloc::BoundaryTag::deallocate):
2521         (bmalloc::BoundaryTag::splitLarge):
2522         (bmalloc::BoundaryTag::allocate):
2523         * bmalloc/Chunk.h:
2524         * bmalloc/Deallocator.cpp:
2525         (bmalloc::Deallocator::processObjectLog):
2526         (bmalloc::Deallocator::deallocateSlowCase):
2527         * bmalloc/Deallocator.h:
2528         (bmalloc::Deallocator::deallocateFastCase):
2529         * bmalloc/FixedVector.h:
2530         (bmalloc::Capacity>::operator):
2531         (bmalloc::Capacity>::push):
2532         (bmalloc::Capacity>::pop):
2533         (bmalloc::Capacity>::shrink):
2534         * bmalloc/Heap.cpp:
2535         (bmalloc::Heap::allocateLarge):
2536         * bmalloc/LargeChunk.h:
2537         (bmalloc::LargeChunk::get):
2538         (bmalloc::LargeChunk::endTag):
2539         * bmalloc/Line.h:
2540         (bmalloc::Line<Traits>::concurrentRef):
2541         (bmalloc::Line<Traits>::deref):
2542         * bmalloc/MediumAllocator.h:
2543         (bmalloc::MediumAllocator::allocate):
2544         * bmalloc/ObjectType.h:
2545         (bmalloc::isSmall):
2546         * bmalloc/Page.h:
2547         (bmalloc::Page<Traits>::ref):
2548         (bmalloc::Page<Traits>::deref):
2549         * bmalloc/PerThread.h:
2550         (bmalloc::PerThread<T>::getSlowCase):
2551         * bmalloc/SegregatedFreeList.cpp:
2552         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2553         (bmalloc::SegregatedFreeList::insert):
2554         * bmalloc/SmallAllocator.h:
2555         (bmalloc::SmallAllocator::allocate):
2556         (bmalloc::SmallAllocator::refill):
2557         * bmalloc/Syscall.h:
2558         * bmalloc/VMAllocate.h:
2559         (bmalloc::vmValidate):
2560         (bmalloc::vmAllocate):
2561         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2562         * bmalloc/Vector.h:
2563         (bmalloc::Vector<T>::operator):
2564         (bmalloc::Vector<T>::pop):
2565         (bmalloc::Vector<T>::shrink):
2566         * bmalloc/XLargeChunk.h:
2567         (bmalloc::XLargeChunk::range):
2568         (bmalloc::XLargeChunk::size):
2569
2570 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
2571
2572         Removed an unused file.
2573
2574         Unreviewed.
2575
2576         * bmalloc/AsyncTask.cpp: Removed.
2577
2578 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2579
2580         Build bmalloc on Mac
2581         https://bugs.webkit.org/show_bug.cgi?id=131333
2582
2583         Reviewed by Mark Rowe.
2584
2585         * Makefile: Added. For make clients.
2586
2587         These files are required for building any project in WebKit. I copied
2588         them from WTF:
2589         * Configurations: Added.
2590         * Configurations/Base.xcconfig: Added.
2591         * Configurations/DebugRelease.xcconfig: Added.
2592         * Configurations/bmalloc.xcconfig: Added.
2593         * Configurations/iOS.xcconfig: Added.
2594         * Configurations/mbmalloc.xcconfig: Added.
2595
2596         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
2597         from here because everything is in .xcconfig files now.
2598
2599         I had to fix a bunch of minor warnings, since they're enabled in our
2600         .xcconfig files:
2601
2602         * bmalloc/AsyncTask.h:
2603         (bmalloc::Function>::AsyncTask):
2604         * bmalloc/BAssert.h:
2605         * bmalloc/BoundaryTagInlines.h:
2606         (bmalloc::validate):
2607         * bmalloc/Heap.cpp:
2608         (bmalloc::Heap::Heap):
2609         (bmalloc::Heap::allocateLarge):
2610         (bmalloc::Heap::deallocateLarge):
2611         * bmalloc/Mutex.h:
2612         (bmalloc::Mutex::Mutex): Deleted.
2613         * bmalloc/VMAllocate.h:
2614         (bmalloc::vmValidate):
2615         * bmalloc/mbmalloc.cpp:
2616
2617 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2618
2619         bmalloc: Fixed a leak in the per-thread cache
2620         https://bugs.webkit.org/show_bug.cgi?id=131330
2621
2622         Reviewed by Andreas Kling.
2623
2624         Remember to deallocate our line caches upon thread exit.
2625
2626         * bmalloc/Deallocator.cpp:
2627         (bmalloc::Deallocator::~Deallocator):
2628
2629 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2630
2631         bmalloc: rolled out the tryLock experiment
2632         https://bugs.webkit.org/show_bug.cgi?id=131328
2633
2634         Reviewed by Andreas Kling.
2635
2636         It wasn't a speedup.
2637
2638         * bmalloc.xcodeproj/project.pbxproj:
2639         * bmalloc/Allocator.cpp:
2640         (bmalloc::Allocator::processSmallAllocatorLog):
2641         (bmalloc::Allocator::processMediumAllocatorLog):
2642         * bmalloc/Deallocator.cpp:
2643         (bmalloc::Deallocator::processObjectLog):
2644         (bmalloc::Deallocator::deallocateSlowCase):
2645         (bmalloc::Deallocator::deallocateSmallLine):
2646         (bmalloc::Deallocator::deallocateMediumLine):
2647         * bmalloc/Deallocator.h:
2648         (bmalloc::Deallocator::deallocateFastCase):
2649         * bmalloc/Heap.h:
2650         (bmalloc::Heap::deallocateSmallLine):
2651         (bmalloc::Heap::deallocateMediumLine):
2652         * bmalloc/Line.h:
2653         (bmalloc::Line<Traits>::deref):
2654         * bmalloc/Page.h:
2655         (bmalloc::Page<Traits>::deref):
2656
2657 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
2658
2659         bmalloc
2660         https://bugs.webkit.org/show_bug.cgi?id=131170
2661
2662         Reviewed by Andreas Kling.
2663
2664         Initial commit.
2665
2666         * bmalloc: Added.
2667         * bmalloc.xcodeproj: Added.
2668         * bmalloc.xcodeproj/project.pbxproj: Added.
2669         * bmalloc/Algorithm.h: Added.
2670         (bmalloc::max):
2671         (bmalloc::min):
2672         (bmalloc::mask):
2673         (bmalloc::test):
2674         (bmalloc::roundUpToMultipleOf):
2675         (bmalloc::roundDownToMultipleOf):
2676         (bmalloc::sizeOf):
2677         (bmalloc::bitCount):
2678         (bmalloc::isPowerOfTwo):
2679         * bmalloc/Allocator.cpp: Added.
2680         (bmalloc::Allocator::Allocator):
2681         (bmalloc::Allocator::~Allocator):
2682         (bmalloc::Allocator::log):
2683         (bmalloc::Allocator::processSmallAllocatorLog):
2684         (bmalloc::Allocator::processMediumAllocatorLog):
2685         (bmalloc::Allocator::allocateLarge):
2686         (bmalloc::Allocator::allocateXLarge):
2687         (bmalloc::Allocator::allocateMedium):
2688         (bmalloc::Allocator::allocateSlowCase):
2689         * bmalloc/Allocator.h: Added.
2690         (bmalloc::Allocator::smallAllocatorFor):
2691         (bmalloc::Allocator::allocateFastCase):
2692         (bmalloc::Allocator::allocate):
2693         * bmalloc/AsyncTask.cpp: Added.
2694         (bmalloc::AsyncTask<Function>::runSlowCase):
2695         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2696         (bmalloc::AsyncTask<Function>::entryPoint):
2697         * bmalloc/AsyncTask.h: Added.
2698         (bmalloc::Function>::AsyncTask):
2699         (bmalloc::Function>::join):
2700         (bmalloc::Function>::run):
2701         (bmalloc::Function>::runSlowCase):
2702         (bmalloc::Function>::pthreadEntryPoint):
2703         (bmalloc::Function>::entryPoint):
2704         * bmalloc/BAssert.h: Added.
2705         * bmalloc/BeginTag.h: Added.
2706         (bmalloc::BeginTag::isInFreeList):
2707         * bmalloc/BoundaryTag.h: Added.
2708         (bmalloc::BoundaryTag::isXLarge):
2709         (bmalloc::BoundaryTag::setXLarge):
2710         (bmalloc::BoundaryTag::isFree):
2711         (bmalloc::BoundaryTag::setFree):
2712         (bmalloc::BoundaryTag::isEnd):
2713         (bmalloc::BoundaryTag::setEnd):
2714         (bmalloc::BoundaryTag::hasPhysicalPages):
2715         (bmalloc::BoundaryTag::setHasPhysicalPages):
2716         (bmalloc::BoundaryTag::isNull):
2717         (bmalloc::BoundaryTag::clear):
2718         (bmalloc::BoundaryTag::size):
2719         (bmalloc::BoundaryTag::setSize):
2720         (bmalloc::BoundaryTag::prev):
2721         (bmalloc::BoundaryTag::next):
2722         * bmalloc/BoundaryTagInlines.h: Added.
2723         (bmalloc::validate):
2724         (bmalloc::validatePrev):
2725         (bmalloc::validateNext):
2726         (bmalloc::BoundaryTag::init):
2727         (bmalloc::BoundaryTag::mergeLargeLeft):
2728         (bmalloc::BoundaryTag::mergeLargeRight):
2729         (bmalloc::BoundaryTag::mergeLarge):
2730         (bmalloc::BoundaryTag::deallocate):
2731         (bmalloc::BoundaryTag::splitLarge):
2732         (bmalloc::BoundaryTag::allocate):
2733         * bmalloc/Cache.cpp: Added.
2734         (bmalloc::Cache::operator new):
2735         (bmalloc::Cache::operator delete):
2736         (bmalloc::Cache::Cache):
2737         (bmalloc::Cache::allocateSlowCase):
2738         (bmalloc::Cache::allocateSlowCaseNullCache):
2739         (bmalloc::Cache::deallocateSlowCase):
2740         (bmalloc::Cache::deallocateSlowCaseNullCache):
2741         * bmalloc/Cache.h: Added.
2742         (bmalloc::Cache::allocator):
2743         (bmalloc::Cache::deallocator):
2744         (bmalloc::Cache::allocateFastCase):
2745         (bmalloc::Cache::deallocateFastCase):
2746         (bmalloc::Cache::allocate):
2747         (bmalloc::Cache::deallocate):
2748         * bmalloc/Chunk.h: Added.
2749         (bmalloc::Chunk::begin):
2750         (bmalloc::Chunk::end):
2751         (bmalloc::Chunk::lines):
2752         (bmalloc::Chunk::pages):
2753         * bmalloc/Deallocator.cpp: Added.
2754         (bmalloc::Deallocator::Deallocator):
2755         (bmalloc::Deallocator::~Deallocator):
2756         (bmalloc::Deallocator::deallocateLarge):
2757         (bmalloc::Deallocator::deallocateXLarge):
2758         (bmalloc::Deallocator::processObjectLog):
2759         (bmalloc::Deallocator::deallocateSlowCase):
2760         (bmalloc::Deallocator::deallocateSmallLine):
2761         (bmalloc::Deallocator::allocateSmallLine):
2762         (bmalloc::Deallocator::deallocateMediumLine):
2763         (bmalloc::Deallocator::allocateMediumLine):
2764         * bmalloc/Deallocator.h: Added.
2765         (bmalloc::Deallocator::deallocateFastCase):
2766         (bmalloc::Deallocator::deallocate):
2767         * bmalloc/EndTag.h: Added.
2768         (bmalloc::EndTag::operator=):
2769         * bmalloc/FixedVector.h: Added.
2770         (bmalloc::FixedVector::begin):
2771         (bmalloc::FixedVector::end):
2772         (bmalloc::FixedVector::size):
2773         (bmalloc::FixedVector::capacity):
2774         (bmalloc::FixedVector::clear):
2775         (bmalloc::FixedVector::isEmpty):
2776         (bmalloc::Capacity>::FixedVector):
2777         (bmalloc::Capacity>::operator):
2778         (bmalloc::Capacity>::push):
2779         (bmalloc::Capacity>::pop):
2780         (bmalloc::Capacity>::shrink):
2781         * bmalloc/Heap.cpp: Added.
2782         (bmalloc::sleep):
2783         (bmalloc::Heap::Heap):
2784         (bmalloc::Heap::concurrentScavenge):
2785         (bmalloc::Heap::scavengeSmallPages):
2786         (bmalloc::Heap::scavengeMediumPages):
2787         (bmalloc::Heap::scavengeLargeRanges):
2788         (bmalloc::Heap::allocateSmallLineSlowCase):
2789         (bmalloc::Heap::allocateMediumLineSlowCase):
2790         (bmalloc::Heap::allocateXLarge):
2791         (bmalloc::Heap::deallocateXLarge):
2792         (bmalloc::Heap::allocateLarge):
2793         (bmalloc::Heap::deallocateLarge):
2794         * bmalloc/Heap.h: Added.
2795         (bmalloc::Heap::deallocateSmallLine):
2796         (bmalloc::Heap::allocateSmallLine):
2797         (bmalloc::Heap::deallocateMediumLine):
2798         (bmalloc::Heap::allocateMediumLine):
2799         * bmalloc/Inline.h: Added.
2800         * bmalloc/LargeChunk.h: Added.
2801         (bmalloc::LargeChunk::begin):
2802         (bmalloc::LargeChunk::end):
2803         (bmalloc::LargeChunk::create):
2804         (bmalloc::LargeChunk::get):
2805         (bmalloc::LargeChunk::beginTag):
2806         (bmalloc::LargeChunk::endTag):
2807         * bmalloc/Line.h: Added.
2808         (bmalloc::Line<Traits>::begin):
2809         (bmalloc::Line<Traits>::end):
2810         (bmalloc::Line<Traits>::concurrentRef):
2811         (bmalloc::Line<Traits>::deref):
2812         * bmalloc/MediumAllocator.h: Added.
2813         (bmalloc::MediumAllocator::isNull):
2814         (bmalloc::MediumAllocator::MediumAllocator):
2815         (bmalloc::MediumAllocator::line):
2816         (bmalloc::MediumAllocator::allocate):
2817         (bmalloc::MediumAllocator::derefCount):
2818         (bmalloc::MediumAllocator::refill):
2819         * bmalloc/MediumChunk.h: Added.
2820         * bmalloc/MediumLine.h: Added.
2821         * bmalloc/MediumPage.h: Added.
2822         * bmalloc/MediumTraits.h: Added.
2823         * bmalloc/Mutex.cpp: Added.
2824         (bmalloc::Mutex::lockSlowCase):
2825         * bmalloc/Mutex.h: Added.
2826         (bmalloc::Mutex::Mutex):
2827         (bmalloc::Mutex::try_lock):
2828         (bmalloc::Mutex::lock):
2829         (bmalloc::Mutex::unlock):
2830         * bmalloc/ObjectType.cpp: Added.
2831         (bmalloc::objectType):
2832         * bmalloc/ObjectType.h: Added.
2833         (bmalloc::isSmallOrMedium):
2834         (bmalloc::isSmall):
2835         * bmalloc/Page.h: Added.
2836         (bmalloc::Page<Traits>::ref):
2837         (bmalloc::Page<Traits>::deref):
2838         (bmalloc::Page<Traits>::refCount):
2839         * bmalloc/PerProcess.h: Added.
2840         (bmalloc::PerProcess::mutex):
2841         (bmalloc::PerProcess<T>::getFastCase):
2842         (bmalloc::PerProcess<T>::get):
2843         (bmalloc::PerProcess<T>::getSlowCase):
2844         * bmalloc/PerThread.h: Added.
2845         (bmalloc::PerThreadStorage<Cache>::get):
2846         (bmalloc::PerThreadStorage<Cache>::init):
2847         (bmalloc::PerThreadStorage::get):
2848         (bmalloc::PerThreadStorage::init):
2849         (bmalloc::PerThread<T>::getFastCase):
2850         (bmalloc::PerThread<T>::get):
2851         (bmalloc::PerThread<T>::destructor):
2852         (bmalloc::PerThread<T>::getSlowCase):
2853         * bmalloc/Range.h: Added.
2854         (bmalloc::Range::Range):
2855         (bmalloc::Range::begin):
2856         (bmalloc::Range::end):
2857         (bmalloc::Range::size):
2858         (bmalloc::Range::operator!):
2859         (bmalloc::Range::operator<):
2860         * bmalloc/SegregatedFreeList.cpp: Added.
2861         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2862         (bmalloc::SegregatedFreeList::insert):
2863         (bmalloc::SegregatedFreeList::takeGreedy):
2864         (bmalloc::SegregatedFreeList::take):
2865         * bmalloc/SegregatedFreeList.h: Added.
2866         * bmalloc/Sizes.h: Added.
2867         * bmalloc/SmallAllocator.h: Added.
2868         (bmalloc::SmallAllocator::isNull):
2869         (bmalloc::SmallAllocator::canAllocate):
2870         (bmalloc::SmallAllocator::SmallAllocator):
2871         (bmalloc::SmallAllocator::line):
2872         (bmalloc::SmallAllocator::allocate):
2873         (bmalloc::SmallAllocator::objectCount):
2874         (bmalloc::SmallAllocator::derefCount):
2875         (bmalloc::SmallAllocator::refill):
2876         * bmalloc/SmallChunk.h: Added.
2877         * bmalloc/SmallLine.h: Added.
2878         * bmalloc/SmallPage.h: Added.
2879         * bmalloc/SmallTraits.h: Added.
2880         * bmalloc/Syscall.h: Added.
2881         * bmalloc/VMAllocate.h: Added.
2882         (bmalloc::vmSize):
2883         (bmalloc::vmValidate):
2884         (bmalloc::vmAllocate):
2885         (bmalloc::vmDeallocate):
2886         (bmalloc::vmDeallocatePhysicalPages):
2887         (bmalloc::vmAllocatePhysicalPages):
2888         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2889         (bmalloc::vmAllocatePhysicalPagesSloppy):
2890         * bmalloc/VMHeap.cpp: Added.
2891         (bmalloc::VMHeap::VMHeap):
2892         (bmalloc::VMHeap::allocateSmallChunk):
2893         (bmalloc::VMHeap::allocateMediumChunk):
2894         (bmalloc::VMHeap::allocateLargeChunk):
2895         * bmalloc/VMHeap.h: Added.
2896         (bmalloc::VMHeap::allocateSmallPage):
2897         (bmalloc::VMHeap::allocateMediumPage):
2898         (bmalloc::VMHeap::allocateLargeRange):
2899         (bmalloc::VMHeap::deallocateSmallPage):
2900         (bmalloc::VMHeap::deallocateMediumPage):
2901         (bmalloc::VMHeap::deallocateLargeRange):
2902         * bmalloc/Vector.h: Added.
2903         (bmalloc::Vector::begin):
2904         (bmalloc::Vector::end):
2905         (bmalloc::Vector::size):
2906         (bmalloc::Vector::capacity):
2907         (bmalloc::Vector::last):
2908         (bmalloc::Vector::pop):
2909         (bmalloc::Vector<T>::Vector):
2910         (bmalloc::Vector<T>::~Vector):
2911         (bmalloc::Vector<T>::operator):
2912         (bmalloc::Vector<T>::push):
2913         (bmalloc::Vector<T>::pop):
2914         (bmalloc::Vector<T>::shrink):
2915         (bmalloc::Vector<T>::reallocateBuffer):
2916         (bmalloc::Vector<T>::shrinkCapacity):
2917         (bmalloc::Vector<T>::growCapacity):
2918         * bmalloc/XLargeChunk.h: Added.
2919         (bmalloc::XLargeChunk::get):
2920         (bmalloc::XLargeChunk::begin):
2921         (bmalloc::XLargeChunk::XLargeChunk):
2922         (bmalloc::XLargeChunk::create):
2923         (bmalloc::XLargeChunk::destroy):
2924         (bmalloc::XLargeChunk::range):
2925         (bmalloc::XLargeChunk::size):
2926         * bmalloc/bmalloc.h: Added.
2927         (bmalloc::api::malloc):
2928         (bmalloc::api::free):
2929         (bmalloc::api::realloc):
2930         * bmalloc/mbmalloc.cpp: Added.
2931