Use the system string for telephone number menu
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc uses 8X more virtual memory than necessary
4         https://bugs.webkit.org/show_bug.cgi?id=138495
5
6         Reviewed by Mark Lam.
7
8         iOS has a per-process virtual memory cap around 1GB, so there's some
9         value to not going totally ham with virtual memory.
10
11         We currently use about 8X the necessary amount:
12             - 2X to align our VM allocation
13             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
14
15         We can cut that down:
16             - Return the unaligned portion of our VM allocation (-2X)
17             - Use all the chunks in a superchunk, instead of allocating one
18               chunk per superchunk (-4X)
19
20         * bmalloc/Algorithm.h:
21         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
22         function so we can call it with getpagesize() at runtime.
23
24         * bmalloc/Chunk.h:
25         * bmalloc/LargeChunk.h:
26         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
27         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
28
29         * bmalloc/VMAllocate.h:
30         (bmalloc::vmValidate):
31         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
32         if it does not succeed. Allocate precisely, and give back the extra.
33
34         * bmalloc/VMHeap.cpp:
35         (bmalloc::VMHeap::allocateSuperChunk):
36         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
37         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
38         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
39         in a superchunk, instead of just one.
40
41         * bmalloc/VMHeap.h:
42         (bmalloc::VMHeap::allocateSmallPage):
43         (bmalloc::VMHeap::allocateMediumPage):
44         (bmalloc::VMHeap::allocateLargeRange):
45         * bmalloc/XLargeChunk.h:
46         (bmalloc::XLargeChunk::create): Updated to match changes above.
47
48 2014-11-01  David Kilzer  <ddkilzer@apple.com>
49
50         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
51         <https://webkit.org/b/138286>
52         <rdar://problem/18847087>
53
54         Reviewed by Dan Bernstein.
55
56         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
57         to NO for the target that produces libbmalloc.a so that the
58         debug symbols will be linked into JavaScriptCore and end up in
59         its dSYM file.
60
61 2014-10-30  Dana Burkart  <dburkart@apple.com>
62
63         <rdar://problem/18821260> Prepare for the mysterious future
64
65         Reviewed by Lucas Forschler.
66
67         * Configurations/Base.xcconfig:
68         * Configurations/DebugRelease.xcconfig:
69
70 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
71
72         bmalloc: cleaned up fast path vs slow path
73         https://bugs.webkit.org/show_bug.cgi?id=137081
74
75         Reviewed by Sam Weinig.
76
77         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
78
79         * bmalloc/Allocator.cpp:
80         (bmalloc::Allocator::Allocator): Merged the small and medium range
81         caches, just like the small and medium allocators. Ranges are abstract
82         objects that don't really care whether they hold small or medium objects,
83         so they don't need to be segregated.
84
85         (bmalloc::Allocator::scavenge): Ditto.
86
87         (bmalloc::Allocator::allocateBumpRangeSlowCase):
88         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
89         we do care a tiny bit, because we need to specify small vs medium when
90         allocating new ranges from the heap, to ensure that the heap allocates
91         from the right segment of VM.
92
93         (bmalloc::Allocator::allocateLarge):
94         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
95         up the fast path. Large allocation performance is dominated by allocation
96         logic and initialization, so inlining it doesn't help.
97
98         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
99         it doesn't need to distinguish small vs medium objects.
100
101         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
102         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
103
104         * bmalloc/Allocator.h:
105         * bmalloc/BumpRange.h:
106
107         * bmalloc/Cache.cpp:
108         (bmalloc::Cache::allocateSlowCase): Deleted.
109         (bmalloc::Cache::deallocateSlowCase): Deleted.
110         * bmalloc/Cache.h:
111         (bmalloc::Cache::allocate):
112         (bmalloc::Cache::deallocate):
113         (bmalloc::Cache::allocateFastCase): Deleted.
114         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
115         paths. The downside to this change is that the fast path branches to two
116         distinct failure cases instead of one. The upside is that the slow path
117         doesn't need to re-read the segment register, which is not as cheap as a
118         normal register, and it doesn't need to do an extra level of function 
119         call. Seems to be worth it.
120
121         * bmalloc/Deallocator.h:
122         * bmalloc/Heap.cpp:
123         (bmalloc::Heap::refillSmallBumpRangeCache):
124         (bmalloc::Heap::refillMediumBumpRangeCache):
125         * bmalloc/Heap.h: Updated for interface changes.
126
127         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
128         small lines in a page / 2, since any other free lines will coalesce
129         with their neighbors.
130
131 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
132
133         Rolled out r173346.
134
135             bmalloc should honor the FastMalloc statistics API
136             https://bugs.webkit.org/show_bug.cgi?id=136592
137
138         This didn't really work. Because we allow ranges with and without
139         physical pages to merge, and we allow double-committing and
140         double-decommitting, we can't rely on commit actions to track memory
141         footprint.
142
143         * bmalloc/Heap.cpp:
144         (bmalloc::Heap::size): Deleted.
145         (bmalloc::Heap::capacity): Deleted.
146         * bmalloc/Heap.h:
147         * bmalloc/VMHeap.cpp:
148         (bmalloc::VMHeap::VMHeap):
149         (bmalloc::VMHeap::allocateSmallChunk):
150         (bmalloc::VMHeap::allocateMediumChunk):
151         (bmalloc::VMHeap::allocateLargeChunk):
152         * bmalloc/VMHeap.h:
153         (bmalloc::VMHeap::allocateSmallPage):
154         (bmalloc::VMHeap::allocateMediumPage):
155         (bmalloc::VMHeap::allocateLargeRange):
156         (bmalloc::VMHeap::deallocateSmallPage):
157         (bmalloc::VMHeap::deallocateMediumPage):
158         (bmalloc::VMHeap::deallocateLargeRange):
159         (bmalloc::VMHeap::size): Deleted.
160         (bmalloc::VMHeap::capacity): Deleted.
161         * bmalloc/bmalloc.h:
162         (bmalloc::api::heapSize): Deleted.
163         (bmalloc::api::heapCapacity): Deleted.
164
165 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
166
167         bmalloc: Allocation should be more precise
168         https://bugs.webkit.org/show_bug.cgi?id=136993
169
170         Reviewed by Gavin Barraclough.
171
172         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
173
174         This patch teaches the allocator to merge adjacent free lines into a
175         single allocatable range. This allows us to shrink the size of an
176         individual line without increasing fragmentation or the rate of allocator
177         slow paths.
178
179         We'll only take more slow paths when available memory is sparse, which
180         is exactly when it's worth it. When available memory is dense, we'll
181         take fewer slow paths.
182
183         * bmalloc.xcodeproj/project.pbxproj:
184         * bmalloc/Algorithm.h:
185         (bmalloc::divideRoundingUp):
186
187         * bmalloc/Allocator.cpp:
188         (bmalloc::Allocator::Allocator): Updated for interface changes.
189
190         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
191         Now that we merge lines, it's not convenient to scavenge by line.
192
193         (bmalloc::Allocator::allocateSmallBumpRange):
194         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
195         instead of individual lines.
196
197         (bmalloc::Allocator::allocateSlowCase):
198         (bmalloc::Allocator::allocateSmallLine): Deleted.
199         (bmalloc::Allocator::allocateMediumLine): Deleted.
200         (bmalloc::Allocator::allocateMedium): Deleted.
201         * bmalloc/Allocator.h:
202         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
203         into the standard fast path with small allocations. Since a BumpAllocator
204         just allocates out of an arbitrary range, it doesn't need to distinguish
205         between small and medium lines.
206
207         * bmalloc/BumpAllocator.h:
208         (bmalloc::BumpAllocator::size):
209         (bmalloc::BumpAllocator::BumpAllocator):
210         (bmalloc::BumpAllocator::init):
211         (bmalloc::BumpAllocator::refill):
212         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
213         anymore: the heap just gives us a pointer and a pre-computed number of
214         objects, and we allocate them.
215
216         * bmalloc/Deallocator.cpp:
217         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
218
219         * bmalloc/Heap.cpp:
220         (bmalloc::Heap::Heap):
221         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
222         detailing where all objects will lie in memory. After we merge two lines,
223         we might allocate an object that spans from one line to the next. This
224         metadata details which bits of memory overlap in that way, and how they
225         overlap.
226
227         (bmalloc::Heap::refillSmallBumpRangeCache):
228         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
229         and merge adjacent free lines into BumpRanges.
230
231         (bmalloc::Heap::allocateSmallPage):
232         (bmalloc::Heap::allocateMediumPage):
233         (bmalloc::Heap::deallocateSmallLine):
234         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
235         since we scan for free memory a page at a time.
236
237         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
238         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
239         fast path.
240
241         * bmalloc/Heap.h:
242         (bmalloc::Heap::derefSmallLine):
243         (bmalloc::Heap::derefMediumLine):
244         (bmalloc::Heap::deallocateSmallLine): Deleted.
245         (bmalloc::Heap::allocateSmallLine): Deleted.
246         (bmalloc::Heap::deallocateMediumLine): Deleted.
247         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
248
249         * bmalloc/Line.h:
250         (bmalloc::Line<Traits>::ref):
251         (bmalloc::Line<Traits>::deref):
252         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
253         anymore, since we only ever deref by 1 now.
254
255         * bmalloc/MediumAllocator.h:
256         (bmalloc::MediumAllocator::isNull): Deleted.
257         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
258         (bmalloc::MediumAllocator::line): Deleted.
259         (bmalloc::MediumAllocator::allocate): Deleted.
260         (bmalloc::MediumAllocator::derefCount): Deleted.
261         (bmalloc::MediumAllocator::refill): Deleted.
262         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
263         been dead for a while, since it doesn't build anymore with this patch.
264
265         * bmalloc/Page.h:
266         (bmalloc::Page::sizeClass):
267         (bmalloc::Page::setSizeClass):
268         (bmalloc::Page::smallSizeClass): Deleted.
269         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
270         to sizeClass, since we use it for medium sizes too.
271
272         * bmalloc/Sizes.h:
273         (bmalloc::Sizes::sizeClass):
274         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
275
276         (bmalloc::Sizes::smallSizeClassFor): Deleted.
277         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
278
279         * bmalloc/bmalloc.h:
280         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
281         can be a bit more precise. It also has to be, since we can't guarantee
282         that an object ends at the end of a line anymore.
283
284 2014-09-19  Daniel Bates  <dabates@apple.com>
285
286         Always assume internal SDK when building configuration Production
287         https://bugs.webkit.org/show_bug.cgi?id=136925
288         <rdar://problem/18362399>
289
290         Reviewed by Dan Bernstein.
291
292         * Configurations/Base.xcconfig:
293
294 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
295
296         bmalloc: moved line caches from the deallocator to the allocator
297         https://bugs.webkit.org/show_bug.cgi?id=136868
298
299         Reviewed by Gavin Barraclough.
300
301         I did this mostly as a simplification, to make it easier to change the
302         allocation strategy.
303
304         No throughput change on MallocBench. Saves about 50kB.
305
306         Since the deallocator needs to lock the heap when freeing lines anyway,
307         there isn't much benefit to giving the deallocator a local cache of
308         deallocated lines.
309
310         We still give the allocator a local cache of lines because that does
311         reduce the frequency at which it needs to lock the heap in order to
312         acquire more lines.
313
314         * bmalloc/Allocator.cpp:
315         (bmalloc::Allocator::scavenge):
316         (bmalloc::Allocator::allocateSmallLine):
317         (bmalloc::Allocator::allocateMediumLine):
318         (bmalloc::Allocator::allocateMedium):
319         (bmalloc::Allocator::allocateSlowCase):
320         * bmalloc/Allocator.h:
321         * bmalloc/Deallocator.cpp:
322         (bmalloc::Deallocator::Deallocator):
323         (bmalloc::Deallocator::scavenge):
324         (bmalloc::Deallocator::processObjectLog):
325         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
326         (bmalloc::Deallocator::allocateSmallLine): Deleted.
327         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
328         (bmalloc::Deallocator::allocateMediumLine): Deleted.
329         * bmalloc/Deallocator.h:
330
331         * bmalloc/Sizes.h:
332         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
333         exactly one page in size. That's about what we were shooting for anyway,
334         and it may make it easier to switch to per-page allocation in future.
335
336 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
337
338         bmalloc: allocate small and medium objects using the same bump pointer class
339         https://bugs.webkit.org/show_bug.cgi?id=136843
340
341         Reviewed by Gavin Barraclough.
342
343         4% speedup on MallocBench.
344
345         Now that medium-sized objects have dedicated per-size allocators, they
346         don't need to use an arbitrary bump pointer allocator. This means that
347         every allocator knows how many objects it will allocate from the start,
348         and we don't need a post-processing step to adjust refcounts based on
349         real allocation count.
350
351         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
352         since it's used for small and medium objects now.
353
354         * bmalloc/Allocator.cpp:
355         (bmalloc::Allocator::Allocator): Updated to use new interface.
356         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
357         to make sure that we finish allocating all the objects in it.
358
359         (bmalloc::Allocator::allocateMedium):
360         (bmalloc::Allocator::allocateSlowCase):
361         (bmalloc::Allocator::retire): Deleted.
362         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
363         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
364         * bmalloc/Allocator.h:
365         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
366         used to post-process an allocator based on how many objects it allocated.
367
368         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
369         (bmalloc::BumpAllocator::BumpAllocator):
370         (bmalloc::BumpAllocator::init):
371         (bmalloc::BumpAllocator::line):
372         (bmalloc::BumpAllocator::validate):
373         (bmalloc::BumpAllocator::allocate):
374         (bmalloc::BumpAllocator::refill):
375         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
376         about the kinds of lines they allocate into. In some cases, the line
377         type must be provided as a template parameter by the caller.
378
379         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
380         (bmalloc::SmallAllocator::line): Deleted.
381         (bmalloc::SmallAllocator::allocate): Deleted.
382         (bmalloc::SmallAllocator::objectCount): Deleted.
383         (bmalloc::SmallAllocator::derefCount): Deleted.
384         (bmalloc::SmallAllocator::refill): Deleted.
385         (bmalloc::SmallAllocator::clear): Deleted.
386
387         * bmalloc/ObjectType.h:
388         (bmalloc::isMedium):
389
390         * bmalloc/SmallAllocator.h:
391         (bmalloc::SmallAllocator::isNull): Deleted.
392         (bmalloc::SmallAllocator::canAllocate): Deleted.
393         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
394         (bmalloc::SmallAllocator::line): Deleted.
395         (bmalloc::SmallAllocator::allocate): Deleted.
396         (bmalloc::SmallAllocator::objectCount): Deleted.
397         (bmalloc::SmallAllocator::derefCount): Deleted.
398         (bmalloc::SmallAllocator::refill): Deleted.
399         (bmalloc::SmallAllocator::clear): Deleted.
400
401 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
402
403         Fixed a goof in bmalloc Vector sizing
404         https://bugs.webkit.org/show_bug.cgi?id=136795
405
406         Reviewed by Gavin Barraclough and Sam Weinig.
407
408         We want our minimum vector to be page-sized since the OS will give us
409         a page no matter what -- but we want that many bytes, and not enough
410         bytes to store that many elements.
411
412         * bmalloc/Vector.h: Math is hard.
413
414 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
415
416         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
417         https://bugs.webkit.org/show_bug.cgi?id=136693
418
419         Reviewed by Gavin Barraclough.
420
421         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
422
423         No throughput change.
424
425         We keep an array of medium allocators, just like our array of small
426         allocators.
427
428         In future, we can simplify the allocation fast path by merging the small
429         and medium allocator arrays. For now, this is the simplest change that
430         gets the win.
431
432         * bmalloc/Allocator.cpp:
433         (bmalloc::Allocator::Allocator):
434         (bmalloc::Allocator::scavenge):
435         (bmalloc::Allocator::allocateMedium):
436         * bmalloc/Allocator.h:
437         * bmalloc/Sizes.h:
438         (bmalloc::Sizes::mediumSizeClassFor):
439
440 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
441
442         Reviewed by Sam Weinig.
443
444         Renamed log => retire for clarity.
445
446         * bmalloc/Allocator.cpp:
447         (bmalloc::Allocator::scavenge):
448         (bmalloc::Allocator::retire):
449         (bmalloc::Allocator::allocateMedium):
450         (bmalloc::Allocator::allocateSlowCase):
451         (bmalloc::Allocator::log): Deleted.
452         * bmalloc/Allocator.h:
453
454 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
455
456         bmalloc: eager scavenge leaves behind a bogus allocator
457         https://bugs.webkit.org/show_bug.cgi?id=136743
458
459         Reviewed by Sam Weinig.
460
461         Be sure to clear the allocator after logging it in the eager scavenge
462         case, so that we don't later try to allocate out of the lines that we
463         have thrown away.
464
465         We didn't need to do this previously because scavenge would only happen
466         at thread exit time, after which no further allocation from the per-thread
467         cache would take place.
468
469         * bmalloc/Allocator.cpp:
470         (bmalloc::Allocator::scavenge):
471         * bmalloc/MediumAllocator.h:
472         (bmalloc::MediumAllocator::clear):
473         * bmalloc/SmallAllocator.h:
474         (bmalloc::SmallAllocator::clear):
475
476 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
477
478         bmalloc should honor the FastMalloc statistics API
479         https://bugs.webkit.org/show_bug.cgi?id=136592
480
481         Reviewed by Gavin Barraclough.
482
483         We do this by tracking "size" and "capacity" in the VM heap.
484
485         The VM heap's "capacity" is all the VM we ever allocated.
486
487         The VM heap's "size" the subset of VM currently held onto by the
488         VM heap (and therefore not in use by the regular heap).
489
490         Somewhat ironically, reducing the process's memory footprint, increases
491         the size of the VM heap, since the VM heap holds the pages that are
492         purely virtual and not physical.
493
494         * bmalloc/Heap.cpp:
495         (bmalloc::Heap::size):
496         (bmalloc::Heap::capacity):
497         * bmalloc/Heap.h:
498         * bmalloc/VMHeap.cpp:
499         (bmalloc::VMHeap::VMHeap):
500         (bmalloc::VMHeap::allocateSmallChunk):
501         (bmalloc::VMHeap::allocateMediumChunk):
502         (bmalloc::VMHeap::allocateLargeChunk):
503         * bmalloc/VMHeap.h:
504         (bmalloc::VMHeap::size):
505         (bmalloc::VMHeap::capacity):
506         (bmalloc::VMHeap::allocateSmallPage):
507         (bmalloc::VMHeap::allocateMediumPage):
508         (bmalloc::VMHeap::allocateLargeRange):
509         (bmalloc::VMHeap::deallocateSmallPage):
510         (bmalloc::VMHeap::deallocateMediumPage):
511         (bmalloc::VMHeap::deallocateLargeRange):
512         * bmalloc/bmalloc.h:
513         (bmalloc::api::heapSize):
514         (bmalloc::api::heapCapacity):
515
516 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
517
518         bmalloc crashes on the EWS bots (due to bad large object allocation)
519         https://bugs.webkit.org/show_bug.cgi?id=136469
520
521         Reviewed by Andreas Kling.
522
523         It's possible to convince bmalloc to perform a bad large object allocation,
524         through these steps:
525
526         (1) Insert object A into freelist F0.
527
528         (2) Split, merge and split again A's neighbors such that object B is
529         inserted into freelist F0, with boundary tag and size equal to object A,
530         but pointer not completely equal to object A. Put object B at the head of F0.
531
532         (3) Allocate some other object from F0, swapping its position in the
533         freelist with object B, such that object A is now ahead of object B.
534
535         --> Now, the next allocation for size A/B will allocate object A, which
536         has a slightly wrong idea about where the object actually begins.
537         Immediately, you'll corrupt a little memory, and over time, you'll also
538         corrupt boundary tag metadata.
539
540         The solution is to store the begin pointer in the boundary tag. Luckily,
541         this doesn't make the tag any bigger, and it's not a noticeable slowdown
542         on MallocBench.
543
544         * bmalloc/Algorithm.h:
545         (bmalloc::rightShift):
546         * bmalloc/BeginTag.h:
547         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
548         validate the start pointer when popping off the free list. Through a
549         very uncommon set of steps, it is possible to have an item in the free
550         list that is valid by all accounts except for its start pointer.
551
552         * bmalloc/BoundaryTag.h:
553         (bmalloc::BoundaryTag::compactBegin):
554         (bmalloc::BoundaryTag::setRange):
555         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
556         start pointer. We don't need the whole pointer -- just the offset, in
557         largeAlignment increments, into the relevant boundary tag bucket.
558
559         * bmalloc/BoundaryTagInlines.h:
560         (bmalloc::validateNext):
561         (bmalloc::BoundaryTag::init):
562         (bmalloc::BoundaryTag::mergeLarge):
563         (bmalloc::BoundaryTag::splitLarge):
564         * bmalloc/SegregatedFreeList.cpp:
565         (bmalloc::SegregatedFreeList::insert):
566         (bmalloc::SegregatedFreeList::takeGreedy):
567         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
568         the size when establishing a boundary tag, as required by the new
569         interface.
570
571         * bmalloc/Sizes.h:
572
573 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
574
575         Fixed a bmalloc crash seen on the EWS bot
576         https://bugs.webkit.org/show_bug.cgi?id=135955
577
578         Reviewed by Andreas Kling.
579
580         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
581         input buffer is a malloc region, that region will get marked Copy-On-Write
582         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
583         on older OS X's. In 10.10, they still fail, but they do not return
584         EINVAL.
585
586         So, we can only ASSERT that our syscalls succeed starting with 10.10.
587
588 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
589
590         Fixed the bmalloc build
591         https://bugs.webkit.org/show_bug.cgi?id=135953
592
593         Reviewed by Andreas Kling.
594
595         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
596         These headers are used, so they must be available outside the project.
597
598 2014-08-13  Daniel Bates  <dabates@apple.com>
599
600         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
601         (https://bugs.webkit.org/show_bug.cgi?id=135895)
602
603         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
604         implementation of PerThread<T>::getFastCase().
605
606         * bmalloc/PerThread.h:
607         (bmalloc::PerThread<T>::getFastCase):
608
609 2014-08-13  Daniel Bates  <dabates@apple.com>
610
611         Make bmalloc::PerThread work without C++ thread local storage
612         https://bugs.webkit.org/show_bug.cgi?id=135895
613
614         Reviewed by Geoffrey Garen.
615
616         Implement support for building bmalloc without C++ thread local storage.
617
618         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
619         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
620         to determine whether the compiler supports C++ thread local storage.
621         * bmalloc/PerThread.h:
622         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
623         without C++ thread local storage.
624         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
625         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
626         PerThreadStorage::initSharedKeyIfNeeded().
627         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
628         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
629         ensure that the shared key has been initialized.
630         (_pthread_setspecific_direct): Deleted.
631         (_pthread_getspecific_direct): Deleted.
632
633 2014-08-13  Daniel Bates  <dabates@apple.com>
634
635         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
636         https://bugs.webkit.org/show_bug.cgi?id=135848
637
638         Reviewed by Geoffrey Garen.
639
640         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
641         when building for the iOS Simulator.
642         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
643         using the public SDK.
644         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
645         using the public SDK.
646         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
647         using the public SDK.
648
649 2014-08-12  Daniel Bates  <dabates@apple.com>
650
651         BPLATFORM(IOS) always evaluates to false
652         https://bugs.webkit.org/show_bug.cgi?id=135843
653
654         Reviewed by Geoffrey Garen.
655
656         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
657         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
658         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
659         building for iOS.
660
661         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
662         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
663
664         * bmalloc/BPlatform.h:
665
666 2014-08-11  Andy Estes  <aestes@apple.com>
667
668         [iOS] Get rid of iOS.xcconfig
669         https://bugs.webkit.org/show_bug.cgi?id=135809
670
671         Reviewed by Joseph Pecoraro.
672
673         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
674
675         * Configurations/Base.xcconfig:
676         * Configurations/iOS.xcconfig: Removed.
677         * bmalloc.xcodeproj/project.pbxproj:
678
679 2014-05-01  Dan Bernstein  <mitz@apple.com>
680
681         Fixed production builds for the iOS Simulator.
682         <rdar://problem/16792221>
683
684         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
685         PRIVATE_HEADERS_FOLDER_PATH when installing.
686
687 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
688
689         bmalloc: Segregate pages by objects size
690         https://bugs.webkit.org/show_bug.cgi?id=131909
691
692         Reviewed by Andreas Kling.
693
694         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
695
696         * bmalloc/Allocator.cpp:
697         (bmalloc::Allocator::allocateSlowCase):
698         * bmalloc/Allocator.h:
699         (bmalloc::Allocator::allocateFastCase):
700         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
701         function for size class calculation.
702
703         * bmalloc/Deallocator.cpp:
704         (bmalloc::Deallocator::Deallocator):
705         (bmalloc::Deallocator::scavenge):
706         (bmalloc::Deallocator::deallocateSmallLine):
707         (bmalloc::Deallocator::allocateSmallLine):
708         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
709         cache can't be shared anymore.
710
711         * bmalloc/Heap.cpp:
712         (bmalloc::Heap::allocateSmallLineSlowCase):
713         * bmalloc/Heap.h:
714         (bmalloc::Heap::deallocateSmallLine): Ditto.
715
716         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
717         page refcount when allocating a line because we might have deallocated
718         the page and the recycled it for another size class.
719
720         (bmalloc::Heap::deallocateMediumLine):
721         (bmalloc::Heap::allocateMediumLine):
722         * bmalloc/Line.h:
723         (bmalloc::Line::refCount):
724         * bmalloc/Page.h:
725         (bmalloc::Page::refCount):
726         (bmalloc::Page::smallSizeClass):
727         (bmalloc::Page::setSmallSizeClass):
728         (bmalloc::Page<Traits>::refCount): Deleted.
729         * bmalloc/Sizes.h:
730         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
731         an index into an array from a size.
732
733 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
734
735         bmalloc: Improved alignment in LargeChunk
736         https://bugs.webkit.org/show_bug.cgi?id=131895
737
738         Reviewed by Andreas Kling.
739
740         * bmalloc/Chunk.h:
741         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
742         Technically, the previous alignment was harmless, but I would prefer,
743         dear reader, not to have to explain the interlocking set of
744         circumstances that made it so.
745
746 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
747
748         Rolled out r167502 because it caused a crash on the facebook benchmark.
749
750         Unreviewed.
751
752             bmalloc: Added an XSmall line size
753             https://bugs.webkit.org/show_bug.cgi?id=131851
754
755             Reviewed by Sam Weinig.
756
757 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
758
759         bmalloc: Mutex should be harder to use wrong
760         https://bugs.webkit.org/show_bug.cgi?id=131879
761
762         Reviewed by Andreas Kling.
763
764         Mutex now has a proper constructor, so you can't deadlock by forgetting
765         to initialize it.
766
767         * bmalloc.xcodeproj/project.pbxproj:
768         * bmalloc/Allocator.cpp:
769         (bmalloc::Allocator::processXSmallAllocatorLog):
770         (bmalloc::Allocator::processSmallAllocatorLog):
771         (bmalloc::Allocator::processMediumAllocatorLog):
772         (bmalloc::Allocator::allocateLarge):
773         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
774         since the Heap mutex is a static.
775
776         * bmalloc/AsyncTask.h:
777         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
778         need for explicit initialization anymore.
779
780         * bmalloc/Deallocator.cpp:
781         (bmalloc::Deallocator::scavenge):
782         (bmalloc::Deallocator::deallocateLarge):
783         (bmalloc::Deallocator::deallocateXLarge):
784         (bmalloc::Deallocator::processObjectLog):
785         (bmalloc::Deallocator::deallocateSmallLine):
786         (bmalloc::Deallocator::deallocateXSmallLine):
787         (bmalloc::Deallocator::allocateSmallLine):
788         (bmalloc::Deallocator::allocateXSmallLine):
789         (bmalloc::Deallocator::deallocateMediumLine):
790         (bmalloc::Deallocator::allocateMediumLine):
791         * bmalloc/Deallocator.h:
792         * bmalloc/Heap.cpp:
793         (bmalloc::sleep):
794         (bmalloc::Heap::Heap):
795         (bmalloc::Heap::concurrentScavenge):
796         (bmalloc::Heap::scavenge):
797         (bmalloc::Heap::scavengeSmallPages):
798         (bmalloc::Heap::scavengeXSmallPages):
799         (bmalloc::Heap::scavengeMediumPages):
800         (bmalloc::Heap::scavengeLargeRanges):
801         (bmalloc::Heap::allocateXSmallLineSlowCase):
802         (bmalloc::Heap::allocateSmallLineSlowCase):
803         (bmalloc::Heap::allocateMediumLineSlowCase):
804         (bmalloc::Heap::allocateXLarge):
805         (bmalloc::Heap::deallocateXLarge):
806         (bmalloc::Heap::allocateLarge):
807         (bmalloc::Heap::deallocateLarge):
808         * bmalloc/Heap.h:
809         (bmalloc::Heap::deallocateXSmallLine):
810         (bmalloc::Heap::allocateXSmallLine):
811         (bmalloc::Heap::deallocateSmallLine):
812         (bmalloc::Heap::allocateSmallLine):
813         (bmalloc::Heap::deallocateMediumLine):
814         (bmalloc::Heap::allocateMediumLine):
815         * bmalloc/Line.h:
816         (bmalloc::Line<Traits>::deref):
817         * bmalloc/Mutex.cpp: Removed.
818         * bmalloc/Mutex.h:
819         (bmalloc::Mutex::Mutex):
820         (bmalloc::Mutex::init): Deleted.
821         (bmalloc::Mutex::try_lock): Deleted.
822         (bmalloc::Mutex::lock): Deleted.
823         (bmalloc::Mutex::unlock): Deleted.
824         * bmalloc/Page.h:
825         (bmalloc::Page<Traits>::ref):
826         (bmalloc::Page<Traits>::deref):
827         (bmalloc::Page<Traits>::refCount):
828         * bmalloc/PerProcess.h:
829         (bmalloc::PerProcess::mutex):
830         (bmalloc::PerProcess<T>::getSlowCase):
831         * bmalloc/StaticMutex.cpp: Added.
832         (bmalloc::StaticMutex::lockSlowCase):
833         * bmalloc/StaticMutex.h: Added.
834         (bmalloc::StaticMutex::init):
835         (bmalloc::StaticMutex::try_lock):
836         (bmalloc::StaticMutex::lock):
837         (bmalloc::StaticMutex::unlock):
838         * bmalloc/VMHeap.h:
839         (bmalloc::VMHeap::deallocateXSmallPage):
840         (bmalloc::VMHeap::deallocateSmallPage):
841         (bmalloc::VMHeap::deallocateMediumPage):
842         (bmalloc::VMHeap::deallocateLargeRange):
843         * bmalloc/bmalloc.h:
844         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
845         since the Heap mutex is a static.
846
847 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
848
849         bmalloc: AsyncTask should use Mutex instead of std::mutex
850         https://bugs.webkit.org/show_bug.cgi?id=131865
851
852         Reviewed by Gavin Barraclough.
853
854         std::mutex is so slow that it makes parallelizing simple tasks through
855         AsyncTask a net regression. Mutex fixes this.
856
857         * bmalloc/AsyncTask.h:
858         (bmalloc::Function>::AsyncTask):
859         (bmalloc::Function>::join):
860         (bmalloc::Function>::runSlowCase):
861         (bmalloc::Function>::entryPoint):
862         * bmalloc/Mutex.h:
863         (bmalloc::Mutex::init):
864
865 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
866
867         bmalloc: Added an XSmall line size
868         https://bugs.webkit.org/show_bug.cgi?id=131851
869
870         Reviewed by Sam Weinig.
871
872         Reduces malloc footprint on Membuster recordings by 10%.
873
874         This is a throughput regression, but we're still way ahead of TCMalloc.
875         I have some ideas for how to recover the regression -- but I wanted to
876         get this win in first.
877
878         Full set of benchmark results:
879
880                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
881                                                                                                 
882                                                                nopatch                      patch                                Δ
883                 Peak Memory:
884                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
885                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
886                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
887
888                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
889                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
890                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
891
892                 Memory at End:
893                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
894                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
895                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
896
897                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
898                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
899                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
900
901                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
902                                                                                                 
903                                                    nopatch                patch                         Δ
904                 Execution Time:
905                     churn                            127ms                151ms            ! 1.19x slower
906                     list_allocate                    130ms                164ms            ! 1.26x slower
907                     tree_allocate                    109ms                127ms            ! 1.17x slower
908                     tree_churn                       115ms                120ms            ! 1.04x slower
909                     facebook                         240ms                259ms            ! 1.08x slower
910                     fragment                          91ms                131ms            ! 1.44x slower
911                     fragment_iterate                 105ms                106ms            ! 1.01x slower
912                     message_one                      260ms                259ms             ^ 1.0x faster
913                     message_many                     149ms                154ms            ! 1.03x slower
914                     medium                           194ms                248ms            ! 1.28x slower
915                     big                              157ms                160ms            ! 1.02x slower
916
917                     <geometric mean>                 144ms                163ms            ! 1.13x slower
918                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
919                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
920
921                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
922                                                                                                 
923                                                                        nopatch                          patch                                     Δ
924                 Execution Time:
925                     churn                                                126ms                          148ms                        ! 1.17x slower
926                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
927                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
928                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
929                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
930                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
931                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
932                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
933                     facebook                                             240ms                          276ms                        ! 1.15x slower
934                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
935                     fragment                                              92ms                          130ms                        ! 1.41x slower
936                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
937                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
938                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
939                     message_one                                          260ms                          260ms                                      
940                     message_many                                         170ms                          238ms                         ! 1.4x slower
941                     medium                                               185ms                          250ms                        ! 1.35x slower
942                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
943                     big                                                  150ms                          169ms                        ! 1.13x slower
944                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
945
946                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
947                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
948                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
949
950                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
951
952                                                                     TC                      patch                                Δ
953                 Peak Memory:
954                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
955                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
956                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
957
958                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
959                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
960                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
961
962                 Memory at End:
963                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
964                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
965                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
966
967                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
968                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
969                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
970
971                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
972                                                                                                 
973                                                         TC                patch                         Δ
974                 Execution Time:
975                     churn                            416ms                148ms            ^ 2.81x faster
976                     list_allocate                    463ms                164ms            ^ 2.82x faster
977                     tree_allocate                    292ms                127ms             ^ 2.3x faster
978                     tree_churn                       157ms                120ms            ^ 1.31x faster
979                     facebook                         327ms                276ms            ^ 1.18x faster
980                     fragment                         335ms                129ms             ^ 2.6x faster
981                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
982                     message_one                      386ms                258ms             ^ 1.5x faster
983                     message_many                     410ms                154ms            ^ 2.66x faster
984                     medium                           391ms                245ms             ^ 1.6x faster
985                     big                              261ms                167ms            ^ 1.56x faster
986
987                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
988                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
989                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
990
991         * bmalloc.xcodeproj/project.pbxproj:
992         * bmalloc/Allocator.cpp:
993         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
994         index corresponds with its size. Instead, use the size selection function
995         explicitly. Now that we have XSmall, some small allocator entries are
996         unused.
997
998         (bmalloc::Allocator::scavenge):
999         (bmalloc::Allocator::log):
1000         (bmalloc::Allocator::processXSmallAllocatorLog):
1001         (bmalloc::Allocator::allocateSlowCase):
1002         * bmalloc/Allocator.h:
1003         (bmalloc::Allocator::xSmallAllocatorFor):
1004         (bmalloc::Allocator::allocateFastCase):
1005         * bmalloc/Chunk.h:
1006         * bmalloc/Deallocator.cpp:
1007         (bmalloc::Deallocator::scavenge):
1008         (bmalloc::Deallocator::processObjectLog):
1009         (bmalloc::Deallocator::deallocateSlowCase):
1010         (bmalloc::Deallocator::deallocateXSmallLine):
1011         (bmalloc::Deallocator::allocateXSmallLine):
1012         * bmalloc/Deallocator.h:
1013         (bmalloc::Deallocator::deallocateFastCase):
1014         * bmalloc/Heap.cpp:
1015         (bmalloc::Heap::scavenge):
1016         (bmalloc::Heap::scavengeXSmallPages):
1017         (bmalloc::Heap::allocateXSmallLineSlowCase):
1018         * bmalloc/Heap.h:
1019         (bmalloc::Heap::deallocateXSmallLine):
1020         (bmalloc::Heap::allocateXSmallLine):
1021         * bmalloc/LargeChunk.h:
1022         (bmalloc::LargeChunk::get):
1023         (bmalloc::LargeChunk::endTag):
1024         * bmalloc/Line.h:
1025         * bmalloc/MediumAllocator.h:
1026         (bmalloc::MediumAllocator::allocate):
1027         (bmalloc::MediumAllocator::refill):
1028         * bmalloc/ObjectType.cpp:
1029         (bmalloc::objectType):
1030         * bmalloc/ObjectType.h:
1031         (bmalloc::isXSmall):
1032         (bmalloc::isSmall):
1033         (bmalloc::isMedium):
1034         (bmalloc::isLarge):
1035         (bmalloc::isSmallOrMedium): Deleted.
1036         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1037         handling small objects. There's probably a reasonable way to share this
1038         code in the future -- I'll look into that once it's stopped changing.
1039
1040         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1041         is the main reason things got slower.
1042
1043         * bmalloc/SmallAllocator.h:
1044         (bmalloc::SmallAllocator::allocate):
1045         * bmalloc/SmallTraits.h:
1046         * bmalloc/VMHeap.cpp:
1047         (bmalloc::VMHeap::allocateXSmallChunk):
1048         * bmalloc/VMHeap.h:
1049         (bmalloc::VMHeap::allocateXSmallPage):
1050         (bmalloc::VMHeap::deallocateXSmallPage):
1051         * bmalloc/XSmallAllocator.h: Added.
1052         (bmalloc::XSmallAllocator::isNull):
1053         (bmalloc::XSmallAllocator::canAllocate):
1054         (bmalloc::XSmallAllocator::XSmallAllocator):
1055         (bmalloc::XSmallAllocator::line):
1056         (bmalloc::XSmallAllocator::allocate):
1057         (bmalloc::XSmallAllocator::objectCount):
1058         (bmalloc::XSmallAllocator::derefCount):
1059         (bmalloc::XSmallAllocator::refill):
1060         * bmalloc/XSmallChunk.h: Added.
1061         * bmalloc/XSmallLine.h: Added.
1062         * bmalloc/XSmallPage.h: Added.
1063         * bmalloc/XSmallTraits.h: Added.
1064         * bmalloc/bmalloc.h:
1065         (bmalloc::api::realloc): Boiler-plate copy, as above.
1066
1067 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1068
1069         MallocBench should scavenge explicitly instead of waiting
1070         https://bugs.webkit.org/show_bug.cgi?id=131661
1071
1072         Reviewed by Andreas Kling.
1073
1074         Added explicit scavenge support to bmalloc. This isn't a memory win,
1075         since bmalloc's per-thread cache is so small. But it makes testing
1076         simpler.
1077
1078         * bmalloc/Allocator.cpp:
1079         (bmalloc::Allocator::~Allocator):
1080         (bmalloc::Allocator::scavenge):
1081         * bmalloc/Allocator.h:
1082         * bmalloc/Cache.cpp:
1083         (bmalloc::Cache::operator new):
1084         (bmalloc::Cache::operator delete):
1085         (bmalloc::Cache::Cache):
1086         (bmalloc::Cache::scavenge):
1087         * bmalloc/Cache.h:
1088         * bmalloc/Deallocator.cpp:
1089         (bmalloc::Deallocator::~Deallocator):
1090         (bmalloc::Deallocator::scavenge):
1091         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1092         functions, for reuse.
1093
1094         * bmalloc/Heap.cpp:
1095         (bmalloc::sleep):
1096         (bmalloc::Heap::concurrentScavenge):
1097         (bmalloc::Heap::scavenge):
1098         (bmalloc::Heap::scavengeSmallPages):
1099         (bmalloc::Heap::scavengeMediumPages):
1100         (bmalloc::Heap::scavengeLargeRanges):
1101         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1102         scavenging -- in response to a benchmark or a low memory warning --
1103         wants to complete as soon as possible, so its sleep duration is 0.
1104
1105         * bmalloc/bmalloc.h:
1106         (bmalloc::api::scavenge):
1107         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1108
1109 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1110
1111         Use 4kB pages on Mac
1112         https://bugs.webkit.org/show_bug.cgi?id=131658
1113
1114         Reviewed by Sam Weinig.
1115
1116         This reduces memory use a lot on Membuster:
1117
1118                                                                   base                      patch                                Δ
1119                 Execution Time:
1120                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1121                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1122                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1123
1124                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1125                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1126                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1127
1128                 Peak Memory:
1129                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1130                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1131                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1132
1133                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1134                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1135                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1136
1137                 Memory at End:
1138                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1139                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1140                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1141
1142                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1143                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1144                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1145
1146
1147         * bmalloc.xcodeproj/project.pbxproj:
1148         * bmalloc/BPlatform.h: Added.
1149         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1150         is 4kB on Mac.
1151
1152 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1153
1154         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1155
1156         * bmalloc.xcodeproj: Modified property svn:ignore.
1157
1158 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1159
1160         Fixed some mbmalloc exports
1161         https://bugs.webkit.org/show_bug.cgi?id=131599
1162
1163         Reviewed by Ryosuke Niwa.
1164
1165         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1166         of the project, so we can call them from API.
1167
1168         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1169         visibility, so they show up as exported in the .dylib.
1170
1171 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1172
1173         Put bmalloc headers in the right place
1174         https://bugs.webkit.org/show_bug.cgi?id=131464
1175
1176         Reviewed by Mark Rowe.
1177
1178         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1179         specify that we don't just want to dump all of our generically-named
1180         headers into /usr/local/include.
1181
1182 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1183
1184         Made bmalloc more #include friendly
1185         https://bugs.webkit.org/show_bug.cgi?id=131386
1186
1187         Reviewed by Andreas Kling.
1188
1189         Marked a bunch of headers private so they can be used from client code
1190         that #includes bmalloc.h.
1191
1192         Renamed ASSERT macros to BASSERT. This matches their header, which already
1193         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1194
1195         * bmalloc.xcodeproj/project.pbxproj:
1196         * bmalloc/Allocator.cpp:
1197         (bmalloc::Allocator::allocateSlowCase):
1198         * bmalloc/AsyncTask.h:
1199         (bmalloc::Function>::runSlowCase):
1200         * bmalloc/BAssert.h:
1201         * bmalloc/BoundaryTag.h:
1202         (bmalloc::BoundaryTag::setSize):
1203         * bmalloc/BoundaryTagInlines.h:
1204         (bmalloc::validate):
1205         (bmalloc::BoundaryTag::init):
1206         (bmalloc::BoundaryTag::deallocate):
1207         (bmalloc::BoundaryTag::splitLarge):
1208         (bmalloc::BoundaryTag::allocate):
1209         * bmalloc/Chunk.h:
1210         * bmalloc/Deallocator.cpp:
1211         (bmalloc::Deallocator::processObjectLog):
1212         (bmalloc::Deallocator::deallocateSlowCase):
1213         * bmalloc/Deallocator.h:
1214         (bmalloc::Deallocator::deallocateFastCase):
1215         * bmalloc/FixedVector.h:
1216         (bmalloc::Capacity>::operator):
1217         (bmalloc::Capacity>::push):
1218         (bmalloc::Capacity>::pop):
1219         (bmalloc::Capacity>::shrink):
1220         * bmalloc/Heap.cpp:
1221         (bmalloc::Heap::allocateLarge):
1222         * bmalloc/LargeChunk.h:
1223         (bmalloc::LargeChunk::get):
1224         (bmalloc::LargeChunk::endTag):
1225         * bmalloc/Line.h:
1226         (bmalloc::Line<Traits>::concurrentRef):
1227         (bmalloc::Line<Traits>::deref):
1228         * bmalloc/MediumAllocator.h:
1229         (bmalloc::MediumAllocator::allocate):
1230         * bmalloc/ObjectType.h:
1231         (bmalloc::isSmall):
1232         * bmalloc/Page.h:
1233         (bmalloc::Page<Traits>::ref):
1234         (bmalloc::Page<Traits>::deref):
1235         * bmalloc/PerThread.h:
1236         (bmalloc::PerThread<T>::getSlowCase):
1237         * bmalloc/SegregatedFreeList.cpp:
1238         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1239         (bmalloc::SegregatedFreeList::insert):
1240         * bmalloc/SmallAllocator.h:
1241         (bmalloc::SmallAllocator::allocate):
1242         (bmalloc::SmallAllocator::refill):
1243         * bmalloc/Syscall.h:
1244         * bmalloc/VMAllocate.h:
1245         (bmalloc::vmValidate):
1246         (bmalloc::vmAllocate):
1247         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1248         * bmalloc/Vector.h:
1249         (bmalloc::Vector<T>::operator):
1250         (bmalloc::Vector<T>::pop):
1251         (bmalloc::Vector<T>::shrink):
1252         * bmalloc/XLargeChunk.h:
1253         (bmalloc::XLargeChunk::range):
1254         (bmalloc::XLargeChunk::size):
1255
1256 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1257
1258         Removed an unused file.
1259
1260         Unreviewed.
1261
1262         * bmalloc/AsyncTask.cpp: Removed.
1263
1264 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1265
1266         Build bmalloc on Mac
1267         https://bugs.webkit.org/show_bug.cgi?id=131333
1268
1269         Reviewed by Mark Rowe.
1270
1271         * Makefile: Added. For make clients.
1272
1273         These files are required for building any project in WebKit. I copied
1274         them from WTF:
1275         * Configurations: Added.
1276         * Configurations/Base.xcconfig: Added.
1277         * Configurations/DebugRelease.xcconfig: Added.
1278         * Configurations/bmalloc.xcconfig: Added.
1279         * Configurations/iOS.xcconfig: Added.
1280         * Configurations/mbmalloc.xcconfig: Added.
1281
1282         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1283         from here because everything is in .xcconfig files now.
1284
1285         I had to fix a bunch of minor warnings, since they're enabled in our
1286         .xcconfig files:
1287
1288         * bmalloc/AsyncTask.h:
1289         (bmalloc::Function>::AsyncTask):
1290         * bmalloc/BAssert.h:
1291         * bmalloc/BoundaryTagInlines.h:
1292         (bmalloc::validate):
1293         * bmalloc/Heap.cpp:
1294         (bmalloc::Heap::Heap):
1295         (bmalloc::Heap::allocateLarge):
1296         (bmalloc::Heap::deallocateLarge):
1297         * bmalloc/Mutex.h:
1298         (bmalloc::Mutex::Mutex): Deleted.
1299         * bmalloc/VMAllocate.h:
1300         (bmalloc::vmValidate):
1301         * bmalloc/mbmalloc.cpp:
1302
1303 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1304
1305         bmalloc: Fixed a leak in the per-thread cache
1306         https://bugs.webkit.org/show_bug.cgi?id=131330
1307
1308         Reviewed by Andreas Kling.
1309
1310         Remember to deallocate our line caches upon thread exit.
1311
1312         * bmalloc/Deallocator.cpp:
1313         (bmalloc::Deallocator::~Deallocator):
1314
1315 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1316
1317         bmalloc: rolled out the tryLock experiment
1318         https://bugs.webkit.org/show_bug.cgi?id=131328
1319
1320         Reviewed by Andreas Kling.
1321
1322         It wasn't a speedup.
1323
1324         * bmalloc.xcodeproj/project.pbxproj:
1325         * bmalloc/Allocator.cpp:
1326         (bmalloc::Allocator::processSmallAllocatorLog):
1327         (bmalloc::Allocator::processMediumAllocatorLog):
1328         * bmalloc/Deallocator.cpp:
1329         (bmalloc::Deallocator::processObjectLog):
1330         (bmalloc::Deallocator::deallocateSlowCase):
1331         (bmalloc::Deallocator::deallocateSmallLine):
1332         (bmalloc::Deallocator::deallocateMediumLine):
1333         * bmalloc/Deallocator.h:
1334         (bmalloc::Deallocator::deallocateFastCase):
1335         * bmalloc/Heap.h:
1336         (bmalloc::Heap::deallocateSmallLine):
1337         (bmalloc::Heap::deallocateMediumLine):
1338         * bmalloc/Line.h:
1339         (bmalloc::Line<Traits>::deref):
1340         * bmalloc/Page.h:
1341         (bmalloc::Page<Traits>::deref):
1342
1343 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1344
1345         bmalloc
1346         https://bugs.webkit.org/show_bug.cgi?id=131170
1347
1348         Reviewed by Andreas Kling.
1349
1350         Initial commit.
1351
1352         * bmalloc: Added.
1353         * bmalloc.xcodeproj: Added.
1354         * bmalloc.xcodeproj/project.pbxproj: Added.
1355         * bmalloc/Algorithm.h: Added.
1356         (bmalloc::max):
1357         (bmalloc::min):
1358         (bmalloc::mask):
1359         (bmalloc::test):
1360         (bmalloc::roundUpToMultipleOf):
1361         (bmalloc::roundDownToMultipleOf):
1362         (bmalloc::sizeOf):
1363         (bmalloc::bitCount):
1364         (bmalloc::isPowerOfTwo):
1365         * bmalloc/Allocator.cpp: Added.
1366         (bmalloc::Allocator::Allocator):
1367         (bmalloc::Allocator::~Allocator):
1368         (bmalloc::Allocator::log):
1369         (bmalloc::Allocator::processSmallAllocatorLog):
1370         (bmalloc::Allocator::processMediumAllocatorLog):
1371         (bmalloc::Allocator::allocateLarge):
1372         (bmalloc::Allocator::allocateXLarge):
1373         (bmalloc::Allocator::allocateMedium):
1374         (bmalloc::Allocator::allocateSlowCase):
1375         * bmalloc/Allocator.h: Added.
1376         (bmalloc::Allocator::smallAllocatorFor):
1377         (bmalloc::Allocator::allocateFastCase):
1378         (bmalloc::Allocator::allocate):
1379         * bmalloc/AsyncTask.cpp: Added.
1380         (bmalloc::AsyncTask<Function>::runSlowCase):
1381         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
1382         (bmalloc::AsyncTask<Function>::entryPoint):
1383         * bmalloc/AsyncTask.h: Added.
1384         (bmalloc::Function>::AsyncTask):
1385         (bmalloc::Function>::join):
1386         (bmalloc::Function>::run):
1387         (bmalloc::Function>::runSlowCase):
1388         (bmalloc::Function>::pthreadEntryPoint):
1389         (bmalloc::Function>::entryPoint):
1390         * bmalloc/BAssert.h: Added.
1391         * bmalloc/BeginTag.h: Added.
1392         (bmalloc::BeginTag::isInFreeList):
1393         * bmalloc/BoundaryTag.h: Added.
1394         (bmalloc::BoundaryTag::isXLarge):
1395         (bmalloc::BoundaryTag::setXLarge):
1396         (bmalloc::BoundaryTag::isFree):
1397         (bmalloc::BoundaryTag::setFree):
1398         (bmalloc::BoundaryTag::isEnd):
1399         (bmalloc::BoundaryTag::setEnd):
1400         (bmalloc::BoundaryTag::hasPhysicalPages):
1401         (bmalloc::BoundaryTag::setHasPhysicalPages):
1402         (bmalloc::BoundaryTag::isNull):
1403         (bmalloc::BoundaryTag::clear):
1404         (bmalloc::BoundaryTag::size):
1405         (bmalloc::BoundaryTag::setSize):
1406         (bmalloc::BoundaryTag::prev):
1407         (bmalloc::BoundaryTag::next):
1408         * bmalloc/BoundaryTagInlines.h: Added.
1409         (bmalloc::validate):
1410         (bmalloc::validatePrev):
1411         (bmalloc::validateNext):
1412         (bmalloc::BoundaryTag::init):
1413         (bmalloc::BoundaryTag::mergeLargeLeft):
1414         (bmalloc::BoundaryTag::mergeLargeRight):
1415         (bmalloc::BoundaryTag::mergeLarge):
1416         (bmalloc::BoundaryTag::deallocate):
1417         (bmalloc::BoundaryTag::splitLarge):
1418         (bmalloc::BoundaryTag::allocate):
1419         * bmalloc/Cache.cpp: Added.
1420         (bmalloc::Cache::operator new):
1421         (bmalloc::Cache::operator delete):
1422         (bmalloc::Cache::Cache):
1423         (bmalloc::Cache::allocateSlowCase):
1424         (bmalloc::Cache::allocateSlowCaseNullCache):
1425         (bmalloc::Cache::deallocateSlowCase):
1426         (bmalloc::Cache::deallocateSlowCaseNullCache):
1427         * bmalloc/Cache.h: Added.
1428         (bmalloc::Cache::allocator):
1429         (bmalloc::Cache::deallocator):
1430         (bmalloc::Cache::allocateFastCase):
1431         (bmalloc::Cache::deallocateFastCase):
1432         (bmalloc::Cache::allocate):
1433         (bmalloc::Cache::deallocate):
1434         * bmalloc/Chunk.h: Added.
1435         (bmalloc::Chunk::begin):
1436         (bmalloc::Chunk::end):
1437         (bmalloc::Chunk::lines):
1438         (bmalloc::Chunk::pages):
1439         * bmalloc/Deallocator.cpp: Added.
1440         (bmalloc::Deallocator::Deallocator):
1441         (bmalloc::Deallocator::~Deallocator):
1442         (bmalloc::Deallocator::deallocateLarge):
1443         (bmalloc::Deallocator::deallocateXLarge):
1444         (bmalloc::Deallocator::processObjectLog):
1445         (bmalloc::Deallocator::deallocateSlowCase):
1446         (bmalloc::Deallocator::deallocateSmallLine):
1447         (bmalloc::Deallocator::allocateSmallLine):
1448         (bmalloc::Deallocator::deallocateMediumLine):
1449         (bmalloc::Deallocator::allocateMediumLine):
1450         * bmalloc/Deallocator.h: Added.
1451         (bmalloc::Deallocator::deallocateFastCase):
1452         (bmalloc::Deallocator::deallocate):
1453         * bmalloc/EndTag.h: Added.
1454         (bmalloc::EndTag::operator=):
1455         * bmalloc/FixedVector.h: Added.
1456         (bmalloc::FixedVector::begin):
1457         (bmalloc::FixedVector::end):
1458         (bmalloc::FixedVector::size):
1459         (bmalloc::FixedVector::capacity):
1460         (bmalloc::FixedVector::clear):
1461         (bmalloc::FixedVector::isEmpty):
1462         (bmalloc::Capacity>::FixedVector):
1463         (bmalloc::Capacity>::operator):
1464         (bmalloc::Capacity>::push):
1465         (bmalloc::Capacity>::pop):
1466         (bmalloc::Capacity>::shrink):
1467         * bmalloc/Heap.cpp: Added.
1468         (bmalloc::sleep):
1469         (bmalloc::Heap::Heap):
1470         (bmalloc::Heap::concurrentScavenge):
1471         (bmalloc::Heap::scavengeSmallPages):
1472         (bmalloc::Heap::scavengeMediumPages):
1473         (bmalloc::Heap::scavengeLargeRanges):
1474         (bmalloc::Heap::allocateSmallLineSlowCase):
1475         (bmalloc::Heap::allocateMediumLineSlowCase):
1476         (bmalloc::Heap::allocateXLarge):
1477         (bmalloc::Heap::deallocateXLarge):
1478         (bmalloc::Heap::allocateLarge):
1479         (bmalloc::Heap::deallocateLarge):
1480         * bmalloc/Heap.h: Added.
1481         (bmalloc::Heap::deallocateSmallLine):
1482         (bmalloc::Heap::allocateSmallLine):
1483         (bmalloc::Heap::deallocateMediumLine):
1484         (bmalloc::Heap::allocateMediumLine):
1485         * bmalloc/Inline.h: Added.
1486         * bmalloc/LargeChunk.h: Added.
1487         (bmalloc::LargeChunk::begin):
1488         (bmalloc::LargeChunk::end):
1489         (bmalloc::LargeChunk::create):
1490         (bmalloc::LargeChunk::get):
1491         (bmalloc::LargeChunk::beginTag):
1492         (bmalloc::LargeChunk::endTag):
1493         * bmalloc/Line.h: Added.
1494         (bmalloc::Line<Traits>::begin):
1495         (bmalloc::Line<Traits>::end):
1496         (bmalloc::Line<Traits>::concurrentRef):
1497         (bmalloc::Line<Traits>::deref):
1498         * bmalloc/MediumAllocator.h: Added.
1499         (bmalloc::MediumAllocator::isNull):
1500         (bmalloc::MediumAllocator::MediumAllocator):
1501         (bmalloc::MediumAllocator::line):
1502         (bmalloc::MediumAllocator::allocate):
1503         (bmalloc::MediumAllocator::derefCount):
1504         (bmalloc::MediumAllocator::refill):
1505         * bmalloc/MediumChunk.h: Added.
1506         * bmalloc/MediumLine.h: Added.
1507         * bmalloc/MediumPage.h: Added.
1508         * bmalloc/MediumTraits.h: Added.
1509         * bmalloc/Mutex.cpp: Added.
1510         (bmalloc::Mutex::lockSlowCase):
1511         * bmalloc/Mutex.h: Added.
1512         (bmalloc::Mutex::Mutex):
1513         (bmalloc::Mutex::try_lock):
1514         (bmalloc::Mutex::lock):
1515         (bmalloc::Mutex::unlock):
1516         * bmalloc/ObjectType.cpp: Added.
1517         (bmalloc::objectType):
1518         * bmalloc/ObjectType.h: Added.
1519         (bmalloc::isSmallOrMedium):
1520         (bmalloc::isSmall):
1521         * bmalloc/Page.h: Added.
1522         (bmalloc::Page<Traits>::ref):
1523         (bmalloc::Page<Traits>::deref):
1524         (bmalloc::Page<Traits>::refCount):
1525         * bmalloc/PerProcess.h: Added.
1526         (bmalloc::PerProcess::mutex):
1527         (bmalloc::PerProcess<T>::getFastCase):
1528         (bmalloc::PerProcess<T>::get):
1529         (bmalloc::PerProcess<T>::getSlowCase):
1530         * bmalloc/PerThread.h: Added.
1531         (bmalloc::PerThreadStorage<Cache>::get):
1532         (bmalloc::PerThreadStorage<Cache>::init):
1533         (bmalloc::PerThreadStorage::get):
1534         (bmalloc::PerThreadStorage::init):
1535         (bmalloc::PerThread<T>::getFastCase):
1536         (bmalloc::PerThread<T>::get):
1537         (bmalloc::PerThread<T>::destructor):
1538         (bmalloc::PerThread<T>::getSlowCase):
1539         * bmalloc/Range.h: Added.
1540         (bmalloc::Range::Range):
1541         (bmalloc::Range::begin):
1542         (bmalloc::Range::end):
1543         (bmalloc::Range::size):
1544         (bmalloc::Range::operator!):
1545         (bmalloc::Range::operator<):
1546         * bmalloc/SegregatedFreeList.cpp: Added.
1547         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1548         (bmalloc::SegregatedFreeList::insert):
1549         (bmalloc::SegregatedFreeList::takeGreedy):
1550         (bmalloc::SegregatedFreeList::take):
1551         * bmalloc/SegregatedFreeList.h: Added.
1552         * bmalloc/Sizes.h: Added.
1553         * bmalloc/SmallAllocator.h: Added.
1554         (bmalloc::SmallAllocator::isNull):
1555         (bmalloc::SmallAllocator::canAllocate):
1556         (bmalloc::SmallAllocator::SmallAllocator):
1557         (bmalloc::SmallAllocator::line):
1558         (bmalloc::SmallAllocator::allocate):
1559         (bmalloc::SmallAllocator::objectCount):
1560         (bmalloc::SmallAllocator::derefCount):
1561         (bmalloc::SmallAllocator::refill):
1562         * bmalloc/SmallChunk.h: Added.
1563         * bmalloc/SmallLine.h: Added.
1564         * bmalloc/SmallPage.h: Added.
1565         * bmalloc/SmallTraits.h: Added.
1566         * bmalloc/Syscall.h: Added.
1567         * bmalloc/VMAllocate.h: Added.
1568         (bmalloc::vmSize):
1569         (bmalloc::vmValidate):
1570         (bmalloc::vmAllocate):
1571         (bmalloc::vmDeallocate):
1572         (bmalloc::vmDeallocatePhysicalPages):
1573         (bmalloc::vmAllocatePhysicalPages):
1574         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1575         (bmalloc::vmAllocatePhysicalPagesSloppy):
1576         * bmalloc/VMHeap.cpp: Added.
1577         (bmalloc::VMHeap::VMHeap):
1578         (bmalloc::VMHeap::allocateSmallChunk):
1579         (bmalloc::VMHeap::allocateMediumChunk):
1580         (bmalloc::VMHeap::allocateLargeChunk):
1581         * bmalloc/VMHeap.h: Added.
1582         (bmalloc::VMHeap::allocateSmallPage):
1583         (bmalloc::VMHeap::allocateMediumPage):
1584         (bmalloc::VMHeap::allocateLargeRange):
1585         (bmalloc::VMHeap::deallocateSmallPage):
1586         (bmalloc::VMHeap::deallocateMediumPage):
1587         (bmalloc::VMHeap::deallocateLargeRange):
1588         * bmalloc/Vector.h: Added.
1589         (bmalloc::Vector::begin):
1590         (bmalloc::Vector::end):
1591         (bmalloc::Vector::size):
1592         (bmalloc::Vector::capacity):
1593         (bmalloc::Vector::last):
1594         (bmalloc::Vector::pop):
1595         (bmalloc::Vector<T>::Vector):
1596         (bmalloc::Vector<T>::~Vector):
1597         (bmalloc::Vector<T>::operator):
1598         (bmalloc::Vector<T>::push):
1599         (bmalloc::Vector<T>::pop):
1600         (bmalloc::Vector<T>::shrink):
1601         (bmalloc::Vector<T>::reallocateBuffer):
1602         (bmalloc::Vector<T>::shrinkCapacity):
1603         (bmalloc::Vector<T>::growCapacity):
1604         * bmalloc/XLargeChunk.h: Added.
1605         (bmalloc::XLargeChunk::get):
1606         (bmalloc::XLargeChunk::begin):
1607         (bmalloc::XLargeChunk::XLargeChunk):
1608         (bmalloc::XLargeChunk::create):
1609         (bmalloc::XLargeChunk::destroy):
1610         (bmalloc::XLargeChunk::range):
1611         (bmalloc::XLargeChunk::size):
1612         * bmalloc/bmalloc.h: Added.
1613         (bmalloc::api::malloc):
1614         (bmalloc::api::free):
1615         (bmalloc::api::realloc):
1616         * bmalloc/mbmalloc.cpp: Added.
1617