bmalloc: allocate small and medium objects using the same bump pointer class
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc: allocate small and medium objects using the same bump pointer class
4         https://bugs.webkit.org/show_bug.cgi?id=136843
5
6         Reviewed by Gavin Barraclough.
7
8         4% speedup on MallocBench.
9
10         Now that medium-sized objects have dedicated per-size allocators, they
11         don't need to use an arbitrary bump pointer allocator. This means that
12         every allocator knows how many objects it will allocate from the start,
13         and we don't need a post-processing step to adjust refcounts based on
14         real allocation count.
15
16         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
17         since it's used for small and medium objects now.
18
19         * bmalloc/Allocator.cpp:
20         (bmalloc::Allocator::Allocator): Updated to use new interface.
21         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
22         to make sure that we finish allocating all the objects in it.
23
24         (bmalloc::Allocator::allocateMedium):
25         (bmalloc::Allocator::allocateSlowCase):
26         (bmalloc::Allocator::retire): Deleted.
27         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
28         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
29         * bmalloc/Allocator.h:
30         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
31         used to post-process an allocator based on how many objects it allocated.
32
33         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
34         (bmalloc::BumpAllocator::BumpAllocator):
35         (bmalloc::BumpAllocator::init):
36         (bmalloc::BumpAllocator::line):
37         (bmalloc::BumpAllocator::validate):
38         (bmalloc::BumpAllocator::allocate):
39         (bmalloc::BumpAllocator::refill):
40         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
41         about the kinds of lines they allocate into. In some cases, the line
42         type must be provided as a template parameter by the caller.
43
44         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
45         (bmalloc::SmallAllocator::line): Deleted.
46         (bmalloc::SmallAllocator::allocate): Deleted.
47         (bmalloc::SmallAllocator::objectCount): Deleted.
48         (bmalloc::SmallAllocator::derefCount): Deleted.
49         (bmalloc::SmallAllocator::refill): Deleted.
50         (bmalloc::SmallAllocator::clear): Deleted.
51
52         * bmalloc/ObjectType.h:
53         (bmalloc::isMedium):
54
55         * bmalloc/SmallAllocator.h:
56         (bmalloc::SmallAllocator::isNull): Deleted.
57         (bmalloc::SmallAllocator::canAllocate): Deleted.
58         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
59         (bmalloc::SmallAllocator::line): Deleted.
60         (bmalloc::SmallAllocator::allocate): Deleted.
61         (bmalloc::SmallAllocator::objectCount): Deleted.
62         (bmalloc::SmallAllocator::derefCount): Deleted.
63         (bmalloc::SmallAllocator::refill): Deleted.
64         (bmalloc::SmallAllocator::clear): Deleted.
65
66 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
67
68         Fixed a goof in bmalloc Vector sizing
69         https://bugs.webkit.org/show_bug.cgi?id=136795
70
71         Reviewed by Gavin Barraclough and Sam Weinig.
72
73         We want our minimum vector to be page-sized since the OS will give us
74         a page no matter what -- but we want that many bytes, and not enough
75         bytes to store that many elements.
76
77         * bmalloc/Vector.h: Math is hard.
78
79 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
80
81         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
82         https://bugs.webkit.org/show_bug.cgi?id=136693
83
84         Reviewed by Gavin Barraclough.
85
86         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
87
88         No throughput change.
89
90         We keep an array of medium allocators, just like our array of small
91         allocators.
92
93         In future, we can simplify the allocation fast path by merging the small
94         and medium allocator arrays. For now, this is the simplest change that
95         gets the win.
96
97         * bmalloc/Allocator.cpp:
98         (bmalloc::Allocator::Allocator):
99         (bmalloc::Allocator::scavenge):
100         (bmalloc::Allocator::allocateMedium):
101         * bmalloc/Allocator.h:
102         * bmalloc/Sizes.h:
103         (bmalloc::Sizes::mediumSizeClassFor):
104
105 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
106
107         Reviewed by Sam Weinig.
108
109         Renamed log => retire for clarity.
110
111         * bmalloc/Allocator.cpp:
112         (bmalloc::Allocator::scavenge):
113         (bmalloc::Allocator::retire):
114         (bmalloc::Allocator::allocateMedium):
115         (bmalloc::Allocator::allocateSlowCase):
116         (bmalloc::Allocator::log): Deleted.
117         * bmalloc/Allocator.h:
118
119 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
120
121         bmalloc: eager scavenge leaves behind a bogus allocator
122         https://bugs.webkit.org/show_bug.cgi?id=136743
123
124         Reviewed by Sam Weinig.
125
126         Be sure to clear the allocator after logging it in the eager scavenge
127         case, so that we don't later try to allocate out of the lines that we
128         have thrown away.
129
130         We didn't need to do this previously because scavenge would only happen
131         at thread exit time, after which no further allocation from the per-thread
132         cache would take place.
133
134         * bmalloc/Allocator.cpp:
135         (bmalloc::Allocator::scavenge):
136         * bmalloc/MediumAllocator.h:
137         (bmalloc::MediumAllocator::clear):
138         * bmalloc/SmallAllocator.h:
139         (bmalloc::SmallAllocator::clear):
140
141 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
142
143         bmalloc should honor the FastMalloc statistics API
144         https://bugs.webkit.org/show_bug.cgi?id=136592
145
146         Reviewed by Gavin Barraclough.
147
148         We do this by tracking "size" and "capacity" in the VM heap.
149
150         The VM heap's "capacity" is all the VM we ever allocated.
151
152         The VM heap's "size" the subset of VM currently held onto by the
153         VM heap (and therefore not in use by the regular heap).
154
155         Somewhat ironically, reducing the process's memory footprint, increases
156         the size of the VM heap, since the VM heap holds the pages that are
157         purely virtual and not physical.
158
159         * bmalloc/Heap.cpp:
160         (bmalloc::Heap::size):
161         (bmalloc::Heap::capacity):
162         * bmalloc/Heap.h:
163         * bmalloc/VMHeap.cpp:
164         (bmalloc::VMHeap::VMHeap):
165         (bmalloc::VMHeap::allocateSmallChunk):
166         (bmalloc::VMHeap::allocateMediumChunk):
167         (bmalloc::VMHeap::allocateLargeChunk):
168         * bmalloc/VMHeap.h:
169         (bmalloc::VMHeap::size):
170         (bmalloc::VMHeap::capacity):
171         (bmalloc::VMHeap::allocateSmallPage):
172         (bmalloc::VMHeap::allocateMediumPage):
173         (bmalloc::VMHeap::allocateLargeRange):
174         (bmalloc::VMHeap::deallocateSmallPage):
175         (bmalloc::VMHeap::deallocateMediumPage):
176         (bmalloc::VMHeap::deallocateLargeRange):
177         * bmalloc/bmalloc.h:
178         (bmalloc::api::heapSize):
179         (bmalloc::api::heapCapacity):
180
181 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
182
183         bmalloc crashes on the EWS bots (due to bad large object allocation)
184         https://bugs.webkit.org/show_bug.cgi?id=136469
185
186         Reviewed by Andreas Kling.
187
188         It's possible to convince bmalloc to perform a bad large object allocation,
189         through these steps:
190
191         (1) Insert object A into freelist F0.
192
193         (2) Split, merge and split again A's neighbors such that object B is
194         inserted into freelist F0, with boundary tag and size equal to object A,
195         but pointer not completely equal to object A. Put object B at the head of F0.
196
197         (3) Allocate some other object from F0, swapping its position in the
198         freelist with object B, such that object A is now ahead of object B.
199
200         --> Now, the next allocation for size A/B will allocate object A, which
201         has a slightly wrong idea about where the object actually begins.
202         Immediately, you'll corrupt a little memory, and over time, you'll also
203         corrupt boundary tag metadata.
204
205         The solution is to store the begin pointer in the boundary tag. Luckily,
206         this doesn't make the tag any bigger, and it's not a noticeable slowdown
207         on MallocBench.
208
209         * bmalloc/Algorithm.h:
210         (bmalloc::rightShift):
211         * bmalloc/BeginTag.h:
212         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
213         validate the start pointer when popping off the free list. Through a
214         very uncommon set of steps, it is possible to have an item in the free
215         list that is valid by all accounts except for its start pointer.
216
217         * bmalloc/BoundaryTag.h:
218         (bmalloc::BoundaryTag::compactBegin):
219         (bmalloc::BoundaryTag::setRange):
220         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
221         start pointer. We don't need the whole pointer -- just the offset, in
222         largeAlignment increments, into the relevant boundary tag bucket.
223
224         * bmalloc/BoundaryTagInlines.h:
225         (bmalloc::validateNext):
226         (bmalloc::BoundaryTag::init):
227         (bmalloc::BoundaryTag::mergeLarge):
228         (bmalloc::BoundaryTag::splitLarge):
229         * bmalloc/SegregatedFreeList.cpp:
230         (bmalloc::SegregatedFreeList::insert):
231         (bmalloc::SegregatedFreeList::takeGreedy):
232         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
233         the size when establishing a boundary tag, as required by the new
234         interface.
235
236         * bmalloc/Sizes.h:
237
238 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
239
240         Fixed a bmalloc crash seen on the EWS bot
241         https://bugs.webkit.org/show_bug.cgi?id=135955
242
243         Reviewed by Andreas Kling.
244
245         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
246         input buffer is a malloc region, that region will get marked Copy-On-Write
247         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
248         on older OS X's. In 10.10, they still fail, but they do not return
249         EINVAL.
250
251         So, we can only ASSERT that our syscalls succeed starting with 10.10.
252
253 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
254
255         Fixed the bmalloc build
256         https://bugs.webkit.org/show_bug.cgi?id=135953
257
258         Reviewed by Andreas Kling.
259
260         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
261         These headers are used, so they must be available outside the project.
262
263 2014-08-13  Daniel Bates  <dabates@apple.com>
264
265         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
266         (https://bugs.webkit.org/show_bug.cgi?id=135895)
267
268         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
269         implementation of PerThread<T>::getFastCase().
270
271         * bmalloc/PerThread.h:
272         (bmalloc::PerThread<T>::getFastCase):
273
274 2014-08-13  Daniel Bates  <dabates@apple.com>
275
276         Make bmalloc::PerThread work without C++ thread local storage
277         https://bugs.webkit.org/show_bug.cgi?id=135895
278
279         Reviewed by Geoffrey Garen.
280
281         Implement support for building bmalloc without C++ thread local storage.
282
283         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
284         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
285         to determine whether the compiler supports C++ thread local storage.
286         * bmalloc/PerThread.h:
287         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
288         without C++ thread local storage.
289         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
290         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
291         PerThreadStorage::initSharedKeyIfNeeded().
292         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
293         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
294         ensure that the shared key has been initialized.
295         (_pthread_setspecific_direct): Deleted.
296         (_pthread_getspecific_direct): Deleted.
297
298 2014-08-13  Daniel Bates  <dabates@apple.com>
299
300         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
301         https://bugs.webkit.org/show_bug.cgi?id=135848
302
303         Reviewed by Geoffrey Garen.
304
305         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
306         when building for the iOS Simulator.
307         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
308         using the public SDK.
309         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
310         using the public SDK.
311         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
312         using the public SDK.
313
314 2014-08-12  Daniel Bates  <dabates@apple.com>
315
316         BPLATFORM(IOS) always evaluates to false
317         https://bugs.webkit.org/show_bug.cgi?id=135843
318
319         Reviewed by Geoffrey Garen.
320
321         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
322         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
323         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
324         building for iOS.
325
326         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
327         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
328
329         * bmalloc/BPlatform.h:
330
331 2014-08-11  Andy Estes  <aestes@apple.com>
332
333         [iOS] Get rid of iOS.xcconfig
334         https://bugs.webkit.org/show_bug.cgi?id=135809
335
336         Reviewed by Joseph Pecoraro.
337
338         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
339
340         * Configurations/Base.xcconfig:
341         * Configurations/iOS.xcconfig: Removed.
342         * bmalloc.xcodeproj/project.pbxproj:
343
344 2014-05-01  Dan Bernstein  <mitz@apple.com>
345
346         Fixed production builds for the iOS Simulator.
347         <rdar://problem/16792221>
348
349         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
350         PRIVATE_HEADERS_FOLDER_PATH when installing.
351
352 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
353
354         bmalloc: Segregate pages by objects size
355         https://bugs.webkit.org/show_bug.cgi?id=131909
356
357         Reviewed by Andreas Kling.
358
359         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
360
361         * bmalloc/Allocator.cpp:
362         (bmalloc::Allocator::allocateSlowCase):
363         * bmalloc/Allocator.h:
364         (bmalloc::Allocator::allocateFastCase):
365         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
366         function for size class calculation.
367
368         * bmalloc/Deallocator.cpp:
369         (bmalloc::Deallocator::Deallocator):
370         (bmalloc::Deallocator::scavenge):
371         (bmalloc::Deallocator::deallocateSmallLine):
372         (bmalloc::Deallocator::allocateSmallLine):
373         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
374         cache can't be shared anymore.
375
376         * bmalloc/Heap.cpp:
377         (bmalloc::Heap::allocateSmallLineSlowCase):
378         * bmalloc/Heap.h:
379         (bmalloc::Heap::deallocateSmallLine): Ditto.
380
381         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
382         page refcount when allocating a line because we might have deallocated
383         the page and the recycled it for another size class.
384
385         (bmalloc::Heap::deallocateMediumLine):
386         (bmalloc::Heap::allocateMediumLine):
387         * bmalloc/Line.h:
388         (bmalloc::Line::refCount):
389         * bmalloc/Page.h:
390         (bmalloc::Page::refCount):
391         (bmalloc::Page::smallSizeClass):
392         (bmalloc::Page::setSmallSizeClass):
393         (bmalloc::Page<Traits>::refCount): Deleted.
394         * bmalloc/Sizes.h:
395         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
396         an index into an array from a size.
397
398 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
399
400         bmalloc: Improved alignment in LargeChunk
401         https://bugs.webkit.org/show_bug.cgi?id=131895
402
403         Reviewed by Andreas Kling.
404
405         * bmalloc/Chunk.h:
406         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
407         Technically, the previous alignment was harmless, but I would prefer,
408         dear reader, not to have to explain the interlocking set of
409         circumstances that made it so.
410
411 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
412
413         Rolled out r167502 because it caused a crash on the facebook benchmark.
414
415         Unreviewed.
416
417             bmalloc: Added an XSmall line size
418             https://bugs.webkit.org/show_bug.cgi?id=131851
419
420             Reviewed by Sam Weinig.
421
422 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
423
424         bmalloc: Mutex should be harder to use wrong
425         https://bugs.webkit.org/show_bug.cgi?id=131879
426
427         Reviewed by Andreas Kling.
428
429         Mutex now has a proper constructor, so you can't deadlock by forgetting
430         to initialize it.
431
432         * bmalloc.xcodeproj/project.pbxproj:
433         * bmalloc/Allocator.cpp:
434         (bmalloc::Allocator::processXSmallAllocatorLog):
435         (bmalloc::Allocator::processSmallAllocatorLog):
436         (bmalloc::Allocator::processMediumAllocatorLog):
437         (bmalloc::Allocator::allocateLarge):
438         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
439         since the Heap mutex is a static.
440
441         * bmalloc/AsyncTask.h:
442         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
443         need for explicit initialization anymore.
444
445         * bmalloc/Deallocator.cpp:
446         (bmalloc::Deallocator::scavenge):
447         (bmalloc::Deallocator::deallocateLarge):
448         (bmalloc::Deallocator::deallocateXLarge):
449         (bmalloc::Deallocator::processObjectLog):
450         (bmalloc::Deallocator::deallocateSmallLine):
451         (bmalloc::Deallocator::deallocateXSmallLine):
452         (bmalloc::Deallocator::allocateSmallLine):
453         (bmalloc::Deallocator::allocateXSmallLine):
454         (bmalloc::Deallocator::deallocateMediumLine):
455         (bmalloc::Deallocator::allocateMediumLine):
456         * bmalloc/Deallocator.h:
457         * bmalloc/Heap.cpp:
458         (bmalloc::sleep):
459         (bmalloc::Heap::Heap):
460         (bmalloc::Heap::concurrentScavenge):
461         (bmalloc::Heap::scavenge):
462         (bmalloc::Heap::scavengeSmallPages):
463         (bmalloc::Heap::scavengeXSmallPages):
464         (bmalloc::Heap::scavengeMediumPages):
465         (bmalloc::Heap::scavengeLargeRanges):
466         (bmalloc::Heap::allocateXSmallLineSlowCase):
467         (bmalloc::Heap::allocateSmallLineSlowCase):
468         (bmalloc::Heap::allocateMediumLineSlowCase):
469         (bmalloc::Heap::allocateXLarge):
470         (bmalloc::Heap::deallocateXLarge):
471         (bmalloc::Heap::allocateLarge):
472         (bmalloc::Heap::deallocateLarge):
473         * bmalloc/Heap.h:
474         (bmalloc::Heap::deallocateXSmallLine):
475         (bmalloc::Heap::allocateXSmallLine):
476         (bmalloc::Heap::deallocateSmallLine):
477         (bmalloc::Heap::allocateSmallLine):
478         (bmalloc::Heap::deallocateMediumLine):
479         (bmalloc::Heap::allocateMediumLine):
480         * bmalloc/Line.h:
481         (bmalloc::Line<Traits>::deref):
482         * bmalloc/Mutex.cpp: Removed.
483         * bmalloc/Mutex.h:
484         (bmalloc::Mutex::Mutex):
485         (bmalloc::Mutex::init): Deleted.
486         (bmalloc::Mutex::try_lock): Deleted.
487         (bmalloc::Mutex::lock): Deleted.
488         (bmalloc::Mutex::unlock): Deleted.
489         * bmalloc/Page.h:
490         (bmalloc::Page<Traits>::ref):
491         (bmalloc::Page<Traits>::deref):
492         (bmalloc::Page<Traits>::refCount):
493         * bmalloc/PerProcess.h:
494         (bmalloc::PerProcess::mutex):
495         (bmalloc::PerProcess<T>::getSlowCase):
496         * bmalloc/StaticMutex.cpp: Added.
497         (bmalloc::StaticMutex::lockSlowCase):
498         * bmalloc/StaticMutex.h: Added.
499         (bmalloc::StaticMutex::init):
500         (bmalloc::StaticMutex::try_lock):
501         (bmalloc::StaticMutex::lock):
502         (bmalloc::StaticMutex::unlock):
503         * bmalloc/VMHeap.h:
504         (bmalloc::VMHeap::deallocateXSmallPage):
505         (bmalloc::VMHeap::deallocateSmallPage):
506         (bmalloc::VMHeap::deallocateMediumPage):
507         (bmalloc::VMHeap::deallocateLargeRange):
508         * bmalloc/bmalloc.h:
509         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
510         since the Heap mutex is a static.
511
512 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
513
514         bmalloc: AsyncTask should use Mutex instead of std::mutex
515         https://bugs.webkit.org/show_bug.cgi?id=131865
516
517         Reviewed by Gavin Barraclough.
518
519         std::mutex is so slow that it makes parallelizing simple tasks through
520         AsyncTask a net regression. Mutex fixes this.
521
522         * bmalloc/AsyncTask.h:
523         (bmalloc::Function>::AsyncTask):
524         (bmalloc::Function>::join):
525         (bmalloc::Function>::runSlowCase):
526         (bmalloc::Function>::entryPoint):
527         * bmalloc/Mutex.h:
528         (bmalloc::Mutex::init):
529
530 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
531
532         bmalloc: Added an XSmall line size
533         https://bugs.webkit.org/show_bug.cgi?id=131851
534
535         Reviewed by Sam Weinig.
536
537         Reduces malloc footprint on Membuster recordings by 10%.
538
539         This is a throughput regression, but we're still way ahead of TCMalloc.
540         I have some ideas for how to recover the regression -- but I wanted to
541         get this win in first.
542
543         Full set of benchmark results:
544
545                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
546                                                                                                 
547                                                                nopatch                      patch                                Δ
548                 Peak Memory:
549                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
550                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
551                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
552
553                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
554                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
555                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
556
557                 Memory at End:
558                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
559                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
560                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
561
562                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
563                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
564                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
565
566                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
567                                                                                                 
568                                                    nopatch                patch                         Δ
569                 Execution Time:
570                     churn                            127ms                151ms            ! 1.19x slower
571                     list_allocate                    130ms                164ms            ! 1.26x slower
572                     tree_allocate                    109ms                127ms            ! 1.17x slower
573                     tree_churn                       115ms                120ms            ! 1.04x slower
574                     facebook                         240ms                259ms            ! 1.08x slower
575                     fragment                          91ms                131ms            ! 1.44x slower
576                     fragment_iterate                 105ms                106ms            ! 1.01x slower
577                     message_one                      260ms                259ms             ^ 1.0x faster
578                     message_many                     149ms                154ms            ! 1.03x slower
579                     medium                           194ms                248ms            ! 1.28x slower
580                     big                              157ms                160ms            ! 1.02x slower
581
582                     <geometric mean>                 144ms                163ms            ! 1.13x slower
583                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
584                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
585
586                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
587                                                                                                 
588                                                                        nopatch                          patch                                     Δ
589                 Execution Time:
590                     churn                                                126ms                          148ms                        ! 1.17x slower
591                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
592                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
593                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
594                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
595                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
596                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
597                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
598                     facebook                                             240ms                          276ms                        ! 1.15x slower
599                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
600                     fragment                                              92ms                          130ms                        ! 1.41x slower
601                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
602                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
603                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
604                     message_one                                          260ms                          260ms                                      
605                     message_many                                         170ms                          238ms                         ! 1.4x slower
606                     medium                                               185ms                          250ms                        ! 1.35x slower
607                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
608                     big                                                  150ms                          169ms                        ! 1.13x slower
609                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
610
611                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
612                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
613                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
614
615                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
616
617                                                                     TC                      patch                                Δ
618                 Peak Memory:
619                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
620                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
621                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
622
623                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
624                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
625                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
626
627                 Memory at End:
628                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
629                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
630                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
631
632                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
633                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
634                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
635
636                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
637                                                                                                 
638                                                         TC                patch                         Δ
639                 Execution Time:
640                     churn                            416ms                148ms            ^ 2.81x faster
641                     list_allocate                    463ms                164ms            ^ 2.82x faster
642                     tree_allocate                    292ms                127ms             ^ 2.3x faster
643                     tree_churn                       157ms                120ms            ^ 1.31x faster
644                     facebook                         327ms                276ms            ^ 1.18x faster
645                     fragment                         335ms                129ms             ^ 2.6x faster
646                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
647                     message_one                      386ms                258ms             ^ 1.5x faster
648                     message_many                     410ms                154ms            ^ 2.66x faster
649                     medium                           391ms                245ms             ^ 1.6x faster
650                     big                              261ms                167ms            ^ 1.56x faster
651
652                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
653                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
654                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
655
656         * bmalloc.xcodeproj/project.pbxproj:
657         * bmalloc/Allocator.cpp:
658         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
659         index corresponds with its size. Instead, use the size selection function
660         explicitly. Now that we have XSmall, some small allocator entries are
661         unused.
662
663         (bmalloc::Allocator::scavenge):
664         (bmalloc::Allocator::log):
665         (bmalloc::Allocator::processXSmallAllocatorLog):
666         (bmalloc::Allocator::allocateSlowCase):
667         * bmalloc/Allocator.h:
668         (bmalloc::Allocator::xSmallAllocatorFor):
669         (bmalloc::Allocator::allocateFastCase):
670         * bmalloc/Chunk.h:
671         * bmalloc/Deallocator.cpp:
672         (bmalloc::Deallocator::scavenge):
673         (bmalloc::Deallocator::processObjectLog):
674         (bmalloc::Deallocator::deallocateSlowCase):
675         (bmalloc::Deallocator::deallocateXSmallLine):
676         (bmalloc::Deallocator::allocateXSmallLine):
677         * bmalloc/Deallocator.h:
678         (bmalloc::Deallocator::deallocateFastCase):
679         * bmalloc/Heap.cpp:
680         (bmalloc::Heap::scavenge):
681         (bmalloc::Heap::scavengeXSmallPages):
682         (bmalloc::Heap::allocateXSmallLineSlowCase):
683         * bmalloc/Heap.h:
684         (bmalloc::Heap::deallocateXSmallLine):
685         (bmalloc::Heap::allocateXSmallLine):
686         * bmalloc/LargeChunk.h:
687         (bmalloc::LargeChunk::get):
688         (bmalloc::LargeChunk::endTag):
689         * bmalloc/Line.h:
690         * bmalloc/MediumAllocator.h:
691         (bmalloc::MediumAllocator::allocate):
692         (bmalloc::MediumAllocator::refill):
693         * bmalloc/ObjectType.cpp:
694         (bmalloc::objectType):
695         * bmalloc/ObjectType.h:
696         (bmalloc::isXSmall):
697         (bmalloc::isSmall):
698         (bmalloc::isMedium):
699         (bmalloc::isLarge):
700         (bmalloc::isSmallOrMedium): Deleted.
701         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
702         handling small objects. There's probably a reasonable way to share this
703         code in the future -- I'll look into that once it's stopped changing.
704
705         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
706         is the main reason things got slower.
707
708         * bmalloc/SmallAllocator.h:
709         (bmalloc::SmallAllocator::allocate):
710         * bmalloc/SmallTraits.h:
711         * bmalloc/VMHeap.cpp:
712         (bmalloc::VMHeap::allocateXSmallChunk):
713         * bmalloc/VMHeap.h:
714         (bmalloc::VMHeap::allocateXSmallPage):
715         (bmalloc::VMHeap::deallocateXSmallPage):
716         * bmalloc/XSmallAllocator.h: Added.
717         (bmalloc::XSmallAllocator::isNull):
718         (bmalloc::XSmallAllocator::canAllocate):
719         (bmalloc::XSmallAllocator::XSmallAllocator):
720         (bmalloc::XSmallAllocator::line):
721         (bmalloc::XSmallAllocator::allocate):
722         (bmalloc::XSmallAllocator::objectCount):
723         (bmalloc::XSmallAllocator::derefCount):
724         (bmalloc::XSmallAllocator::refill):
725         * bmalloc/XSmallChunk.h: Added.
726         * bmalloc/XSmallLine.h: Added.
727         * bmalloc/XSmallPage.h: Added.
728         * bmalloc/XSmallTraits.h: Added.
729         * bmalloc/bmalloc.h:
730         (bmalloc::api::realloc): Boiler-plate copy, as above.
731
732 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
733
734         MallocBench should scavenge explicitly instead of waiting
735         https://bugs.webkit.org/show_bug.cgi?id=131661
736
737         Reviewed by Andreas Kling.
738
739         Added explicit scavenge support to bmalloc. This isn't a memory win,
740         since bmalloc's per-thread cache is so small. But it makes testing
741         simpler.
742
743         * bmalloc/Allocator.cpp:
744         (bmalloc::Allocator::~Allocator):
745         (bmalloc::Allocator::scavenge):
746         * bmalloc/Allocator.h:
747         * bmalloc/Cache.cpp:
748         (bmalloc::Cache::operator new):
749         (bmalloc::Cache::operator delete):
750         (bmalloc::Cache::Cache):
751         (bmalloc::Cache::scavenge):
752         * bmalloc/Cache.h:
753         * bmalloc/Deallocator.cpp:
754         (bmalloc::Deallocator::~Deallocator):
755         (bmalloc::Deallocator::scavenge):
756         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
757         functions, for reuse.
758
759         * bmalloc/Heap.cpp:
760         (bmalloc::sleep):
761         (bmalloc::Heap::concurrentScavenge):
762         (bmalloc::Heap::scavenge):
763         (bmalloc::Heap::scavengeSmallPages):
764         (bmalloc::Heap::scavengeMediumPages):
765         (bmalloc::Heap::scavengeLargeRanges):
766         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
767         scavenging -- in response to a benchmark or a low memory warning --
768         wants to complete as soon as possible, so its sleep duration is 0.
769
770         * bmalloc/bmalloc.h:
771         (bmalloc::api::scavenge):
772         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
773
774 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
775
776         Use 4kB pages on Mac
777         https://bugs.webkit.org/show_bug.cgi?id=131658
778
779         Reviewed by Sam Weinig.
780
781         This reduces memory use a lot on Membuster:
782
783                                                                   base                      patch                                Δ
784                 Execution Time:
785                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
786                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
787                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
788
789                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
790                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
791                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
792
793                 Peak Memory:
794                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
795                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
796                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
797
798                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
799                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
800                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
801
802                 Memory at End:
803                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
804                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
805                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
806
807                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
808                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
809                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
810
811
812         * bmalloc.xcodeproj/project.pbxproj:
813         * bmalloc/BPlatform.h: Added.
814         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
815         is 4kB on Mac.
816
817 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
818
819         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
820
821         * bmalloc.xcodeproj: Modified property svn:ignore.
822
823 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
824
825         Fixed some mbmalloc exports
826         https://bugs.webkit.org/show_bug.cgi?id=131599
827
828         Reviewed by Ryosuke Niwa.
829
830         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
831         of the project, so we can call them from API.
832
833         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
834         visibility, so they show up as exported in the .dylib.
835
836 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
837
838         Put bmalloc headers in the right place
839         https://bugs.webkit.org/show_bug.cgi?id=131464
840
841         Reviewed by Mark Rowe.
842
843         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
844         specify that we don't just want to dump all of our generically-named
845         headers into /usr/local/include.
846
847 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
848
849         Made bmalloc more #include friendly
850         https://bugs.webkit.org/show_bug.cgi?id=131386
851
852         Reviewed by Andreas Kling.
853
854         Marked a bunch of headers private so they can be used from client code
855         that #includes bmalloc.h.
856
857         Renamed ASSERT macros to BASSERT. This matches their header, which already
858         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
859
860         * bmalloc.xcodeproj/project.pbxproj:
861         * bmalloc/Allocator.cpp:
862         (bmalloc::Allocator::allocateSlowCase):
863         * bmalloc/AsyncTask.h:
864         (bmalloc::Function>::runSlowCase):
865         * bmalloc/BAssert.h:
866         * bmalloc/BoundaryTag.h:
867         (bmalloc::BoundaryTag::setSize):
868         * bmalloc/BoundaryTagInlines.h:
869         (bmalloc::validate):
870         (bmalloc::BoundaryTag::init):
871         (bmalloc::BoundaryTag::deallocate):
872         (bmalloc::BoundaryTag::splitLarge):
873         (bmalloc::BoundaryTag::allocate):
874         * bmalloc/Chunk.h:
875         * bmalloc/Deallocator.cpp:
876         (bmalloc::Deallocator::processObjectLog):
877         (bmalloc::Deallocator::deallocateSlowCase):
878         * bmalloc/Deallocator.h:
879         (bmalloc::Deallocator::deallocateFastCase):
880         * bmalloc/FixedVector.h:
881         (bmalloc::Capacity>::operator):
882         (bmalloc::Capacity>::push):
883         (bmalloc::Capacity>::pop):
884         (bmalloc::Capacity>::shrink):
885         * bmalloc/Heap.cpp:
886         (bmalloc::Heap::allocateLarge):
887         * bmalloc/LargeChunk.h:
888         (bmalloc::LargeChunk::get):
889         (bmalloc::LargeChunk::endTag):
890         * bmalloc/Line.h:
891         (bmalloc::Line<Traits>::concurrentRef):
892         (bmalloc::Line<Traits>::deref):
893         * bmalloc/MediumAllocator.h:
894         (bmalloc::MediumAllocator::allocate):
895         * bmalloc/ObjectType.h:
896         (bmalloc::isSmall):
897         * bmalloc/Page.h:
898         (bmalloc::Page<Traits>::ref):
899         (bmalloc::Page<Traits>::deref):
900         * bmalloc/PerThread.h:
901         (bmalloc::PerThread<T>::getSlowCase):
902         * bmalloc/SegregatedFreeList.cpp:
903         (bmalloc::SegregatedFreeList::SegregatedFreeList):
904         (bmalloc::SegregatedFreeList::insert):
905         * bmalloc/SmallAllocator.h:
906         (bmalloc::SmallAllocator::allocate):
907         (bmalloc::SmallAllocator::refill):
908         * bmalloc/Syscall.h:
909         * bmalloc/VMAllocate.h:
910         (bmalloc::vmValidate):
911         (bmalloc::vmAllocate):
912         (bmalloc::vmDeallocatePhysicalPagesSloppy):
913         * bmalloc/Vector.h:
914         (bmalloc::Vector<T>::operator):
915         (bmalloc::Vector<T>::pop):
916         (bmalloc::Vector<T>::shrink):
917         * bmalloc/XLargeChunk.h:
918         (bmalloc::XLargeChunk::range):
919         (bmalloc::XLargeChunk::size):
920
921 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
922
923         Removed an unused file.
924
925         Unreviewed.
926
927         * bmalloc/AsyncTask.cpp: Removed.
928
929 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
930
931         Build bmalloc on Mac
932         https://bugs.webkit.org/show_bug.cgi?id=131333
933
934         Reviewed by Mark Rowe.
935
936         * Makefile: Added. For make clients.
937
938         These files are required for building any project in WebKit. I copied
939         them from WTF:
940         * Configurations: Added.
941         * Configurations/Base.xcconfig: Added.
942         * Configurations/DebugRelease.xcconfig: Added.
943         * Configurations/bmalloc.xcconfig: Added.
944         * Configurations/iOS.xcconfig: Added.
945         * Configurations/mbmalloc.xcconfig: Added.
946
947         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
948         from here because everything is in .xcconfig files now.
949
950         I had to fix a bunch of minor warnings, since they're enabled in our
951         .xcconfig files:
952
953         * bmalloc/AsyncTask.h:
954         (bmalloc::Function>::AsyncTask):
955         * bmalloc/BAssert.h:
956         * bmalloc/BoundaryTagInlines.h:
957         (bmalloc::validate):
958         * bmalloc/Heap.cpp:
959         (bmalloc::Heap::Heap):
960         (bmalloc::Heap::allocateLarge):
961         (bmalloc::Heap::deallocateLarge):
962         * bmalloc/Mutex.h:
963         (bmalloc::Mutex::Mutex): Deleted.
964         * bmalloc/VMAllocate.h:
965         (bmalloc::vmValidate):
966         * bmalloc/mbmalloc.cpp:
967
968 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
969
970         bmalloc: Fixed a leak in the per-thread cache
971         https://bugs.webkit.org/show_bug.cgi?id=131330
972
973         Reviewed by Andreas Kling.
974
975         Remember to deallocate our line caches upon thread exit.
976
977         * bmalloc/Deallocator.cpp:
978         (bmalloc::Deallocator::~Deallocator):
979
980 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
981
982         bmalloc: rolled out the tryLock experiment
983         https://bugs.webkit.org/show_bug.cgi?id=131328
984
985         Reviewed by Andreas Kling.
986
987         It wasn't a speedup.
988
989         * bmalloc.xcodeproj/project.pbxproj:
990         * bmalloc/Allocator.cpp:
991         (bmalloc::Allocator::processSmallAllocatorLog):
992         (bmalloc::Allocator::processMediumAllocatorLog):
993         * bmalloc/Deallocator.cpp:
994         (bmalloc::Deallocator::processObjectLog):
995         (bmalloc::Deallocator::deallocateSlowCase):
996         (bmalloc::Deallocator::deallocateSmallLine):
997         (bmalloc::Deallocator::deallocateMediumLine):
998         * bmalloc/Deallocator.h:
999         (bmalloc::Deallocator::deallocateFastCase):
1000         * bmalloc/Heap.h:
1001         (bmalloc::Heap::deallocateSmallLine):
1002         (bmalloc::Heap::deallocateMediumLine):
1003         * bmalloc/Line.h:
1004         (bmalloc::Line<Traits>::deref):
1005         * bmalloc/Page.h:
1006         (bmalloc::Page<Traits>::deref):
1007
1008 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1009
1010         bmalloc
1011         https://bugs.webkit.org/show_bug.cgi?id=131170
1012
1013         Reviewed by Andreas Kling.
1014
1015         Initial commit.
1016
1017         * bmalloc: Added.
1018         * bmalloc.xcodeproj: Added.
1019         * bmalloc.xcodeproj/project.pbxproj: Added.
1020         * bmalloc/Algorithm.h: Added.
1021         (bmalloc::max):
1022         (bmalloc::min):
1023         (bmalloc::mask):
1024         (bmalloc::test):
1025         (bmalloc::roundUpToMultipleOf):
1026         (bmalloc::roundDownToMultipleOf):
1027         (bmalloc::sizeOf):
1028         (bmalloc::bitCount):
1029         (bmalloc::isPowerOfTwo):
1030         * bmalloc/Allocator.cpp: Added.
1031         (bmalloc::Allocator::Allocator):
1032         (bmalloc::Allocator::~Allocator):
1033         (bmalloc::Allocator::log):
1034         (bmalloc::Allocator::processSmallAllocatorLog):
1035         (bmalloc::Allocator::processMediumAllocatorLog):
1036         (bmalloc::Allocator::allocateLarge):
1037         (bmalloc::Allocator::allocateXLarge):
1038         (bmalloc::Allocator::allocateMedium):
1039         (bmalloc::Allocator::allocateSlowCase):
1040         * bmalloc/Allocator.h: Added.
1041         (bmalloc::Allocator::smallAllocatorFor):
1042         (bmalloc::Allocator::allocateFastCase):
1043         (bmalloc::Allocator::allocate):
1044         * bmalloc/AsyncTask.cpp: Added.
1045         (bmalloc::AsyncTask<Function>::runSlowCase):
1046         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
1047         (bmalloc::AsyncTask<Function>::entryPoint):
1048         * bmalloc/AsyncTask.h: Added.
1049         (bmalloc::Function>::AsyncTask):
1050         (bmalloc::Function>::join):
1051         (bmalloc::Function>::run):
1052         (bmalloc::Function>::runSlowCase):
1053         (bmalloc::Function>::pthreadEntryPoint):
1054         (bmalloc::Function>::entryPoint):
1055         * bmalloc/BAssert.h: Added.
1056         * bmalloc/BeginTag.h: Added.
1057         (bmalloc::BeginTag::isInFreeList):
1058         * bmalloc/BoundaryTag.h: Added.
1059         (bmalloc::BoundaryTag::isXLarge):
1060         (bmalloc::BoundaryTag::setXLarge):
1061         (bmalloc::BoundaryTag::isFree):
1062         (bmalloc::BoundaryTag::setFree):
1063         (bmalloc::BoundaryTag::isEnd):
1064         (bmalloc::BoundaryTag::setEnd):
1065         (bmalloc::BoundaryTag::hasPhysicalPages):
1066         (bmalloc::BoundaryTag::setHasPhysicalPages):
1067         (bmalloc::BoundaryTag::isNull):
1068         (bmalloc::BoundaryTag::clear):
1069         (bmalloc::BoundaryTag::size):
1070         (bmalloc::BoundaryTag::setSize):
1071         (bmalloc::BoundaryTag::prev):
1072         (bmalloc::BoundaryTag::next):
1073         * bmalloc/BoundaryTagInlines.h: Added.
1074         (bmalloc::validate):
1075         (bmalloc::validatePrev):
1076         (bmalloc::validateNext):
1077         (bmalloc::BoundaryTag::init):
1078         (bmalloc::BoundaryTag::mergeLargeLeft):
1079         (bmalloc::BoundaryTag::mergeLargeRight):
1080         (bmalloc::BoundaryTag::mergeLarge):
1081         (bmalloc::BoundaryTag::deallocate):
1082         (bmalloc::BoundaryTag::splitLarge):
1083         (bmalloc::BoundaryTag::allocate):
1084         * bmalloc/Cache.cpp: Added.
1085         (bmalloc::Cache::operator new):
1086         (bmalloc::Cache::operator delete):
1087         (bmalloc::Cache::Cache):
1088         (bmalloc::Cache::allocateSlowCase):
1089         (bmalloc::Cache::allocateSlowCaseNullCache):
1090         (bmalloc::Cache::deallocateSlowCase):
1091         (bmalloc::Cache::deallocateSlowCaseNullCache):
1092         * bmalloc/Cache.h: Added.
1093         (bmalloc::Cache::allocator):
1094         (bmalloc::Cache::deallocator):
1095         (bmalloc::Cache::allocateFastCase):
1096         (bmalloc::Cache::deallocateFastCase):
1097         (bmalloc::Cache::allocate):
1098         (bmalloc::Cache::deallocate):
1099         * bmalloc/Chunk.h: Added.
1100         (bmalloc::Chunk::begin):
1101         (bmalloc::Chunk::end):
1102         (bmalloc::Chunk::lines):
1103         (bmalloc::Chunk::pages):
1104         * bmalloc/Deallocator.cpp: Added.
1105         (bmalloc::Deallocator::Deallocator):
1106         (bmalloc::Deallocator::~Deallocator):
1107         (bmalloc::Deallocator::deallocateLarge):
1108         (bmalloc::Deallocator::deallocateXLarge):
1109         (bmalloc::Deallocator::processObjectLog):
1110         (bmalloc::Deallocator::deallocateSlowCase):
1111         (bmalloc::Deallocator::deallocateSmallLine):
1112         (bmalloc::Deallocator::allocateSmallLine):
1113         (bmalloc::Deallocator::deallocateMediumLine):
1114         (bmalloc::Deallocator::allocateMediumLine):
1115         * bmalloc/Deallocator.h: Added.
1116         (bmalloc::Deallocator::deallocateFastCase):
1117         (bmalloc::Deallocator::deallocate):
1118         * bmalloc/EndTag.h: Added.
1119         (bmalloc::EndTag::operator=):
1120         * bmalloc/FixedVector.h: Added.
1121         (bmalloc::FixedVector::begin):
1122         (bmalloc::FixedVector::end):
1123         (bmalloc::FixedVector::size):
1124         (bmalloc::FixedVector::capacity):
1125         (bmalloc::FixedVector::clear):
1126         (bmalloc::FixedVector::isEmpty):
1127         (bmalloc::Capacity>::FixedVector):
1128         (bmalloc::Capacity>::operator):
1129         (bmalloc::Capacity>::push):
1130         (bmalloc::Capacity>::pop):
1131         (bmalloc::Capacity>::shrink):
1132         * bmalloc/Heap.cpp: Added.
1133         (bmalloc::sleep):
1134         (bmalloc::Heap::Heap):
1135         (bmalloc::Heap::concurrentScavenge):
1136         (bmalloc::Heap::scavengeSmallPages):
1137         (bmalloc::Heap::scavengeMediumPages):
1138         (bmalloc::Heap::scavengeLargeRanges):
1139         (bmalloc::Heap::allocateSmallLineSlowCase):
1140         (bmalloc::Heap::allocateMediumLineSlowCase):
1141         (bmalloc::Heap::allocateXLarge):
1142         (bmalloc::Heap::deallocateXLarge):
1143         (bmalloc::Heap::allocateLarge):
1144         (bmalloc::Heap::deallocateLarge):
1145         * bmalloc/Heap.h: Added.
1146         (bmalloc::Heap::deallocateSmallLine):
1147         (bmalloc::Heap::allocateSmallLine):
1148         (bmalloc::Heap::deallocateMediumLine):
1149         (bmalloc::Heap::allocateMediumLine):
1150         * bmalloc/Inline.h: Added.
1151         * bmalloc/LargeChunk.h: Added.
1152         (bmalloc::LargeChunk::begin):
1153         (bmalloc::LargeChunk::end):
1154         (bmalloc::LargeChunk::create):
1155         (bmalloc::LargeChunk::get):
1156         (bmalloc::LargeChunk::beginTag):
1157         (bmalloc::LargeChunk::endTag):
1158         * bmalloc/Line.h: Added.
1159         (bmalloc::Line<Traits>::begin):
1160         (bmalloc::Line<Traits>::end):
1161         (bmalloc::Line<Traits>::concurrentRef):
1162         (bmalloc::Line<Traits>::deref):
1163         * bmalloc/MediumAllocator.h: Added.
1164         (bmalloc::MediumAllocator::isNull):
1165         (bmalloc::MediumAllocator::MediumAllocator):
1166         (bmalloc::MediumAllocator::line):
1167         (bmalloc::MediumAllocator::allocate):
1168         (bmalloc::MediumAllocator::derefCount):
1169         (bmalloc::MediumAllocator::refill):
1170         * bmalloc/MediumChunk.h: Added.
1171         * bmalloc/MediumLine.h: Added.
1172         * bmalloc/MediumPage.h: Added.
1173         * bmalloc/MediumTraits.h: Added.
1174         * bmalloc/Mutex.cpp: Added.
1175         (bmalloc::Mutex::lockSlowCase):
1176         * bmalloc/Mutex.h: Added.
1177         (bmalloc::Mutex::Mutex):
1178         (bmalloc::Mutex::try_lock):
1179         (bmalloc::Mutex::lock):
1180         (bmalloc::Mutex::unlock):
1181         * bmalloc/ObjectType.cpp: Added.
1182         (bmalloc::objectType):
1183         * bmalloc/ObjectType.h: Added.
1184         (bmalloc::isSmallOrMedium):
1185         (bmalloc::isSmall):
1186         * bmalloc/Page.h: Added.
1187         (bmalloc::Page<Traits>::ref):
1188         (bmalloc::Page<Traits>::deref):
1189         (bmalloc::Page<Traits>::refCount):
1190         * bmalloc/PerProcess.h: Added.
1191         (bmalloc::PerProcess::mutex):
1192         (bmalloc::PerProcess<T>::getFastCase):
1193         (bmalloc::PerProcess<T>::get):
1194         (bmalloc::PerProcess<T>::getSlowCase):
1195         * bmalloc/PerThread.h: Added.
1196         (bmalloc::PerThreadStorage<Cache>::get):
1197         (bmalloc::PerThreadStorage<Cache>::init):
1198         (bmalloc::PerThreadStorage::get):
1199         (bmalloc::PerThreadStorage::init):
1200         (bmalloc::PerThread<T>::getFastCase):
1201         (bmalloc::PerThread<T>::get):
1202         (bmalloc::PerThread<T>::destructor):
1203         (bmalloc::PerThread<T>::getSlowCase):
1204         * bmalloc/Range.h: Added.
1205         (bmalloc::Range::Range):
1206         (bmalloc::Range::begin):
1207         (bmalloc::Range::end):
1208         (bmalloc::Range::size):
1209         (bmalloc::Range::operator!):
1210         (bmalloc::Range::operator<):
1211         * bmalloc/SegregatedFreeList.cpp: Added.
1212         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1213         (bmalloc::SegregatedFreeList::insert):
1214         (bmalloc::SegregatedFreeList::takeGreedy):
1215         (bmalloc::SegregatedFreeList::take):
1216         * bmalloc/SegregatedFreeList.h: Added.
1217         * bmalloc/Sizes.h: Added.
1218         * bmalloc/SmallAllocator.h: Added.
1219         (bmalloc::SmallAllocator::isNull):
1220         (bmalloc::SmallAllocator::canAllocate):
1221         (bmalloc::SmallAllocator::SmallAllocator):
1222         (bmalloc::SmallAllocator::line):
1223         (bmalloc::SmallAllocator::allocate):
1224         (bmalloc::SmallAllocator::objectCount):
1225         (bmalloc::SmallAllocator::derefCount):
1226         (bmalloc::SmallAllocator::refill):
1227         * bmalloc/SmallChunk.h: Added.
1228         * bmalloc/SmallLine.h: Added.
1229         * bmalloc/SmallPage.h: Added.
1230         * bmalloc/SmallTraits.h: Added.
1231         * bmalloc/Syscall.h: Added.
1232         * bmalloc/VMAllocate.h: Added.
1233         (bmalloc::vmSize):
1234         (bmalloc::vmValidate):
1235         (bmalloc::vmAllocate):
1236         (bmalloc::vmDeallocate):
1237         (bmalloc::vmDeallocatePhysicalPages):
1238         (bmalloc::vmAllocatePhysicalPages):
1239         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1240         (bmalloc::vmAllocatePhysicalPagesSloppy):
1241         * bmalloc/VMHeap.cpp: Added.
1242         (bmalloc::VMHeap::VMHeap):
1243         (bmalloc::VMHeap::allocateSmallChunk):
1244         (bmalloc::VMHeap::allocateMediumChunk):
1245         (bmalloc::VMHeap::allocateLargeChunk):
1246         * bmalloc/VMHeap.h: Added.
1247         (bmalloc::VMHeap::allocateSmallPage):
1248         (bmalloc::VMHeap::allocateMediumPage):
1249         (bmalloc::VMHeap::allocateLargeRange):
1250         (bmalloc::VMHeap::deallocateSmallPage):
1251         (bmalloc::VMHeap::deallocateMediumPage):
1252         (bmalloc::VMHeap::deallocateLargeRange):
1253         * bmalloc/Vector.h: Added.
1254         (bmalloc::Vector::begin):
1255         (bmalloc::Vector::end):
1256         (bmalloc::Vector::size):
1257         (bmalloc::Vector::capacity):
1258         (bmalloc::Vector::last):
1259         (bmalloc::Vector::pop):
1260         (bmalloc::Vector<T>::Vector):
1261         (bmalloc::Vector<T>::~Vector):
1262         (bmalloc::Vector<T>::operator):
1263         (bmalloc::Vector<T>::push):
1264         (bmalloc::Vector<T>::pop):
1265         (bmalloc::Vector<T>::shrink):
1266         (bmalloc::Vector<T>::reallocateBuffer):
1267         (bmalloc::Vector<T>::shrinkCapacity):
1268         (bmalloc::Vector<T>::growCapacity):
1269         * bmalloc/XLargeChunk.h: Added.
1270         (bmalloc::XLargeChunk::get):
1271         (bmalloc::XLargeChunk::begin):
1272         (bmalloc::XLargeChunk::XLargeChunk):
1273         (bmalloc::XLargeChunk::create):
1274         (bmalloc::XLargeChunk::destroy):
1275         (bmalloc::XLargeChunk::range):
1276         (bmalloc::XLargeChunk::size):
1277         * bmalloc/bmalloc.h: Added.
1278         (bmalloc::api::malloc):
1279         (bmalloc::api::free):
1280         (bmalloc::api::realloc):
1281         * bmalloc/mbmalloc.cpp: Added.
1282