Fix up bmalloc's PerThread for use on Linux
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
2
3         Fix up bmalloc's PerThread for use on Linux
4         https://bugs.webkit.org/show_bug.cgi?id=139804
5
6         Reviewed by Anders Carlsson.
7
8         The previous implementation was a bit slow.
9
10         * bmalloc/PerThread.h:
11         (bmalloc::PerThreadStorage<Cache>::get):
12         (bmalloc::PerThreadStorage::get):
13         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
14         way to do fast per-thread access without taking a lock every time. This
15         probably works on all the platforms we care about, and it matches other
16         techniques we use elsewhere in WebKit.
17
18         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
19         this class because PerThreadStorage now encapsulates everything that
20         needs to be conditional.
21
22         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
23
24 2014-12-26  Dan Bernstein  <mitz@apple.com>
25
26         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
27         https://bugs.webkit.org/show_bug.cgi?id=139950
28
29         Reviewed by David Kilzer.
30
31         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
32         in a manner that works with Xcode 5.1.1.
33
34 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
35
36         Safari crashes when you set Malloc environment variables
37         https://bugs.webkit.org/show_bug.cgi?id=139656
38
39         Reviewed by Michael Saboff.
40
41         I forgot to cover the realloc() case. Whoops. (OoPS?)
42
43         This time around, I ran the full MallocBench test suite in Malloc=1
44         mode, and it passed.
45
46         * bmalloc/Allocator.cpp:
47         (bmalloc::Allocator::reallocate):
48         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
49         It needs to be down there so that we can do the short-circuiting check
50         for whether bmalloc is enabled first.
51
52         Also added the check.
53
54         * bmalloc/Cache.cpp:
55         (bmalloc::Cache::scavenge):
56         (bmalloc::Cache::Cache):
57         (bmalloc::Cache::reallocateSlowCaseNullCache):
58         * bmalloc/Cache.h:
59         (bmalloc::Cache::deallocator):
60         (bmalloc::Cache::reallocate): Ditto.
61
62         * bmalloc/bmalloc.h:
63         (bmalloc::api::free):
64         (bmalloc::api::realloc): Ditto.
65
66         (bmalloc::api::scavenge): Pushed this down into Cache to match the
67         surrounding functions.
68
69 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
70
71         bmalloc should support system memory analysis tools (part 2)
72         https://bugs.webkit.org/show_bug.cgi?id=139565
73
74         Reviewed by Mark Lam.
75
76         This patch actually queries the environment to see if memory analysis
77         tools have been enabled.
78
79         * bmalloc/Deallocator.cpp:
80         (bmalloc::Deallocator::scavenge): Don't process the object log if
81         we've disabled bmalloc because it will be full of invalid nullptrs.
82
83         * bmalloc/Environment.cpp:
84         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
85         Malloc debugging flags. I also added a plain "Malloc" catch-all for
86         when you want to disable bmalloc without enabling any kind of funny
87         business.
88
89         It would be slightly nicer just to iterate the list of environment
90         variables and strstr them, but getenv is the more portable option,
91         and performance here doesn't really matter.
92
93         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
94         environment variable.
95
96         (bmalloc::Environment::computeIsBmallocEnabled):
97
98 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
99
100         Try to fix the iOS simulator build.
101
102         #include the declaration of malloc / free.
103
104         * bmalloc/Allocator.cpp:
105         * bmalloc/Deallocator.cpp:
106
107 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
108
109         Try to fix the build.
110
111         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
112
113 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
114
115         bmalloc should support system memory analysis tools (part 1)
116         https://bugs.webkit.org/show_bug.cgi?id=139559
117
118         Reviewed by Mark Lam.
119
120         This patch adds the hooks to disable bmalloc at runtime if certain
121         environment variables are set, but doesn't actually read from the
122         environment yet.
123
124         No performance change.
125
126         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
127         we'll use to read environment variables and see if memory analysis tools
128         have been enabled.
129
130         * bmalloc/Allocator.cpp:
131         (bmalloc::Allocator::Allocator):
132         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
133         on the allocation path. We cache the setting to make the check fast.
134
135         * bmalloc/Allocator.h: Interface changes.
136
137         * bmalloc/Cache.cpp:
138         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
139         and deallocator. This main purpose is to enable them to query the
140         environment for whether bmalloc is enabled; but this is also a slightly
141         cleaner way to guarantee to them that the Heap has been pre-initialized.
142
143         * bmalloc/Deallocator.cpp:
144         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
145         fill the object log to force us to take the slow path on all deallocations.
146
147         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
148
149         * bmalloc/Deallocator.h: Interface changes.
150
151         * bmalloc/Environment.cpp: Added.
152         (bmalloc::Environment::Environment):
153         (bmalloc::Environment::computeIsBmallocEnabled):
154         * bmalloc/Environment.h: Added.
155         (bmalloc::Environment::isBmallocEnabled): This is the class that will
156         encapsulate looking for environment variables that turn on heap
157         analysis tools.
158
159         * bmalloc/Heap.h:
160         (bmalloc::Heap::environment):
161
162         * bmalloc/Mutex.h:
163         (bmalloc::Mutex::Mutex):
164         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
165         since I got super confused about them while writing this patch.
166
167         * bmalloc/VMHeap.cpp: Fixed an #include.
168
169 2014-12-09  David Kilzer  <ddkilzer@apple.com>
170
171         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
172         <http://webkit.org/b/139212>
173
174         Reviewed by Joseph Pecoraro.
175
176         * Configurations/Base.xcconfig:
177         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
178           on OS X.
179         * Configurations/DebugRelease.xcconfig:
180         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
181
182 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
183
184         bmalloc uses 8X more virtual memory than necessary
185         https://bugs.webkit.org/show_bug.cgi?id=138495
186
187         Reviewed by Mark Lam.
188
189         iOS has a per-process virtual memory cap around 1GB, so there's some
190         value to not going totally ham with virtual memory.
191
192         We currently use about 8X the necessary amount:
193             - 2X to align our VM allocation
194             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
195
196         We can cut that down:
197             - Return the unaligned portion of our VM allocation (-2X)
198             - Use all the chunks in a superchunk, instead of allocating one
199               chunk per superchunk (-4X)
200
201         * bmalloc/Algorithm.h:
202         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
203         function so we can call it with getpagesize() at runtime.
204
205         * bmalloc/Chunk.h:
206         * bmalloc/LargeChunk.h:
207         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
208         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
209
210         * bmalloc/VMAllocate.h:
211         (bmalloc::vmValidate):
212         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
213         if it does not succeed. Allocate precisely, and give back the extra.
214
215         * bmalloc/VMHeap.cpp:
216         (bmalloc::VMHeap::allocateSuperChunk):
217         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
218         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
219         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
220         in a superchunk, instead of just one.
221
222         * bmalloc/VMHeap.h:
223         (bmalloc::VMHeap::allocateSmallPage):
224         (bmalloc::VMHeap::allocateMediumPage):
225         (bmalloc::VMHeap::allocateLargeRange):
226         * bmalloc/XLargeChunk.h:
227         (bmalloc::XLargeChunk::create): Updated to match changes above.
228
229 2014-11-01  David Kilzer  <ddkilzer@apple.com>
230
231         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
232         <https://webkit.org/b/138286>
233         <rdar://problem/18847087>
234
235         Reviewed by Dan Bernstein.
236
237         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
238         to NO for the target that produces libbmalloc.a so that the
239         debug symbols will be linked into JavaScriptCore and end up in
240         its dSYM file.
241
242 2014-10-30  Dana Burkart  <dburkart@apple.com>
243
244         <rdar://problem/18821260> Prepare for the mysterious future
245
246         Reviewed by Lucas Forschler.
247
248         * Configurations/Base.xcconfig:
249         * Configurations/DebugRelease.xcconfig:
250
251 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
252
253         bmalloc: cleaned up fast path vs slow path
254         https://bugs.webkit.org/show_bug.cgi?id=137081
255
256         Reviewed by Sam Weinig.
257
258         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
259
260         * bmalloc/Allocator.cpp:
261         (bmalloc::Allocator::Allocator): Merged the small and medium range
262         caches, just like the small and medium allocators. Ranges are abstract
263         objects that don't really care whether they hold small or medium objects,
264         so they don't need to be segregated.
265
266         (bmalloc::Allocator::scavenge): Ditto.
267
268         (bmalloc::Allocator::allocateBumpRangeSlowCase):
269         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
270         we do care a tiny bit, because we need to specify small vs medium when
271         allocating new ranges from the heap, to ensure that the heap allocates
272         from the right segment of VM.
273
274         (bmalloc::Allocator::allocateLarge):
275         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
276         up the fast path. Large allocation performance is dominated by allocation
277         logic and initialization, so inlining it doesn't help.
278
279         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
280         it doesn't need to distinguish small vs medium objects.
281
282         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
283         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
284
285         * bmalloc/Allocator.h:
286         * bmalloc/BumpRange.h:
287
288         * bmalloc/Cache.cpp:
289         (bmalloc::Cache::allocateSlowCase): Deleted.
290         (bmalloc::Cache::deallocateSlowCase): Deleted.
291         * bmalloc/Cache.h:
292         (bmalloc::Cache::allocate):
293         (bmalloc::Cache::deallocate):
294         (bmalloc::Cache::allocateFastCase): Deleted.
295         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
296         paths. The downside to this change is that the fast path branches to two
297         distinct failure cases instead of one. The upside is that the slow path
298         doesn't need to re-read the segment register, which is not as cheap as a
299         normal register, and it doesn't need to do an extra level of function 
300         call. Seems to be worth it.
301
302         * bmalloc/Deallocator.h:
303         * bmalloc/Heap.cpp:
304         (bmalloc::Heap::refillSmallBumpRangeCache):
305         (bmalloc::Heap::refillMediumBumpRangeCache):
306         * bmalloc/Heap.h: Updated for interface changes.
307
308         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
309         small lines in a page / 2, since any other free lines will coalesce
310         with their neighbors.
311
312 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
313
314         Rolled out r173346.
315
316             bmalloc should honor the FastMalloc statistics API
317             https://bugs.webkit.org/show_bug.cgi?id=136592
318
319         This didn't really work. Because we allow ranges with and without
320         physical pages to merge, and we allow double-committing and
321         double-decommitting, we can't rely on commit actions to track memory
322         footprint.
323
324         * bmalloc/Heap.cpp:
325         (bmalloc::Heap::size): Deleted.
326         (bmalloc::Heap::capacity): Deleted.
327         * bmalloc/Heap.h:
328         * bmalloc/VMHeap.cpp:
329         (bmalloc::VMHeap::VMHeap):
330         (bmalloc::VMHeap::allocateSmallChunk):
331         (bmalloc::VMHeap::allocateMediumChunk):
332         (bmalloc::VMHeap::allocateLargeChunk):
333         * bmalloc/VMHeap.h:
334         (bmalloc::VMHeap::allocateSmallPage):
335         (bmalloc::VMHeap::allocateMediumPage):
336         (bmalloc::VMHeap::allocateLargeRange):
337         (bmalloc::VMHeap::deallocateSmallPage):
338         (bmalloc::VMHeap::deallocateMediumPage):
339         (bmalloc::VMHeap::deallocateLargeRange):
340         (bmalloc::VMHeap::size): Deleted.
341         (bmalloc::VMHeap::capacity): Deleted.
342         * bmalloc/bmalloc.h:
343         (bmalloc::api::heapSize): Deleted.
344         (bmalloc::api::heapCapacity): Deleted.
345
346 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
347
348         bmalloc: Allocation should be more precise
349         https://bugs.webkit.org/show_bug.cgi?id=136993
350
351         Reviewed by Gavin Barraclough.
352
353         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
354
355         This patch teaches the allocator to merge adjacent free lines into a
356         single allocatable range. This allows us to shrink the size of an
357         individual line without increasing fragmentation or the rate of allocator
358         slow paths.
359
360         We'll only take more slow paths when available memory is sparse, which
361         is exactly when it's worth it. When available memory is dense, we'll
362         take fewer slow paths.
363
364         * bmalloc.xcodeproj/project.pbxproj:
365         * bmalloc/Algorithm.h:
366         (bmalloc::divideRoundingUp):
367
368         * bmalloc/Allocator.cpp:
369         (bmalloc::Allocator::Allocator): Updated for interface changes.
370
371         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
372         Now that we merge lines, it's not convenient to scavenge by line.
373
374         (bmalloc::Allocator::allocateSmallBumpRange):
375         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
376         instead of individual lines.
377
378         (bmalloc::Allocator::allocateSlowCase):
379         (bmalloc::Allocator::allocateSmallLine): Deleted.
380         (bmalloc::Allocator::allocateMediumLine): Deleted.
381         (bmalloc::Allocator::allocateMedium): Deleted.
382         * bmalloc/Allocator.h:
383         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
384         into the standard fast path with small allocations. Since a BumpAllocator
385         just allocates out of an arbitrary range, it doesn't need to distinguish
386         between small and medium lines.
387
388         * bmalloc/BumpAllocator.h:
389         (bmalloc::BumpAllocator::size):
390         (bmalloc::BumpAllocator::BumpAllocator):
391         (bmalloc::BumpAllocator::init):
392         (bmalloc::BumpAllocator::refill):
393         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
394         anymore: the heap just gives us a pointer and a pre-computed number of
395         objects, and we allocate them.
396
397         * bmalloc/Deallocator.cpp:
398         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
399
400         * bmalloc/Heap.cpp:
401         (bmalloc::Heap::Heap):
402         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
403         detailing where all objects will lie in memory. After we merge two lines,
404         we might allocate an object that spans from one line to the next. This
405         metadata details which bits of memory overlap in that way, and how they
406         overlap.
407
408         (bmalloc::Heap::refillSmallBumpRangeCache):
409         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
410         and merge adjacent free lines into BumpRanges.
411
412         (bmalloc::Heap::allocateSmallPage):
413         (bmalloc::Heap::allocateMediumPage):
414         (bmalloc::Heap::deallocateSmallLine):
415         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
416         since we scan for free memory a page at a time.
417
418         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
419         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
420         fast path.
421
422         * bmalloc/Heap.h:
423         (bmalloc::Heap::derefSmallLine):
424         (bmalloc::Heap::derefMediumLine):
425         (bmalloc::Heap::deallocateSmallLine): Deleted.
426         (bmalloc::Heap::allocateSmallLine): Deleted.
427         (bmalloc::Heap::deallocateMediumLine): Deleted.
428         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
429
430         * bmalloc/Line.h:
431         (bmalloc::Line<Traits>::ref):
432         (bmalloc::Line<Traits>::deref):
433         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
434         anymore, since we only ever deref by 1 now.
435
436         * bmalloc/MediumAllocator.h:
437         (bmalloc::MediumAllocator::isNull): Deleted.
438         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
439         (bmalloc::MediumAllocator::line): Deleted.
440         (bmalloc::MediumAllocator::allocate): Deleted.
441         (bmalloc::MediumAllocator::derefCount): Deleted.
442         (bmalloc::MediumAllocator::refill): Deleted.
443         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
444         been dead for a while, since it doesn't build anymore with this patch.
445
446         * bmalloc/Page.h:
447         (bmalloc::Page::sizeClass):
448         (bmalloc::Page::setSizeClass):
449         (bmalloc::Page::smallSizeClass): Deleted.
450         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
451         to sizeClass, since we use it for medium sizes too.
452
453         * bmalloc/Sizes.h:
454         (bmalloc::Sizes::sizeClass):
455         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
456
457         (bmalloc::Sizes::smallSizeClassFor): Deleted.
458         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
459
460         * bmalloc/bmalloc.h:
461         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
462         can be a bit more precise. It also has to be, since we can't guarantee
463         that an object ends at the end of a line anymore.
464
465 2014-09-19  Daniel Bates  <dabates@apple.com>
466
467         Always assume internal SDK when building configuration Production
468         https://bugs.webkit.org/show_bug.cgi?id=136925
469         <rdar://problem/18362399>
470
471         Reviewed by Dan Bernstein.
472
473         * Configurations/Base.xcconfig:
474
475 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
476
477         bmalloc: moved line caches from the deallocator to the allocator
478         https://bugs.webkit.org/show_bug.cgi?id=136868
479
480         Reviewed by Gavin Barraclough.
481
482         I did this mostly as a simplification, to make it easier to change the
483         allocation strategy.
484
485         No throughput change on MallocBench. Saves about 50kB.
486
487         Since the deallocator needs to lock the heap when freeing lines anyway,
488         there isn't much benefit to giving the deallocator a local cache of
489         deallocated lines.
490
491         We still give the allocator a local cache of lines because that does
492         reduce the frequency at which it needs to lock the heap in order to
493         acquire more lines.
494
495         * bmalloc/Allocator.cpp:
496         (bmalloc::Allocator::scavenge):
497         (bmalloc::Allocator::allocateSmallLine):
498         (bmalloc::Allocator::allocateMediumLine):
499         (bmalloc::Allocator::allocateMedium):
500         (bmalloc::Allocator::allocateSlowCase):
501         * bmalloc/Allocator.h:
502         * bmalloc/Deallocator.cpp:
503         (bmalloc::Deallocator::Deallocator):
504         (bmalloc::Deallocator::scavenge):
505         (bmalloc::Deallocator::processObjectLog):
506         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
507         (bmalloc::Deallocator::allocateSmallLine): Deleted.
508         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
509         (bmalloc::Deallocator::allocateMediumLine): Deleted.
510         * bmalloc/Deallocator.h:
511
512         * bmalloc/Sizes.h:
513         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
514         exactly one page in size. That's about what we were shooting for anyway,
515         and it may make it easier to switch to per-page allocation in future.
516
517 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
518
519         bmalloc: allocate small and medium objects using the same bump pointer class
520         https://bugs.webkit.org/show_bug.cgi?id=136843
521
522         Reviewed by Gavin Barraclough.
523
524         4% speedup on MallocBench.
525
526         Now that medium-sized objects have dedicated per-size allocators, they
527         don't need to use an arbitrary bump pointer allocator. This means that
528         every allocator knows how many objects it will allocate from the start,
529         and we don't need a post-processing step to adjust refcounts based on
530         real allocation count.
531
532         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
533         since it's used for small and medium objects now.
534
535         * bmalloc/Allocator.cpp:
536         (bmalloc::Allocator::Allocator): Updated to use new interface.
537         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
538         to make sure that we finish allocating all the objects in it.
539
540         (bmalloc::Allocator::allocateMedium):
541         (bmalloc::Allocator::allocateSlowCase):
542         (bmalloc::Allocator::retire): Deleted.
543         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
544         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
545         * bmalloc/Allocator.h:
546         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
547         used to post-process an allocator based on how many objects it allocated.
548
549         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
550         (bmalloc::BumpAllocator::BumpAllocator):
551         (bmalloc::BumpAllocator::init):
552         (bmalloc::BumpAllocator::line):
553         (bmalloc::BumpAllocator::validate):
554         (bmalloc::BumpAllocator::allocate):
555         (bmalloc::BumpAllocator::refill):
556         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
557         about the kinds of lines they allocate into. In some cases, the line
558         type must be provided as a template parameter by the caller.
559
560         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
561         (bmalloc::SmallAllocator::line): Deleted.
562         (bmalloc::SmallAllocator::allocate): Deleted.
563         (bmalloc::SmallAllocator::objectCount): Deleted.
564         (bmalloc::SmallAllocator::derefCount): Deleted.
565         (bmalloc::SmallAllocator::refill): Deleted.
566         (bmalloc::SmallAllocator::clear): Deleted.
567
568         * bmalloc/ObjectType.h:
569         (bmalloc::isMedium):
570
571         * bmalloc/SmallAllocator.h:
572         (bmalloc::SmallAllocator::isNull): Deleted.
573         (bmalloc::SmallAllocator::canAllocate): Deleted.
574         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
575         (bmalloc::SmallAllocator::line): Deleted.
576         (bmalloc::SmallAllocator::allocate): Deleted.
577         (bmalloc::SmallAllocator::objectCount): Deleted.
578         (bmalloc::SmallAllocator::derefCount): Deleted.
579         (bmalloc::SmallAllocator::refill): Deleted.
580         (bmalloc::SmallAllocator::clear): Deleted.
581
582 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
583
584         Fixed a goof in bmalloc Vector sizing
585         https://bugs.webkit.org/show_bug.cgi?id=136795
586
587         Reviewed by Gavin Barraclough and Sam Weinig.
588
589         We want our minimum vector to be page-sized since the OS will give us
590         a page no matter what -- but we want that many bytes, and not enough
591         bytes to store that many elements.
592
593         * bmalloc/Vector.h: Math is hard.
594
595 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
596
597         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
598         https://bugs.webkit.org/show_bug.cgi?id=136693
599
600         Reviewed by Gavin Barraclough.
601
602         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
603
604         No throughput change.
605
606         We keep an array of medium allocators, just like our array of small
607         allocators.
608
609         In future, we can simplify the allocation fast path by merging the small
610         and medium allocator arrays. For now, this is the simplest change that
611         gets the win.
612
613         * bmalloc/Allocator.cpp:
614         (bmalloc::Allocator::Allocator):
615         (bmalloc::Allocator::scavenge):
616         (bmalloc::Allocator::allocateMedium):
617         * bmalloc/Allocator.h:
618         * bmalloc/Sizes.h:
619         (bmalloc::Sizes::mediumSizeClassFor):
620
621 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
622
623         Reviewed by Sam Weinig.
624
625         Renamed log => retire for clarity.
626
627         * bmalloc/Allocator.cpp:
628         (bmalloc::Allocator::scavenge):
629         (bmalloc::Allocator::retire):
630         (bmalloc::Allocator::allocateMedium):
631         (bmalloc::Allocator::allocateSlowCase):
632         (bmalloc::Allocator::log): Deleted.
633         * bmalloc/Allocator.h:
634
635 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
636
637         bmalloc: eager scavenge leaves behind a bogus allocator
638         https://bugs.webkit.org/show_bug.cgi?id=136743
639
640         Reviewed by Sam Weinig.
641
642         Be sure to clear the allocator after logging it in the eager scavenge
643         case, so that we don't later try to allocate out of the lines that we
644         have thrown away.
645
646         We didn't need to do this previously because scavenge would only happen
647         at thread exit time, after which no further allocation from the per-thread
648         cache would take place.
649
650         * bmalloc/Allocator.cpp:
651         (bmalloc::Allocator::scavenge):
652         * bmalloc/MediumAllocator.h:
653         (bmalloc::MediumAllocator::clear):
654         * bmalloc/SmallAllocator.h:
655         (bmalloc::SmallAllocator::clear):
656
657 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
658
659         bmalloc should honor the FastMalloc statistics API
660         https://bugs.webkit.org/show_bug.cgi?id=136592
661
662         Reviewed by Gavin Barraclough.
663
664         We do this by tracking "size" and "capacity" in the VM heap.
665
666         The VM heap's "capacity" is all the VM we ever allocated.
667
668         The VM heap's "size" the subset of VM currently held onto by the
669         VM heap (and therefore not in use by the regular heap).
670
671         Somewhat ironically, reducing the process's memory footprint, increases
672         the size of the VM heap, since the VM heap holds the pages that are
673         purely virtual and not physical.
674
675         * bmalloc/Heap.cpp:
676         (bmalloc::Heap::size):
677         (bmalloc::Heap::capacity):
678         * bmalloc/Heap.h:
679         * bmalloc/VMHeap.cpp:
680         (bmalloc::VMHeap::VMHeap):
681         (bmalloc::VMHeap::allocateSmallChunk):
682         (bmalloc::VMHeap::allocateMediumChunk):
683         (bmalloc::VMHeap::allocateLargeChunk):
684         * bmalloc/VMHeap.h:
685         (bmalloc::VMHeap::size):
686         (bmalloc::VMHeap::capacity):
687         (bmalloc::VMHeap::allocateSmallPage):
688         (bmalloc::VMHeap::allocateMediumPage):
689         (bmalloc::VMHeap::allocateLargeRange):
690         (bmalloc::VMHeap::deallocateSmallPage):
691         (bmalloc::VMHeap::deallocateMediumPage):
692         (bmalloc::VMHeap::deallocateLargeRange):
693         * bmalloc/bmalloc.h:
694         (bmalloc::api::heapSize):
695         (bmalloc::api::heapCapacity):
696
697 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
698
699         bmalloc crashes on the EWS bots (due to bad large object allocation)
700         https://bugs.webkit.org/show_bug.cgi?id=136469
701
702         Reviewed by Andreas Kling.
703
704         It's possible to convince bmalloc to perform a bad large object allocation,
705         through these steps:
706
707         (1) Insert object A into freelist F0.
708
709         (2) Split, merge and split again A's neighbors such that object B is
710         inserted into freelist F0, with boundary tag and size equal to object A,
711         but pointer not completely equal to object A. Put object B at the head of F0.
712
713         (3) Allocate some other object from F0, swapping its position in the
714         freelist with object B, such that object A is now ahead of object B.
715
716         --> Now, the next allocation for size A/B will allocate object A, which
717         has a slightly wrong idea about where the object actually begins.
718         Immediately, you'll corrupt a little memory, and over time, you'll also
719         corrupt boundary tag metadata.
720
721         The solution is to store the begin pointer in the boundary tag. Luckily,
722         this doesn't make the tag any bigger, and it's not a noticeable slowdown
723         on MallocBench.
724
725         * bmalloc/Algorithm.h:
726         (bmalloc::rightShift):
727         * bmalloc/BeginTag.h:
728         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
729         validate the start pointer when popping off the free list. Through a
730         very uncommon set of steps, it is possible to have an item in the free
731         list that is valid by all accounts except for its start pointer.
732
733         * bmalloc/BoundaryTag.h:
734         (bmalloc::BoundaryTag::compactBegin):
735         (bmalloc::BoundaryTag::setRange):
736         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
737         start pointer. We don't need the whole pointer -- just the offset, in
738         largeAlignment increments, into the relevant boundary tag bucket.
739
740         * bmalloc/BoundaryTagInlines.h:
741         (bmalloc::validateNext):
742         (bmalloc::BoundaryTag::init):
743         (bmalloc::BoundaryTag::mergeLarge):
744         (bmalloc::BoundaryTag::splitLarge):
745         * bmalloc/SegregatedFreeList.cpp:
746         (bmalloc::SegregatedFreeList::insert):
747         (bmalloc::SegregatedFreeList::takeGreedy):
748         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
749         the size when establishing a boundary tag, as required by the new
750         interface.
751
752         * bmalloc/Sizes.h:
753
754 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
755
756         Fixed a bmalloc crash seen on the EWS bot
757         https://bugs.webkit.org/show_bug.cgi?id=135955
758
759         Reviewed by Andreas Kling.
760
761         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
762         input buffer is a malloc region, that region will get marked Copy-On-Write
763         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
764         on older OS X's. In 10.10, they still fail, but they do not return
765         EINVAL.
766
767         So, we can only ASSERT that our syscalls succeed starting with 10.10.
768
769 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
770
771         Fixed the bmalloc build
772         https://bugs.webkit.org/show_bug.cgi?id=135953
773
774         Reviewed by Andreas Kling.
775
776         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
777         These headers are used, so they must be available outside the project.
778
779 2014-08-13  Daniel Bates  <dabates@apple.com>
780
781         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
782         (https://bugs.webkit.org/show_bug.cgi?id=135895)
783
784         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
785         implementation of PerThread<T>::getFastCase().
786
787         * bmalloc/PerThread.h:
788         (bmalloc::PerThread<T>::getFastCase):
789
790 2014-08-13  Daniel Bates  <dabates@apple.com>
791
792         Make bmalloc::PerThread work without C++ thread local storage
793         https://bugs.webkit.org/show_bug.cgi?id=135895
794
795         Reviewed by Geoffrey Garen.
796
797         Implement support for building bmalloc without C++ thread local storage.
798
799         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
800         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
801         to determine whether the compiler supports C++ thread local storage.
802         * bmalloc/PerThread.h:
803         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
804         without C++ thread local storage.
805         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
806         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
807         PerThreadStorage::initSharedKeyIfNeeded().
808         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
809         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
810         ensure that the shared key has been initialized.
811         (_pthread_setspecific_direct): Deleted.
812         (_pthread_getspecific_direct): Deleted.
813
814 2014-08-13  Daniel Bates  <dabates@apple.com>
815
816         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
817         https://bugs.webkit.org/show_bug.cgi?id=135848
818
819         Reviewed by Geoffrey Garen.
820
821         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
822         when building for the iOS Simulator.
823         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
824         using the public SDK.
825         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
826         using the public SDK.
827         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
828         using the public SDK.
829
830 2014-08-12  Daniel Bates  <dabates@apple.com>
831
832         BPLATFORM(IOS) always evaluates to false
833         https://bugs.webkit.org/show_bug.cgi?id=135843
834
835         Reviewed by Geoffrey Garen.
836
837         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
838         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
839         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
840         building for iOS.
841
842         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
843         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
844
845         * bmalloc/BPlatform.h:
846
847 2014-08-11  Andy Estes  <aestes@apple.com>
848
849         [iOS] Get rid of iOS.xcconfig
850         https://bugs.webkit.org/show_bug.cgi?id=135809
851
852         Reviewed by Joseph Pecoraro.
853
854         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
855
856         * Configurations/Base.xcconfig:
857         * Configurations/iOS.xcconfig: Removed.
858         * bmalloc.xcodeproj/project.pbxproj:
859
860 2014-05-01  Dan Bernstein  <mitz@apple.com>
861
862         Fixed production builds for the iOS Simulator.
863         <rdar://problem/16792221>
864
865         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
866         PRIVATE_HEADERS_FOLDER_PATH when installing.
867
868 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
869
870         bmalloc: Segregate pages by objects size
871         https://bugs.webkit.org/show_bug.cgi?id=131909
872
873         Reviewed by Andreas Kling.
874
875         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
876
877         * bmalloc/Allocator.cpp:
878         (bmalloc::Allocator::allocateSlowCase):
879         * bmalloc/Allocator.h:
880         (bmalloc::Allocator::allocateFastCase):
881         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
882         function for size class calculation.
883
884         * bmalloc/Deallocator.cpp:
885         (bmalloc::Deallocator::Deallocator):
886         (bmalloc::Deallocator::scavenge):
887         (bmalloc::Deallocator::deallocateSmallLine):
888         (bmalloc::Deallocator::allocateSmallLine):
889         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
890         cache can't be shared anymore.
891
892         * bmalloc/Heap.cpp:
893         (bmalloc::Heap::allocateSmallLineSlowCase):
894         * bmalloc/Heap.h:
895         (bmalloc::Heap::deallocateSmallLine): Ditto.
896
897         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
898         page refcount when allocating a line because we might have deallocated
899         the page and the recycled it for another size class.
900
901         (bmalloc::Heap::deallocateMediumLine):
902         (bmalloc::Heap::allocateMediumLine):
903         * bmalloc/Line.h:
904         (bmalloc::Line::refCount):
905         * bmalloc/Page.h:
906         (bmalloc::Page::refCount):
907         (bmalloc::Page::smallSizeClass):
908         (bmalloc::Page::setSmallSizeClass):
909         (bmalloc::Page<Traits>::refCount): Deleted.
910         * bmalloc/Sizes.h:
911         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
912         an index into an array from a size.
913
914 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
915
916         bmalloc: Improved alignment in LargeChunk
917         https://bugs.webkit.org/show_bug.cgi?id=131895
918
919         Reviewed by Andreas Kling.
920
921         * bmalloc/Chunk.h:
922         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
923         Technically, the previous alignment was harmless, but I would prefer,
924         dear reader, not to have to explain the interlocking set of
925         circumstances that made it so.
926
927 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
928
929         Rolled out r167502 because it caused a crash on the facebook benchmark.
930
931         Unreviewed.
932
933             bmalloc: Added an XSmall line size
934             https://bugs.webkit.org/show_bug.cgi?id=131851
935
936             Reviewed by Sam Weinig.
937
938 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
939
940         bmalloc: Mutex should be harder to use wrong
941         https://bugs.webkit.org/show_bug.cgi?id=131879
942
943         Reviewed by Andreas Kling.
944
945         Mutex now has a proper constructor, so you can't deadlock by forgetting
946         to initialize it.
947
948         * bmalloc.xcodeproj/project.pbxproj:
949         * bmalloc/Allocator.cpp:
950         (bmalloc::Allocator::processXSmallAllocatorLog):
951         (bmalloc::Allocator::processSmallAllocatorLog):
952         (bmalloc::Allocator::processMediumAllocatorLog):
953         (bmalloc::Allocator::allocateLarge):
954         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
955         since the Heap mutex is a static.
956
957         * bmalloc/AsyncTask.h:
958         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
959         need for explicit initialization anymore.
960
961         * bmalloc/Deallocator.cpp:
962         (bmalloc::Deallocator::scavenge):
963         (bmalloc::Deallocator::deallocateLarge):
964         (bmalloc::Deallocator::deallocateXLarge):
965         (bmalloc::Deallocator::processObjectLog):
966         (bmalloc::Deallocator::deallocateSmallLine):
967         (bmalloc::Deallocator::deallocateXSmallLine):
968         (bmalloc::Deallocator::allocateSmallLine):
969         (bmalloc::Deallocator::allocateXSmallLine):
970         (bmalloc::Deallocator::deallocateMediumLine):
971         (bmalloc::Deallocator::allocateMediumLine):
972         * bmalloc/Deallocator.h:
973         * bmalloc/Heap.cpp:
974         (bmalloc::sleep):
975         (bmalloc::Heap::Heap):
976         (bmalloc::Heap::concurrentScavenge):
977         (bmalloc::Heap::scavenge):
978         (bmalloc::Heap::scavengeSmallPages):
979         (bmalloc::Heap::scavengeXSmallPages):
980         (bmalloc::Heap::scavengeMediumPages):
981         (bmalloc::Heap::scavengeLargeRanges):
982         (bmalloc::Heap::allocateXSmallLineSlowCase):
983         (bmalloc::Heap::allocateSmallLineSlowCase):
984         (bmalloc::Heap::allocateMediumLineSlowCase):
985         (bmalloc::Heap::allocateXLarge):
986         (bmalloc::Heap::deallocateXLarge):
987         (bmalloc::Heap::allocateLarge):
988         (bmalloc::Heap::deallocateLarge):
989         * bmalloc/Heap.h:
990         (bmalloc::Heap::deallocateXSmallLine):
991         (bmalloc::Heap::allocateXSmallLine):
992         (bmalloc::Heap::deallocateSmallLine):
993         (bmalloc::Heap::allocateSmallLine):
994         (bmalloc::Heap::deallocateMediumLine):
995         (bmalloc::Heap::allocateMediumLine):
996         * bmalloc/Line.h:
997         (bmalloc::Line<Traits>::deref):
998         * bmalloc/Mutex.cpp: Removed.
999         * bmalloc/Mutex.h:
1000         (bmalloc::Mutex::Mutex):
1001         (bmalloc::Mutex::init): Deleted.
1002         (bmalloc::Mutex::try_lock): Deleted.
1003         (bmalloc::Mutex::lock): Deleted.
1004         (bmalloc::Mutex::unlock): Deleted.
1005         * bmalloc/Page.h:
1006         (bmalloc::Page<Traits>::ref):
1007         (bmalloc::Page<Traits>::deref):
1008         (bmalloc::Page<Traits>::refCount):
1009         * bmalloc/PerProcess.h:
1010         (bmalloc::PerProcess::mutex):
1011         (bmalloc::PerProcess<T>::getSlowCase):
1012         * bmalloc/StaticMutex.cpp: Added.
1013         (bmalloc::StaticMutex::lockSlowCase):
1014         * bmalloc/StaticMutex.h: Added.
1015         (bmalloc::StaticMutex::init):
1016         (bmalloc::StaticMutex::try_lock):
1017         (bmalloc::StaticMutex::lock):
1018         (bmalloc::StaticMutex::unlock):
1019         * bmalloc/VMHeap.h:
1020         (bmalloc::VMHeap::deallocateXSmallPage):
1021         (bmalloc::VMHeap::deallocateSmallPage):
1022         (bmalloc::VMHeap::deallocateMediumPage):
1023         (bmalloc::VMHeap::deallocateLargeRange):
1024         * bmalloc/bmalloc.h:
1025         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1026         since the Heap mutex is a static.
1027
1028 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1029
1030         bmalloc: AsyncTask should use Mutex instead of std::mutex
1031         https://bugs.webkit.org/show_bug.cgi?id=131865
1032
1033         Reviewed by Gavin Barraclough.
1034
1035         std::mutex is so slow that it makes parallelizing simple tasks through
1036         AsyncTask a net regression. Mutex fixes this.
1037
1038         * bmalloc/AsyncTask.h:
1039         (bmalloc::Function>::AsyncTask):
1040         (bmalloc::Function>::join):
1041         (bmalloc::Function>::runSlowCase):
1042         (bmalloc::Function>::entryPoint):
1043         * bmalloc/Mutex.h:
1044         (bmalloc::Mutex::init):
1045
1046 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1047
1048         bmalloc: Added an XSmall line size
1049         https://bugs.webkit.org/show_bug.cgi?id=131851
1050
1051         Reviewed by Sam Weinig.
1052
1053         Reduces malloc footprint on Membuster recordings by 10%.
1054
1055         This is a throughput regression, but we're still way ahead of TCMalloc.
1056         I have some ideas for how to recover the regression -- but I wanted to
1057         get this win in first.
1058
1059         Full set of benchmark results:
1060
1061                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1062                                                                                                 
1063                                                                nopatch                      patch                                Δ
1064                 Peak Memory:
1065                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1066                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1067                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1068
1069                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1070                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1071                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1072
1073                 Memory at End:
1074                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1075                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1076                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1077
1078                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1079                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1080                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1081
1082                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1083                                                                                                 
1084                                                    nopatch                patch                         Δ
1085                 Execution Time:
1086                     churn                            127ms                151ms            ! 1.19x slower
1087                     list_allocate                    130ms                164ms            ! 1.26x slower
1088                     tree_allocate                    109ms                127ms            ! 1.17x slower
1089                     tree_churn                       115ms                120ms            ! 1.04x slower
1090                     facebook                         240ms                259ms            ! 1.08x slower
1091                     fragment                          91ms                131ms            ! 1.44x slower
1092                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1093                     message_one                      260ms                259ms             ^ 1.0x faster
1094                     message_many                     149ms                154ms            ! 1.03x slower
1095                     medium                           194ms                248ms            ! 1.28x slower
1096                     big                              157ms                160ms            ! 1.02x slower
1097
1098                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1099                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1100                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1101
1102                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1103                                                                                                 
1104                                                                        nopatch                          patch                                     Δ
1105                 Execution Time:
1106                     churn                                                126ms                          148ms                        ! 1.17x slower
1107                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1108                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1109                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1110                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1111                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1112                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1113                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1114                     facebook                                             240ms                          276ms                        ! 1.15x slower
1115                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1116                     fragment                                              92ms                          130ms                        ! 1.41x slower
1117                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1118                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1119                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1120                     message_one                                          260ms                          260ms                                      
1121                     message_many                                         170ms                          238ms                         ! 1.4x slower
1122                     medium                                               185ms                          250ms                        ! 1.35x slower
1123                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1124                     big                                                  150ms                          169ms                        ! 1.13x slower
1125                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1126
1127                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1128                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1129                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1130
1131                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1132
1133                                                                     TC                      patch                                Δ
1134                 Peak Memory:
1135                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1136                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1137                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1138
1139                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1140                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1141                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1142
1143                 Memory at End:
1144                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
1145                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
1146                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
1147
1148                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
1149                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
1150                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
1151
1152                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1153                                                                                                 
1154                                                         TC                patch                         Δ
1155                 Execution Time:
1156                     churn                            416ms                148ms            ^ 2.81x faster
1157                     list_allocate                    463ms                164ms            ^ 2.82x faster
1158                     tree_allocate                    292ms                127ms             ^ 2.3x faster
1159                     tree_churn                       157ms                120ms            ^ 1.31x faster
1160                     facebook                         327ms                276ms            ^ 1.18x faster
1161                     fragment                         335ms                129ms             ^ 2.6x faster
1162                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
1163                     message_one                      386ms                258ms             ^ 1.5x faster
1164                     message_many                     410ms                154ms            ^ 2.66x faster
1165                     medium                           391ms                245ms             ^ 1.6x faster
1166                     big                              261ms                167ms            ^ 1.56x faster
1167
1168                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
1169                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
1170                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
1171
1172         * bmalloc.xcodeproj/project.pbxproj:
1173         * bmalloc/Allocator.cpp:
1174         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
1175         index corresponds with its size. Instead, use the size selection function
1176         explicitly. Now that we have XSmall, some small allocator entries are
1177         unused.
1178
1179         (bmalloc::Allocator::scavenge):
1180         (bmalloc::Allocator::log):
1181         (bmalloc::Allocator::processXSmallAllocatorLog):
1182         (bmalloc::Allocator::allocateSlowCase):
1183         * bmalloc/Allocator.h:
1184         (bmalloc::Allocator::xSmallAllocatorFor):
1185         (bmalloc::Allocator::allocateFastCase):
1186         * bmalloc/Chunk.h:
1187         * bmalloc/Deallocator.cpp:
1188         (bmalloc::Deallocator::scavenge):
1189         (bmalloc::Deallocator::processObjectLog):
1190         (bmalloc::Deallocator::deallocateSlowCase):
1191         (bmalloc::Deallocator::deallocateXSmallLine):
1192         (bmalloc::Deallocator::allocateXSmallLine):
1193         * bmalloc/Deallocator.h:
1194         (bmalloc::Deallocator::deallocateFastCase):
1195         * bmalloc/Heap.cpp:
1196         (bmalloc::Heap::scavenge):
1197         (bmalloc::Heap::scavengeXSmallPages):
1198         (bmalloc::Heap::allocateXSmallLineSlowCase):
1199         * bmalloc/Heap.h:
1200         (bmalloc::Heap::deallocateXSmallLine):
1201         (bmalloc::Heap::allocateXSmallLine):
1202         * bmalloc/LargeChunk.h:
1203         (bmalloc::LargeChunk::get):
1204         (bmalloc::LargeChunk::endTag):
1205         * bmalloc/Line.h:
1206         * bmalloc/MediumAllocator.h:
1207         (bmalloc::MediumAllocator::allocate):
1208         (bmalloc::MediumAllocator::refill):
1209         * bmalloc/ObjectType.cpp:
1210         (bmalloc::objectType):
1211         * bmalloc/ObjectType.h:
1212         (bmalloc::isXSmall):
1213         (bmalloc::isSmall):
1214         (bmalloc::isMedium):
1215         (bmalloc::isLarge):
1216         (bmalloc::isSmallOrMedium): Deleted.
1217         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1218         handling small objects. There's probably a reasonable way to share this
1219         code in the future -- I'll look into that once it's stopped changing.
1220
1221         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1222         is the main reason things got slower.
1223
1224         * bmalloc/SmallAllocator.h:
1225         (bmalloc::SmallAllocator::allocate):
1226         * bmalloc/SmallTraits.h:
1227         * bmalloc/VMHeap.cpp:
1228         (bmalloc::VMHeap::allocateXSmallChunk):
1229         * bmalloc/VMHeap.h:
1230         (bmalloc::VMHeap::allocateXSmallPage):
1231         (bmalloc::VMHeap::deallocateXSmallPage):
1232         * bmalloc/XSmallAllocator.h: Added.
1233         (bmalloc::XSmallAllocator::isNull):
1234         (bmalloc::XSmallAllocator::canAllocate):
1235         (bmalloc::XSmallAllocator::XSmallAllocator):
1236         (bmalloc::XSmallAllocator::line):
1237         (bmalloc::XSmallAllocator::allocate):
1238         (bmalloc::XSmallAllocator::objectCount):
1239         (bmalloc::XSmallAllocator::derefCount):
1240         (bmalloc::XSmallAllocator::refill):
1241         * bmalloc/XSmallChunk.h: Added.
1242         * bmalloc/XSmallLine.h: Added.
1243         * bmalloc/XSmallPage.h: Added.
1244         * bmalloc/XSmallTraits.h: Added.
1245         * bmalloc/bmalloc.h:
1246         (bmalloc::api::realloc): Boiler-plate copy, as above.
1247
1248 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1249
1250         MallocBench should scavenge explicitly instead of waiting
1251         https://bugs.webkit.org/show_bug.cgi?id=131661
1252
1253         Reviewed by Andreas Kling.
1254
1255         Added explicit scavenge support to bmalloc. This isn't a memory win,
1256         since bmalloc's per-thread cache is so small. But it makes testing
1257         simpler.
1258
1259         * bmalloc/Allocator.cpp:
1260         (bmalloc::Allocator::~Allocator):
1261         (bmalloc::Allocator::scavenge):
1262         * bmalloc/Allocator.h:
1263         * bmalloc/Cache.cpp:
1264         (bmalloc::Cache::operator new):
1265         (bmalloc::Cache::operator delete):
1266         (bmalloc::Cache::Cache):
1267         (bmalloc::Cache::scavenge):
1268         * bmalloc/Cache.h:
1269         * bmalloc/Deallocator.cpp:
1270         (bmalloc::Deallocator::~Deallocator):
1271         (bmalloc::Deallocator::scavenge):
1272         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1273         functions, for reuse.
1274
1275         * bmalloc/Heap.cpp:
1276         (bmalloc::sleep):
1277         (bmalloc::Heap::concurrentScavenge):
1278         (bmalloc::Heap::scavenge):
1279         (bmalloc::Heap::scavengeSmallPages):
1280         (bmalloc::Heap::scavengeMediumPages):
1281         (bmalloc::Heap::scavengeLargeRanges):
1282         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1283         scavenging -- in response to a benchmark or a low memory warning --
1284         wants to complete as soon as possible, so its sleep duration is 0.
1285
1286         * bmalloc/bmalloc.h:
1287         (bmalloc::api::scavenge):
1288         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1289
1290 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1291
1292         Use 4kB pages on Mac
1293         https://bugs.webkit.org/show_bug.cgi?id=131658
1294
1295         Reviewed by Sam Weinig.
1296
1297         This reduces memory use a lot on Membuster:
1298
1299                                                                   base                      patch                                Δ
1300                 Execution Time:
1301                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1302                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1303                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1304
1305                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1306                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1307                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1308
1309                 Peak Memory:
1310                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1311                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1312                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1313
1314                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1315                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1316                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1317
1318                 Memory at End:
1319                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1320                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1321                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1322
1323                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1324                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1325                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1326
1327
1328         * bmalloc.xcodeproj/project.pbxproj:
1329         * bmalloc/BPlatform.h: Added.
1330         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1331         is 4kB on Mac.
1332
1333 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1334
1335         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1336
1337         * bmalloc.xcodeproj: Modified property svn:ignore.
1338
1339 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1340
1341         Fixed some mbmalloc exports
1342         https://bugs.webkit.org/show_bug.cgi?id=131599
1343
1344         Reviewed by Ryosuke Niwa.
1345
1346         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1347         of the project, so we can call them from API.
1348
1349         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1350         visibility, so they show up as exported in the .dylib.
1351
1352 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1353
1354         Put bmalloc headers in the right place
1355         https://bugs.webkit.org/show_bug.cgi?id=131464
1356
1357         Reviewed by Mark Rowe.
1358
1359         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1360         specify that we don't just want to dump all of our generically-named
1361         headers into /usr/local/include.
1362
1363 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1364
1365         Made bmalloc more #include friendly
1366         https://bugs.webkit.org/show_bug.cgi?id=131386
1367
1368         Reviewed by Andreas Kling.
1369
1370         Marked a bunch of headers private so they can be used from client code
1371         that #includes bmalloc.h.
1372
1373         Renamed ASSERT macros to BASSERT. This matches their header, which already
1374         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1375
1376         * bmalloc.xcodeproj/project.pbxproj:
1377         * bmalloc/Allocator.cpp:
1378         (bmalloc::Allocator::allocateSlowCase):
1379         * bmalloc/AsyncTask.h:
1380         (bmalloc::Function>::runSlowCase):
1381         * bmalloc/BAssert.h:
1382         * bmalloc/BoundaryTag.h:
1383         (bmalloc::BoundaryTag::setSize):
1384         * bmalloc/BoundaryTagInlines.h:
1385         (bmalloc::validate):
1386         (bmalloc::BoundaryTag::init):
1387         (bmalloc::BoundaryTag::deallocate):
1388         (bmalloc::BoundaryTag::splitLarge):
1389         (bmalloc::BoundaryTag::allocate):
1390         * bmalloc/Chunk.h:
1391         * bmalloc/Deallocator.cpp:
1392         (bmalloc::Deallocator::processObjectLog):
1393         (bmalloc::Deallocator::deallocateSlowCase):
1394         * bmalloc/Deallocator.h:
1395         (bmalloc::Deallocator::deallocateFastCase):
1396         * bmalloc/FixedVector.h:
1397         (bmalloc::Capacity>::operator):
1398         (bmalloc::Capacity>::push):
1399         (bmalloc::Capacity>::pop):
1400         (bmalloc::Capacity>::shrink):
1401         * bmalloc/Heap.cpp:
1402         (bmalloc::Heap::allocateLarge):
1403         * bmalloc/LargeChunk.h:
1404         (bmalloc::LargeChunk::get):
1405         (bmalloc::LargeChunk::endTag):
1406         * bmalloc/Line.h:
1407         (bmalloc::Line<Traits>::concurrentRef):
1408         (bmalloc::Line<Traits>::deref):
1409         * bmalloc/MediumAllocator.h:
1410         (bmalloc::MediumAllocator::allocate):
1411         * bmalloc/ObjectType.h:
1412         (bmalloc::isSmall):
1413         * bmalloc/Page.h:
1414         (bmalloc::Page<Traits>::ref):
1415         (bmalloc::Page<Traits>::deref):
1416         * bmalloc/PerThread.h:
1417         (bmalloc::PerThread<T>::getSlowCase):
1418         * bmalloc/SegregatedFreeList.cpp:
1419         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1420         (bmalloc::SegregatedFreeList::insert):
1421         * bmalloc/SmallAllocator.h:
1422         (bmalloc::SmallAllocator::allocate):
1423         (bmalloc::SmallAllocator::refill):
1424         * bmalloc/Syscall.h:
1425         * bmalloc/VMAllocate.h:
1426         (bmalloc::vmValidate):
1427         (bmalloc::vmAllocate):
1428         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1429         * bmalloc/Vector.h:
1430         (bmalloc::Vector<T>::operator):
1431         (bmalloc::Vector<T>::pop):
1432         (bmalloc::Vector<T>::shrink):
1433         * bmalloc/XLargeChunk.h:
1434         (bmalloc::XLargeChunk::range):
1435         (bmalloc::XLargeChunk::size):
1436
1437 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1438
1439         Removed an unused file.
1440
1441         Unreviewed.
1442
1443         * bmalloc/AsyncTask.cpp: Removed.
1444
1445 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1446
1447         Build bmalloc on Mac
1448         https://bugs.webkit.org/show_bug.cgi?id=131333
1449
1450         Reviewed by Mark Rowe.
1451
1452         * Makefile: Added. For make clients.
1453
1454         These files are required for building any project in WebKit. I copied
1455         them from WTF:
1456         * Configurations: Added.
1457         * Configurations/Base.xcconfig: Added.
1458         * Configurations/DebugRelease.xcconfig: Added.
1459         * Configurations/bmalloc.xcconfig: Added.
1460         * Configurations/iOS.xcconfig: Added.
1461         * Configurations/mbmalloc.xcconfig: Added.
1462
1463         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1464         from here because everything is in .xcconfig files now.
1465
1466         I had to fix a bunch of minor warnings, since they're enabled in our
1467         .xcconfig files:
1468
1469         * bmalloc/AsyncTask.h:
1470         (bmalloc::Function>::AsyncTask):
1471         * bmalloc/BAssert.h:
1472         * bmalloc/BoundaryTagInlines.h:
1473         (bmalloc::validate):
1474         * bmalloc/Heap.cpp:
1475         (bmalloc::Heap::Heap):
1476         (bmalloc::Heap::allocateLarge):
1477         (bmalloc::Heap::deallocateLarge):
1478         * bmalloc/Mutex.h:
1479         (bmalloc::Mutex::Mutex): Deleted.
1480         * bmalloc/VMAllocate.h:
1481         (bmalloc::vmValidate):
1482         * bmalloc/mbmalloc.cpp:
1483
1484 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1485
1486         bmalloc: Fixed a leak in the per-thread cache
1487         https://bugs.webkit.org/show_bug.cgi?id=131330
1488
1489         Reviewed by Andreas Kling.
1490
1491         Remember to deallocate our line caches upon thread exit.
1492
1493         * bmalloc/Deallocator.cpp:
1494         (bmalloc::Deallocator::~Deallocator):
1495
1496 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1497
1498         bmalloc: rolled out the tryLock experiment
1499         https://bugs.webkit.org/show_bug.cgi?id=131328
1500
1501         Reviewed by Andreas Kling.
1502
1503         It wasn't a speedup.
1504
1505         * bmalloc.xcodeproj/project.pbxproj:
1506         * bmalloc/Allocator.cpp:
1507         (bmalloc::Allocator::processSmallAllocatorLog):
1508         (bmalloc::Allocator::processMediumAllocatorLog):
1509         * bmalloc/Deallocator.cpp:
1510         (bmalloc::Deallocator::processObjectLog):
1511         (bmalloc::Deallocator::deallocateSlowCase):
1512         (bmalloc::Deallocator::deallocateSmallLine):
1513         (bmalloc::Deallocator::deallocateMediumLine):
1514         * bmalloc/Deallocator.h:
1515         (bmalloc::Deallocator::deallocateFastCase):
1516         * bmalloc/Heap.h:
1517         (bmalloc::Heap::deallocateSmallLine):
1518         (bmalloc::Heap::deallocateMediumLine):
1519         * bmalloc/Line.h:
1520         (bmalloc::Line<Traits>::deref):
1521         * bmalloc/Page.h:
1522         (bmalloc::Page<Traits>::deref):
1523
1524 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1525
1526         bmalloc
1527         https://bugs.webkit.org/show_bug.cgi?id=131170
1528
1529         Reviewed by Andreas Kling.
1530
1531         Initial commit.
1532
1533         * bmalloc: Added.
1534         * bmalloc.xcodeproj: Added.
1535         * bmalloc.xcodeproj/project.pbxproj: Added.
1536         * bmalloc/Algorithm.h: Added.
1537         (bmalloc::max):
1538         (bmalloc::min):
1539         (bmalloc::mask):
1540         (bmalloc::test):
1541         (bmalloc::roundUpToMultipleOf):
1542         (bmalloc::roundDownToMultipleOf):
1543         (bmalloc::sizeOf):
1544         (bmalloc::bitCount):
1545         (bmalloc::isPowerOfTwo):
1546         * bmalloc/Allocator.cpp: Added.
1547         (bmalloc::Allocator::Allocator):
1548         (bmalloc::Allocator::~Allocator):
1549         (bmalloc::Allocator::log):
1550         (bmalloc::Allocator::processSmallAllocatorLog):
1551         (bmalloc::Allocator::processMediumAllocatorLog):
1552         (bmalloc::Allocator::allocateLarge):
1553         (bmalloc::Allocator::allocateXLarge):
1554         (bmalloc::Allocator::allocateMedium):
1555         (bmalloc::Allocator::allocateSlowCase):
1556         * bmalloc/Allocator.h: Added.
1557         (bmalloc::Allocator::smallAllocatorFor):
1558         (bmalloc::Allocator::allocateFastCase):
1559         (bmalloc::Allocator::allocate):
1560         * bmalloc/AsyncTask.cpp: Added.
1561         (bmalloc::AsyncTask<Function>::runSlowCase):
1562         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
1563         (bmalloc::AsyncTask<Function>::entryPoint):
1564         * bmalloc/AsyncTask.h: Added.
1565         (bmalloc::Function>::AsyncTask):
1566         (bmalloc::Function>::join):
1567         (bmalloc::Function>::run):
1568         (bmalloc::Function>::runSlowCase):
1569         (bmalloc::Function>::pthreadEntryPoint):
1570         (bmalloc::Function>::entryPoint):
1571         * bmalloc/BAssert.h: Added.
1572         * bmalloc/BeginTag.h: Added.
1573         (bmalloc::BeginTag::isInFreeList):
1574         * bmalloc/BoundaryTag.h: Added.
1575         (bmalloc::BoundaryTag::isXLarge):
1576         (bmalloc::BoundaryTag::setXLarge):
1577         (bmalloc::BoundaryTag::isFree):
1578         (bmalloc::BoundaryTag::setFree):
1579         (bmalloc::BoundaryTag::isEnd):
1580         (bmalloc::BoundaryTag::setEnd):
1581         (bmalloc::BoundaryTag::hasPhysicalPages):
1582         (bmalloc::BoundaryTag::setHasPhysicalPages):
1583         (bmalloc::BoundaryTag::isNull):
1584         (bmalloc::BoundaryTag::clear):
1585         (bmalloc::BoundaryTag::size):
1586         (bmalloc::BoundaryTag::setSize):
1587         (bmalloc::BoundaryTag::prev):
1588         (bmalloc::BoundaryTag::next):
1589         * bmalloc/BoundaryTagInlines.h: Added.
1590         (bmalloc::validate):
1591         (bmalloc::validatePrev):
1592         (bmalloc::validateNext):
1593         (bmalloc::BoundaryTag::init):
1594         (bmalloc::BoundaryTag::mergeLargeLeft):
1595         (bmalloc::BoundaryTag::mergeLargeRight):
1596         (bmalloc::BoundaryTag::mergeLarge):
1597         (bmalloc::BoundaryTag::deallocate):
1598         (bmalloc::BoundaryTag::splitLarge):
1599         (bmalloc::BoundaryTag::allocate):
1600         * bmalloc/Cache.cpp: Added.
1601         (bmalloc::Cache::operator new):
1602         (bmalloc::Cache::operator delete):
1603         (bmalloc::Cache::Cache):
1604         (bmalloc::Cache::allocateSlowCase):
1605         (bmalloc::Cache::allocateSlowCaseNullCache):
1606         (bmalloc::Cache::deallocateSlowCase):
1607         (bmalloc::Cache::deallocateSlowCaseNullCache):
1608         * bmalloc/Cache.h: Added.
1609         (bmalloc::Cache::allocator):
1610         (bmalloc::Cache::deallocator):
1611         (bmalloc::Cache::allocateFastCase):
1612         (bmalloc::Cache::deallocateFastCase):
1613         (bmalloc::Cache::allocate):
1614         (bmalloc::Cache::deallocate):
1615         * bmalloc/Chunk.h: Added.
1616         (bmalloc::Chunk::begin):
1617         (bmalloc::Chunk::end):
1618         (bmalloc::Chunk::lines):
1619         (bmalloc::Chunk::pages):
1620         * bmalloc/Deallocator.cpp: Added.
1621         (bmalloc::Deallocator::Deallocator):
1622         (bmalloc::Deallocator::~Deallocator):
1623         (bmalloc::Deallocator::deallocateLarge):
1624         (bmalloc::Deallocator::deallocateXLarge):
1625         (bmalloc::Deallocator::processObjectLog):
1626         (bmalloc::Deallocator::deallocateSlowCase):
1627         (bmalloc::Deallocator::deallocateSmallLine):
1628         (bmalloc::Deallocator::allocateSmallLine):
1629         (bmalloc::Deallocator::deallocateMediumLine):
1630         (bmalloc::Deallocator::allocateMediumLine):
1631         * bmalloc/Deallocator.h: Added.
1632         (bmalloc::Deallocator::deallocateFastCase):
1633         (bmalloc::Deallocator::deallocate):
1634         * bmalloc/EndTag.h: Added.
1635         (bmalloc::EndTag::operator=):
1636         * bmalloc/FixedVector.h: Added.
1637         (bmalloc::FixedVector::begin):
1638         (bmalloc::FixedVector::end):
1639         (bmalloc::FixedVector::size):
1640         (bmalloc::FixedVector::capacity):
1641         (bmalloc::FixedVector::clear):
1642         (bmalloc::FixedVector::isEmpty):
1643         (bmalloc::Capacity>::FixedVector):
1644         (bmalloc::Capacity>::operator):
1645         (bmalloc::Capacity>::push):
1646         (bmalloc::Capacity>::pop):
1647         (bmalloc::Capacity>::shrink):
1648         * bmalloc/Heap.cpp: Added.
1649         (bmalloc::sleep):
1650         (bmalloc::Heap::Heap):
1651         (bmalloc::Heap::concurrentScavenge):
1652         (bmalloc::Heap::scavengeSmallPages):
1653         (bmalloc::Heap::scavengeMediumPages):
1654         (bmalloc::Heap::scavengeLargeRanges):
1655         (bmalloc::Heap::allocateSmallLineSlowCase):
1656         (bmalloc::Heap::allocateMediumLineSlowCase):
1657         (bmalloc::Heap::allocateXLarge):
1658         (bmalloc::Heap::deallocateXLarge):
1659         (bmalloc::Heap::allocateLarge):
1660         (bmalloc::Heap::deallocateLarge):
1661         * bmalloc/Heap.h: Added.
1662         (bmalloc::Heap::deallocateSmallLine):
1663         (bmalloc::Heap::allocateSmallLine):
1664         (bmalloc::Heap::deallocateMediumLine):
1665         (bmalloc::Heap::allocateMediumLine):
1666         * bmalloc/Inline.h: Added.
1667         * bmalloc/LargeChunk.h: Added.
1668         (bmalloc::LargeChunk::begin):
1669         (bmalloc::LargeChunk::end):
1670         (bmalloc::LargeChunk::create):
1671         (bmalloc::LargeChunk::get):
1672         (bmalloc::LargeChunk::beginTag):
1673         (bmalloc::LargeChunk::endTag):
1674         * bmalloc/Line.h: Added.
1675         (bmalloc::Line<Traits>::begin):
1676         (bmalloc::Line<Traits>::end):
1677         (bmalloc::Line<Traits>::concurrentRef):
1678         (bmalloc::Line<Traits>::deref):
1679         * bmalloc/MediumAllocator.h: Added.
1680         (bmalloc::MediumAllocator::isNull):
1681         (bmalloc::MediumAllocator::MediumAllocator):
1682         (bmalloc::MediumAllocator::line):
1683         (bmalloc::MediumAllocator::allocate):
1684         (bmalloc::MediumAllocator::derefCount):
1685         (bmalloc::MediumAllocator::refill):
1686         * bmalloc/MediumChunk.h: Added.
1687         * bmalloc/MediumLine.h: Added.
1688         * bmalloc/MediumPage.h: Added.
1689         * bmalloc/MediumTraits.h: Added.
1690         * bmalloc/Mutex.cpp: Added.
1691         (bmalloc::Mutex::lockSlowCase):
1692         * bmalloc/Mutex.h: Added.
1693         (bmalloc::Mutex::Mutex):
1694         (bmalloc::Mutex::try_lock):
1695         (bmalloc::Mutex::lock):
1696         (bmalloc::Mutex::unlock):
1697         * bmalloc/ObjectType.cpp: Added.
1698         (bmalloc::objectType):
1699         * bmalloc/ObjectType.h: Added.
1700         (bmalloc::isSmallOrMedium):
1701         (bmalloc::isSmall):
1702         * bmalloc/Page.h: Added.
1703         (bmalloc::Page<Traits>::ref):
1704         (bmalloc::Page<Traits>::deref):
1705         (bmalloc::Page<Traits>::refCount):
1706         * bmalloc/PerProcess.h: Added.
1707         (bmalloc::PerProcess::mutex):
1708         (bmalloc::PerProcess<T>::getFastCase):
1709         (bmalloc::PerProcess<T>::get):
1710         (bmalloc::PerProcess<T>::getSlowCase):
1711         * bmalloc/PerThread.h: Added.
1712         (bmalloc::PerThreadStorage<Cache>::get):
1713         (bmalloc::PerThreadStorage<Cache>::init):
1714         (bmalloc::PerThreadStorage::get):
1715         (bmalloc::PerThreadStorage::init):
1716         (bmalloc::PerThread<T>::getFastCase):
1717         (bmalloc::PerThread<T>::get):
1718         (bmalloc::PerThread<T>::destructor):
1719         (bmalloc::PerThread<T>::getSlowCase):
1720         * bmalloc/Range.h: Added.
1721         (bmalloc::Range::Range):
1722         (bmalloc::Range::begin):
1723         (bmalloc::Range::end):
1724         (bmalloc::Range::size):
1725         (bmalloc::Range::operator!):
1726         (bmalloc::Range::operator<):
1727         * bmalloc/SegregatedFreeList.cpp: Added.
1728         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1729         (bmalloc::SegregatedFreeList::insert):
1730         (bmalloc::SegregatedFreeList::takeGreedy):
1731         (bmalloc::SegregatedFreeList::take):
1732         * bmalloc/SegregatedFreeList.h: Added.
1733         * bmalloc/Sizes.h: Added.
1734         * bmalloc/SmallAllocator.h: Added.
1735         (bmalloc::SmallAllocator::isNull):
1736         (bmalloc::SmallAllocator::canAllocate):
1737         (bmalloc::SmallAllocator::SmallAllocator):
1738         (bmalloc::SmallAllocator::line):
1739         (bmalloc::SmallAllocator::allocate):
1740         (bmalloc::SmallAllocator::objectCount):
1741         (bmalloc::SmallAllocator::derefCount):
1742         (bmalloc::SmallAllocator::refill):
1743         * bmalloc/SmallChunk.h: Added.
1744         * bmalloc/SmallLine.h: Added.
1745         * bmalloc/SmallPage.h: Added.
1746         * bmalloc/SmallTraits.h: Added.
1747         * bmalloc/Syscall.h: Added.
1748         * bmalloc/VMAllocate.h: Added.
1749         (bmalloc::vmSize):
1750         (bmalloc::vmValidate):
1751         (bmalloc::vmAllocate):
1752         (bmalloc::vmDeallocate):
1753         (bmalloc::vmDeallocatePhysicalPages):
1754         (bmalloc::vmAllocatePhysicalPages):
1755         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1756         (bmalloc::vmAllocatePhysicalPagesSloppy):
1757         * bmalloc/VMHeap.cpp: Added.
1758         (bmalloc::VMHeap::VMHeap):
1759         (bmalloc::VMHeap::allocateSmallChunk):
1760         (bmalloc::VMHeap::allocateMediumChunk):
1761         (bmalloc::VMHeap::allocateLargeChunk):
1762         * bmalloc/VMHeap.h: Added.
1763         (bmalloc::VMHeap::allocateSmallPage):
1764         (bmalloc::VMHeap::allocateMediumPage):
1765         (bmalloc::VMHeap::allocateLargeRange):
1766         (bmalloc::VMHeap::deallocateSmallPage):
1767         (bmalloc::VMHeap::deallocateMediumPage):
1768         (bmalloc::VMHeap::deallocateLargeRange):
1769         * bmalloc/Vector.h: Added.
1770         (bmalloc::Vector::begin):
1771         (bmalloc::Vector::end):
1772         (bmalloc::Vector::size):
1773         (bmalloc::Vector::capacity):
1774         (bmalloc::Vector::last):
1775         (bmalloc::Vector::pop):
1776         (bmalloc::Vector<T>::Vector):
1777         (bmalloc::Vector<T>::~Vector):
1778         (bmalloc::Vector<T>::operator):
1779         (bmalloc::Vector<T>::push):
1780         (bmalloc::Vector<T>::pop):
1781         (bmalloc::Vector<T>::shrink):
1782         (bmalloc::Vector<T>::reallocateBuffer):
1783         (bmalloc::Vector<T>::shrinkCapacity):
1784         (bmalloc::Vector<T>::growCapacity):
1785         * bmalloc/XLargeChunk.h: Added.
1786         (bmalloc::XLargeChunk::get):
1787         (bmalloc::XLargeChunk::begin):
1788         (bmalloc::XLargeChunk::XLargeChunk):
1789         (bmalloc::XLargeChunk::create):
1790         (bmalloc::XLargeChunk::destroy):
1791         (bmalloc::XLargeChunk::range):
1792         (bmalloc::XLargeChunk::size):
1793         * bmalloc/bmalloc.h: Added.
1794         (bmalloc::api::malloc):
1795         (bmalloc::api::free):
1796         (bmalloc::api::realloc):
1797         * bmalloc/mbmalloc.cpp: Added.
1798