Geoff is organized, but he is not an organization.
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-01-11  Dan Bernstein  <mitz@apple.com>
2
3         Geoff is organized, but he is not an organization.
4
5         Rubber-stamped by Anders Carlsson.
6
7         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
8
9 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
10
11         Make bmalloc work with ASan
12         https://bugs.webkit.org/show_bug.cgi?id=140194
13
14         Reviewed by Mark Lam.
15
16         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
17         an OS-specific API to test for loaded runtime libraries.
18
19         * bmalloc/Environment.cpp:
20         (bmalloc::isASanEnabled):
21         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
22         ASan is enabled, since system malloc has the Asan hooks we need.
23
24         You could check for the ASan compile-time flag instead, but doing this
25         check at runtime prepares bmalloc for a world where it is a dynamic
26         library that might be loaded into projects it did not compile with.
27
28 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
29
30         Fix up bmalloc's PerThread for use on Linux
31         https://bugs.webkit.org/show_bug.cgi?id=139804
32
33         Reviewed by Anders Carlsson.
34
35         The previous implementation was a bit slow.
36
37         * bmalloc/PerThread.h:
38         (bmalloc::PerThreadStorage<Cache>::get):
39         (bmalloc::PerThreadStorage::get):
40         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
41         way to do fast per-thread access without taking a lock every time. This
42         probably works on all the platforms we care about, and it matches other
43         techniques we use elsewhere in WebKit.
44
45         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
46         this class because PerThreadStorage now encapsulates everything that
47         needs to be conditional.
48
49         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
50
51 2014-12-26  Dan Bernstein  <mitz@apple.com>
52
53         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
54         https://bugs.webkit.org/show_bug.cgi?id=139950
55
56         Reviewed by David Kilzer.
57
58         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
59         in a manner that works with Xcode 5.1.1.
60
61 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
62
63         Safari crashes when you set Malloc environment variables
64         https://bugs.webkit.org/show_bug.cgi?id=139656
65
66         Reviewed by Michael Saboff.
67
68         I forgot to cover the realloc() case. Whoops. (OoPS?)
69
70         This time around, I ran the full MallocBench test suite in Malloc=1
71         mode, and it passed.
72
73         * bmalloc/Allocator.cpp:
74         (bmalloc::Allocator::reallocate):
75         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
76         It needs to be down there so that we can do the short-circuiting check
77         for whether bmalloc is enabled first.
78
79         Also added the check.
80
81         * bmalloc/Cache.cpp:
82         (bmalloc::Cache::scavenge):
83         (bmalloc::Cache::Cache):
84         (bmalloc::Cache::reallocateSlowCaseNullCache):
85         * bmalloc/Cache.h:
86         (bmalloc::Cache::deallocator):
87         (bmalloc::Cache::reallocate): Ditto.
88
89         * bmalloc/bmalloc.h:
90         (bmalloc::api::free):
91         (bmalloc::api::realloc): Ditto.
92
93         (bmalloc::api::scavenge): Pushed this down into Cache to match the
94         surrounding functions.
95
96 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
97
98         bmalloc should support system memory analysis tools (part 2)
99         https://bugs.webkit.org/show_bug.cgi?id=139565
100
101         Reviewed by Mark Lam.
102
103         This patch actually queries the environment to see if memory analysis
104         tools have been enabled.
105
106         * bmalloc/Deallocator.cpp:
107         (bmalloc::Deallocator::scavenge): Don't process the object log if
108         we've disabled bmalloc because it will be full of invalid nullptrs.
109
110         * bmalloc/Environment.cpp:
111         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
112         Malloc debugging flags. I also added a plain "Malloc" catch-all for
113         when you want to disable bmalloc without enabling any kind of funny
114         business.
115
116         It would be slightly nicer just to iterate the list of environment
117         variables and strstr them, but getenv is the more portable option,
118         and performance here doesn't really matter.
119
120         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
121         environment variable.
122
123         (bmalloc::Environment::computeIsBmallocEnabled):
124
125 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
126
127         Try to fix the iOS simulator build.
128
129         #include the declaration of malloc / free.
130
131         * bmalloc/Allocator.cpp:
132         * bmalloc/Deallocator.cpp:
133
134 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
135
136         Try to fix the build.
137
138         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
139
140 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
141
142         bmalloc should support system memory analysis tools (part 1)
143         https://bugs.webkit.org/show_bug.cgi?id=139559
144
145         Reviewed by Mark Lam.
146
147         This patch adds the hooks to disable bmalloc at runtime if certain
148         environment variables are set, but doesn't actually read from the
149         environment yet.
150
151         No performance change.
152
153         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
154         we'll use to read environment variables and see if memory analysis tools
155         have been enabled.
156
157         * bmalloc/Allocator.cpp:
158         (bmalloc::Allocator::Allocator):
159         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
160         on the allocation path. We cache the setting to make the check fast.
161
162         * bmalloc/Allocator.h: Interface changes.
163
164         * bmalloc/Cache.cpp:
165         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
166         and deallocator. This main purpose is to enable them to query the
167         environment for whether bmalloc is enabled; but this is also a slightly
168         cleaner way to guarantee to them that the Heap has been pre-initialized.
169
170         * bmalloc/Deallocator.cpp:
171         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
172         fill the object log to force us to take the slow path on all deallocations.
173
174         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
175
176         * bmalloc/Deallocator.h: Interface changes.
177
178         * bmalloc/Environment.cpp: Added.
179         (bmalloc::Environment::Environment):
180         (bmalloc::Environment::computeIsBmallocEnabled):
181         * bmalloc/Environment.h: Added.
182         (bmalloc::Environment::isBmallocEnabled): This is the class that will
183         encapsulate looking for environment variables that turn on heap
184         analysis tools.
185
186         * bmalloc/Heap.h:
187         (bmalloc::Heap::environment):
188
189         * bmalloc/Mutex.h:
190         (bmalloc::Mutex::Mutex):
191         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
192         since I got super confused about them while writing this patch.
193
194         * bmalloc/VMHeap.cpp: Fixed an #include.
195
196 2014-12-09  David Kilzer  <ddkilzer@apple.com>
197
198         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
199         <http://webkit.org/b/139212>
200
201         Reviewed by Joseph Pecoraro.
202
203         * Configurations/Base.xcconfig:
204         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
205           on OS X.
206         * Configurations/DebugRelease.xcconfig:
207         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
208
209 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
210
211         bmalloc uses 8X more virtual memory than necessary
212         https://bugs.webkit.org/show_bug.cgi?id=138495
213
214         Reviewed by Mark Lam.
215
216         iOS has a per-process virtual memory cap around 1GB, so there's some
217         value to not going totally ham with virtual memory.
218
219         We currently use about 8X the necessary amount:
220             - 2X to align our VM allocation
221             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
222
223         We can cut that down:
224             - Return the unaligned portion of our VM allocation (-2X)
225             - Use all the chunks in a superchunk, instead of allocating one
226               chunk per superchunk (-4X)
227
228         * bmalloc/Algorithm.h:
229         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
230         function so we can call it with getpagesize() at runtime.
231
232         * bmalloc/Chunk.h:
233         * bmalloc/LargeChunk.h:
234         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
235         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
236
237         * bmalloc/VMAllocate.h:
238         (bmalloc::vmValidate):
239         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
240         if it does not succeed. Allocate precisely, and give back the extra.
241
242         * bmalloc/VMHeap.cpp:
243         (bmalloc::VMHeap::allocateSuperChunk):
244         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
245         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
246         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
247         in a superchunk, instead of just one.
248
249         * bmalloc/VMHeap.h:
250         (bmalloc::VMHeap::allocateSmallPage):
251         (bmalloc::VMHeap::allocateMediumPage):
252         (bmalloc::VMHeap::allocateLargeRange):
253         * bmalloc/XLargeChunk.h:
254         (bmalloc::XLargeChunk::create): Updated to match changes above.
255
256 2014-11-01  David Kilzer  <ddkilzer@apple.com>
257
258         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
259         <https://webkit.org/b/138286>
260         <rdar://problem/18847087>
261
262         Reviewed by Dan Bernstein.
263
264         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
265         to NO for the target that produces libbmalloc.a so that the
266         debug symbols will be linked into JavaScriptCore and end up in
267         its dSYM file.
268
269 2014-10-30  Dana Burkart  <dburkart@apple.com>
270
271         <rdar://problem/18821260> Prepare for the mysterious future
272
273         Reviewed by Lucas Forschler.
274
275         * Configurations/Base.xcconfig:
276         * Configurations/DebugRelease.xcconfig:
277
278 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
279
280         bmalloc: cleaned up fast path vs slow path
281         https://bugs.webkit.org/show_bug.cgi?id=137081
282
283         Reviewed by Sam Weinig.
284
285         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
286
287         * bmalloc/Allocator.cpp:
288         (bmalloc::Allocator::Allocator): Merged the small and medium range
289         caches, just like the small and medium allocators. Ranges are abstract
290         objects that don't really care whether they hold small or medium objects,
291         so they don't need to be segregated.
292
293         (bmalloc::Allocator::scavenge): Ditto.
294
295         (bmalloc::Allocator::allocateBumpRangeSlowCase):
296         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
297         we do care a tiny bit, because we need to specify small vs medium when
298         allocating new ranges from the heap, to ensure that the heap allocates
299         from the right segment of VM.
300
301         (bmalloc::Allocator::allocateLarge):
302         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
303         up the fast path. Large allocation performance is dominated by allocation
304         logic and initialization, so inlining it doesn't help.
305
306         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
307         it doesn't need to distinguish small vs medium objects.
308
309         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
310         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
311
312         * bmalloc/Allocator.h:
313         * bmalloc/BumpRange.h:
314
315         * bmalloc/Cache.cpp:
316         (bmalloc::Cache::allocateSlowCase): Deleted.
317         (bmalloc::Cache::deallocateSlowCase): Deleted.
318         * bmalloc/Cache.h:
319         (bmalloc::Cache::allocate):
320         (bmalloc::Cache::deallocate):
321         (bmalloc::Cache::allocateFastCase): Deleted.
322         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
323         paths. The downside to this change is that the fast path branches to two
324         distinct failure cases instead of one. The upside is that the slow path
325         doesn't need to re-read the segment register, which is not as cheap as a
326         normal register, and it doesn't need to do an extra level of function 
327         call. Seems to be worth it.
328
329         * bmalloc/Deallocator.h:
330         * bmalloc/Heap.cpp:
331         (bmalloc::Heap::refillSmallBumpRangeCache):
332         (bmalloc::Heap::refillMediumBumpRangeCache):
333         * bmalloc/Heap.h: Updated for interface changes.
334
335         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
336         small lines in a page / 2, since any other free lines will coalesce
337         with their neighbors.
338
339 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
340
341         Rolled out r173346.
342
343             bmalloc should honor the FastMalloc statistics API
344             https://bugs.webkit.org/show_bug.cgi?id=136592
345
346         This didn't really work. Because we allow ranges with and without
347         physical pages to merge, and we allow double-committing and
348         double-decommitting, we can't rely on commit actions to track memory
349         footprint.
350
351         * bmalloc/Heap.cpp:
352         (bmalloc::Heap::size): Deleted.
353         (bmalloc::Heap::capacity): Deleted.
354         * bmalloc/Heap.h:
355         * bmalloc/VMHeap.cpp:
356         (bmalloc::VMHeap::VMHeap):
357         (bmalloc::VMHeap::allocateSmallChunk):
358         (bmalloc::VMHeap::allocateMediumChunk):
359         (bmalloc::VMHeap::allocateLargeChunk):
360         * bmalloc/VMHeap.h:
361         (bmalloc::VMHeap::allocateSmallPage):
362         (bmalloc::VMHeap::allocateMediumPage):
363         (bmalloc::VMHeap::allocateLargeRange):
364         (bmalloc::VMHeap::deallocateSmallPage):
365         (bmalloc::VMHeap::deallocateMediumPage):
366         (bmalloc::VMHeap::deallocateLargeRange):
367         (bmalloc::VMHeap::size): Deleted.
368         (bmalloc::VMHeap::capacity): Deleted.
369         * bmalloc/bmalloc.h:
370         (bmalloc::api::heapSize): Deleted.
371         (bmalloc::api::heapCapacity): Deleted.
372
373 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
374
375         bmalloc: Allocation should be more precise
376         https://bugs.webkit.org/show_bug.cgi?id=136993
377
378         Reviewed by Gavin Barraclough.
379
380         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
381
382         This patch teaches the allocator to merge adjacent free lines into a
383         single allocatable range. This allows us to shrink the size of an
384         individual line without increasing fragmentation or the rate of allocator
385         slow paths.
386
387         We'll only take more slow paths when available memory is sparse, which
388         is exactly when it's worth it. When available memory is dense, we'll
389         take fewer slow paths.
390
391         * bmalloc.xcodeproj/project.pbxproj:
392         * bmalloc/Algorithm.h:
393         (bmalloc::divideRoundingUp):
394
395         * bmalloc/Allocator.cpp:
396         (bmalloc::Allocator::Allocator): Updated for interface changes.
397
398         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
399         Now that we merge lines, it's not convenient to scavenge by line.
400
401         (bmalloc::Allocator::allocateSmallBumpRange):
402         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
403         instead of individual lines.
404
405         (bmalloc::Allocator::allocateSlowCase):
406         (bmalloc::Allocator::allocateSmallLine): Deleted.
407         (bmalloc::Allocator::allocateMediumLine): Deleted.
408         (bmalloc::Allocator::allocateMedium): Deleted.
409         * bmalloc/Allocator.h:
410         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
411         into the standard fast path with small allocations. Since a BumpAllocator
412         just allocates out of an arbitrary range, it doesn't need to distinguish
413         between small and medium lines.
414
415         * bmalloc/BumpAllocator.h:
416         (bmalloc::BumpAllocator::size):
417         (bmalloc::BumpAllocator::BumpAllocator):
418         (bmalloc::BumpAllocator::init):
419         (bmalloc::BumpAllocator::refill):
420         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
421         anymore: the heap just gives us a pointer and a pre-computed number of
422         objects, and we allocate them.
423
424         * bmalloc/Deallocator.cpp:
425         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
426
427         * bmalloc/Heap.cpp:
428         (bmalloc::Heap::Heap):
429         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
430         detailing where all objects will lie in memory. After we merge two lines,
431         we might allocate an object that spans from one line to the next. This
432         metadata details which bits of memory overlap in that way, and how they
433         overlap.
434
435         (bmalloc::Heap::refillSmallBumpRangeCache):
436         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
437         and merge adjacent free lines into BumpRanges.
438
439         (bmalloc::Heap::allocateSmallPage):
440         (bmalloc::Heap::allocateMediumPage):
441         (bmalloc::Heap::deallocateSmallLine):
442         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
443         since we scan for free memory a page at a time.
444
445         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
446         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
447         fast path.
448
449         * bmalloc/Heap.h:
450         (bmalloc::Heap::derefSmallLine):
451         (bmalloc::Heap::derefMediumLine):
452         (bmalloc::Heap::deallocateSmallLine): Deleted.
453         (bmalloc::Heap::allocateSmallLine): Deleted.
454         (bmalloc::Heap::deallocateMediumLine): Deleted.
455         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
456
457         * bmalloc/Line.h:
458         (bmalloc::Line<Traits>::ref):
459         (bmalloc::Line<Traits>::deref):
460         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
461         anymore, since we only ever deref by 1 now.
462
463         * bmalloc/MediumAllocator.h:
464         (bmalloc::MediumAllocator::isNull): Deleted.
465         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
466         (bmalloc::MediumAllocator::line): Deleted.
467         (bmalloc::MediumAllocator::allocate): Deleted.
468         (bmalloc::MediumAllocator::derefCount): Deleted.
469         (bmalloc::MediumAllocator::refill): Deleted.
470         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
471         been dead for a while, since it doesn't build anymore with this patch.
472
473         * bmalloc/Page.h:
474         (bmalloc::Page::sizeClass):
475         (bmalloc::Page::setSizeClass):
476         (bmalloc::Page::smallSizeClass): Deleted.
477         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
478         to sizeClass, since we use it for medium sizes too.
479
480         * bmalloc/Sizes.h:
481         (bmalloc::Sizes::sizeClass):
482         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
483
484         (bmalloc::Sizes::smallSizeClassFor): Deleted.
485         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
486
487         * bmalloc/bmalloc.h:
488         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
489         can be a bit more precise. It also has to be, since we can't guarantee
490         that an object ends at the end of a line anymore.
491
492 2014-09-19  Daniel Bates  <dabates@apple.com>
493
494         Always assume internal SDK when building configuration Production
495         https://bugs.webkit.org/show_bug.cgi?id=136925
496         <rdar://problem/18362399>
497
498         Reviewed by Dan Bernstein.
499
500         * Configurations/Base.xcconfig:
501
502 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
503
504         bmalloc: moved line caches from the deallocator to the allocator
505         https://bugs.webkit.org/show_bug.cgi?id=136868
506
507         Reviewed by Gavin Barraclough.
508
509         I did this mostly as a simplification, to make it easier to change the
510         allocation strategy.
511
512         No throughput change on MallocBench. Saves about 50kB.
513
514         Since the deallocator needs to lock the heap when freeing lines anyway,
515         there isn't much benefit to giving the deallocator a local cache of
516         deallocated lines.
517
518         We still give the allocator a local cache of lines because that does
519         reduce the frequency at which it needs to lock the heap in order to
520         acquire more lines.
521
522         * bmalloc/Allocator.cpp:
523         (bmalloc::Allocator::scavenge):
524         (bmalloc::Allocator::allocateSmallLine):
525         (bmalloc::Allocator::allocateMediumLine):
526         (bmalloc::Allocator::allocateMedium):
527         (bmalloc::Allocator::allocateSlowCase):
528         * bmalloc/Allocator.h:
529         * bmalloc/Deallocator.cpp:
530         (bmalloc::Deallocator::Deallocator):
531         (bmalloc::Deallocator::scavenge):
532         (bmalloc::Deallocator::processObjectLog):
533         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
534         (bmalloc::Deallocator::allocateSmallLine): Deleted.
535         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
536         (bmalloc::Deallocator::allocateMediumLine): Deleted.
537         * bmalloc/Deallocator.h:
538
539         * bmalloc/Sizes.h:
540         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
541         exactly one page in size. That's about what we were shooting for anyway,
542         and it may make it easier to switch to per-page allocation in future.
543
544 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
545
546         bmalloc: allocate small and medium objects using the same bump pointer class
547         https://bugs.webkit.org/show_bug.cgi?id=136843
548
549         Reviewed by Gavin Barraclough.
550
551         4% speedup on MallocBench.
552
553         Now that medium-sized objects have dedicated per-size allocators, they
554         don't need to use an arbitrary bump pointer allocator. This means that
555         every allocator knows how many objects it will allocate from the start,
556         and we don't need a post-processing step to adjust refcounts based on
557         real allocation count.
558
559         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
560         since it's used for small and medium objects now.
561
562         * bmalloc/Allocator.cpp:
563         (bmalloc::Allocator::Allocator): Updated to use new interface.
564         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
565         to make sure that we finish allocating all the objects in it.
566
567         (bmalloc::Allocator::allocateMedium):
568         (bmalloc::Allocator::allocateSlowCase):
569         (bmalloc::Allocator::retire): Deleted.
570         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
571         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
572         * bmalloc/Allocator.h:
573         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
574         used to post-process an allocator based on how many objects it allocated.
575
576         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
577         (bmalloc::BumpAllocator::BumpAllocator):
578         (bmalloc::BumpAllocator::init):
579         (bmalloc::BumpAllocator::line):
580         (bmalloc::BumpAllocator::validate):
581         (bmalloc::BumpAllocator::allocate):
582         (bmalloc::BumpAllocator::refill):
583         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
584         about the kinds of lines they allocate into. In some cases, the line
585         type must be provided as a template parameter by the caller.
586
587         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
588         (bmalloc::SmallAllocator::line): Deleted.
589         (bmalloc::SmallAllocator::allocate): Deleted.
590         (bmalloc::SmallAllocator::objectCount): Deleted.
591         (bmalloc::SmallAllocator::derefCount): Deleted.
592         (bmalloc::SmallAllocator::refill): Deleted.
593         (bmalloc::SmallAllocator::clear): Deleted.
594
595         * bmalloc/ObjectType.h:
596         (bmalloc::isMedium):
597
598         * bmalloc/SmallAllocator.h:
599         (bmalloc::SmallAllocator::isNull): Deleted.
600         (bmalloc::SmallAllocator::canAllocate): Deleted.
601         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
602         (bmalloc::SmallAllocator::line): Deleted.
603         (bmalloc::SmallAllocator::allocate): Deleted.
604         (bmalloc::SmallAllocator::objectCount): Deleted.
605         (bmalloc::SmallAllocator::derefCount): Deleted.
606         (bmalloc::SmallAllocator::refill): Deleted.
607         (bmalloc::SmallAllocator::clear): Deleted.
608
609 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
610
611         Fixed a goof in bmalloc Vector sizing
612         https://bugs.webkit.org/show_bug.cgi?id=136795
613
614         Reviewed by Gavin Barraclough and Sam Weinig.
615
616         We want our minimum vector to be page-sized since the OS will give us
617         a page no matter what -- but we want that many bytes, and not enough
618         bytes to store that many elements.
619
620         * bmalloc/Vector.h: Math is hard.
621
622 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
623
624         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
625         https://bugs.webkit.org/show_bug.cgi?id=136693
626
627         Reviewed by Gavin Barraclough.
628
629         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
630
631         No throughput change.
632
633         We keep an array of medium allocators, just like our array of small
634         allocators.
635
636         In future, we can simplify the allocation fast path by merging the small
637         and medium allocator arrays. For now, this is the simplest change that
638         gets the win.
639
640         * bmalloc/Allocator.cpp:
641         (bmalloc::Allocator::Allocator):
642         (bmalloc::Allocator::scavenge):
643         (bmalloc::Allocator::allocateMedium):
644         * bmalloc/Allocator.h:
645         * bmalloc/Sizes.h:
646         (bmalloc::Sizes::mediumSizeClassFor):
647
648 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
649
650         Reviewed by Sam Weinig.
651
652         Renamed log => retire for clarity.
653
654         * bmalloc/Allocator.cpp:
655         (bmalloc::Allocator::scavenge):
656         (bmalloc::Allocator::retire):
657         (bmalloc::Allocator::allocateMedium):
658         (bmalloc::Allocator::allocateSlowCase):
659         (bmalloc::Allocator::log): Deleted.
660         * bmalloc/Allocator.h:
661
662 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
663
664         bmalloc: eager scavenge leaves behind a bogus allocator
665         https://bugs.webkit.org/show_bug.cgi?id=136743
666
667         Reviewed by Sam Weinig.
668
669         Be sure to clear the allocator after logging it in the eager scavenge
670         case, so that we don't later try to allocate out of the lines that we
671         have thrown away.
672
673         We didn't need to do this previously because scavenge would only happen
674         at thread exit time, after which no further allocation from the per-thread
675         cache would take place.
676
677         * bmalloc/Allocator.cpp:
678         (bmalloc::Allocator::scavenge):
679         * bmalloc/MediumAllocator.h:
680         (bmalloc::MediumAllocator::clear):
681         * bmalloc/SmallAllocator.h:
682         (bmalloc::SmallAllocator::clear):
683
684 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
685
686         bmalloc should honor the FastMalloc statistics API
687         https://bugs.webkit.org/show_bug.cgi?id=136592
688
689         Reviewed by Gavin Barraclough.
690
691         We do this by tracking "size" and "capacity" in the VM heap.
692
693         The VM heap's "capacity" is all the VM we ever allocated.
694
695         The VM heap's "size" the subset of VM currently held onto by the
696         VM heap (and therefore not in use by the regular heap).
697
698         Somewhat ironically, reducing the process's memory footprint, increases
699         the size of the VM heap, since the VM heap holds the pages that are
700         purely virtual and not physical.
701
702         * bmalloc/Heap.cpp:
703         (bmalloc::Heap::size):
704         (bmalloc::Heap::capacity):
705         * bmalloc/Heap.h:
706         * bmalloc/VMHeap.cpp:
707         (bmalloc::VMHeap::VMHeap):
708         (bmalloc::VMHeap::allocateSmallChunk):
709         (bmalloc::VMHeap::allocateMediumChunk):
710         (bmalloc::VMHeap::allocateLargeChunk):
711         * bmalloc/VMHeap.h:
712         (bmalloc::VMHeap::size):
713         (bmalloc::VMHeap::capacity):
714         (bmalloc::VMHeap::allocateSmallPage):
715         (bmalloc::VMHeap::allocateMediumPage):
716         (bmalloc::VMHeap::allocateLargeRange):
717         (bmalloc::VMHeap::deallocateSmallPage):
718         (bmalloc::VMHeap::deallocateMediumPage):
719         (bmalloc::VMHeap::deallocateLargeRange):
720         * bmalloc/bmalloc.h:
721         (bmalloc::api::heapSize):
722         (bmalloc::api::heapCapacity):
723
724 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
725
726         bmalloc crashes on the EWS bots (due to bad large object allocation)
727         https://bugs.webkit.org/show_bug.cgi?id=136469
728
729         Reviewed by Andreas Kling.
730
731         It's possible to convince bmalloc to perform a bad large object allocation,
732         through these steps:
733
734         (1) Insert object A into freelist F0.
735
736         (2) Split, merge and split again A's neighbors such that object B is
737         inserted into freelist F0, with boundary tag and size equal to object A,
738         but pointer not completely equal to object A. Put object B at the head of F0.
739
740         (3) Allocate some other object from F0, swapping its position in the
741         freelist with object B, such that object A is now ahead of object B.
742
743         --> Now, the next allocation for size A/B will allocate object A, which
744         has a slightly wrong idea about where the object actually begins.
745         Immediately, you'll corrupt a little memory, and over time, you'll also
746         corrupt boundary tag metadata.
747
748         The solution is to store the begin pointer in the boundary tag. Luckily,
749         this doesn't make the tag any bigger, and it's not a noticeable slowdown
750         on MallocBench.
751
752         * bmalloc/Algorithm.h:
753         (bmalloc::rightShift):
754         * bmalloc/BeginTag.h:
755         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
756         validate the start pointer when popping off the free list. Through a
757         very uncommon set of steps, it is possible to have an item in the free
758         list that is valid by all accounts except for its start pointer.
759
760         * bmalloc/BoundaryTag.h:
761         (bmalloc::BoundaryTag::compactBegin):
762         (bmalloc::BoundaryTag::setRange):
763         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
764         start pointer. We don't need the whole pointer -- just the offset, in
765         largeAlignment increments, into the relevant boundary tag bucket.
766
767         * bmalloc/BoundaryTagInlines.h:
768         (bmalloc::validateNext):
769         (bmalloc::BoundaryTag::init):
770         (bmalloc::BoundaryTag::mergeLarge):
771         (bmalloc::BoundaryTag::splitLarge):
772         * bmalloc/SegregatedFreeList.cpp:
773         (bmalloc::SegregatedFreeList::insert):
774         (bmalloc::SegregatedFreeList::takeGreedy):
775         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
776         the size when establishing a boundary tag, as required by the new
777         interface.
778
779         * bmalloc/Sizes.h:
780
781 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
782
783         Fixed a bmalloc crash seen on the EWS bot
784         https://bugs.webkit.org/show_bug.cgi?id=135955
785
786         Reviewed by Andreas Kling.
787
788         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
789         input buffer is a malloc region, that region will get marked Copy-On-Write
790         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
791         on older OS X's. In 10.10, they still fail, but they do not return
792         EINVAL.
793
794         So, we can only ASSERT that our syscalls succeed starting with 10.10.
795
796 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
797
798         Fixed the bmalloc build
799         https://bugs.webkit.org/show_bug.cgi?id=135953
800
801         Reviewed by Andreas Kling.
802
803         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
804         These headers are used, so they must be available outside the project.
805
806 2014-08-13  Daniel Bates  <dabates@apple.com>
807
808         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
809         (https://bugs.webkit.org/show_bug.cgi?id=135895)
810
811         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
812         implementation of PerThread<T>::getFastCase().
813
814         * bmalloc/PerThread.h:
815         (bmalloc::PerThread<T>::getFastCase):
816
817 2014-08-13  Daniel Bates  <dabates@apple.com>
818
819         Make bmalloc::PerThread work without C++ thread local storage
820         https://bugs.webkit.org/show_bug.cgi?id=135895
821
822         Reviewed by Geoffrey Garen.
823
824         Implement support for building bmalloc without C++ thread local storage.
825
826         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
827         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
828         to determine whether the compiler supports C++ thread local storage.
829         * bmalloc/PerThread.h:
830         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
831         without C++ thread local storage.
832         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
833         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
834         PerThreadStorage::initSharedKeyIfNeeded().
835         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
836         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
837         ensure that the shared key has been initialized.
838         (_pthread_setspecific_direct): Deleted.
839         (_pthread_getspecific_direct): Deleted.
840
841 2014-08-13  Daniel Bates  <dabates@apple.com>
842
843         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
844         https://bugs.webkit.org/show_bug.cgi?id=135848
845
846         Reviewed by Geoffrey Garen.
847
848         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
849         when building for the iOS Simulator.
850         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
851         using the public SDK.
852         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
853         using the public SDK.
854         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
855         using the public SDK.
856
857 2014-08-12  Daniel Bates  <dabates@apple.com>
858
859         BPLATFORM(IOS) always evaluates to false
860         https://bugs.webkit.org/show_bug.cgi?id=135843
861
862         Reviewed by Geoffrey Garen.
863
864         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
865         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
866         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
867         building for iOS.
868
869         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
870         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
871
872         * bmalloc/BPlatform.h:
873
874 2014-08-11  Andy Estes  <aestes@apple.com>
875
876         [iOS] Get rid of iOS.xcconfig
877         https://bugs.webkit.org/show_bug.cgi?id=135809
878
879         Reviewed by Joseph Pecoraro.
880
881         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
882
883         * Configurations/Base.xcconfig:
884         * Configurations/iOS.xcconfig: Removed.
885         * bmalloc.xcodeproj/project.pbxproj:
886
887 2014-05-01  Dan Bernstein  <mitz@apple.com>
888
889         Fixed production builds for the iOS Simulator.
890         <rdar://problem/16792221>
891
892         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
893         PRIVATE_HEADERS_FOLDER_PATH when installing.
894
895 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
896
897         bmalloc: Segregate pages by objects size
898         https://bugs.webkit.org/show_bug.cgi?id=131909
899
900         Reviewed by Andreas Kling.
901
902         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
903
904         * bmalloc/Allocator.cpp:
905         (bmalloc::Allocator::allocateSlowCase):
906         * bmalloc/Allocator.h:
907         (bmalloc::Allocator::allocateFastCase):
908         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
909         function for size class calculation.
910
911         * bmalloc/Deallocator.cpp:
912         (bmalloc::Deallocator::Deallocator):
913         (bmalloc::Deallocator::scavenge):
914         (bmalloc::Deallocator::deallocateSmallLine):
915         (bmalloc::Deallocator::allocateSmallLine):
916         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
917         cache can't be shared anymore.
918
919         * bmalloc/Heap.cpp:
920         (bmalloc::Heap::allocateSmallLineSlowCase):
921         * bmalloc/Heap.h:
922         (bmalloc::Heap::deallocateSmallLine): Ditto.
923
924         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
925         page refcount when allocating a line because we might have deallocated
926         the page and the recycled it for another size class.
927
928         (bmalloc::Heap::deallocateMediumLine):
929         (bmalloc::Heap::allocateMediumLine):
930         * bmalloc/Line.h:
931         (bmalloc::Line::refCount):
932         * bmalloc/Page.h:
933         (bmalloc::Page::refCount):
934         (bmalloc::Page::smallSizeClass):
935         (bmalloc::Page::setSmallSizeClass):
936         (bmalloc::Page<Traits>::refCount): Deleted.
937         * bmalloc/Sizes.h:
938         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
939         an index into an array from a size.
940
941 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
942
943         bmalloc: Improved alignment in LargeChunk
944         https://bugs.webkit.org/show_bug.cgi?id=131895
945
946         Reviewed by Andreas Kling.
947
948         * bmalloc/Chunk.h:
949         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
950         Technically, the previous alignment was harmless, but I would prefer,
951         dear reader, not to have to explain the interlocking set of
952         circumstances that made it so.
953
954 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
955
956         Rolled out r167502 because it caused a crash on the facebook benchmark.
957
958         Unreviewed.
959
960             bmalloc: Added an XSmall line size
961             https://bugs.webkit.org/show_bug.cgi?id=131851
962
963             Reviewed by Sam Weinig.
964
965 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
966
967         bmalloc: Mutex should be harder to use wrong
968         https://bugs.webkit.org/show_bug.cgi?id=131879
969
970         Reviewed by Andreas Kling.
971
972         Mutex now has a proper constructor, so you can't deadlock by forgetting
973         to initialize it.
974
975         * bmalloc.xcodeproj/project.pbxproj:
976         * bmalloc/Allocator.cpp:
977         (bmalloc::Allocator::processXSmallAllocatorLog):
978         (bmalloc::Allocator::processSmallAllocatorLog):
979         (bmalloc::Allocator::processMediumAllocatorLog):
980         (bmalloc::Allocator::allocateLarge):
981         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
982         since the Heap mutex is a static.
983
984         * bmalloc/AsyncTask.h:
985         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
986         need for explicit initialization anymore.
987
988         * bmalloc/Deallocator.cpp:
989         (bmalloc::Deallocator::scavenge):
990         (bmalloc::Deallocator::deallocateLarge):
991         (bmalloc::Deallocator::deallocateXLarge):
992         (bmalloc::Deallocator::processObjectLog):
993         (bmalloc::Deallocator::deallocateSmallLine):
994         (bmalloc::Deallocator::deallocateXSmallLine):
995         (bmalloc::Deallocator::allocateSmallLine):
996         (bmalloc::Deallocator::allocateXSmallLine):
997         (bmalloc::Deallocator::deallocateMediumLine):
998         (bmalloc::Deallocator::allocateMediumLine):
999         * bmalloc/Deallocator.h:
1000         * bmalloc/Heap.cpp:
1001         (bmalloc::sleep):
1002         (bmalloc::Heap::Heap):
1003         (bmalloc::Heap::concurrentScavenge):
1004         (bmalloc::Heap::scavenge):
1005         (bmalloc::Heap::scavengeSmallPages):
1006         (bmalloc::Heap::scavengeXSmallPages):
1007         (bmalloc::Heap::scavengeMediumPages):
1008         (bmalloc::Heap::scavengeLargeRanges):
1009         (bmalloc::Heap::allocateXSmallLineSlowCase):
1010         (bmalloc::Heap::allocateSmallLineSlowCase):
1011         (bmalloc::Heap::allocateMediumLineSlowCase):
1012         (bmalloc::Heap::allocateXLarge):
1013         (bmalloc::Heap::deallocateXLarge):
1014         (bmalloc::Heap::allocateLarge):
1015         (bmalloc::Heap::deallocateLarge):
1016         * bmalloc/Heap.h:
1017         (bmalloc::Heap::deallocateXSmallLine):
1018         (bmalloc::Heap::allocateXSmallLine):
1019         (bmalloc::Heap::deallocateSmallLine):
1020         (bmalloc::Heap::allocateSmallLine):
1021         (bmalloc::Heap::deallocateMediumLine):
1022         (bmalloc::Heap::allocateMediumLine):
1023         * bmalloc/Line.h:
1024         (bmalloc::Line<Traits>::deref):
1025         * bmalloc/Mutex.cpp: Removed.
1026         * bmalloc/Mutex.h:
1027         (bmalloc::Mutex::Mutex):
1028         (bmalloc::Mutex::init): Deleted.
1029         (bmalloc::Mutex::try_lock): Deleted.
1030         (bmalloc::Mutex::lock): Deleted.
1031         (bmalloc::Mutex::unlock): Deleted.
1032         * bmalloc/Page.h:
1033         (bmalloc::Page<Traits>::ref):
1034         (bmalloc::Page<Traits>::deref):
1035         (bmalloc::Page<Traits>::refCount):
1036         * bmalloc/PerProcess.h:
1037         (bmalloc::PerProcess::mutex):
1038         (bmalloc::PerProcess<T>::getSlowCase):
1039         * bmalloc/StaticMutex.cpp: Added.
1040         (bmalloc::StaticMutex::lockSlowCase):
1041         * bmalloc/StaticMutex.h: Added.
1042         (bmalloc::StaticMutex::init):
1043         (bmalloc::StaticMutex::try_lock):
1044         (bmalloc::StaticMutex::lock):
1045         (bmalloc::StaticMutex::unlock):
1046         * bmalloc/VMHeap.h:
1047         (bmalloc::VMHeap::deallocateXSmallPage):
1048         (bmalloc::VMHeap::deallocateSmallPage):
1049         (bmalloc::VMHeap::deallocateMediumPage):
1050         (bmalloc::VMHeap::deallocateLargeRange):
1051         * bmalloc/bmalloc.h:
1052         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1053         since the Heap mutex is a static.
1054
1055 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1056
1057         bmalloc: AsyncTask should use Mutex instead of std::mutex
1058         https://bugs.webkit.org/show_bug.cgi?id=131865
1059
1060         Reviewed by Gavin Barraclough.
1061
1062         std::mutex is so slow that it makes parallelizing simple tasks through
1063         AsyncTask a net regression. Mutex fixes this.
1064
1065         * bmalloc/AsyncTask.h:
1066         (bmalloc::Function>::AsyncTask):
1067         (bmalloc::Function>::join):
1068         (bmalloc::Function>::runSlowCase):
1069         (bmalloc::Function>::entryPoint):
1070         * bmalloc/Mutex.h:
1071         (bmalloc::Mutex::init):
1072
1073 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1074
1075         bmalloc: Added an XSmall line size
1076         https://bugs.webkit.org/show_bug.cgi?id=131851
1077
1078         Reviewed by Sam Weinig.
1079
1080         Reduces malloc footprint on Membuster recordings by 10%.
1081
1082         This is a throughput regression, but we're still way ahead of TCMalloc.
1083         I have some ideas for how to recover the regression -- but I wanted to
1084         get this win in first.
1085
1086         Full set of benchmark results:
1087
1088                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1089                                                                                                 
1090                                                                nopatch                      patch                                Δ
1091                 Peak Memory:
1092                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1093                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1094                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1095
1096                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1097                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1098                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1099
1100                 Memory at End:
1101                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1102                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1103                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1104
1105                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1106                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1107                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1108
1109                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1110                                                                                                 
1111                                                    nopatch                patch                         Δ
1112                 Execution Time:
1113                     churn                            127ms                151ms            ! 1.19x slower
1114                     list_allocate                    130ms                164ms            ! 1.26x slower
1115                     tree_allocate                    109ms                127ms            ! 1.17x slower
1116                     tree_churn                       115ms                120ms            ! 1.04x slower
1117                     facebook                         240ms                259ms            ! 1.08x slower
1118                     fragment                          91ms                131ms            ! 1.44x slower
1119                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1120                     message_one                      260ms                259ms             ^ 1.0x faster
1121                     message_many                     149ms                154ms            ! 1.03x slower
1122                     medium                           194ms                248ms            ! 1.28x slower
1123                     big                              157ms                160ms            ! 1.02x slower
1124
1125                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1126                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1127                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1128
1129                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1130                                                                                                 
1131                                                                        nopatch                          patch                                     Δ
1132                 Execution Time:
1133                     churn                                                126ms                          148ms                        ! 1.17x slower
1134                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1135                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1136                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1137                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1138                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1139                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1140                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1141                     facebook                                             240ms                          276ms                        ! 1.15x slower
1142                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1143                     fragment                                              92ms                          130ms                        ! 1.41x slower
1144                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1145                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1146                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1147                     message_one                                          260ms                          260ms                                      
1148                     message_many                                         170ms                          238ms                         ! 1.4x slower
1149                     medium                                               185ms                          250ms                        ! 1.35x slower
1150                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1151                     big                                                  150ms                          169ms                        ! 1.13x slower
1152                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1153
1154                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1155                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1156                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1157
1158                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1159
1160                                                                     TC                      patch                                Δ
1161                 Peak Memory:
1162                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1163                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1164                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1165
1166                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1167                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1168                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1169
1170                 Memory at End:
1171                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
1172                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
1173                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
1174
1175                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
1176                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
1177                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
1178
1179                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1180                                                                                                 
1181                                                         TC                patch                         Δ
1182                 Execution Time:
1183                     churn                            416ms                148ms            ^ 2.81x faster
1184                     list_allocate                    463ms                164ms            ^ 2.82x faster
1185                     tree_allocate                    292ms                127ms             ^ 2.3x faster
1186                     tree_churn                       157ms                120ms            ^ 1.31x faster
1187                     facebook                         327ms                276ms            ^ 1.18x faster
1188                     fragment                         335ms                129ms             ^ 2.6x faster
1189                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
1190                     message_one                      386ms                258ms             ^ 1.5x faster
1191                     message_many                     410ms                154ms            ^ 2.66x faster
1192                     medium                           391ms                245ms             ^ 1.6x faster
1193                     big                              261ms                167ms            ^ 1.56x faster
1194
1195                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
1196                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
1197                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
1198
1199         * bmalloc.xcodeproj/project.pbxproj:
1200         * bmalloc/Allocator.cpp:
1201         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
1202         index corresponds with its size. Instead, use the size selection function
1203         explicitly. Now that we have XSmall, some small allocator entries are
1204         unused.
1205
1206         (bmalloc::Allocator::scavenge):
1207         (bmalloc::Allocator::log):
1208         (bmalloc::Allocator::processXSmallAllocatorLog):
1209         (bmalloc::Allocator::allocateSlowCase):
1210         * bmalloc/Allocator.h:
1211         (bmalloc::Allocator::xSmallAllocatorFor):
1212         (bmalloc::Allocator::allocateFastCase):
1213         * bmalloc/Chunk.h:
1214         * bmalloc/Deallocator.cpp:
1215         (bmalloc::Deallocator::scavenge):
1216         (bmalloc::Deallocator::processObjectLog):
1217         (bmalloc::Deallocator::deallocateSlowCase):
1218         (bmalloc::Deallocator::deallocateXSmallLine):
1219         (bmalloc::Deallocator::allocateXSmallLine):
1220         * bmalloc/Deallocator.h:
1221         (bmalloc::Deallocator::deallocateFastCase):
1222         * bmalloc/Heap.cpp:
1223         (bmalloc::Heap::scavenge):
1224         (bmalloc::Heap::scavengeXSmallPages):
1225         (bmalloc::Heap::allocateXSmallLineSlowCase):
1226         * bmalloc/Heap.h:
1227         (bmalloc::Heap::deallocateXSmallLine):
1228         (bmalloc::Heap::allocateXSmallLine):
1229         * bmalloc/LargeChunk.h:
1230         (bmalloc::LargeChunk::get):
1231         (bmalloc::LargeChunk::endTag):
1232         * bmalloc/Line.h:
1233         * bmalloc/MediumAllocator.h:
1234         (bmalloc::MediumAllocator::allocate):
1235         (bmalloc::MediumAllocator::refill):
1236         * bmalloc/ObjectType.cpp:
1237         (bmalloc::objectType):
1238         * bmalloc/ObjectType.h:
1239         (bmalloc::isXSmall):
1240         (bmalloc::isSmall):
1241         (bmalloc::isMedium):
1242         (bmalloc::isLarge):
1243         (bmalloc::isSmallOrMedium): Deleted.
1244         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1245         handling small objects. There's probably a reasonable way to share this
1246         code in the future -- I'll look into that once it's stopped changing.
1247
1248         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1249         is the main reason things got slower.
1250
1251         * bmalloc/SmallAllocator.h:
1252         (bmalloc::SmallAllocator::allocate):
1253         * bmalloc/SmallTraits.h:
1254         * bmalloc/VMHeap.cpp:
1255         (bmalloc::VMHeap::allocateXSmallChunk):
1256         * bmalloc/VMHeap.h:
1257         (bmalloc::VMHeap::allocateXSmallPage):
1258         (bmalloc::VMHeap::deallocateXSmallPage):
1259         * bmalloc/XSmallAllocator.h: Added.
1260         (bmalloc::XSmallAllocator::isNull):
1261         (bmalloc::XSmallAllocator::canAllocate):
1262         (bmalloc::XSmallAllocator::XSmallAllocator):
1263         (bmalloc::XSmallAllocator::line):
1264         (bmalloc::XSmallAllocator::allocate):
1265         (bmalloc::XSmallAllocator::objectCount):
1266         (bmalloc::XSmallAllocator::derefCount):
1267         (bmalloc::XSmallAllocator::refill):
1268         * bmalloc/XSmallChunk.h: Added.
1269         * bmalloc/XSmallLine.h: Added.
1270         * bmalloc/XSmallPage.h: Added.
1271         * bmalloc/XSmallTraits.h: Added.
1272         * bmalloc/bmalloc.h:
1273         (bmalloc::api::realloc): Boiler-plate copy, as above.
1274
1275 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1276
1277         MallocBench should scavenge explicitly instead of waiting
1278         https://bugs.webkit.org/show_bug.cgi?id=131661
1279
1280         Reviewed by Andreas Kling.
1281
1282         Added explicit scavenge support to bmalloc. This isn't a memory win,
1283         since bmalloc's per-thread cache is so small. But it makes testing
1284         simpler.
1285
1286         * bmalloc/Allocator.cpp:
1287         (bmalloc::Allocator::~Allocator):
1288         (bmalloc::Allocator::scavenge):
1289         * bmalloc/Allocator.h:
1290         * bmalloc/Cache.cpp:
1291         (bmalloc::Cache::operator new):
1292         (bmalloc::Cache::operator delete):
1293         (bmalloc::Cache::Cache):
1294         (bmalloc::Cache::scavenge):
1295         * bmalloc/Cache.h:
1296         * bmalloc/Deallocator.cpp:
1297         (bmalloc::Deallocator::~Deallocator):
1298         (bmalloc::Deallocator::scavenge):
1299         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1300         functions, for reuse.
1301
1302         * bmalloc/Heap.cpp:
1303         (bmalloc::sleep):
1304         (bmalloc::Heap::concurrentScavenge):
1305         (bmalloc::Heap::scavenge):
1306         (bmalloc::Heap::scavengeSmallPages):
1307         (bmalloc::Heap::scavengeMediumPages):
1308         (bmalloc::Heap::scavengeLargeRanges):
1309         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1310         scavenging -- in response to a benchmark or a low memory warning --
1311         wants to complete as soon as possible, so its sleep duration is 0.
1312
1313         * bmalloc/bmalloc.h:
1314         (bmalloc::api::scavenge):
1315         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1316
1317 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1318
1319         Use 4kB pages on Mac
1320         https://bugs.webkit.org/show_bug.cgi?id=131658
1321
1322         Reviewed by Sam Weinig.
1323
1324         This reduces memory use a lot on Membuster:
1325
1326                                                                   base                      patch                                Δ
1327                 Execution Time:
1328                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1329                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1330                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1331
1332                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1333                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1334                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1335
1336                 Peak Memory:
1337                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1338                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1339                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1340
1341                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1342                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1343                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1344
1345                 Memory at End:
1346                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1347                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1348                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1349
1350                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1351                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1352                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1353
1354
1355         * bmalloc.xcodeproj/project.pbxproj:
1356         * bmalloc/BPlatform.h: Added.
1357         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1358         is 4kB on Mac.
1359
1360 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1361
1362         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1363
1364         * bmalloc.xcodeproj: Modified property svn:ignore.
1365
1366 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1367
1368         Fixed some mbmalloc exports
1369         https://bugs.webkit.org/show_bug.cgi?id=131599
1370
1371         Reviewed by Ryosuke Niwa.
1372
1373         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1374         of the project, so we can call them from API.
1375
1376         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1377         visibility, so they show up as exported in the .dylib.
1378
1379 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1380
1381         Put bmalloc headers in the right place
1382         https://bugs.webkit.org/show_bug.cgi?id=131464
1383
1384         Reviewed by Mark Rowe.
1385
1386         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1387         specify that we don't just want to dump all of our generically-named
1388         headers into /usr/local/include.
1389
1390 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1391
1392         Made bmalloc more #include friendly
1393         https://bugs.webkit.org/show_bug.cgi?id=131386
1394
1395         Reviewed by Andreas Kling.
1396
1397         Marked a bunch of headers private so they can be used from client code
1398         that #includes bmalloc.h.
1399
1400         Renamed ASSERT macros to BASSERT. This matches their header, which already
1401         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1402
1403         * bmalloc.xcodeproj/project.pbxproj:
1404         * bmalloc/Allocator.cpp:
1405         (bmalloc::Allocator::allocateSlowCase):
1406         * bmalloc/AsyncTask.h:
1407         (bmalloc::Function>::runSlowCase):
1408         * bmalloc/BAssert.h:
1409         * bmalloc/BoundaryTag.h:
1410         (bmalloc::BoundaryTag::setSize):
1411         * bmalloc/BoundaryTagInlines.h:
1412         (bmalloc::validate):
1413         (bmalloc::BoundaryTag::init):
1414         (bmalloc::BoundaryTag::deallocate):
1415         (bmalloc::BoundaryTag::splitLarge):
1416         (bmalloc::BoundaryTag::allocate):
1417         * bmalloc/Chunk.h:
1418         * bmalloc/Deallocator.cpp:
1419         (bmalloc::Deallocator::processObjectLog):
1420         (bmalloc::Deallocator::deallocateSlowCase):
1421         * bmalloc/Deallocator.h:
1422         (bmalloc::Deallocator::deallocateFastCase):
1423         * bmalloc/FixedVector.h:
1424         (bmalloc::Capacity>::operator):
1425         (bmalloc::Capacity>::push):
1426         (bmalloc::Capacity>::pop):
1427         (bmalloc::Capacity>::shrink):
1428         * bmalloc/Heap.cpp:
1429         (bmalloc::Heap::allocateLarge):
1430         * bmalloc/LargeChunk.h:
1431         (bmalloc::LargeChunk::get):
1432         (bmalloc::LargeChunk::endTag):
1433         * bmalloc/Line.h:
1434         (bmalloc::Line<Traits>::concurrentRef):
1435         (bmalloc::Line<Traits>::deref):
1436         * bmalloc/MediumAllocator.h:
1437         (bmalloc::MediumAllocator::allocate):
1438         * bmalloc/ObjectType.h:
1439         (bmalloc::isSmall):
1440         * bmalloc/Page.h:
1441         (bmalloc::Page<Traits>::ref):
1442         (bmalloc::Page<Traits>::deref):
1443         * bmalloc/PerThread.h:
1444         (bmalloc::PerThread<T>::getSlowCase):
1445         * bmalloc/SegregatedFreeList.cpp:
1446         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1447         (bmalloc::SegregatedFreeList::insert):
1448         * bmalloc/SmallAllocator.h:
1449         (bmalloc::SmallAllocator::allocate):
1450         (bmalloc::SmallAllocator::refill):
1451         * bmalloc/Syscall.h:
1452         * bmalloc/VMAllocate.h:
1453         (bmalloc::vmValidate):
1454         (bmalloc::vmAllocate):
1455         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1456         * bmalloc/Vector.h:
1457         (bmalloc::Vector<T>::operator):
1458         (bmalloc::Vector<T>::pop):
1459         (bmalloc::Vector<T>::shrink):
1460         * bmalloc/XLargeChunk.h:
1461         (bmalloc::XLargeChunk::range):
1462         (bmalloc::XLargeChunk::size):
1463
1464 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1465
1466         Removed an unused file.
1467
1468         Unreviewed.
1469
1470         * bmalloc/AsyncTask.cpp: Removed.
1471
1472 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1473
1474         Build bmalloc on Mac
1475         https://bugs.webkit.org/show_bug.cgi?id=131333
1476
1477         Reviewed by Mark Rowe.
1478
1479         * Makefile: Added. For make clients.
1480
1481         These files are required for building any project in WebKit. I copied
1482         them from WTF:
1483         * Configurations: Added.
1484         * Configurations/Base.xcconfig: Added.
1485         * Configurations/DebugRelease.xcconfig: Added.
1486         * Configurations/bmalloc.xcconfig: Added.
1487         * Configurations/iOS.xcconfig: Added.
1488         * Configurations/mbmalloc.xcconfig: Added.
1489
1490         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1491         from here because everything is in .xcconfig files now.
1492
1493         I had to fix a bunch of minor warnings, since they're enabled in our
1494         .xcconfig files:
1495
1496         * bmalloc/AsyncTask.h:
1497         (bmalloc::Function>::AsyncTask):
1498         * bmalloc/BAssert.h:
1499         * bmalloc/BoundaryTagInlines.h:
1500         (bmalloc::validate):
1501         * bmalloc/Heap.cpp:
1502         (bmalloc::Heap::Heap):
1503         (bmalloc::Heap::allocateLarge):
1504         (bmalloc::Heap::deallocateLarge):
1505         * bmalloc/Mutex.h:
1506         (bmalloc::Mutex::Mutex): Deleted.
1507         * bmalloc/VMAllocate.h:
1508         (bmalloc::vmValidate):
1509         * bmalloc/mbmalloc.cpp:
1510
1511 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1512
1513         bmalloc: Fixed a leak in the per-thread cache
1514         https://bugs.webkit.org/show_bug.cgi?id=131330
1515
1516         Reviewed by Andreas Kling.
1517
1518         Remember to deallocate our line caches upon thread exit.
1519
1520         * bmalloc/Deallocator.cpp:
1521         (bmalloc::Deallocator::~Deallocator):
1522
1523 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1524
1525         bmalloc: rolled out the tryLock experiment
1526         https://bugs.webkit.org/show_bug.cgi?id=131328
1527
1528         Reviewed by Andreas Kling.
1529
1530         It wasn't a speedup.
1531
1532         * bmalloc.xcodeproj/project.pbxproj:
1533         * bmalloc/Allocator.cpp:
1534         (bmalloc::Allocator::processSmallAllocatorLog):
1535         (bmalloc::Allocator::processMediumAllocatorLog):
1536         * bmalloc/Deallocator.cpp:
1537         (bmalloc::Deallocator::processObjectLog):
1538         (bmalloc::Deallocator::deallocateSlowCase):
1539         (bmalloc::Deallocator::deallocateSmallLine):
1540         (bmalloc::Deallocator::deallocateMediumLine):
1541         * bmalloc/Deallocator.h:
1542         (bmalloc::Deallocator::deallocateFastCase):
1543         * bmalloc/Heap.h:
1544         (bmalloc::Heap::deallocateSmallLine):
1545         (bmalloc::Heap::deallocateMediumLine):
1546         * bmalloc/Line.h:
1547         (bmalloc::Line<Traits>::deref):
1548         * bmalloc/Page.h:
1549         (bmalloc::Page<Traits>::deref):
1550
1551 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1552
1553         bmalloc
1554         https://bugs.webkit.org/show_bug.cgi?id=131170
1555
1556         Reviewed by Andreas Kling.
1557
1558         Initial commit.
1559
1560         * bmalloc: Added.
1561         * bmalloc.xcodeproj: Added.
1562         * bmalloc.xcodeproj/project.pbxproj: Added.
1563         * bmalloc/Algorithm.h: Added.
1564         (bmalloc::max):
1565         (bmalloc::min):
1566         (bmalloc::mask):
1567         (bmalloc::test):
1568         (bmalloc::roundUpToMultipleOf):
1569         (bmalloc::roundDownToMultipleOf):
1570         (bmalloc::sizeOf):
1571         (bmalloc::bitCount):
1572         (bmalloc::isPowerOfTwo):
1573         * bmalloc/Allocator.cpp: Added.
1574         (bmalloc::Allocator::Allocator):
1575         (bmalloc::Allocator::~Allocator):
1576         (bmalloc::Allocator::log):
1577         (bmalloc::Allocator::processSmallAllocatorLog):
1578         (bmalloc::Allocator::processMediumAllocatorLog):
1579         (bmalloc::Allocator::allocateLarge):
1580         (bmalloc::Allocator::allocateXLarge):
1581         (bmalloc::Allocator::allocateMedium):
1582         (bmalloc::Allocator::allocateSlowCase):
1583         * bmalloc/Allocator.h: Added.
1584         (bmalloc::Allocator::smallAllocatorFor):
1585         (bmalloc::Allocator::allocateFastCase):
1586         (bmalloc::Allocator::allocate):
1587         * bmalloc/AsyncTask.cpp: Added.
1588         (bmalloc::AsyncTask<Function>::runSlowCase):
1589         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
1590         (bmalloc::AsyncTask<Function>::entryPoint):
1591         * bmalloc/AsyncTask.h: Added.
1592         (bmalloc::Function>::AsyncTask):
1593         (bmalloc::Function>::join):
1594         (bmalloc::Function>::run):
1595         (bmalloc::Function>::runSlowCase):
1596         (bmalloc::Function>::pthreadEntryPoint):
1597         (bmalloc::Function>::entryPoint):
1598         * bmalloc/BAssert.h: Added.
1599         * bmalloc/BeginTag.h: Added.
1600         (bmalloc::BeginTag::isInFreeList):
1601         * bmalloc/BoundaryTag.h: Added.
1602         (bmalloc::BoundaryTag::isXLarge):
1603         (bmalloc::BoundaryTag::setXLarge):
1604         (bmalloc::BoundaryTag::isFree):
1605         (bmalloc::BoundaryTag::setFree):
1606         (bmalloc::BoundaryTag::isEnd):
1607         (bmalloc::BoundaryTag::setEnd):
1608         (bmalloc::BoundaryTag::hasPhysicalPages):
1609         (bmalloc::BoundaryTag::setHasPhysicalPages):
1610         (bmalloc::BoundaryTag::isNull):
1611         (bmalloc::BoundaryTag::clear):
1612         (bmalloc::BoundaryTag::size):
1613         (bmalloc::BoundaryTag::setSize):
1614         (bmalloc::BoundaryTag::prev):
1615         (bmalloc::BoundaryTag::next):
1616         * bmalloc/BoundaryTagInlines.h: Added.
1617         (bmalloc::validate):
1618         (bmalloc::validatePrev):
1619         (bmalloc::validateNext):
1620         (bmalloc::BoundaryTag::init):
1621         (bmalloc::BoundaryTag::mergeLargeLeft):
1622         (bmalloc::BoundaryTag::mergeLargeRight):
1623         (bmalloc::BoundaryTag::mergeLarge):
1624         (bmalloc::BoundaryTag::deallocate):
1625         (bmalloc::BoundaryTag::splitLarge):
1626         (bmalloc::BoundaryTag::allocate):
1627         * bmalloc/Cache.cpp: Added.
1628         (bmalloc::Cache::operator new):
1629         (bmalloc::Cache::operator delete):
1630         (bmalloc::Cache::Cache):
1631         (bmalloc::Cache::allocateSlowCase):
1632         (bmalloc::Cache::allocateSlowCaseNullCache):
1633         (bmalloc::Cache::deallocateSlowCase):
1634         (bmalloc::Cache::deallocateSlowCaseNullCache):
1635         * bmalloc/Cache.h: Added.
1636         (bmalloc::Cache::allocator):
1637         (bmalloc::Cache::deallocator):
1638         (bmalloc::Cache::allocateFastCase):
1639         (bmalloc::Cache::deallocateFastCase):
1640         (bmalloc::Cache::allocate):
1641         (bmalloc::Cache::deallocate):
1642         * bmalloc/Chunk.h: Added.
1643         (bmalloc::Chunk::begin):
1644         (bmalloc::Chunk::end):
1645         (bmalloc::Chunk::lines):
1646         (bmalloc::Chunk::pages):
1647         * bmalloc/Deallocator.cpp: Added.
1648         (bmalloc::Deallocator::Deallocator):
1649         (bmalloc::Deallocator::~Deallocator):
1650         (bmalloc::Deallocator::deallocateLarge):
1651         (bmalloc::Deallocator::deallocateXLarge):
1652         (bmalloc::Deallocator::processObjectLog):
1653         (bmalloc::Deallocator::deallocateSlowCase):
1654         (bmalloc::Deallocator::deallocateSmallLine):
1655         (bmalloc::Deallocator::allocateSmallLine):
1656         (bmalloc::Deallocator::deallocateMediumLine):
1657         (bmalloc::Deallocator::allocateMediumLine):
1658         * bmalloc/Deallocator.h: Added.
1659         (bmalloc::Deallocator::deallocateFastCase):
1660         (bmalloc::Deallocator::deallocate):
1661         * bmalloc/EndTag.h: Added.
1662         (bmalloc::EndTag::operator=):
1663         * bmalloc/FixedVector.h: Added.
1664         (bmalloc::FixedVector::begin):
1665         (bmalloc::FixedVector::end):
1666         (bmalloc::FixedVector::size):
1667         (bmalloc::FixedVector::capacity):
1668         (bmalloc::FixedVector::clear):
1669         (bmalloc::FixedVector::isEmpty):
1670         (bmalloc::Capacity>::FixedVector):
1671         (bmalloc::Capacity>::operator):
1672         (bmalloc::Capacity>::push):
1673         (bmalloc::Capacity>::pop):
1674         (bmalloc::Capacity>::shrink):
1675         * bmalloc/Heap.cpp: Added.
1676         (bmalloc::sleep):
1677         (bmalloc::Heap::Heap):
1678         (bmalloc::Heap::concurrentScavenge):
1679         (bmalloc::Heap::scavengeSmallPages):
1680         (bmalloc::Heap::scavengeMediumPages):
1681         (bmalloc::Heap::scavengeLargeRanges):
1682         (bmalloc::Heap::allocateSmallLineSlowCase):
1683         (bmalloc::Heap::allocateMediumLineSlowCase):
1684         (bmalloc::Heap::allocateXLarge):
1685         (bmalloc::Heap::deallocateXLarge):
1686         (bmalloc::Heap::allocateLarge):
1687         (bmalloc::Heap::deallocateLarge):
1688         * bmalloc/Heap.h: Added.
1689         (bmalloc::Heap::deallocateSmallLine):
1690         (bmalloc::Heap::allocateSmallLine):
1691         (bmalloc::Heap::deallocateMediumLine):
1692         (bmalloc::Heap::allocateMediumLine):
1693         * bmalloc/Inline.h: Added.
1694         * bmalloc/LargeChunk.h: Added.
1695         (bmalloc::LargeChunk::begin):
1696         (bmalloc::LargeChunk::end):
1697         (bmalloc::LargeChunk::create):
1698         (bmalloc::LargeChunk::get):
1699         (bmalloc::LargeChunk::beginTag):
1700         (bmalloc::LargeChunk::endTag):
1701         * bmalloc/Line.h: Added.
1702         (bmalloc::Line<Traits>::begin):
1703         (bmalloc::Line<Traits>::end):
1704         (bmalloc::Line<Traits>::concurrentRef):
1705         (bmalloc::Line<Traits>::deref):
1706         * bmalloc/MediumAllocator.h: Added.
1707         (bmalloc::MediumAllocator::isNull):
1708         (bmalloc::MediumAllocator::MediumAllocator):
1709         (bmalloc::MediumAllocator::line):
1710         (bmalloc::MediumAllocator::allocate):
1711         (bmalloc::MediumAllocator::derefCount):
1712         (bmalloc::MediumAllocator::refill):
1713         * bmalloc/MediumChunk.h: Added.
1714         * bmalloc/MediumLine.h: Added.
1715         * bmalloc/MediumPage.h: Added.
1716         * bmalloc/MediumTraits.h: Added.
1717         * bmalloc/Mutex.cpp: Added.
1718         (bmalloc::Mutex::lockSlowCase):
1719         * bmalloc/Mutex.h: Added.
1720         (bmalloc::Mutex::Mutex):
1721         (bmalloc::Mutex::try_lock):
1722         (bmalloc::Mutex::lock):
1723         (bmalloc::Mutex::unlock):
1724         * bmalloc/ObjectType.cpp: Added.
1725         (bmalloc::objectType):
1726         * bmalloc/ObjectType.h: Added.
1727         (bmalloc::isSmallOrMedium):
1728         (bmalloc::isSmall):
1729         * bmalloc/Page.h: Added.
1730         (bmalloc::Page<Traits>::ref):
1731         (bmalloc::Page<Traits>::deref):
1732         (bmalloc::Page<Traits>::refCount):
1733         * bmalloc/PerProcess.h: Added.
1734         (bmalloc::PerProcess::mutex):
1735         (bmalloc::PerProcess<T>::getFastCase):
1736         (bmalloc::PerProcess<T>::get):
1737         (bmalloc::PerProcess<T>::getSlowCase):
1738         * bmalloc/PerThread.h: Added.
1739         (bmalloc::PerThreadStorage<Cache>::get):
1740         (bmalloc::PerThreadStorage<Cache>::init):
1741         (bmalloc::PerThreadStorage::get):
1742         (bmalloc::PerThreadStorage::init):
1743         (bmalloc::PerThread<T>::getFastCase):
1744         (bmalloc::PerThread<T>::get):
1745         (bmalloc::PerThread<T>::destructor):
1746         (bmalloc::PerThread<T>::getSlowCase):
1747         * bmalloc/Range.h: Added.
1748         (bmalloc::Range::Range):
1749         (bmalloc::Range::begin):
1750         (bmalloc::Range::end):
1751         (bmalloc::Range::size):
1752         (bmalloc::Range::operator!):
1753         (bmalloc::Range::operator<):
1754         * bmalloc/SegregatedFreeList.cpp: Added.
1755         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1756         (bmalloc::SegregatedFreeList::insert):
1757         (bmalloc::SegregatedFreeList::takeGreedy):
1758         (bmalloc::SegregatedFreeList::take):
1759         * bmalloc/SegregatedFreeList.h: Added.
1760         * bmalloc/Sizes.h: Added.
1761         * bmalloc/SmallAllocator.h: Added.
1762         (bmalloc::SmallAllocator::isNull):
1763         (bmalloc::SmallAllocator::canAllocate):
1764         (bmalloc::SmallAllocator::SmallAllocator):
1765         (bmalloc::SmallAllocator::line):
1766         (bmalloc::SmallAllocator::allocate):
1767         (bmalloc::SmallAllocator::objectCount):
1768         (bmalloc::SmallAllocator::derefCount):
1769         (bmalloc::SmallAllocator::refill):
1770         * bmalloc/SmallChunk.h: Added.
1771         * bmalloc/SmallLine.h: Added.
1772         * bmalloc/SmallPage.h: Added.
1773         * bmalloc/SmallTraits.h: Added.
1774         * bmalloc/Syscall.h: Added.
1775         * bmalloc/VMAllocate.h: Added.
1776         (bmalloc::vmSize):
1777         (bmalloc::vmValidate):
1778         (bmalloc::vmAllocate):
1779         (bmalloc::vmDeallocate):
1780         (bmalloc::vmDeallocatePhysicalPages):
1781         (bmalloc::vmAllocatePhysicalPages):
1782         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1783         (bmalloc::vmAllocatePhysicalPagesSloppy):
1784         * bmalloc/VMHeap.cpp: Added.
1785         (bmalloc::VMHeap::VMHeap):
1786         (bmalloc::VMHeap::allocateSmallChunk):
1787         (bmalloc::VMHeap::allocateMediumChunk):
1788         (bmalloc::VMHeap::allocateLargeChunk):
1789         * bmalloc/VMHeap.h: Added.
1790         (bmalloc::VMHeap::allocateSmallPage):
1791         (bmalloc::VMHeap::allocateMediumPage):
1792         (bmalloc::VMHeap::allocateLargeRange):
1793         (bmalloc::VMHeap::deallocateSmallPage):
1794         (bmalloc::VMHeap::deallocateMediumPage):
1795         (bmalloc::VMHeap::deallocateLargeRange):
1796         * bmalloc/Vector.h: Added.
1797         (bmalloc::Vector::begin):
1798         (bmalloc::Vector::end):
1799         (bmalloc::Vector::size):
1800         (bmalloc::Vector::capacity):
1801         (bmalloc::Vector::last):
1802         (bmalloc::Vector::pop):
1803         (bmalloc::Vector<T>::Vector):
1804         (bmalloc::Vector<T>::~Vector):
1805         (bmalloc::Vector<T>::operator):
1806         (bmalloc::Vector<T>::push):
1807         (bmalloc::Vector<T>::pop):
1808         (bmalloc::Vector<T>::shrink):
1809         (bmalloc::Vector<T>::reallocateBuffer):
1810         (bmalloc::Vector<T>::shrinkCapacity):
1811         (bmalloc::Vector<T>::growCapacity):
1812         * bmalloc/XLargeChunk.h: Added.
1813         (bmalloc::XLargeChunk::get):
1814         (bmalloc::XLargeChunk::begin):
1815         (bmalloc::XLargeChunk::XLargeChunk):
1816         (bmalloc::XLargeChunk::create):
1817         (bmalloc::XLargeChunk::destroy):
1818         (bmalloc::XLargeChunk::range):
1819         (bmalloc::XLargeChunk::size):
1820         * bmalloc/bmalloc.h: Added.
1821         (bmalloc::api::malloc):
1822         (bmalloc::api::free):
1823         (bmalloc::api::realloc):
1824         * bmalloc/mbmalloc.cpp: Added.
1825