bmalloc should implement malloc introspection (to stop false-positive leaks when...
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-02-20  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
4         https://bugs.webkit.org/show_bug.cgi?id=141802
5
6         Reviewed by Andreas Kling.
7
8         Rolling back in with a fix for a crash seen while using GuardMalloc.
9
10         * bmalloc/VMHeap.cpp:
11         (bmalloc::VMHeap::grow):
12         * bmalloc/VMHeap.h:
13         * bmalloc/Zone.cpp: Re-land the old patch.
14
15         (bmalloc::Zone::size): Be sure to implement the size() function since
16         it's accessible indirectly via the malloc_zone_from_ptr public API --
17         and GuardMalloc calls it all the time.
18
19         (bmalloc::Zone::Zone):
20         * bmalloc/Zone.h: Re-land the old patch.
21
22 2015-02-19  Commit Queue  <commit-queue@webkit.org>
23
24         Unreviewed, rolling out r180363.
25         https://bugs.webkit.org/show_bug.cgi?id=141814
26
27         Caused >50 crashes when running LayoutTests in GuardMalloc or
28         ASAN modes. (Requested by jernoble on #webkit).
29
30         Reverted changeset:
31
32         "bmalloc should implement malloc introspection (to stop false-
33         positive leaks when MallocStackLogging is off)"
34         https://bugs.webkit.org/show_bug.cgi?id=141802
35         http://trac.webkit.org/changeset/180363
36
37 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
38
39         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
40         https://bugs.webkit.org/show_bug.cgi?id=141802
41
42         Reviewed by Andreas Kling.
43
44         Fixed a last-minute type.
45
46         The macro is OS, not PLATFORM.
47
48         * bmalloc/VMHeap.cpp:
49         (bmalloc::VMHeap::grow):
50         * bmalloc/VMHeap.h:
51         * bmalloc/Zone.h:
52
53 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
54
55         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
56         https://bugs.webkit.org/show_bug.cgi?id=141802
57
58         Reviewed by Andreas Kling.
59
60         This patch does the bare minimum to stop false positive leaks from
61         being reported by the Darwin leaks tool. We register each super chunk
62         as a single object, and then request that the leaks tool scan it.
63
64         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
65         zone introspection API.
66
67         * bmalloc/Algorithm.h: Missing #include.
68
69         * bmalloc/VMHeap.cpp:
70         (bmalloc::VMHeap::grow):
71         * bmalloc/VMHeap.h: Adopt the new abstraction.
72
73         * bmalloc/Zone.cpp: Added.
74         (bmalloc::remoteRead): Helper for reading an object out of another process.
75         (bmalloc::Zone::enumerator):
76         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
77         in introspection.
78
79         * bmalloc/Zone.h: Added.
80         (bmalloc::Zone::superChunks):
81         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
82         since our dynamic allocations will not be scanned by leaks since they
83         will have the malloc VM tag.
84
85 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
86
87         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
88         https://bugs.webkit.org/show_bug.cgi?id=141759
89
90         Reviewed by Andreas Kling.
91
92         * bmalloc.xcodeproj/project.pbxproj:
93         * bmalloc/SuperChunk.h: Added.
94         (bmalloc::SuperChunk::create):
95         (bmalloc::SuperChunk::SuperChunk):
96         (bmalloc::SuperChunk::smallChunk):
97         (bmalloc::SuperChunk::mediumChunk):
98         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
99         into a separate class, for clarity and type safety.
100
101         * bmalloc/VMHeap.cpp:
102         (bmalloc::VMHeap::grow):
103         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
104         "grow" because Andreas found "allocateSuperChunk" to be unclear.
105
106         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
107         for malloc introspection.
108
109         (bmalloc::VMHeap::allocateSmallPage):
110         (bmalloc::VMHeap::allocateMediumPage):
111         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
112
113 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
114
115         Build bmalloc through CMake as a static library. It's then linked either
116         into the WTF library (if built as a shared library) or into the JSC and
117         WebKit2 libraries. There's no need to build it as a standalone shared library.
118
119         Rubber-stamped by Carlos Garcia Campos.
120
121         * CMakeLists.txt:
122
123 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
124
125         [BMalloc] Add a FIXME comment for memory alignas
126         https://bugs.webkit.org/show_bug.cgi?id=141556
127
128         Reviewed by Csaba Osztrogonác.
129
130         * bmalloc/Chunk.h: Add a FIXME comment.
131         * bmalloc/LargeChunk.h: ditto.
132
133 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
134
135         bmalloc buildfix on 32 bit Linux (x86/ARM)
136         https://bugs.webkit.org/show_bug.cgi?id=141472
137
138         Reviewed by Gyuyoung Kim.
139
140         * bmalloc/Algorithm.h:
141         (bmalloc::roundUpToMultipleOf):
142         * bmalloc/FixedVector.h:
143         (bmalloc::FixedVector::clear):
144         * bmalloc/Sizes.h:
145         (bmalloc::Sizes::sizeClass):
146
147 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
148
149         [EFL][GTK] Use bmalloc instead of tcmalloc
150         https://bugs.webkit.org/show_bug.cgi?id=140162
151
152         Reviewed by Carlos Garcia Campos.
153
154         Support to use bmalloc on EFL and GTK ports.
155
156         * CMakeLists.txt: Added.
157         * bmalloc/Allocator.cpp:
158         (bmalloc::Allocator::allocate):
159             Fix unused return value caused by posix_memalign().
160         * bmalloc/AsyncTask.h:
161         * bmalloc/BoundaryTag.h:
162         (bmalloc::BoundaryTag::clear):
163         * bmalloc/Chunk.h:
164             Change Traits::Page with Traits::PageType in order to fix
165             -fpermitive build error on EFL and GTK port. 
166         * bmalloc/EndTag.h:
167         (bmalloc::EndTag::operator=):
168         * bmalloc/Line.h: ditto.
169         * bmalloc/MediumTraits.h:
170         * bmalloc/Page.h: ditto.
171         * bmalloc/PerThread.h:
172             EFL port doesn't support __has_include definition yet.
173             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
174         * bmalloc/SmallTraits.h: ditto.
175         * bmalloc/VMAllocate.h:
176         (bmalloc::vmDeallocatePhysicalPages):
177         (bmalloc::vmAllocatePhysicalPages):
178         * bmalloc/Vector.h:
179         (bmalloc::Vector<T>::push):
180         (bmalloc::Vector<T>::reallocateBuffer):
181
182 2015-01-31  Sam Weinig  <sam@webkit.org>
183
184         Remove even more Mountain Lion support
185         https://bugs.webkit.org/show_bug.cgi?id=141124
186
187         Reviewed by Alexey Proskuryakov.
188
189         * Configurations/Base.xcconfig:
190         * Configurations/DebugRelease.xcconfig:
191
192 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
193
194         GC marking threads should clear malloc caches
195         https://bugs.webkit.org/show_bug.cgi?id=141097
196
197         Reviewed by Andreas Kling.
198
199         Split the scavenging API into per-thread vs global, so that you can
200         request to scavenge your own thread without scavenging the whole heap.
201
202         * bmalloc/Cache.cpp:
203         (bmalloc::Cache::scavenge):
204         * bmalloc/bmalloc.h:
205         (bmalloc::api::scavengeThisThread):
206         (bmalloc::api::scavenge):
207
208 2015-01-28  Dana Burkart  <dburkart@apple.com>
209
210         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
211         https://bugs.webkit.org/show_bug.cgi?id=136765
212
213         Reviewed by Alexey Proskuryakov.
214
215         * Configurations/Base.xcconfig:
216         * Configurations/DebugRelease.xcconfig:
217
218 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
219
220         bmalloc: support aligned allocation
221         https://bugs.webkit.org/show_bug.cgi?id=140732
222
223         Reviewed by Andreas Kling.
224
225         * bmalloc/Allocator.cpp:
226         (bmalloc::Allocator::allocate): New function for aligned allocation.
227
228         Small and medium requests just allocate and free until they find an
229         aligned pointer. This is slightly inefficient in the worst case, but
230         still constant-time with little-to-no space overhead.
231
232         Large requests use a new API that requires the client to specify both
233         its ideal size and alignment, and the worst-case size you would have to
234         allocate in order to produce some interior pointer of the requested size
235         and alignment. We put the burden of this calculation on the client
236         because it simplifies things if we guarantee that allocation won't fail.
237
238         XLarge requests are easy: we just forward them to vmAllocate, which
239         already supported aligned requests.
240
241         * bmalloc/BoundaryTag.h:
242         * bmalloc/BoundaryTagInlines.h:
243         (bmalloc::BoundaryTag::mergeLeft):
244         (bmalloc::BoundaryTag::mergeRight):
245         (bmalloc::BoundaryTag::merge):
246         (bmalloc::BoundaryTag::deallocate):
247         (bmalloc::BoundaryTag::split):
248         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
249         refactored the interface to remove some reference out parameters in
250         order to clarify what changes and what doesn't.
251
252         * bmalloc/Heap.cpp:
253         (bmalloc::Heap::allocateXLarge): Added an alignment API.
254
255         (bmalloc::Heap::allocateLarge):
256         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
257         a few variants, so aligned and unaligned allocation could share some code.
258
259         * bmalloc/SegregatedFreeList.cpp:
260         (bmalloc::SegregatedFreeList::take):
261         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
262         for aligned allocation. It turns out that the aligned path is pretty
263         different, since it ends up searching for two potential ways to satisfy
264         an allocation: either large enough and aligned, or large enough to split
265         into something not aligned and something large enough and aligned.
266
267         * bmalloc/VMAllocate.h:
268         (bmalloc::vmAllocate): Switched alignment to come before size because
269         that's how the memalign API specifies it.
270
271         * bmalloc/VMHeap.h:
272         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
273
274 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
275
276         bmalloc: a little bit of cleanup
277         https://bugs.webkit.org/show_bug.cgi?id=140687
278
279         Reviewed by Anders Carlsson.
280
281         * bmalloc/Algorithm.h:
282         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
283         of code.
284
285         * bmalloc/BoundaryTag.h:
286         * bmalloc/BoundaryTagInlines.h:
287         (bmalloc::BoundaryTag::mergeLeft):
288         (bmalloc::BoundaryTag::mergeRight):
289         (bmalloc::BoundaryTag::merge):
290         (bmalloc::BoundaryTag::deallocate):
291         (bmalloc::BoundaryTag::split):
292         (bmalloc::BoundaryTag::allocate):
293         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
294         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
295         (bmalloc::BoundaryTag::mergeLarge): Deleted.
296         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
297         from all these functions, since boundary tags always pertain to large
298         objects, and putting the word "Large" everywhere wasn't helping to
299         explain that.
300
301         * bmalloc/Heap.cpp:
302         (bmalloc::Heap::allocateXLarge):
303         (bmalloc::Heap::findXLarge):
304         (bmalloc::Heap::deallocateXLarge):
305         * bmalloc/Heap.h:
306         * bmalloc/VMHeap.h:
307         (bmalloc::VMHeap::allocateXLarge): Deleted.
308         (bmalloc::VMHeap::findXLarge): Deleted.
309         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
310         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
311         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
312         really make sense for the VMHeap to be involved.
313
314 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
315
316         bmalloc: refactored XLarge allocation for better alignment
317         https://bugs.webkit.org/show_bug.cgi?id=140582
318
319         Reviewed by Andreas Kling.
320
321         XLarge objects used to be Large objects with an extra bit of metadata
322         that said "actually, I'm not large -- I'm extra large".
323
324         The metadata header in an XLarge allocation made it impossible for the
325         XLarge object to honor a very large alignment request.
326
327         The solution is to stop using a metadata header for XLarge objects, and
328         instead to store explicit metadata on the side.
329
330         This is a bit less astonishing, which is also nice.
331
332         Finding XLarge metadata is now a linear search. That's probably OK, since
333         it was always so in TCMalloc, and the usual number of XLarge allocations
334         in a process is 0.
335
336         This design makes it possible for the heap to cache XLarge allocations
337         with and/or without physical pages. I haven't actually done that yet
338         because the tradeoffs are subtle, so I don't want to do anything without
339         a motivating test case.
340
341         * bmalloc.xcodeproj/project.pbxproj:
342         * bmalloc/Allocator.cpp:
343         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
344         since an XLarge allocation is now just a naked buffer without a header.
345
346         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
347         XLarge alignment, since XLargeChunk won't give this to us implicitly
348         anymore.
349
350         * bmalloc/BoundaryTag.h:
351         (bmalloc::BoundaryTag::setRange):
352         (bmalloc::BoundaryTag::isXLarge): Deleted.
353         (bmalloc::BoundaryTag::setXLarge): Deleted.
354         * bmalloc/BoundaryTagInlines.h:
355         (bmalloc::validate):
356         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
357
358         * bmalloc/Deallocator.cpp:
359         (bmalloc::Deallocator::deallocateXLarge):
360         (bmalloc::Deallocator::deallocateSlowCase):
361         * bmalloc/Heap.cpp:
362         (bmalloc::Heap::findXLarge):
363         (bmalloc::Heap::allocateXLarge):
364         (bmalloc::Heap::deallocateXLarge):
365         * bmalloc/Heap.h: Updated for interface changes.
366
367         * bmalloc/ObjectType.cpp:
368         (bmalloc::objectType):
369         * bmalloc/ObjectType.h:
370         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
371         examining its bit pattern -- just like we do for other kinds of
372         allocations -- which is nice.
373
374         * bmalloc/Sizes.h:
375         * bmalloc/VMHeap.h:
376         (bmalloc::VMHeap::allocateXLarge):
377         (bmalloc::VMHeap::findXLarge):
378         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
379         for XLarge allocations.
380
381         * bmalloc/XLargeChunk.h: Removed.
382
383 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
384
385         bmalloc: added some infrastructure for aligned allocation
386         https://bugs.webkit.org/show_bug.cgi?id=140572
387
388         Reviewed by Andreas Kling.
389
390         * bmalloc/Algorithm.h:
391         (bmalloc::isPowerOfTwo):
392         (bmalloc::roundUpToMultipleOf):
393         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
394         isPowerOfTwo helper function.
395
396         * bmalloc/Allocator.cpp:
397         (bmalloc::Allocator::allocate):
398         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
399         Doesn't do anything yet, but does correctly forward to system malloc
400         when bmalloc is disabled.
401
402         * bmalloc/Cache.cpp:
403         (bmalloc::Cache::allocateSlowCaseNullCache):
404         * bmalloc/Cache.h:
405         (bmalloc::Cache::allocate):
406         * bmalloc/bmalloc.h:
407         (bmalloc::api::memalign):
408         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
409
410 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
411
412         Consider alignment when allocating from a SegregatedFreeList
413         https://bugs.webkit.org/show_bug.cgi?id=140408
414
415         Reviewed by Sam Weinig.
416
417         In preparation for supporting aligned allocation.
418
419         No performance change.
420
421         Since this is just one extra branch in an already expensive function,
422         I decided not to duplicate the function just to avoid the branch in
423         the un-aligned case.
424
425         * bmalloc/SegregatedFreeList.cpp:
426         (bmalloc::SegregatedFreeList::take):
427         * bmalloc/SegregatedFreeList.h:
428
429 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
430
431         Renamed minimum to size in SegregatedFreeList
432         https://bugs.webkit.org/show_bug.cgi?id=140406
433
434         Reviewed by Sam Weinig.
435
436         In preparation for supporting aligned allocation.
437
438         * bmalloc/SegregatedFreeList.cpp:
439         (bmalloc::SegregatedFreeList::takeGreedy):
440         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
441         really just a minimum. Let's not imply that this value is special.
442
443 2015-01-11  Dan Bernstein  <mitz@apple.com>
444
445         Geoff is organized, but he is not an organization.
446
447         Rubber-stamped by Anders Carlsson.
448
449         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
450
451 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
452
453         Make bmalloc work with ASan
454         https://bugs.webkit.org/show_bug.cgi?id=140194
455
456         Reviewed by Mark Lam.
457
458         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
459         an OS-specific API to test for loaded runtime libraries.
460
461         * bmalloc/Environment.cpp:
462         (bmalloc::isASanEnabled):
463         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
464         ASan is enabled, since system malloc has the Asan hooks we need.
465
466         You could check for the ASan compile-time flag instead, but doing this
467         check at runtime prepares bmalloc for a world where it is a dynamic
468         library that might be loaded into projects it did not compile with.
469
470 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
471
472         Fix up bmalloc's PerThread for use on Linux
473         https://bugs.webkit.org/show_bug.cgi?id=139804
474
475         Reviewed by Anders Carlsson.
476
477         The previous implementation was a bit slow.
478
479         * bmalloc/PerThread.h:
480         (bmalloc::PerThreadStorage<Cache>::get):
481         (bmalloc::PerThreadStorage::get):
482         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
483         way to do fast per-thread access without taking a lock every time. This
484         probably works on all the platforms we care about, and it matches other
485         techniques we use elsewhere in WebKit.
486
487         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
488         this class because PerThreadStorage now encapsulates everything that
489         needs to be conditional.
490
491         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
492
493 2014-12-26  Dan Bernstein  <mitz@apple.com>
494
495         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
496         https://bugs.webkit.org/show_bug.cgi?id=139950
497
498         Reviewed by David Kilzer.
499
500         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
501         in a manner that works with Xcode 5.1.1.
502
503 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
504
505         Safari crashes when you set Malloc environment variables
506         https://bugs.webkit.org/show_bug.cgi?id=139656
507
508         Reviewed by Michael Saboff.
509
510         I forgot to cover the realloc() case. Whoops. (OoPS?)
511
512         This time around, I ran the full MallocBench test suite in Malloc=1
513         mode, and it passed.
514
515         * bmalloc/Allocator.cpp:
516         (bmalloc::Allocator::reallocate):
517         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
518         It needs to be down there so that we can do the short-circuiting check
519         for whether bmalloc is enabled first.
520
521         Also added the check.
522
523         * bmalloc/Cache.cpp:
524         (bmalloc::Cache::scavenge):
525         (bmalloc::Cache::Cache):
526         (bmalloc::Cache::reallocateSlowCaseNullCache):
527         * bmalloc/Cache.h:
528         (bmalloc::Cache::deallocator):
529         (bmalloc::Cache::reallocate): Ditto.
530
531         * bmalloc/bmalloc.h:
532         (bmalloc::api::free):
533         (bmalloc::api::realloc): Ditto.
534
535         (bmalloc::api::scavenge): Pushed this down into Cache to match the
536         surrounding functions.
537
538 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
539
540         bmalloc should support system memory analysis tools (part 2)
541         https://bugs.webkit.org/show_bug.cgi?id=139565
542
543         Reviewed by Mark Lam.
544
545         This patch actually queries the environment to see if memory analysis
546         tools have been enabled.
547
548         * bmalloc/Deallocator.cpp:
549         (bmalloc::Deallocator::scavenge): Don't process the object log if
550         we've disabled bmalloc because it will be full of invalid nullptrs.
551
552         * bmalloc/Environment.cpp:
553         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
554         Malloc debugging flags. I also added a plain "Malloc" catch-all for
555         when you want to disable bmalloc without enabling any kind of funny
556         business.
557
558         It would be slightly nicer just to iterate the list of environment
559         variables and strstr them, but getenv is the more portable option,
560         and performance here doesn't really matter.
561
562         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
563         environment variable.
564
565         (bmalloc::Environment::computeIsBmallocEnabled):
566
567 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
568
569         Try to fix the iOS simulator build.
570
571         #include the declaration of malloc / free.
572
573         * bmalloc/Allocator.cpp:
574         * bmalloc/Deallocator.cpp:
575
576 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
577
578         Try to fix the build.
579
580         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
581
582 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
583
584         bmalloc should support system memory analysis tools (part 1)
585         https://bugs.webkit.org/show_bug.cgi?id=139559
586
587         Reviewed by Mark Lam.
588
589         This patch adds the hooks to disable bmalloc at runtime if certain
590         environment variables are set, but doesn't actually read from the
591         environment yet.
592
593         No performance change.
594
595         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
596         we'll use to read environment variables and see if memory analysis tools
597         have been enabled.
598
599         * bmalloc/Allocator.cpp:
600         (bmalloc::Allocator::Allocator):
601         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
602         on the allocation path. We cache the setting to make the check fast.
603
604         * bmalloc/Allocator.h: Interface changes.
605
606         * bmalloc/Cache.cpp:
607         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
608         and deallocator. This main purpose is to enable them to query the
609         environment for whether bmalloc is enabled; but this is also a slightly
610         cleaner way to guarantee to them that the Heap has been pre-initialized.
611
612         * bmalloc/Deallocator.cpp:
613         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
614         fill the object log to force us to take the slow path on all deallocations.
615
616         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
617
618         * bmalloc/Deallocator.h: Interface changes.
619
620         * bmalloc/Environment.cpp: Added.
621         (bmalloc::Environment::Environment):
622         (bmalloc::Environment::computeIsBmallocEnabled):
623         * bmalloc/Environment.h: Added.
624         (bmalloc::Environment::isBmallocEnabled): This is the class that will
625         encapsulate looking for environment variables that turn on heap
626         analysis tools.
627
628         * bmalloc/Heap.h:
629         (bmalloc::Heap::environment):
630
631         * bmalloc/Mutex.h:
632         (bmalloc::Mutex::Mutex):
633         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
634         since I got super confused about them while writing this patch.
635
636         * bmalloc/VMHeap.cpp: Fixed an #include.
637
638 2014-12-09  David Kilzer  <ddkilzer@apple.com>
639
640         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
641         <http://webkit.org/b/139212>
642
643         Reviewed by Joseph Pecoraro.
644
645         * Configurations/Base.xcconfig:
646         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
647           on OS X.
648         * Configurations/DebugRelease.xcconfig:
649         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
650
651 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
652
653         bmalloc uses 8X more virtual memory than necessary
654         https://bugs.webkit.org/show_bug.cgi?id=138495
655
656         Reviewed by Mark Lam.
657
658         iOS has a per-process virtual memory cap around 1GB, so there's some
659         value to not going totally ham with virtual memory.
660
661         We currently use about 8X the necessary amount:
662             - 2X to align our VM allocation
663             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
664
665         We can cut that down:
666             - Return the unaligned portion of our VM allocation (-2X)
667             - Use all the chunks in a superchunk, instead of allocating one
668               chunk per superchunk (-4X)
669
670         * bmalloc/Algorithm.h:
671         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
672         function so we can call it with getpagesize() at runtime.
673
674         * bmalloc/Chunk.h:
675         * bmalloc/LargeChunk.h:
676         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
677         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
678
679         * bmalloc/VMAllocate.h:
680         (bmalloc::vmValidate):
681         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
682         if it does not succeed. Allocate precisely, and give back the extra.
683
684         * bmalloc/VMHeap.cpp:
685         (bmalloc::VMHeap::allocateSuperChunk):
686         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
687         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
688         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
689         in a superchunk, instead of just one.
690
691         * bmalloc/VMHeap.h:
692         (bmalloc::VMHeap::allocateSmallPage):
693         (bmalloc::VMHeap::allocateMediumPage):
694         (bmalloc::VMHeap::allocateLargeRange):
695         * bmalloc/XLargeChunk.h:
696         (bmalloc::XLargeChunk::create): Updated to match changes above.
697
698 2014-11-01  David Kilzer  <ddkilzer@apple.com>
699
700         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
701         <https://webkit.org/b/138286>
702         <rdar://problem/18847087>
703
704         Reviewed by Dan Bernstein.
705
706         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
707         to NO for the target that produces libbmalloc.a so that the
708         debug symbols will be linked into JavaScriptCore and end up in
709         its dSYM file.
710
711 2014-10-30  Dana Burkart  <dburkart@apple.com>
712
713         <rdar://problem/18821260> Prepare for the mysterious future
714
715         Reviewed by Lucas Forschler.
716
717         * Configurations/Base.xcconfig:
718         * Configurations/DebugRelease.xcconfig:
719
720 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
721
722         bmalloc: cleaned up fast path vs slow path
723         https://bugs.webkit.org/show_bug.cgi?id=137081
724
725         Reviewed by Sam Weinig.
726
727         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
728
729         * bmalloc/Allocator.cpp:
730         (bmalloc::Allocator::Allocator): Merged the small and medium range
731         caches, just like the small and medium allocators. Ranges are abstract
732         objects that don't really care whether they hold small or medium objects,
733         so they don't need to be segregated.
734
735         (bmalloc::Allocator::scavenge): Ditto.
736
737         (bmalloc::Allocator::allocateBumpRangeSlowCase):
738         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
739         we do care a tiny bit, because we need to specify small vs medium when
740         allocating new ranges from the heap, to ensure that the heap allocates
741         from the right segment of VM.
742
743         (bmalloc::Allocator::allocateLarge):
744         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
745         up the fast path. Large allocation performance is dominated by allocation
746         logic and initialization, so inlining it doesn't help.
747
748         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
749         it doesn't need to distinguish small vs medium objects.
750
751         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
752         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
753
754         * bmalloc/Allocator.h:
755         * bmalloc/BumpRange.h:
756
757         * bmalloc/Cache.cpp:
758         (bmalloc::Cache::allocateSlowCase): Deleted.
759         (bmalloc::Cache::deallocateSlowCase): Deleted.
760         * bmalloc/Cache.h:
761         (bmalloc::Cache::allocate):
762         (bmalloc::Cache::deallocate):
763         (bmalloc::Cache::allocateFastCase): Deleted.
764         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
765         paths. The downside to this change is that the fast path branches to two
766         distinct failure cases instead of one. The upside is that the slow path
767         doesn't need to re-read the segment register, which is not as cheap as a
768         normal register, and it doesn't need to do an extra level of function 
769         call. Seems to be worth it.
770
771         * bmalloc/Deallocator.h:
772         * bmalloc/Heap.cpp:
773         (bmalloc::Heap::refillSmallBumpRangeCache):
774         (bmalloc::Heap::refillMediumBumpRangeCache):
775         * bmalloc/Heap.h: Updated for interface changes.
776
777         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
778         small lines in a page / 2, since any other free lines will coalesce
779         with their neighbors.
780
781 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
782
783         Rolled out r173346.
784
785             bmalloc should honor the FastMalloc statistics API
786             https://bugs.webkit.org/show_bug.cgi?id=136592
787
788         This didn't really work. Because we allow ranges with and without
789         physical pages to merge, and we allow double-committing and
790         double-decommitting, we can't rely on commit actions to track memory
791         footprint.
792
793         * bmalloc/Heap.cpp:
794         (bmalloc::Heap::size): Deleted.
795         (bmalloc::Heap::capacity): Deleted.
796         * bmalloc/Heap.h:
797         * bmalloc/VMHeap.cpp:
798         (bmalloc::VMHeap::VMHeap):
799         (bmalloc::VMHeap::allocateSmallChunk):
800         (bmalloc::VMHeap::allocateMediumChunk):
801         (bmalloc::VMHeap::allocateLargeChunk):
802         * bmalloc/VMHeap.h:
803         (bmalloc::VMHeap::allocateSmallPage):
804         (bmalloc::VMHeap::allocateMediumPage):
805         (bmalloc::VMHeap::allocateLargeRange):
806         (bmalloc::VMHeap::deallocateSmallPage):
807         (bmalloc::VMHeap::deallocateMediumPage):
808         (bmalloc::VMHeap::deallocateLargeRange):
809         (bmalloc::VMHeap::size): Deleted.
810         (bmalloc::VMHeap::capacity): Deleted.
811         * bmalloc/bmalloc.h:
812         (bmalloc::api::heapSize): Deleted.
813         (bmalloc::api::heapCapacity): Deleted.
814
815 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
816
817         bmalloc: Allocation should be more precise
818         https://bugs.webkit.org/show_bug.cgi?id=136993
819
820         Reviewed by Gavin Barraclough.
821
822         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
823
824         This patch teaches the allocator to merge adjacent free lines into a
825         single allocatable range. This allows us to shrink the size of an
826         individual line without increasing fragmentation or the rate of allocator
827         slow paths.
828
829         We'll only take more slow paths when available memory is sparse, which
830         is exactly when it's worth it. When available memory is dense, we'll
831         take fewer slow paths.
832
833         * bmalloc.xcodeproj/project.pbxproj:
834         * bmalloc/Algorithm.h:
835         (bmalloc::divideRoundingUp):
836
837         * bmalloc/Allocator.cpp:
838         (bmalloc::Allocator::Allocator): Updated for interface changes.
839
840         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
841         Now that we merge lines, it's not convenient to scavenge by line.
842
843         (bmalloc::Allocator::allocateSmallBumpRange):
844         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
845         instead of individual lines.
846
847         (bmalloc::Allocator::allocateSlowCase):
848         (bmalloc::Allocator::allocateSmallLine): Deleted.
849         (bmalloc::Allocator::allocateMediumLine): Deleted.
850         (bmalloc::Allocator::allocateMedium): Deleted.
851         * bmalloc/Allocator.h:
852         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
853         into the standard fast path with small allocations. Since a BumpAllocator
854         just allocates out of an arbitrary range, it doesn't need to distinguish
855         between small and medium lines.
856
857         * bmalloc/BumpAllocator.h:
858         (bmalloc::BumpAllocator::size):
859         (bmalloc::BumpAllocator::BumpAllocator):
860         (bmalloc::BumpAllocator::init):
861         (bmalloc::BumpAllocator::refill):
862         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
863         anymore: the heap just gives us a pointer and a pre-computed number of
864         objects, and we allocate them.
865
866         * bmalloc/Deallocator.cpp:
867         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
868
869         * bmalloc/Heap.cpp:
870         (bmalloc::Heap::Heap):
871         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
872         detailing where all objects will lie in memory. After we merge two lines,
873         we might allocate an object that spans from one line to the next. This
874         metadata details which bits of memory overlap in that way, and how they
875         overlap.
876
877         (bmalloc::Heap::refillSmallBumpRangeCache):
878         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
879         and merge adjacent free lines into BumpRanges.
880
881         (bmalloc::Heap::allocateSmallPage):
882         (bmalloc::Heap::allocateMediumPage):
883         (bmalloc::Heap::deallocateSmallLine):
884         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
885         since we scan for free memory a page at a time.
886
887         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
888         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
889         fast path.
890
891         * bmalloc/Heap.h:
892         (bmalloc::Heap::derefSmallLine):
893         (bmalloc::Heap::derefMediumLine):
894         (bmalloc::Heap::deallocateSmallLine): Deleted.
895         (bmalloc::Heap::allocateSmallLine): Deleted.
896         (bmalloc::Heap::deallocateMediumLine): Deleted.
897         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
898
899         * bmalloc/Line.h:
900         (bmalloc::Line<Traits>::ref):
901         (bmalloc::Line<Traits>::deref):
902         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
903         anymore, since we only ever deref by 1 now.
904
905         * bmalloc/MediumAllocator.h:
906         (bmalloc::MediumAllocator::isNull): Deleted.
907         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
908         (bmalloc::MediumAllocator::line): Deleted.
909         (bmalloc::MediumAllocator::allocate): Deleted.
910         (bmalloc::MediumAllocator::derefCount): Deleted.
911         (bmalloc::MediumAllocator::refill): Deleted.
912         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
913         been dead for a while, since it doesn't build anymore with this patch.
914
915         * bmalloc/Page.h:
916         (bmalloc::Page::sizeClass):
917         (bmalloc::Page::setSizeClass):
918         (bmalloc::Page::smallSizeClass): Deleted.
919         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
920         to sizeClass, since we use it for medium sizes too.
921
922         * bmalloc/Sizes.h:
923         (bmalloc::Sizes::sizeClass):
924         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
925
926         (bmalloc::Sizes::smallSizeClassFor): Deleted.
927         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
928
929         * bmalloc/bmalloc.h:
930         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
931         can be a bit more precise. It also has to be, since we can't guarantee
932         that an object ends at the end of a line anymore.
933
934 2014-09-19  Daniel Bates  <dabates@apple.com>
935
936         Always assume internal SDK when building configuration Production
937         https://bugs.webkit.org/show_bug.cgi?id=136925
938         <rdar://problem/18362399>
939
940         Reviewed by Dan Bernstein.
941
942         * Configurations/Base.xcconfig:
943
944 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
945
946         bmalloc: moved line caches from the deallocator to the allocator
947         https://bugs.webkit.org/show_bug.cgi?id=136868
948
949         Reviewed by Gavin Barraclough.
950
951         I did this mostly as a simplification, to make it easier to change the
952         allocation strategy.
953
954         No throughput change on MallocBench. Saves about 50kB.
955
956         Since the deallocator needs to lock the heap when freeing lines anyway,
957         there isn't much benefit to giving the deallocator a local cache of
958         deallocated lines.
959
960         We still give the allocator a local cache of lines because that does
961         reduce the frequency at which it needs to lock the heap in order to
962         acquire more lines.
963
964         * bmalloc/Allocator.cpp:
965         (bmalloc::Allocator::scavenge):
966         (bmalloc::Allocator::allocateSmallLine):
967         (bmalloc::Allocator::allocateMediumLine):
968         (bmalloc::Allocator::allocateMedium):
969         (bmalloc::Allocator::allocateSlowCase):
970         * bmalloc/Allocator.h:
971         * bmalloc/Deallocator.cpp:
972         (bmalloc::Deallocator::Deallocator):
973         (bmalloc::Deallocator::scavenge):
974         (bmalloc::Deallocator::processObjectLog):
975         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
976         (bmalloc::Deallocator::allocateSmallLine): Deleted.
977         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
978         (bmalloc::Deallocator::allocateMediumLine): Deleted.
979         * bmalloc/Deallocator.h:
980
981         * bmalloc/Sizes.h:
982         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
983         exactly one page in size. That's about what we were shooting for anyway,
984         and it may make it easier to switch to per-page allocation in future.
985
986 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
987
988         bmalloc: allocate small and medium objects using the same bump pointer class
989         https://bugs.webkit.org/show_bug.cgi?id=136843
990
991         Reviewed by Gavin Barraclough.
992
993         4% speedup on MallocBench.
994
995         Now that medium-sized objects have dedicated per-size allocators, they
996         don't need to use an arbitrary bump pointer allocator. This means that
997         every allocator knows how many objects it will allocate from the start,
998         and we don't need a post-processing step to adjust refcounts based on
999         real allocation count.
1000
1001         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
1002         since it's used for small and medium objects now.
1003
1004         * bmalloc/Allocator.cpp:
1005         (bmalloc::Allocator::Allocator): Updated to use new interface.
1006         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
1007         to make sure that we finish allocating all the objects in it.
1008
1009         (bmalloc::Allocator::allocateMedium):
1010         (bmalloc::Allocator::allocateSlowCase):
1011         (bmalloc::Allocator::retire): Deleted.
1012         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
1013         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
1014         * bmalloc/Allocator.h:
1015         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
1016         used to post-process an allocator based on how many objects it allocated.
1017
1018         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
1019         (bmalloc::BumpAllocator::BumpAllocator):
1020         (bmalloc::BumpAllocator::init):
1021         (bmalloc::BumpAllocator::line):
1022         (bmalloc::BumpAllocator::validate):
1023         (bmalloc::BumpAllocator::allocate):
1024         (bmalloc::BumpAllocator::refill):
1025         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1026         about the kinds of lines they allocate into. In some cases, the line
1027         type must be provided as a template parameter by the caller.
1028
1029         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1030         (bmalloc::SmallAllocator::line): Deleted.
1031         (bmalloc::SmallAllocator::allocate): Deleted.
1032         (bmalloc::SmallAllocator::objectCount): Deleted.
1033         (bmalloc::SmallAllocator::derefCount): Deleted.
1034         (bmalloc::SmallAllocator::refill): Deleted.
1035         (bmalloc::SmallAllocator::clear): Deleted.
1036
1037         * bmalloc/ObjectType.h:
1038         (bmalloc::isMedium):
1039
1040         * bmalloc/SmallAllocator.h:
1041         (bmalloc::SmallAllocator::isNull): Deleted.
1042         (bmalloc::SmallAllocator::canAllocate): Deleted.
1043         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1044         (bmalloc::SmallAllocator::line): Deleted.
1045         (bmalloc::SmallAllocator::allocate): Deleted.
1046         (bmalloc::SmallAllocator::objectCount): Deleted.
1047         (bmalloc::SmallAllocator::derefCount): Deleted.
1048         (bmalloc::SmallAllocator::refill): Deleted.
1049         (bmalloc::SmallAllocator::clear): Deleted.
1050
1051 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1052
1053         Fixed a goof in bmalloc Vector sizing
1054         https://bugs.webkit.org/show_bug.cgi?id=136795
1055
1056         Reviewed by Gavin Barraclough and Sam Weinig.
1057
1058         We want our minimum vector to be page-sized since the OS will give us
1059         a page no matter what -- but we want that many bytes, and not enough
1060         bytes to store that many elements.
1061
1062         * bmalloc/Vector.h: Math is hard.
1063
1064 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1065
1066         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1067         https://bugs.webkit.org/show_bug.cgi?id=136693
1068
1069         Reviewed by Gavin Barraclough.
1070
1071         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1072
1073         No throughput change.
1074
1075         We keep an array of medium allocators, just like our array of small
1076         allocators.
1077
1078         In future, we can simplify the allocation fast path by merging the small
1079         and medium allocator arrays. For now, this is the simplest change that
1080         gets the win.
1081
1082         * bmalloc/Allocator.cpp:
1083         (bmalloc::Allocator::Allocator):
1084         (bmalloc::Allocator::scavenge):
1085         (bmalloc::Allocator::allocateMedium):
1086         * bmalloc/Allocator.h:
1087         * bmalloc/Sizes.h:
1088         (bmalloc::Sizes::mediumSizeClassFor):
1089
1090 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1091
1092         Reviewed by Sam Weinig.
1093
1094         Renamed log => retire for clarity.
1095
1096         * bmalloc/Allocator.cpp:
1097         (bmalloc::Allocator::scavenge):
1098         (bmalloc::Allocator::retire):
1099         (bmalloc::Allocator::allocateMedium):
1100         (bmalloc::Allocator::allocateSlowCase):
1101         (bmalloc::Allocator::log): Deleted.
1102         * bmalloc/Allocator.h:
1103
1104 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1105
1106         bmalloc: eager scavenge leaves behind a bogus allocator
1107         https://bugs.webkit.org/show_bug.cgi?id=136743
1108
1109         Reviewed by Sam Weinig.
1110
1111         Be sure to clear the allocator after logging it in the eager scavenge
1112         case, so that we don't later try to allocate out of the lines that we
1113         have thrown away.
1114
1115         We didn't need to do this previously because scavenge would only happen
1116         at thread exit time, after which no further allocation from the per-thread
1117         cache would take place.
1118
1119         * bmalloc/Allocator.cpp:
1120         (bmalloc::Allocator::scavenge):
1121         * bmalloc/MediumAllocator.h:
1122         (bmalloc::MediumAllocator::clear):
1123         * bmalloc/SmallAllocator.h:
1124         (bmalloc::SmallAllocator::clear):
1125
1126 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1127
1128         bmalloc should honor the FastMalloc statistics API
1129         https://bugs.webkit.org/show_bug.cgi?id=136592
1130
1131         Reviewed by Gavin Barraclough.
1132
1133         We do this by tracking "size" and "capacity" in the VM heap.
1134
1135         The VM heap's "capacity" is all the VM we ever allocated.
1136
1137         The VM heap's "size" the subset of VM currently held onto by the
1138         VM heap (and therefore not in use by the regular heap).
1139
1140         Somewhat ironically, reducing the process's memory footprint, increases
1141         the size of the VM heap, since the VM heap holds the pages that are
1142         purely virtual and not physical.
1143
1144         * bmalloc/Heap.cpp:
1145         (bmalloc::Heap::size):
1146         (bmalloc::Heap::capacity):
1147         * bmalloc/Heap.h:
1148         * bmalloc/VMHeap.cpp:
1149         (bmalloc::VMHeap::VMHeap):
1150         (bmalloc::VMHeap::allocateSmallChunk):
1151         (bmalloc::VMHeap::allocateMediumChunk):
1152         (bmalloc::VMHeap::allocateLargeChunk):
1153         * bmalloc/VMHeap.h:
1154         (bmalloc::VMHeap::size):
1155         (bmalloc::VMHeap::capacity):
1156         (bmalloc::VMHeap::allocateSmallPage):
1157         (bmalloc::VMHeap::allocateMediumPage):
1158         (bmalloc::VMHeap::allocateLargeRange):
1159         (bmalloc::VMHeap::deallocateSmallPage):
1160         (bmalloc::VMHeap::deallocateMediumPage):
1161         (bmalloc::VMHeap::deallocateLargeRange):
1162         * bmalloc/bmalloc.h:
1163         (bmalloc::api::heapSize):
1164         (bmalloc::api::heapCapacity):
1165
1166 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1167
1168         bmalloc crashes on the EWS bots (due to bad large object allocation)
1169         https://bugs.webkit.org/show_bug.cgi?id=136469
1170
1171         Reviewed by Andreas Kling.
1172
1173         It's possible to convince bmalloc to perform a bad large object allocation,
1174         through these steps:
1175
1176         (1) Insert object A into freelist F0.
1177
1178         (2) Split, merge and split again A's neighbors such that object B is
1179         inserted into freelist F0, with boundary tag and size equal to object A,
1180         but pointer not completely equal to object A. Put object B at the head of F0.
1181
1182         (3) Allocate some other object from F0, swapping its position in the
1183         freelist with object B, such that object A is now ahead of object B.
1184
1185         --> Now, the next allocation for size A/B will allocate object A, which
1186         has a slightly wrong idea about where the object actually begins.
1187         Immediately, you'll corrupt a little memory, and over time, you'll also
1188         corrupt boundary tag metadata.
1189
1190         The solution is to store the begin pointer in the boundary tag. Luckily,
1191         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1192         on MallocBench.
1193
1194         * bmalloc/Algorithm.h:
1195         (bmalloc::rightShift):
1196         * bmalloc/BeginTag.h:
1197         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1198         validate the start pointer when popping off the free list. Through a
1199         very uncommon set of steps, it is possible to have an item in the free
1200         list that is valid by all accounts except for its start pointer.
1201
1202         * bmalloc/BoundaryTag.h:
1203         (bmalloc::BoundaryTag::compactBegin):
1204         (bmalloc::BoundaryTag::setRange):
1205         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1206         start pointer. We don't need the whole pointer -- just the offset, in
1207         largeAlignment increments, into the relevant boundary tag bucket.
1208
1209         * bmalloc/BoundaryTagInlines.h:
1210         (bmalloc::validateNext):
1211         (bmalloc::BoundaryTag::init):
1212         (bmalloc::BoundaryTag::mergeLarge):
1213         (bmalloc::BoundaryTag::splitLarge):
1214         * bmalloc/SegregatedFreeList.cpp:
1215         (bmalloc::SegregatedFreeList::insert):
1216         (bmalloc::SegregatedFreeList::takeGreedy):
1217         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1218         the size when establishing a boundary tag, as required by the new
1219         interface.
1220
1221         * bmalloc/Sizes.h:
1222
1223 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1224
1225         Fixed a bmalloc crash seen on the EWS bot
1226         https://bugs.webkit.org/show_bug.cgi?id=135955
1227
1228         Reviewed by Andreas Kling.
1229
1230         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1231         input buffer is a malloc region, that region will get marked Copy-On-Write
1232         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1233         on older OS X's. In 10.10, they still fail, but they do not return
1234         EINVAL.
1235
1236         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1237
1238 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1239
1240         Fixed the bmalloc build
1241         https://bugs.webkit.org/show_bug.cgi?id=135953
1242
1243         Reviewed by Andreas Kling.
1244
1245         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1246         These headers are used, so they must be available outside the project.
1247
1248 2014-08-13  Daniel Bates  <dabates@apple.com>
1249
1250         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1251         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1252
1253         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1254         implementation of PerThread<T>::getFastCase().
1255
1256         * bmalloc/PerThread.h:
1257         (bmalloc::PerThread<T>::getFastCase):
1258
1259 2014-08-13  Daniel Bates  <dabates@apple.com>
1260
1261         Make bmalloc::PerThread work without C++ thread local storage
1262         https://bugs.webkit.org/show_bug.cgi?id=135895
1263
1264         Reviewed by Geoffrey Garen.
1265
1266         Implement support for building bmalloc without C++ thread local storage.
1267
1268         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1269         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1270         to determine whether the compiler supports C++ thread local storage.
1271         * bmalloc/PerThread.h:
1272         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1273         without C++ thread local storage.
1274         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1275         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1276         PerThreadStorage::initSharedKeyIfNeeded().
1277         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1278         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1279         ensure that the shared key has been initialized.
1280         (_pthread_setspecific_direct): Deleted.
1281         (_pthread_getspecific_direct): Deleted.
1282
1283 2014-08-13  Daniel Bates  <dabates@apple.com>
1284
1285         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1286         https://bugs.webkit.org/show_bug.cgi?id=135848
1287
1288         Reviewed by Geoffrey Garen.
1289
1290         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1291         when building for the iOS Simulator.
1292         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1293         using the public SDK.
1294         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1295         using the public SDK.
1296         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1297         using the public SDK.
1298
1299 2014-08-12  Daniel Bates  <dabates@apple.com>
1300
1301         BPLATFORM(IOS) always evaluates to false
1302         https://bugs.webkit.org/show_bug.cgi?id=135843
1303
1304         Reviewed by Geoffrey Garen.
1305
1306         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1307         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1308         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1309         building for iOS.
1310
1311         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1312         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1313
1314         * bmalloc/BPlatform.h:
1315
1316 2014-08-11  Andy Estes  <aestes@apple.com>
1317
1318         [iOS] Get rid of iOS.xcconfig
1319         https://bugs.webkit.org/show_bug.cgi?id=135809
1320
1321         Reviewed by Joseph Pecoraro.
1322
1323         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1324
1325         * Configurations/Base.xcconfig:
1326         * Configurations/iOS.xcconfig: Removed.
1327         * bmalloc.xcodeproj/project.pbxproj:
1328
1329 2014-05-01  Dan Bernstein  <mitz@apple.com>
1330
1331         Fixed production builds for the iOS Simulator.
1332         <rdar://problem/16792221>
1333
1334         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1335         PRIVATE_HEADERS_FOLDER_PATH when installing.
1336
1337 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1338
1339         bmalloc: Segregate pages by objects size
1340         https://bugs.webkit.org/show_bug.cgi?id=131909
1341
1342         Reviewed by Andreas Kling.
1343
1344         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1345
1346         * bmalloc/Allocator.cpp:
1347         (bmalloc::Allocator::allocateSlowCase):
1348         * bmalloc/Allocator.h:
1349         (bmalloc::Allocator::allocateFastCase):
1350         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1351         function for size class calculation.
1352
1353         * bmalloc/Deallocator.cpp:
1354         (bmalloc::Deallocator::Deallocator):
1355         (bmalloc::Deallocator::scavenge):
1356         (bmalloc::Deallocator::deallocateSmallLine):
1357         (bmalloc::Deallocator::allocateSmallLine):
1358         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1359         cache can't be shared anymore.
1360
1361         * bmalloc/Heap.cpp:
1362         (bmalloc::Heap::allocateSmallLineSlowCase):
1363         * bmalloc/Heap.h:
1364         (bmalloc::Heap::deallocateSmallLine): Ditto.
1365
1366         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1367         page refcount when allocating a line because we might have deallocated
1368         the page and the recycled it for another size class.
1369
1370         (bmalloc::Heap::deallocateMediumLine):
1371         (bmalloc::Heap::allocateMediumLine):
1372         * bmalloc/Line.h:
1373         (bmalloc::Line::refCount):
1374         * bmalloc/Page.h:
1375         (bmalloc::Page::refCount):
1376         (bmalloc::Page::smallSizeClass):
1377         (bmalloc::Page::setSmallSizeClass):
1378         (bmalloc::Page<Traits>::refCount): Deleted.
1379         * bmalloc/Sizes.h:
1380         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1381         an index into an array from a size.
1382
1383 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1384
1385         bmalloc: Improved alignment in LargeChunk
1386         https://bugs.webkit.org/show_bug.cgi?id=131895
1387
1388         Reviewed by Andreas Kling.
1389
1390         * bmalloc/Chunk.h:
1391         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1392         Technically, the previous alignment was harmless, but I would prefer,
1393         dear reader, not to have to explain the interlocking set of
1394         circumstances that made it so.
1395
1396 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1397
1398         Rolled out r167502 because it caused a crash on the facebook benchmark.
1399
1400         Unreviewed.
1401
1402             bmalloc: Added an XSmall line size
1403             https://bugs.webkit.org/show_bug.cgi?id=131851
1404
1405             Reviewed by Sam Weinig.
1406
1407 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1408
1409         bmalloc: Mutex should be harder to use wrong
1410         https://bugs.webkit.org/show_bug.cgi?id=131879
1411
1412         Reviewed by Andreas Kling.
1413
1414         Mutex now has a proper constructor, so you can't deadlock by forgetting
1415         to initialize it.
1416
1417         * bmalloc.xcodeproj/project.pbxproj:
1418         * bmalloc/Allocator.cpp:
1419         (bmalloc::Allocator::processXSmallAllocatorLog):
1420         (bmalloc::Allocator::processSmallAllocatorLog):
1421         (bmalloc::Allocator::processMediumAllocatorLog):
1422         (bmalloc::Allocator::allocateLarge):
1423         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1424         since the Heap mutex is a static.
1425
1426         * bmalloc/AsyncTask.h:
1427         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1428         need for explicit initialization anymore.
1429
1430         * bmalloc/Deallocator.cpp:
1431         (bmalloc::Deallocator::scavenge):
1432         (bmalloc::Deallocator::deallocateLarge):
1433         (bmalloc::Deallocator::deallocateXLarge):
1434         (bmalloc::Deallocator::processObjectLog):
1435         (bmalloc::Deallocator::deallocateSmallLine):
1436         (bmalloc::Deallocator::deallocateXSmallLine):
1437         (bmalloc::Deallocator::allocateSmallLine):
1438         (bmalloc::Deallocator::allocateXSmallLine):
1439         (bmalloc::Deallocator::deallocateMediumLine):
1440         (bmalloc::Deallocator::allocateMediumLine):
1441         * bmalloc/Deallocator.h:
1442         * bmalloc/Heap.cpp:
1443         (bmalloc::sleep):
1444         (bmalloc::Heap::Heap):
1445         (bmalloc::Heap::concurrentScavenge):
1446         (bmalloc::Heap::scavenge):
1447         (bmalloc::Heap::scavengeSmallPages):
1448         (bmalloc::Heap::scavengeXSmallPages):
1449         (bmalloc::Heap::scavengeMediumPages):
1450         (bmalloc::Heap::scavengeLargeRanges):
1451         (bmalloc::Heap::allocateXSmallLineSlowCase):
1452         (bmalloc::Heap::allocateSmallLineSlowCase):
1453         (bmalloc::Heap::allocateMediumLineSlowCase):
1454         (bmalloc::Heap::allocateXLarge):
1455         (bmalloc::Heap::deallocateXLarge):
1456         (bmalloc::Heap::allocateLarge):
1457         (bmalloc::Heap::deallocateLarge):
1458         * bmalloc/Heap.h:
1459         (bmalloc::Heap::deallocateXSmallLine):
1460         (bmalloc::Heap::allocateXSmallLine):
1461         (bmalloc::Heap::deallocateSmallLine):
1462         (bmalloc::Heap::allocateSmallLine):
1463         (bmalloc::Heap::deallocateMediumLine):
1464         (bmalloc::Heap::allocateMediumLine):
1465         * bmalloc/Line.h:
1466         (bmalloc::Line<Traits>::deref):
1467         * bmalloc/Mutex.cpp: Removed.
1468         * bmalloc/Mutex.h:
1469         (bmalloc::Mutex::Mutex):
1470         (bmalloc::Mutex::init): Deleted.
1471         (bmalloc::Mutex::try_lock): Deleted.
1472         (bmalloc::Mutex::lock): Deleted.
1473         (bmalloc::Mutex::unlock): Deleted.
1474         * bmalloc/Page.h:
1475         (bmalloc::Page<Traits>::ref):
1476         (bmalloc::Page<Traits>::deref):
1477         (bmalloc::Page<Traits>::refCount):
1478         * bmalloc/PerProcess.h:
1479         (bmalloc::PerProcess::mutex):
1480         (bmalloc::PerProcess<T>::getSlowCase):
1481         * bmalloc/StaticMutex.cpp: Added.
1482         (bmalloc::StaticMutex::lockSlowCase):
1483         * bmalloc/StaticMutex.h: Added.
1484         (bmalloc::StaticMutex::init):
1485         (bmalloc::StaticMutex::try_lock):
1486         (bmalloc::StaticMutex::lock):
1487         (bmalloc::StaticMutex::unlock):
1488         * bmalloc/VMHeap.h:
1489         (bmalloc::VMHeap::deallocateXSmallPage):
1490         (bmalloc::VMHeap::deallocateSmallPage):
1491         (bmalloc::VMHeap::deallocateMediumPage):
1492         (bmalloc::VMHeap::deallocateLargeRange):
1493         * bmalloc/bmalloc.h:
1494         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1495         since the Heap mutex is a static.
1496
1497 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1498
1499         bmalloc: AsyncTask should use Mutex instead of std::mutex
1500         https://bugs.webkit.org/show_bug.cgi?id=131865
1501
1502         Reviewed by Gavin Barraclough.
1503
1504         std::mutex is so slow that it makes parallelizing simple tasks through
1505         AsyncTask a net regression. Mutex fixes this.
1506
1507         * bmalloc/AsyncTask.h:
1508         (bmalloc::Function>::AsyncTask):
1509         (bmalloc::Function>::join):
1510         (bmalloc::Function>::runSlowCase):
1511         (bmalloc::Function>::entryPoint):
1512         * bmalloc/Mutex.h:
1513         (bmalloc::Mutex::init):
1514
1515 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1516
1517         bmalloc: Added an XSmall line size
1518         https://bugs.webkit.org/show_bug.cgi?id=131851
1519
1520         Reviewed by Sam Weinig.
1521
1522         Reduces malloc footprint on Membuster recordings by 10%.
1523
1524         This is a throughput regression, but we're still way ahead of TCMalloc.
1525         I have some ideas for how to recover the regression -- but I wanted to
1526         get this win in first.
1527
1528         Full set of benchmark results:
1529
1530                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1531                                                                                                 
1532                                                                nopatch                      patch                                Δ
1533                 Peak Memory:
1534                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1535                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1536                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1537
1538                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1539                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1540                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1541
1542                 Memory at End:
1543                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1544                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1545                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1546
1547                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1548                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1549                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1550
1551                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1552                                                                                                 
1553                                                    nopatch                patch                         Δ
1554                 Execution Time:
1555                     churn                            127ms                151ms            ! 1.19x slower
1556                     list_allocate                    130ms                164ms            ! 1.26x slower
1557                     tree_allocate                    109ms                127ms            ! 1.17x slower
1558                     tree_churn                       115ms                120ms            ! 1.04x slower
1559                     facebook                         240ms                259ms            ! 1.08x slower
1560                     fragment                          91ms                131ms            ! 1.44x slower
1561                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1562                     message_one                      260ms                259ms             ^ 1.0x faster
1563                     message_many                     149ms                154ms            ! 1.03x slower
1564                     medium                           194ms                248ms            ! 1.28x slower
1565                     big                              157ms                160ms            ! 1.02x slower
1566
1567                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1568                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1569                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1570
1571                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1572                                                                                                 
1573                                                                        nopatch                          patch                                     Δ
1574                 Execution Time:
1575                     churn                                                126ms                          148ms                        ! 1.17x slower
1576                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1577                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1578                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1579                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1580                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1581                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1582                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1583                     facebook                                             240ms                          276ms                        ! 1.15x slower
1584                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1585                     fragment                                              92ms                          130ms                        ! 1.41x slower
1586                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1587                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1588                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1589                     message_one                                          260ms                          260ms                                      
1590                     message_many                                         170ms                          238ms                         ! 1.4x slower
1591                     medium                                               185ms                          250ms                        ! 1.35x slower
1592                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1593                     big                                                  150ms                          169ms                        ! 1.13x slower
1594                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1595
1596                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1597                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1598                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1599
1600                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1601
1602                                                                     TC                      patch                                Δ
1603                 Peak Memory:
1604                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1605                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1606                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1607
1608                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1609                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1610                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1611
1612                 Memory at End:
1613                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
1614                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
1615                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
1616
1617                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
1618                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
1619                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
1620
1621                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1622                                                                                                 
1623                                                         TC                patch                         Δ
1624                 Execution Time:
1625                     churn                            416ms                148ms            ^ 2.81x faster
1626                     list_allocate                    463ms                164ms            ^ 2.82x faster
1627                     tree_allocate                    292ms                127ms             ^ 2.3x faster
1628                     tree_churn                       157ms                120ms            ^ 1.31x faster
1629                     facebook                         327ms                276ms            ^ 1.18x faster
1630                     fragment                         335ms                129ms             ^ 2.6x faster
1631                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
1632                     message_one                      386ms                258ms             ^ 1.5x faster
1633                     message_many                     410ms                154ms            ^ 2.66x faster
1634                     medium                           391ms                245ms             ^ 1.6x faster
1635                     big                              261ms                167ms            ^ 1.56x faster
1636
1637                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
1638                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
1639                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
1640
1641         * bmalloc.xcodeproj/project.pbxproj:
1642         * bmalloc/Allocator.cpp:
1643         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
1644         index corresponds with its size. Instead, use the size selection function
1645         explicitly. Now that we have XSmall, some small allocator entries are
1646         unused.
1647
1648         (bmalloc::Allocator::scavenge):
1649         (bmalloc::Allocator::log):
1650         (bmalloc::Allocator::processXSmallAllocatorLog):
1651         (bmalloc::Allocator::allocateSlowCase):
1652         * bmalloc/Allocator.h:
1653         (bmalloc::Allocator::xSmallAllocatorFor):
1654         (bmalloc::Allocator::allocateFastCase):
1655         * bmalloc/Chunk.h:
1656         * bmalloc/Deallocator.cpp:
1657         (bmalloc::Deallocator::scavenge):
1658         (bmalloc::Deallocator::processObjectLog):
1659         (bmalloc::Deallocator::deallocateSlowCase):
1660         (bmalloc::Deallocator::deallocateXSmallLine):
1661         (bmalloc::Deallocator::allocateXSmallLine):
1662         * bmalloc/Deallocator.h:
1663         (bmalloc::Deallocator::deallocateFastCase):
1664         * bmalloc/Heap.cpp:
1665         (bmalloc::Heap::scavenge):
1666         (bmalloc::Heap::scavengeXSmallPages):
1667         (bmalloc::Heap::allocateXSmallLineSlowCase):
1668         * bmalloc/Heap.h:
1669         (bmalloc::Heap::deallocateXSmallLine):
1670         (bmalloc::Heap::allocateXSmallLine):
1671         * bmalloc/LargeChunk.h:
1672         (bmalloc::LargeChunk::get):
1673         (bmalloc::LargeChunk::endTag):
1674         * bmalloc/Line.h:
1675         * bmalloc/MediumAllocator.h:
1676         (bmalloc::MediumAllocator::allocate):
1677         (bmalloc::MediumAllocator::refill):
1678         * bmalloc/ObjectType.cpp:
1679         (bmalloc::objectType):
1680         * bmalloc/ObjectType.h:
1681         (bmalloc::isXSmall):
1682         (bmalloc::isSmall):
1683         (bmalloc::isMedium):
1684         (bmalloc::isLarge):
1685         (bmalloc::isSmallOrMedium): Deleted.
1686         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1687         handling small objects. There's probably a reasonable way to share this
1688         code in the future -- I'll look into that once it's stopped changing.
1689
1690         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1691         is the main reason things got slower.
1692
1693         * bmalloc/SmallAllocator.h:
1694         (bmalloc::SmallAllocator::allocate):
1695         * bmalloc/SmallTraits.h:
1696         * bmalloc/VMHeap.cpp:
1697         (bmalloc::VMHeap::allocateXSmallChunk):
1698         * bmalloc/VMHeap.h:
1699         (bmalloc::VMHeap::allocateXSmallPage):
1700         (bmalloc::VMHeap::deallocateXSmallPage):
1701         * bmalloc/XSmallAllocator.h: Added.
1702         (bmalloc::XSmallAllocator::isNull):
1703         (bmalloc::XSmallAllocator::canAllocate):
1704         (bmalloc::XSmallAllocator::XSmallAllocator):
1705         (bmalloc::XSmallAllocator::line):
1706         (bmalloc::XSmallAllocator::allocate):
1707         (bmalloc::XSmallAllocator::objectCount):
1708         (bmalloc::XSmallAllocator::derefCount):
1709         (bmalloc::XSmallAllocator::refill):
1710         * bmalloc/XSmallChunk.h: Added.
1711         * bmalloc/XSmallLine.h: Added.
1712         * bmalloc/XSmallPage.h: Added.
1713         * bmalloc/XSmallTraits.h: Added.
1714         * bmalloc/bmalloc.h:
1715         (bmalloc::api::realloc): Boiler-plate copy, as above.
1716
1717 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1718
1719         MallocBench should scavenge explicitly instead of waiting
1720         https://bugs.webkit.org/show_bug.cgi?id=131661
1721
1722         Reviewed by Andreas Kling.
1723
1724         Added explicit scavenge support to bmalloc. This isn't a memory win,
1725         since bmalloc's per-thread cache is so small. But it makes testing
1726         simpler.
1727
1728         * bmalloc/Allocator.cpp:
1729         (bmalloc::Allocator::~Allocator):
1730         (bmalloc::Allocator::scavenge):
1731         * bmalloc/Allocator.h:
1732         * bmalloc/Cache.cpp:
1733         (bmalloc::Cache::operator new):
1734         (bmalloc::Cache::operator delete):
1735         (bmalloc::Cache::Cache):
1736         (bmalloc::Cache::scavenge):
1737         * bmalloc/Cache.h:
1738         * bmalloc/Deallocator.cpp:
1739         (bmalloc::Deallocator::~Deallocator):
1740         (bmalloc::Deallocator::scavenge):
1741         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1742         functions, for reuse.
1743
1744         * bmalloc/Heap.cpp:
1745         (bmalloc::sleep):
1746         (bmalloc::Heap::concurrentScavenge):
1747         (bmalloc::Heap::scavenge):
1748         (bmalloc::Heap::scavengeSmallPages):
1749         (bmalloc::Heap::scavengeMediumPages):
1750         (bmalloc::Heap::scavengeLargeRanges):
1751         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1752         scavenging -- in response to a benchmark or a low memory warning --
1753         wants to complete as soon as possible, so its sleep duration is 0.
1754
1755         * bmalloc/bmalloc.h:
1756         (bmalloc::api::scavenge):
1757         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1758
1759 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1760
1761         Use 4kB pages on Mac
1762         https://bugs.webkit.org/show_bug.cgi?id=131658
1763
1764         Reviewed by Sam Weinig.
1765
1766         This reduces memory use a lot on Membuster:
1767
1768                                                                   base                      patch                                Δ
1769                 Execution Time:
1770                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1771                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1772                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1773
1774                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1775                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1776                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1777
1778                 Peak Memory:
1779                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1780                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1781                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1782
1783                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1784                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1785                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1786
1787                 Memory at End:
1788                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1789                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1790                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1791
1792                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1793                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1794                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1795
1796
1797         * bmalloc.xcodeproj/project.pbxproj:
1798         * bmalloc/BPlatform.h: Added.
1799         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1800         is 4kB on Mac.
1801
1802 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1803
1804         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1805
1806         * bmalloc.xcodeproj: Modified property svn:ignore.
1807
1808 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1809
1810         Fixed some mbmalloc exports
1811         https://bugs.webkit.org/show_bug.cgi?id=131599
1812
1813         Reviewed by Ryosuke Niwa.
1814
1815         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1816         of the project, so we can call them from API.
1817
1818         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1819         visibility, so they show up as exported in the .dylib.
1820
1821 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1822
1823         Put bmalloc headers in the right place
1824         https://bugs.webkit.org/show_bug.cgi?id=131464
1825
1826         Reviewed by Mark Rowe.
1827
1828         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1829         specify that we don't just want to dump all of our generically-named
1830         headers into /usr/local/include.
1831
1832 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1833
1834         Made bmalloc more #include friendly
1835         https://bugs.webkit.org/show_bug.cgi?id=131386
1836
1837         Reviewed by Andreas Kling.
1838
1839         Marked a bunch of headers private so they can be used from client code
1840         that #includes bmalloc.h.
1841
1842         Renamed ASSERT macros to BASSERT. This matches their header, which already
1843         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1844
1845         * bmalloc.xcodeproj/project.pbxproj:
1846         * bmalloc/Allocator.cpp:
1847         (bmalloc::Allocator::allocateSlowCase):
1848         * bmalloc/AsyncTask.h:
1849         (bmalloc::Function>::runSlowCase):
1850         * bmalloc/BAssert.h:
1851         * bmalloc/BoundaryTag.h:
1852         (bmalloc::BoundaryTag::setSize):
1853         * bmalloc/BoundaryTagInlines.h:
1854         (bmalloc::validate):
1855         (bmalloc::BoundaryTag::init):
1856         (bmalloc::BoundaryTag::deallocate):
1857         (bmalloc::BoundaryTag::splitLarge):
1858         (bmalloc::BoundaryTag::allocate):
1859         * bmalloc/Chunk.h:
1860         * bmalloc/Deallocator.cpp:
1861         (bmalloc::Deallocator::processObjectLog):
1862         (bmalloc::Deallocator::deallocateSlowCase):
1863         * bmalloc/Deallocator.h:
1864         (bmalloc::Deallocator::deallocateFastCase):
1865         * bmalloc/FixedVector.h:
1866         (bmalloc::Capacity>::operator):
1867         (bmalloc::Capacity>::push):
1868         (bmalloc::Capacity>::pop):
1869         (bmalloc::Capacity>::shrink):
1870         * bmalloc/Heap.cpp:
1871         (bmalloc::Heap::allocateLarge):
1872         * bmalloc/LargeChunk.h:
1873         (bmalloc::LargeChunk::get):
1874         (bmalloc::LargeChunk::endTag):
1875         * bmalloc/Line.h:
1876         (bmalloc::Line<Traits>::concurrentRef):
1877         (bmalloc::Line<Traits>::deref):
1878         * bmalloc/MediumAllocator.h:
1879         (bmalloc::MediumAllocator::allocate):
1880         * bmalloc/ObjectType.h:
1881         (bmalloc::isSmall):
1882         * bmalloc/Page.h:
1883         (bmalloc::Page<Traits>::ref):
1884         (bmalloc::Page<Traits>::deref):
1885         * bmalloc/PerThread.h:
1886         (bmalloc::PerThread<T>::getSlowCase):
1887         * bmalloc/SegregatedFreeList.cpp:
1888         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1889         (bmalloc::SegregatedFreeList::insert):
1890         * bmalloc/SmallAllocator.h:
1891         (bmalloc::SmallAllocator::allocate):
1892         (bmalloc::SmallAllocator::refill):
1893         * bmalloc/Syscall.h:
1894         * bmalloc/VMAllocate.h:
1895         (bmalloc::vmValidate):
1896         (bmalloc::vmAllocate):
1897         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1898         * bmalloc/Vector.h:
1899         (bmalloc::Vector<T>::operator):
1900         (bmalloc::Vector<T>::pop):
1901         (bmalloc::Vector<T>::shrink):
1902         * bmalloc/XLargeChunk.h:
1903         (bmalloc::XLargeChunk::range):
1904         (bmalloc::XLargeChunk::size):
1905
1906 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1907
1908         Removed an unused file.
1909
1910         Unreviewed.
1911
1912         * bmalloc/AsyncTask.cpp: Removed.
1913
1914 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1915
1916         Build bmalloc on Mac
1917         https://bugs.webkit.org/show_bug.cgi?id=131333
1918
1919         Reviewed by Mark Rowe.
1920
1921         * Makefile: Added. For make clients.
1922
1923         These files are required for building any project in WebKit. I copied
1924         them from WTF:
1925         * Configurations: Added.
1926         * Configurations/Base.xcconfig: Added.
1927         * Configurations/DebugRelease.xcconfig: Added.
1928         * Configurations/bmalloc.xcconfig: Added.
1929         * Configurations/iOS.xcconfig: Added.
1930         * Configurations/mbmalloc.xcconfig: Added.
1931
1932         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1933         from here because everything is in .xcconfig files now.
1934
1935         I had to fix a bunch of minor warnings, since they're enabled in our
1936         .xcconfig files:
1937
1938         * bmalloc/AsyncTask.h:
1939         (bmalloc::Function>::AsyncTask):
1940         * bmalloc/BAssert.h:
1941         * bmalloc/BoundaryTagInlines.h:
1942         (bmalloc::validate):
1943         * bmalloc/Heap.cpp:
1944         (bmalloc::Heap::Heap):
1945         (bmalloc::Heap::allocateLarge):
1946         (bmalloc::Heap::deallocateLarge):
1947         * bmalloc/Mutex.h:
1948         (bmalloc::Mutex::Mutex): Deleted.
1949         * bmalloc/VMAllocate.h:
1950         (bmalloc::vmValidate):
1951         * bmalloc/mbmalloc.cpp:
1952
1953 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1954
1955         bmalloc: Fixed a leak in the per-thread cache
1956         https://bugs.webkit.org/show_bug.cgi?id=131330
1957
1958         Reviewed by Andreas Kling.
1959
1960         Remember to deallocate our line caches upon thread exit.
1961
1962         * bmalloc/Deallocator.cpp:
1963         (bmalloc::Deallocator::~Deallocator):
1964
1965 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1966
1967         bmalloc: rolled out the tryLock experiment
1968         https://bugs.webkit.org/show_bug.cgi?id=131328
1969
1970         Reviewed by Andreas Kling.
1971
1972         It wasn't a speedup.
1973
1974         * bmalloc.xcodeproj/project.pbxproj:
1975         * bmalloc/Allocator.cpp:
1976         (bmalloc::Allocator::processSmallAllocatorLog):
1977         (bmalloc::Allocator::processMediumAllocatorLog):
1978         * bmalloc/Deallocator.cpp:
1979         (bmalloc::Deallocator::processObjectLog):
1980         (bmalloc::Deallocator::deallocateSlowCase):
1981         (bmalloc::Deallocator::deallocateSmallLine):
1982         (bmalloc::Deallocator::deallocateMediumLine):
1983         * bmalloc/Deallocator.h:
1984         (bmalloc::Deallocator::deallocateFastCase):
1985         * bmalloc/Heap.h:
1986         (bmalloc::Heap::deallocateSmallLine):
1987         (bmalloc::Heap::deallocateMediumLine):
1988         * bmalloc/Line.h:
1989         (bmalloc::Line<Traits>::deref):
1990         * bmalloc/Page.h:
1991         (bmalloc::Page<Traits>::deref):
1992
1993 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1994
1995         bmalloc
1996         https://bugs.webkit.org/show_bug.cgi?id=131170
1997
1998         Reviewed by Andreas Kling.
1999
2000         Initial commit.
2001
2002         * bmalloc: Added.
2003         * bmalloc.xcodeproj: Added.
2004         * bmalloc.xcodeproj/project.pbxproj: Added.
2005         * bmalloc/Algorithm.h: Added.
2006         (bmalloc::max):
2007         (bmalloc::min):
2008         (bmalloc::mask):
2009         (bmalloc::test):
2010         (bmalloc::roundUpToMultipleOf):
2011         (bmalloc::roundDownToMultipleOf):
2012         (bmalloc::sizeOf):
2013         (bmalloc::bitCount):
2014         (bmalloc::isPowerOfTwo):
2015         * bmalloc/Allocator.cpp: Added.
2016         (bmalloc::Allocator::Allocator):
2017         (bmalloc::Allocator::~Allocator):
2018         (bmalloc::Allocator::log):
2019         (bmalloc::Allocator::processSmallAllocatorLog):
2020         (bmalloc::Allocator::processMediumAllocatorLog):
2021         (bmalloc::Allocator::allocateLarge):
2022         (bmalloc::Allocator::allocateXLarge):
2023         (bmalloc::Allocator::allocateMedium):
2024         (bmalloc::Allocator::allocateSlowCase):
2025         * bmalloc/Allocator.h: Added.
2026         (bmalloc::Allocator::smallAllocatorFor):
2027         (bmalloc::Allocator::allocateFastCase):
2028         (bmalloc::Allocator::allocate):
2029         * bmalloc/AsyncTask.cpp: Added.
2030         (bmalloc::AsyncTask<Function>::runSlowCase):
2031         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2032         (bmalloc::AsyncTask<Function>::entryPoint):
2033         * bmalloc/AsyncTask.h: Added.
2034         (bmalloc::Function>::AsyncTask):
2035         (bmalloc::Function>::join):
2036         (bmalloc::Function>::run):
2037         (bmalloc::Function>::runSlowCase):
2038         (bmalloc::Function>::pthreadEntryPoint):
2039         (bmalloc::Function>::entryPoint):
2040         * bmalloc/BAssert.h: Added.
2041         * bmalloc/BeginTag.h: Added.
2042         (bmalloc::BeginTag::isInFreeList):
2043         * bmalloc/BoundaryTag.h: Added.
2044         (bmalloc::BoundaryTag::isXLarge):
2045         (bmalloc::BoundaryTag::setXLarge):
2046         (bmalloc::BoundaryTag::isFree):
2047         (bmalloc::BoundaryTag::setFree):
2048         (bmalloc::BoundaryTag::isEnd):
2049         (bmalloc::BoundaryTag::setEnd):
2050         (bmalloc::BoundaryTag::hasPhysicalPages):
2051         (bmalloc::BoundaryTag::setHasPhysicalPages):
2052         (bmalloc::BoundaryTag::isNull):
2053         (bmalloc::BoundaryTag::clear):
2054         (bmalloc::BoundaryTag::size):
2055         (bmalloc::BoundaryTag::setSize):
2056         (bmalloc::BoundaryTag::prev):
2057         (bmalloc::BoundaryTag::next):
2058         * bmalloc/BoundaryTagInlines.h: Added.
2059         (bmalloc::validate):
2060         (bmalloc::validatePrev):
2061         (bmalloc::validateNext):
2062         (bmalloc::BoundaryTag::init):
2063         (bmalloc::BoundaryTag::mergeLargeLeft):
2064         (bmalloc::BoundaryTag::mergeLargeRight):
2065         (bmalloc::BoundaryTag::mergeLarge):
2066         (bmalloc::BoundaryTag::deallocate):
2067         (bmalloc::BoundaryTag::splitLarge):
2068         (bmalloc::BoundaryTag::allocate):
2069         * bmalloc/Cache.cpp: Added.
2070         (bmalloc::Cache::operator new):
2071         (bmalloc::Cache::operator delete):
2072         (bmalloc::Cache::Cache):
2073         (bmalloc::Cache::allocateSlowCase):
2074         (bmalloc::Cache::allocateSlowCaseNullCache):
2075         (bmalloc::Cache::deallocateSlowCase):
2076         (bmalloc::Cache::deallocateSlowCaseNullCache):
2077         * bmalloc/Cache.h: Added.
2078         (bmalloc::Cache::allocator):
2079         (bmalloc::Cache::deallocator):
2080         (bmalloc::Cache::allocateFastCase):
2081         (bmalloc::Cache::deallocateFastCase):
2082         (bmalloc::Cache::allocate):
2083         (bmalloc::Cache::deallocate):
2084         * bmalloc/Chunk.h: Added.
2085         (bmalloc::Chunk::begin):
2086         (bmalloc::Chunk::end):
2087         (bmalloc::Chunk::lines):
2088         (bmalloc::Chunk::pages):
2089         * bmalloc/Deallocator.cpp: Added.
2090         (bmalloc::Deallocator::Deallocator):
2091         (bmalloc::Deallocator::~Deallocator):
2092         (bmalloc::Deallocator::deallocateLarge):
2093         (bmalloc::Deallocator::deallocateXLarge):
2094         (bmalloc::Deallocator::processObjectLog):
2095         (bmalloc::Deallocator::deallocateSlowCase):
2096         (bmalloc::Deallocator::deallocateSmallLine):
2097         (bmalloc::Deallocator::allocateSmallLine):
2098         (bmalloc::Deallocator::deallocateMediumLine):
2099         (bmalloc::Deallocator::allocateMediumLine):
2100         * bmalloc/Deallocator.h: Added.
2101         (bmalloc::Deallocator::deallocateFastCase):
2102         (bmalloc::Deallocator::deallocate):
2103         * bmalloc/EndTag.h: Added.
2104         (bmalloc::EndTag::operator=):
2105         * bmalloc/FixedVector.h: Added.
2106         (bmalloc::FixedVector::begin):
2107         (bmalloc::FixedVector::end):
2108         (bmalloc::FixedVector::size):
2109         (bmalloc::FixedVector::capacity):
2110         (bmalloc::FixedVector::clear):
2111         (bmalloc::FixedVector::isEmpty):
2112         (bmalloc::Capacity>::FixedVector):
2113         (bmalloc::Capacity>::operator):
2114         (bmalloc::Capacity>::push):
2115         (bmalloc::Capacity>::pop):
2116         (bmalloc::Capacity>::shrink):
2117         * bmalloc/Heap.cpp: Added.
2118         (bmalloc::sleep):
2119         (bmalloc::Heap::Heap):
2120         (bmalloc::Heap::concurrentScavenge):
2121         (bmalloc::Heap::scavengeSmallPages):
2122         (bmalloc::Heap::scavengeMediumPages):
2123         (bmalloc::Heap::scavengeLargeRanges):
2124         (bmalloc::Heap::allocateSmallLineSlowCase):
2125         (bmalloc::Heap::allocateMediumLineSlowCase):
2126         (bmalloc::Heap::allocateXLarge):
2127         (bmalloc::Heap::deallocateXLarge):
2128         (bmalloc::Heap::allocateLarge):
2129         (bmalloc::Heap::deallocateLarge):
2130         * bmalloc/Heap.h: Added.
2131         (bmalloc::Heap::deallocateSmallLine):
2132         (bmalloc::Heap::allocateSmallLine):
2133         (bmalloc::Heap::deallocateMediumLine):
2134         (bmalloc::Heap::allocateMediumLine):
2135         * bmalloc/Inline.h: Added.
2136         * bmalloc/LargeChunk.h: Added.
2137         (bmalloc::LargeChunk::begin):
2138         (bmalloc::LargeChunk::end):
2139         (bmalloc::LargeChunk::create):
2140         (bmalloc::LargeChunk::get):
2141         (bmalloc::LargeChunk::beginTag):
2142         (bmalloc::LargeChunk::endTag):
2143         * bmalloc/Line.h: Added.
2144         (bmalloc::Line<Traits>::begin):
2145         (bmalloc::Line<Traits>::end):
2146         (bmalloc::Line<Traits>::concurrentRef):
2147         (bmalloc::Line<Traits>::deref):
2148         * bmalloc/MediumAllocator.h: Added.
2149         (bmalloc::MediumAllocator::isNull):
2150         (bmalloc::MediumAllocator::MediumAllocator):
2151         (bmalloc::MediumAllocator::line):
2152         (bmalloc::MediumAllocator::allocate):
2153         (bmalloc::MediumAllocator::derefCount):
2154         (bmalloc::MediumAllocator::refill):
2155         * bmalloc/MediumChunk.h: Added.
2156         * bmalloc/MediumLine.h: Added.
2157         * bmalloc/MediumPage.h: Added.
2158         * bmalloc/MediumTraits.h: Added.
2159         * bmalloc/Mutex.cpp: Added.
2160         (bmalloc::Mutex::lockSlowCase):
2161         * bmalloc/Mutex.h: Added.
2162         (bmalloc::Mutex::Mutex):
2163         (bmalloc::Mutex::try_lock):
2164         (bmalloc::Mutex::lock):
2165         (bmalloc::Mutex::unlock):
2166         * bmalloc/ObjectType.cpp: Added.
2167         (bmalloc::objectType):
2168         * bmalloc/ObjectType.h: Added.
2169         (bmalloc::isSmallOrMedium):
2170         (bmalloc::isSmall):
2171         * bmalloc/Page.h: Added.
2172         (bmalloc::Page<Traits>::ref):
2173         (bmalloc::Page<Traits>::deref):
2174         (bmalloc::Page<Traits>::refCount):
2175         * bmalloc/PerProcess.h: Added.
2176         (bmalloc::PerProcess::mutex):
2177         (bmalloc::PerProcess<T>::getFastCase):
2178         (bmalloc::PerProcess<T>::get):
2179         (bmalloc::PerProcess<T>::getSlowCase):
2180         * bmalloc/PerThread.h: Added.
2181         (bmalloc::PerThreadStorage<Cache>::get):
2182         (bmalloc::PerThreadStorage<Cache>::init):
2183         (bmalloc::PerThreadStorage::get):
2184         (bmalloc::PerThreadStorage::init):
2185         (bmalloc::PerThread<T>::getFastCase):
2186         (bmalloc::PerThread<T>::get):
2187         (bmalloc::PerThread<T>::destructor):
2188         (bmalloc::PerThread<T>::getSlowCase):
2189         * bmalloc/Range.h: Added.
2190         (bmalloc::Range::Range):
2191         (bmalloc::Range::begin):
2192         (bmalloc::Range::end):
2193         (bmalloc::Range::size):
2194         (bmalloc::Range::operator!):
2195         (bmalloc::Range::operator<):
2196         * bmalloc/SegregatedFreeList.cpp: Added.
2197         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2198         (bmalloc::SegregatedFreeList::insert):
2199         (bmalloc::SegregatedFreeList::takeGreedy):
2200         (bmalloc::SegregatedFreeList::take):
2201         * bmalloc/SegregatedFreeList.h: Added.
2202         * bmalloc/Sizes.h: Added.
2203         * bmalloc/SmallAllocator.h: Added.
2204         (bmalloc::SmallAllocator::isNull):
2205         (bmalloc::SmallAllocator::canAllocate):
2206         (bmalloc::SmallAllocator::SmallAllocator):
2207         (bmalloc::SmallAllocator::line):
2208         (bmalloc::SmallAllocator::allocate):
2209         (bmalloc::SmallAllocator::objectCount):
2210         (bmalloc::SmallAllocator::derefCount):
2211         (bmalloc::SmallAllocator::refill):
2212         * bmalloc/SmallChunk.h: Added.
2213         * bmalloc/SmallLine.h: Added.
2214         * bmalloc/SmallPage.h: Added.
2215         * bmalloc/SmallTraits.h: Added.
2216         * bmalloc/Syscall.h: Added.
2217         * bmalloc/VMAllocate.h: Added.
2218         (bmalloc::vmSize):
2219         (bmalloc::vmValidate):
2220         (bmalloc::vmAllocate):
2221         (bmalloc::vmDeallocate):
2222         (bmalloc::vmDeallocatePhysicalPages):
2223         (bmalloc::vmAllocatePhysicalPages):
2224         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2225         (bmalloc::vmAllocatePhysicalPagesSloppy):
2226         * bmalloc/VMHeap.cpp: Added.
2227         (bmalloc::VMHeap::VMHeap):
2228         (bmalloc::VMHeap::allocateSmallChunk):
2229         (bmalloc::VMHeap::allocateMediumChunk):
2230         (bmalloc::VMHeap::allocateLargeChunk):
2231         * bmalloc/VMHeap.h: Added.
2232         (bmalloc::VMHeap::allocateSmallPage):
2233         (bmalloc::VMHeap::allocateMediumPage):
2234         (bmalloc::VMHeap::allocateLargeRange):
2235         (bmalloc::VMHeap::deallocateSmallPage):
2236         (bmalloc::VMHeap::deallocateMediumPage):
2237         (bmalloc::VMHeap::deallocateLargeRange):
2238         * bmalloc/Vector.h: Added.
2239         (bmalloc::Vector::begin):
2240         (bmalloc::Vector::end):
2241         (bmalloc::Vector::size):
2242         (bmalloc::Vector::capacity):
2243         (bmalloc::Vector::last):
2244         (bmalloc::Vector::pop):
2245         (bmalloc::Vector<T>::Vector):
2246         (bmalloc::Vector<T>::~Vector):
2247         (bmalloc::Vector<T>::operator):
2248         (bmalloc::Vector<T>::push):
2249         (bmalloc::Vector<T>::pop):
2250         (bmalloc::Vector<T>::shrink):
2251         (bmalloc::Vector<T>::reallocateBuffer):
2252         (bmalloc::Vector<T>::shrinkCapacity):
2253         (bmalloc::Vector<T>::growCapacity):
2254         * bmalloc/XLargeChunk.h: Added.
2255         (bmalloc::XLargeChunk::get):
2256         (bmalloc::XLargeChunk::begin):
2257         (bmalloc::XLargeChunk::XLargeChunk):
2258         (bmalloc::XLargeChunk::create):
2259         (bmalloc::XLargeChunk::destroy):
2260         (bmalloc::XLargeChunk::range):
2261         (bmalloc::XLargeChunk::size):
2262         * bmalloc/bmalloc.h: Added.
2263         (bmalloc::api::malloc):
2264         (bmalloc::api::free):
2265         (bmalloc::api::realloc):
2266         * bmalloc/mbmalloc.cpp: Added.
2267