Unreviewed, rolling out r180363.
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-02-19  Commit Queue  <commit-queue@webkit.org>
2
3         Unreviewed, rolling out r180363.
4         https://bugs.webkit.org/show_bug.cgi?id=141814
5
6         Caused >50 crashes when running LayoutTests in GuardMalloc or
7         ASAN modes. (Requested by jernoble on #webkit).
8
9         Reverted changeset:
10
11         "bmalloc should implement malloc introspection (to stop false-
12         positive leaks when MallocStackLogging is off)"
13         https://bugs.webkit.org/show_bug.cgi?id=141802
14         http://trac.webkit.org/changeset/180363
15
16 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
17
18         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
19         https://bugs.webkit.org/show_bug.cgi?id=141802
20
21         Reviewed by Andreas Kling.
22
23         Fixed a last-minute type.
24
25         The macro is OS, not PLATFORM.
26
27         * bmalloc/VMHeap.cpp:
28         (bmalloc::VMHeap::grow):
29         * bmalloc/VMHeap.h:
30         * bmalloc/Zone.h:
31
32 2015-02-19  Geoffrey Garen  <ggaren@apple.com>
33
34         bmalloc should implement malloc introspection (to stop false-positive leaks when MallocStackLogging is off)
35         https://bugs.webkit.org/show_bug.cgi?id=141802
36
37         Reviewed by Andreas Kling.
38
39         This patch does the bare minimum to stop false positive leaks from
40         being reported by the Darwin leaks tool. We register each super chunk
41         as a single object, and then request that the leaks tool scan it.
42
43         * bmalloc.xcodeproj/project.pbxproj: Added an abstraction for the malloc
44         zone introspection API.
45
46         * bmalloc/Algorithm.h: Missing #include.
47
48         * bmalloc/VMHeap.cpp:
49         (bmalloc::VMHeap::grow):
50         * bmalloc/VMHeap.h: Adopt the new abstraction.
51
52         * bmalloc/Zone.cpp: Added.
53         (bmalloc::remoteRead): Helper for reading an object out of another process.
54         (bmalloc::Zone::enumerator):
55         (bmalloc::Zone::Zone): Register a malloc zone so that we will participate
56         in introspection.
57
58         * bmalloc/Zone.h: Added.
59         (bmalloc::Zone::superChunks):
60         (bmalloc::Zone::addSuperChunk): Use a non-dynamically-allocated vector
61         since our dynamic allocations will not be scanned by leaks since they
62         will have the malloc VM tag.
63
64 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
65
66         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
67         https://bugs.webkit.org/show_bug.cgi?id=141759
68
69         Reviewed by Andreas Kling.
70
71         * bmalloc.xcodeproj/project.pbxproj:
72         * bmalloc/SuperChunk.h: Added.
73         (bmalloc::SuperChunk::create):
74         (bmalloc::SuperChunk::SuperChunk):
75         (bmalloc::SuperChunk::smallChunk):
76         (bmalloc::SuperChunk::mediumChunk):
77         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
78         into a separate class, for clarity and type safety.
79
80         * bmalloc/VMHeap.cpp:
81         (bmalloc::VMHeap::grow):
82         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
83         "grow" because Andreas found "allocateSuperChunk" to be unclear.
84
85         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
86         for malloc introspection.
87
88         (bmalloc::VMHeap::allocateSmallPage):
89         (bmalloc::VMHeap::allocateMediumPage):
90         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
91
92 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
93
94         Build bmalloc through CMake as a static library. It's then linked either
95         into the WTF library (if built as a shared library) or into the JSC and
96         WebKit2 libraries. There's no need to build it as a standalone shared library.
97
98         Rubber-stamped by Carlos Garcia Campos.
99
100         * CMakeLists.txt:
101
102 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
103
104         [BMalloc] Add a FIXME comment for memory alignas
105         https://bugs.webkit.org/show_bug.cgi?id=141556
106
107         Reviewed by Csaba Osztrogonác.
108
109         * bmalloc/Chunk.h: Add a FIXME comment.
110         * bmalloc/LargeChunk.h: ditto.
111
112 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
113
114         bmalloc buildfix on 32 bit Linux (x86/ARM)
115         https://bugs.webkit.org/show_bug.cgi?id=141472
116
117         Reviewed by Gyuyoung Kim.
118
119         * bmalloc/Algorithm.h:
120         (bmalloc::roundUpToMultipleOf):
121         * bmalloc/FixedVector.h:
122         (bmalloc::FixedVector::clear):
123         * bmalloc/Sizes.h:
124         (bmalloc::Sizes::sizeClass):
125
126 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
127
128         [EFL][GTK] Use bmalloc instead of tcmalloc
129         https://bugs.webkit.org/show_bug.cgi?id=140162
130
131         Reviewed by Carlos Garcia Campos.
132
133         Support to use bmalloc on EFL and GTK ports.
134
135         * CMakeLists.txt: Added.
136         * bmalloc/Allocator.cpp:
137         (bmalloc::Allocator::allocate):
138             Fix unused return value caused by posix_memalign().
139         * bmalloc/AsyncTask.h:
140         * bmalloc/BoundaryTag.h:
141         (bmalloc::BoundaryTag::clear):
142         * bmalloc/Chunk.h:
143             Change Traits::Page with Traits::PageType in order to fix
144             -fpermitive build error on EFL and GTK port. 
145         * bmalloc/EndTag.h:
146         (bmalloc::EndTag::operator=):
147         * bmalloc/Line.h: ditto.
148         * bmalloc/MediumTraits.h:
149         * bmalloc/Page.h: ditto.
150         * bmalloc/PerThread.h:
151             EFL port doesn't support __has_include definition yet.
152             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
153         * bmalloc/SmallTraits.h: ditto.
154         * bmalloc/VMAllocate.h:
155         (bmalloc::vmDeallocatePhysicalPages):
156         (bmalloc::vmAllocatePhysicalPages):
157         * bmalloc/Vector.h:
158         (bmalloc::Vector<T>::push):
159         (bmalloc::Vector<T>::reallocateBuffer):
160
161 2015-01-31  Sam Weinig  <sam@webkit.org>
162
163         Remove even more Mountain Lion support
164         https://bugs.webkit.org/show_bug.cgi?id=141124
165
166         Reviewed by Alexey Proskuryakov.
167
168         * Configurations/Base.xcconfig:
169         * Configurations/DebugRelease.xcconfig:
170
171 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
172
173         GC marking threads should clear malloc caches
174         https://bugs.webkit.org/show_bug.cgi?id=141097
175
176         Reviewed by Andreas Kling.
177
178         Split the scavenging API into per-thread vs global, so that you can
179         request to scavenge your own thread without scavenging the whole heap.
180
181         * bmalloc/Cache.cpp:
182         (bmalloc::Cache::scavenge):
183         * bmalloc/bmalloc.h:
184         (bmalloc::api::scavengeThisThread):
185         (bmalloc::api::scavenge):
186
187 2015-01-28  Dana Burkart  <dburkart@apple.com>
188
189         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
190         https://bugs.webkit.org/show_bug.cgi?id=136765
191
192         Reviewed by Alexey Proskuryakov.
193
194         * Configurations/Base.xcconfig:
195         * Configurations/DebugRelease.xcconfig:
196
197 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
198
199         bmalloc: support aligned allocation
200         https://bugs.webkit.org/show_bug.cgi?id=140732
201
202         Reviewed by Andreas Kling.
203
204         * bmalloc/Allocator.cpp:
205         (bmalloc::Allocator::allocate): New function for aligned allocation.
206
207         Small and medium requests just allocate and free until they find an
208         aligned pointer. This is slightly inefficient in the worst case, but
209         still constant-time with little-to-no space overhead.
210
211         Large requests use a new API that requires the client to specify both
212         its ideal size and alignment, and the worst-case size you would have to
213         allocate in order to produce some interior pointer of the requested size
214         and alignment. We put the burden of this calculation on the client
215         because it simplifies things if we guarantee that allocation won't fail.
216
217         XLarge requests are easy: we just forward them to vmAllocate, which
218         already supported aligned requests.
219
220         * bmalloc/BoundaryTag.h:
221         * bmalloc/BoundaryTagInlines.h:
222         (bmalloc::BoundaryTag::mergeLeft):
223         (bmalloc::BoundaryTag::mergeRight):
224         (bmalloc::BoundaryTag::merge):
225         (bmalloc::BoundaryTag::deallocate):
226         (bmalloc::BoundaryTag::split):
227         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
228         refactored the interface to remove some reference out parameters in
229         order to clarify what changes and what doesn't.
230
231         * bmalloc/Heap.cpp:
232         (bmalloc::Heap::allocateXLarge): Added an alignment API.
233
234         (bmalloc::Heap::allocateLarge):
235         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
236         a few variants, so aligned and unaligned allocation could share some code.
237
238         * bmalloc/SegregatedFreeList.cpp:
239         (bmalloc::SegregatedFreeList::take):
240         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
241         for aligned allocation. It turns out that the aligned path is pretty
242         different, since it ends up searching for two potential ways to satisfy
243         an allocation: either large enough and aligned, or large enough to split
244         into something not aligned and something large enough and aligned.
245
246         * bmalloc/VMAllocate.h:
247         (bmalloc::vmAllocate): Switched alignment to come before size because
248         that's how the memalign API specifies it.
249
250         * bmalloc/VMHeap.h:
251         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
252
253 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
254
255         bmalloc: a little bit of cleanup
256         https://bugs.webkit.org/show_bug.cgi?id=140687
257
258         Reviewed by Anders Carlsson.
259
260         * bmalloc/Algorithm.h:
261         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
262         of code.
263
264         * bmalloc/BoundaryTag.h:
265         * bmalloc/BoundaryTagInlines.h:
266         (bmalloc::BoundaryTag::mergeLeft):
267         (bmalloc::BoundaryTag::mergeRight):
268         (bmalloc::BoundaryTag::merge):
269         (bmalloc::BoundaryTag::deallocate):
270         (bmalloc::BoundaryTag::split):
271         (bmalloc::BoundaryTag::allocate):
272         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
273         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
274         (bmalloc::BoundaryTag::mergeLarge): Deleted.
275         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
276         from all these functions, since boundary tags always pertain to large
277         objects, and putting the word "Large" everywhere wasn't helping to
278         explain that.
279
280         * bmalloc/Heap.cpp:
281         (bmalloc::Heap::allocateXLarge):
282         (bmalloc::Heap::findXLarge):
283         (bmalloc::Heap::deallocateXLarge):
284         * bmalloc/Heap.h:
285         * bmalloc/VMHeap.h:
286         (bmalloc::VMHeap::allocateXLarge): Deleted.
287         (bmalloc::VMHeap::findXLarge): Deleted.
288         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
289         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
290         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
291         really make sense for the VMHeap to be involved.
292
293 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
294
295         bmalloc: refactored XLarge allocation for better alignment
296         https://bugs.webkit.org/show_bug.cgi?id=140582
297
298         Reviewed by Andreas Kling.
299
300         XLarge objects used to be Large objects with an extra bit of metadata
301         that said "actually, I'm not large -- I'm extra large".
302
303         The metadata header in an XLarge allocation made it impossible for the
304         XLarge object to honor a very large alignment request.
305
306         The solution is to stop using a metadata header for XLarge objects, and
307         instead to store explicit metadata on the side.
308
309         This is a bit less astonishing, which is also nice.
310
311         Finding XLarge metadata is now a linear search. That's probably OK, since
312         it was always so in TCMalloc, and the usual number of XLarge allocations
313         in a process is 0.
314
315         This design makes it possible for the heap to cache XLarge allocations
316         with and/or without physical pages. I haven't actually done that yet
317         because the tradeoffs are subtle, so I don't want to do anything without
318         a motivating test case.
319
320         * bmalloc.xcodeproj/project.pbxproj:
321         * bmalloc/Allocator.cpp:
322         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
323         since an XLarge allocation is now just a naked buffer without a header.
324
325         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
326         XLarge alignment, since XLargeChunk won't give this to us implicitly
327         anymore.
328
329         * bmalloc/BoundaryTag.h:
330         (bmalloc::BoundaryTag::setRange):
331         (bmalloc::BoundaryTag::isXLarge): Deleted.
332         (bmalloc::BoundaryTag::setXLarge): Deleted.
333         * bmalloc/BoundaryTagInlines.h:
334         (bmalloc::validate):
335         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
336
337         * bmalloc/Deallocator.cpp:
338         (bmalloc::Deallocator::deallocateXLarge):
339         (bmalloc::Deallocator::deallocateSlowCase):
340         * bmalloc/Heap.cpp:
341         (bmalloc::Heap::findXLarge):
342         (bmalloc::Heap::allocateXLarge):
343         (bmalloc::Heap::deallocateXLarge):
344         * bmalloc/Heap.h: Updated for interface changes.
345
346         * bmalloc/ObjectType.cpp:
347         (bmalloc::objectType):
348         * bmalloc/ObjectType.h:
349         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
350         examining its bit pattern -- just like we do for other kinds of
351         allocations -- which is nice.
352
353         * bmalloc/Sizes.h:
354         * bmalloc/VMHeap.h:
355         (bmalloc::VMHeap::allocateXLarge):
356         (bmalloc::VMHeap::findXLarge):
357         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
358         for XLarge allocations.
359
360         * bmalloc/XLargeChunk.h: Removed.
361
362 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
363
364         bmalloc: added some infrastructure for aligned allocation
365         https://bugs.webkit.org/show_bug.cgi?id=140572
366
367         Reviewed by Andreas Kling.
368
369         * bmalloc/Algorithm.h:
370         (bmalloc::isPowerOfTwo):
371         (bmalloc::roundUpToMultipleOf):
372         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
373         isPowerOfTwo helper function.
374
375         * bmalloc/Allocator.cpp:
376         (bmalloc::Allocator::allocate):
377         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
378         Doesn't do anything yet, but does correctly forward to system malloc
379         when bmalloc is disabled.
380
381         * bmalloc/Cache.cpp:
382         (bmalloc::Cache::allocateSlowCaseNullCache):
383         * bmalloc/Cache.h:
384         (bmalloc::Cache::allocate):
385         * bmalloc/bmalloc.h:
386         (bmalloc::api::memalign):
387         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
388
389 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
390
391         Consider alignment when allocating from a SegregatedFreeList
392         https://bugs.webkit.org/show_bug.cgi?id=140408
393
394         Reviewed by Sam Weinig.
395
396         In preparation for supporting aligned allocation.
397
398         No performance change.
399
400         Since this is just one extra branch in an already expensive function,
401         I decided not to duplicate the function just to avoid the branch in
402         the un-aligned case.
403
404         * bmalloc/SegregatedFreeList.cpp:
405         (bmalloc::SegregatedFreeList::take):
406         * bmalloc/SegregatedFreeList.h:
407
408 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
409
410         Renamed minimum to size in SegregatedFreeList
411         https://bugs.webkit.org/show_bug.cgi?id=140406
412
413         Reviewed by Sam Weinig.
414
415         In preparation for supporting aligned allocation.
416
417         * bmalloc/SegregatedFreeList.cpp:
418         (bmalloc::SegregatedFreeList::takeGreedy):
419         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
420         really just a minimum. Let's not imply that this value is special.
421
422 2015-01-11  Dan Bernstein  <mitz@apple.com>
423
424         Geoff is organized, but he is not an organization.
425
426         Rubber-stamped by Anders Carlsson.
427
428         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
429
430 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
431
432         Make bmalloc work with ASan
433         https://bugs.webkit.org/show_bug.cgi?id=140194
434
435         Reviewed by Mark Lam.
436
437         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
438         an OS-specific API to test for loaded runtime libraries.
439
440         * bmalloc/Environment.cpp:
441         (bmalloc::isASanEnabled):
442         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
443         ASan is enabled, since system malloc has the Asan hooks we need.
444
445         You could check for the ASan compile-time flag instead, but doing this
446         check at runtime prepares bmalloc for a world where it is a dynamic
447         library that might be loaded into projects it did not compile with.
448
449 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
450
451         Fix up bmalloc's PerThread for use on Linux
452         https://bugs.webkit.org/show_bug.cgi?id=139804
453
454         Reviewed by Anders Carlsson.
455
456         The previous implementation was a bit slow.
457
458         * bmalloc/PerThread.h:
459         (bmalloc::PerThreadStorage<Cache>::get):
460         (bmalloc::PerThreadStorage::get):
461         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
462         way to do fast per-thread access without taking a lock every time. This
463         probably works on all the platforms we care about, and it matches other
464         techniques we use elsewhere in WebKit.
465
466         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
467         this class because PerThreadStorage now encapsulates everything that
468         needs to be conditional.
469
470         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
471
472 2014-12-26  Dan Bernstein  <mitz@apple.com>
473
474         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
475         https://bugs.webkit.org/show_bug.cgi?id=139950
476
477         Reviewed by David Kilzer.
478
479         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
480         in a manner that works with Xcode 5.1.1.
481
482 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
483
484         Safari crashes when you set Malloc environment variables
485         https://bugs.webkit.org/show_bug.cgi?id=139656
486
487         Reviewed by Michael Saboff.
488
489         I forgot to cover the realloc() case. Whoops. (OoPS?)
490
491         This time around, I ran the full MallocBench test suite in Malloc=1
492         mode, and it passed.
493
494         * bmalloc/Allocator.cpp:
495         (bmalloc::Allocator::reallocate):
496         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
497         It needs to be down there so that we can do the short-circuiting check
498         for whether bmalloc is enabled first.
499
500         Also added the check.
501
502         * bmalloc/Cache.cpp:
503         (bmalloc::Cache::scavenge):
504         (bmalloc::Cache::Cache):
505         (bmalloc::Cache::reallocateSlowCaseNullCache):
506         * bmalloc/Cache.h:
507         (bmalloc::Cache::deallocator):
508         (bmalloc::Cache::reallocate): Ditto.
509
510         * bmalloc/bmalloc.h:
511         (bmalloc::api::free):
512         (bmalloc::api::realloc): Ditto.
513
514         (bmalloc::api::scavenge): Pushed this down into Cache to match the
515         surrounding functions.
516
517 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
518
519         bmalloc should support system memory analysis tools (part 2)
520         https://bugs.webkit.org/show_bug.cgi?id=139565
521
522         Reviewed by Mark Lam.
523
524         This patch actually queries the environment to see if memory analysis
525         tools have been enabled.
526
527         * bmalloc/Deallocator.cpp:
528         (bmalloc::Deallocator::scavenge): Don't process the object log if
529         we've disabled bmalloc because it will be full of invalid nullptrs.
530
531         * bmalloc/Environment.cpp:
532         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
533         Malloc debugging flags. I also added a plain "Malloc" catch-all for
534         when you want to disable bmalloc without enabling any kind of funny
535         business.
536
537         It would be slightly nicer just to iterate the list of environment
538         variables and strstr them, but getenv is the more portable option,
539         and performance here doesn't really matter.
540
541         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
542         environment variable.
543
544         (bmalloc::Environment::computeIsBmallocEnabled):
545
546 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
547
548         Try to fix the iOS simulator build.
549
550         #include the declaration of malloc / free.
551
552         * bmalloc/Allocator.cpp:
553         * bmalloc/Deallocator.cpp:
554
555 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
556
557         Try to fix the build.
558
559         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
560
561 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
562
563         bmalloc should support system memory analysis tools (part 1)
564         https://bugs.webkit.org/show_bug.cgi?id=139559
565
566         Reviewed by Mark Lam.
567
568         This patch adds the hooks to disable bmalloc at runtime if certain
569         environment variables are set, but doesn't actually read from the
570         environment yet.
571
572         No performance change.
573
574         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
575         we'll use to read environment variables and see if memory analysis tools
576         have been enabled.
577
578         * bmalloc/Allocator.cpp:
579         (bmalloc::Allocator::Allocator):
580         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
581         on the allocation path. We cache the setting to make the check fast.
582
583         * bmalloc/Allocator.h: Interface changes.
584
585         * bmalloc/Cache.cpp:
586         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
587         and deallocator. This main purpose is to enable them to query the
588         environment for whether bmalloc is enabled; but this is also a slightly
589         cleaner way to guarantee to them that the Heap has been pre-initialized.
590
591         * bmalloc/Deallocator.cpp:
592         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
593         fill the object log to force us to take the slow path on all deallocations.
594
595         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
596
597         * bmalloc/Deallocator.h: Interface changes.
598
599         * bmalloc/Environment.cpp: Added.
600         (bmalloc::Environment::Environment):
601         (bmalloc::Environment::computeIsBmallocEnabled):
602         * bmalloc/Environment.h: Added.
603         (bmalloc::Environment::isBmallocEnabled): This is the class that will
604         encapsulate looking for environment variables that turn on heap
605         analysis tools.
606
607         * bmalloc/Heap.h:
608         (bmalloc::Heap::environment):
609
610         * bmalloc/Mutex.h:
611         (bmalloc::Mutex::Mutex):
612         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
613         since I got super confused about them while writing this patch.
614
615         * bmalloc/VMHeap.cpp: Fixed an #include.
616
617 2014-12-09  David Kilzer  <ddkilzer@apple.com>
618
619         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
620         <http://webkit.org/b/139212>
621
622         Reviewed by Joseph Pecoraro.
623
624         * Configurations/Base.xcconfig:
625         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
626           on OS X.
627         * Configurations/DebugRelease.xcconfig:
628         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
629
630 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
631
632         bmalloc uses 8X more virtual memory than necessary
633         https://bugs.webkit.org/show_bug.cgi?id=138495
634
635         Reviewed by Mark Lam.
636
637         iOS has a per-process virtual memory cap around 1GB, so there's some
638         value to not going totally ham with virtual memory.
639
640         We currently use about 8X the necessary amount:
641             - 2X to align our VM allocation
642             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
643
644         We can cut that down:
645             - Return the unaligned portion of our VM allocation (-2X)
646             - Use all the chunks in a superchunk, instead of allocating one
647               chunk per superchunk (-4X)
648
649         * bmalloc/Algorithm.h:
650         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
651         function so we can call it with getpagesize() at runtime.
652
653         * bmalloc/Chunk.h:
654         * bmalloc/LargeChunk.h:
655         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
656         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
657
658         * bmalloc/VMAllocate.h:
659         (bmalloc::vmValidate):
660         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
661         if it does not succeed. Allocate precisely, and give back the extra.
662
663         * bmalloc/VMHeap.cpp:
664         (bmalloc::VMHeap::allocateSuperChunk):
665         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
666         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
667         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
668         in a superchunk, instead of just one.
669
670         * bmalloc/VMHeap.h:
671         (bmalloc::VMHeap::allocateSmallPage):
672         (bmalloc::VMHeap::allocateMediumPage):
673         (bmalloc::VMHeap::allocateLargeRange):
674         * bmalloc/XLargeChunk.h:
675         (bmalloc::XLargeChunk::create): Updated to match changes above.
676
677 2014-11-01  David Kilzer  <ddkilzer@apple.com>
678
679         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
680         <https://webkit.org/b/138286>
681         <rdar://problem/18847087>
682
683         Reviewed by Dan Bernstein.
684
685         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
686         to NO for the target that produces libbmalloc.a so that the
687         debug symbols will be linked into JavaScriptCore and end up in
688         its dSYM file.
689
690 2014-10-30  Dana Burkart  <dburkart@apple.com>
691
692         <rdar://problem/18821260> Prepare for the mysterious future
693
694         Reviewed by Lucas Forschler.
695
696         * Configurations/Base.xcconfig:
697         * Configurations/DebugRelease.xcconfig:
698
699 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
700
701         bmalloc: cleaned up fast path vs slow path
702         https://bugs.webkit.org/show_bug.cgi?id=137081
703
704         Reviewed by Sam Weinig.
705
706         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
707
708         * bmalloc/Allocator.cpp:
709         (bmalloc::Allocator::Allocator): Merged the small and medium range
710         caches, just like the small and medium allocators. Ranges are abstract
711         objects that don't really care whether they hold small or medium objects,
712         so they don't need to be segregated.
713
714         (bmalloc::Allocator::scavenge): Ditto.
715
716         (bmalloc::Allocator::allocateBumpRangeSlowCase):
717         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
718         we do care a tiny bit, because we need to specify small vs medium when
719         allocating new ranges from the heap, to ensure that the heap allocates
720         from the right segment of VM.
721
722         (bmalloc::Allocator::allocateLarge):
723         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
724         up the fast path. Large allocation performance is dominated by allocation
725         logic and initialization, so inlining it doesn't help.
726
727         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
728         it doesn't need to distinguish small vs medium objects.
729
730         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
731         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
732
733         * bmalloc/Allocator.h:
734         * bmalloc/BumpRange.h:
735
736         * bmalloc/Cache.cpp:
737         (bmalloc::Cache::allocateSlowCase): Deleted.
738         (bmalloc::Cache::deallocateSlowCase): Deleted.
739         * bmalloc/Cache.h:
740         (bmalloc::Cache::allocate):
741         (bmalloc::Cache::deallocate):
742         (bmalloc::Cache::allocateFastCase): Deleted.
743         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
744         paths. The downside to this change is that the fast path branches to two
745         distinct failure cases instead of one. The upside is that the slow path
746         doesn't need to re-read the segment register, which is not as cheap as a
747         normal register, and it doesn't need to do an extra level of function 
748         call. Seems to be worth it.
749
750         * bmalloc/Deallocator.h:
751         * bmalloc/Heap.cpp:
752         (bmalloc::Heap::refillSmallBumpRangeCache):
753         (bmalloc::Heap::refillMediumBumpRangeCache):
754         * bmalloc/Heap.h: Updated for interface changes.
755
756         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
757         small lines in a page / 2, since any other free lines will coalesce
758         with their neighbors.
759
760 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
761
762         Rolled out r173346.
763
764             bmalloc should honor the FastMalloc statistics API
765             https://bugs.webkit.org/show_bug.cgi?id=136592
766
767         This didn't really work. Because we allow ranges with and without
768         physical pages to merge, and we allow double-committing and
769         double-decommitting, we can't rely on commit actions to track memory
770         footprint.
771
772         * bmalloc/Heap.cpp:
773         (bmalloc::Heap::size): Deleted.
774         (bmalloc::Heap::capacity): Deleted.
775         * bmalloc/Heap.h:
776         * bmalloc/VMHeap.cpp:
777         (bmalloc::VMHeap::VMHeap):
778         (bmalloc::VMHeap::allocateSmallChunk):
779         (bmalloc::VMHeap::allocateMediumChunk):
780         (bmalloc::VMHeap::allocateLargeChunk):
781         * bmalloc/VMHeap.h:
782         (bmalloc::VMHeap::allocateSmallPage):
783         (bmalloc::VMHeap::allocateMediumPage):
784         (bmalloc::VMHeap::allocateLargeRange):
785         (bmalloc::VMHeap::deallocateSmallPage):
786         (bmalloc::VMHeap::deallocateMediumPage):
787         (bmalloc::VMHeap::deallocateLargeRange):
788         (bmalloc::VMHeap::size): Deleted.
789         (bmalloc::VMHeap::capacity): Deleted.
790         * bmalloc/bmalloc.h:
791         (bmalloc::api::heapSize): Deleted.
792         (bmalloc::api::heapCapacity): Deleted.
793
794 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
795
796         bmalloc: Allocation should be more precise
797         https://bugs.webkit.org/show_bug.cgi?id=136993
798
799         Reviewed by Gavin Barraclough.
800
801         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
802
803         This patch teaches the allocator to merge adjacent free lines into a
804         single allocatable range. This allows us to shrink the size of an
805         individual line without increasing fragmentation or the rate of allocator
806         slow paths.
807
808         We'll only take more slow paths when available memory is sparse, which
809         is exactly when it's worth it. When available memory is dense, we'll
810         take fewer slow paths.
811
812         * bmalloc.xcodeproj/project.pbxproj:
813         * bmalloc/Algorithm.h:
814         (bmalloc::divideRoundingUp):
815
816         * bmalloc/Allocator.cpp:
817         (bmalloc::Allocator::Allocator): Updated for interface changes.
818
819         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
820         Now that we merge lines, it's not convenient to scavenge by line.
821
822         (bmalloc::Allocator::allocateSmallBumpRange):
823         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
824         instead of individual lines.
825
826         (bmalloc::Allocator::allocateSlowCase):
827         (bmalloc::Allocator::allocateSmallLine): Deleted.
828         (bmalloc::Allocator::allocateMediumLine): Deleted.
829         (bmalloc::Allocator::allocateMedium): Deleted.
830         * bmalloc/Allocator.h:
831         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
832         into the standard fast path with small allocations. Since a BumpAllocator
833         just allocates out of an arbitrary range, it doesn't need to distinguish
834         between small and medium lines.
835
836         * bmalloc/BumpAllocator.h:
837         (bmalloc::BumpAllocator::size):
838         (bmalloc::BumpAllocator::BumpAllocator):
839         (bmalloc::BumpAllocator::init):
840         (bmalloc::BumpAllocator::refill):
841         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
842         anymore: the heap just gives us a pointer and a pre-computed number of
843         objects, and we allocate them.
844
845         * bmalloc/Deallocator.cpp:
846         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
847
848         * bmalloc/Heap.cpp:
849         (bmalloc::Heap::Heap):
850         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
851         detailing where all objects will lie in memory. After we merge two lines,
852         we might allocate an object that spans from one line to the next. This
853         metadata details which bits of memory overlap in that way, and how they
854         overlap.
855
856         (bmalloc::Heap::refillSmallBumpRangeCache):
857         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
858         and merge adjacent free lines into BumpRanges.
859
860         (bmalloc::Heap::allocateSmallPage):
861         (bmalloc::Heap::allocateMediumPage):
862         (bmalloc::Heap::deallocateSmallLine):
863         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
864         since we scan for free memory a page at a time.
865
866         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
867         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
868         fast path.
869
870         * bmalloc/Heap.h:
871         (bmalloc::Heap::derefSmallLine):
872         (bmalloc::Heap::derefMediumLine):
873         (bmalloc::Heap::deallocateSmallLine): Deleted.
874         (bmalloc::Heap::allocateSmallLine): Deleted.
875         (bmalloc::Heap::deallocateMediumLine): Deleted.
876         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
877
878         * bmalloc/Line.h:
879         (bmalloc::Line<Traits>::ref):
880         (bmalloc::Line<Traits>::deref):
881         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
882         anymore, since we only ever deref by 1 now.
883
884         * bmalloc/MediumAllocator.h:
885         (bmalloc::MediumAllocator::isNull): Deleted.
886         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
887         (bmalloc::MediumAllocator::line): Deleted.
888         (bmalloc::MediumAllocator::allocate): Deleted.
889         (bmalloc::MediumAllocator::derefCount): Deleted.
890         (bmalloc::MediumAllocator::refill): Deleted.
891         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
892         been dead for a while, since it doesn't build anymore with this patch.
893
894         * bmalloc/Page.h:
895         (bmalloc::Page::sizeClass):
896         (bmalloc::Page::setSizeClass):
897         (bmalloc::Page::smallSizeClass): Deleted.
898         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
899         to sizeClass, since we use it for medium sizes too.
900
901         * bmalloc/Sizes.h:
902         (bmalloc::Sizes::sizeClass):
903         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
904
905         (bmalloc::Sizes::smallSizeClassFor): Deleted.
906         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
907
908         * bmalloc/bmalloc.h:
909         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
910         can be a bit more precise. It also has to be, since we can't guarantee
911         that an object ends at the end of a line anymore.
912
913 2014-09-19  Daniel Bates  <dabates@apple.com>
914
915         Always assume internal SDK when building configuration Production
916         https://bugs.webkit.org/show_bug.cgi?id=136925
917         <rdar://problem/18362399>
918
919         Reviewed by Dan Bernstein.
920
921         * Configurations/Base.xcconfig:
922
923 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
924
925         bmalloc: moved line caches from the deallocator to the allocator
926         https://bugs.webkit.org/show_bug.cgi?id=136868
927
928         Reviewed by Gavin Barraclough.
929
930         I did this mostly as a simplification, to make it easier to change the
931         allocation strategy.
932
933         No throughput change on MallocBench. Saves about 50kB.
934
935         Since the deallocator needs to lock the heap when freeing lines anyway,
936         there isn't much benefit to giving the deallocator a local cache of
937         deallocated lines.
938
939         We still give the allocator a local cache of lines because that does
940         reduce the frequency at which it needs to lock the heap in order to
941         acquire more lines.
942
943         * bmalloc/Allocator.cpp:
944         (bmalloc::Allocator::scavenge):
945         (bmalloc::Allocator::allocateSmallLine):
946         (bmalloc::Allocator::allocateMediumLine):
947         (bmalloc::Allocator::allocateMedium):
948         (bmalloc::Allocator::allocateSlowCase):
949         * bmalloc/Allocator.h:
950         * bmalloc/Deallocator.cpp:
951         (bmalloc::Deallocator::Deallocator):
952         (bmalloc::Deallocator::scavenge):
953         (bmalloc::Deallocator::processObjectLog):
954         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
955         (bmalloc::Deallocator::allocateSmallLine): Deleted.
956         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
957         (bmalloc::Deallocator::allocateMediumLine): Deleted.
958         * bmalloc/Deallocator.h:
959
960         * bmalloc/Sizes.h:
961         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
962         exactly one page in size. That's about what we were shooting for anyway,
963         and it may make it easier to switch to per-page allocation in future.
964
965 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
966
967         bmalloc: allocate small and medium objects using the same bump pointer class
968         https://bugs.webkit.org/show_bug.cgi?id=136843
969
970         Reviewed by Gavin Barraclough.
971
972         4% speedup on MallocBench.
973
974         Now that medium-sized objects have dedicated per-size allocators, they
975         don't need to use an arbitrary bump pointer allocator. This means that
976         every allocator knows how many objects it will allocate from the start,
977         and we don't need a post-processing step to adjust refcounts based on
978         real allocation count.
979
980         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
981         since it's used for small and medium objects now.
982
983         * bmalloc/Allocator.cpp:
984         (bmalloc::Allocator::Allocator): Updated to use new interface.
985         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
986         to make sure that we finish allocating all the objects in it.
987
988         (bmalloc::Allocator::allocateMedium):
989         (bmalloc::Allocator::allocateSlowCase):
990         (bmalloc::Allocator::retire): Deleted.
991         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
992         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
993         * bmalloc/Allocator.h:
994         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
995         used to post-process an allocator based on how many objects it allocated.
996
997         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
998         (bmalloc::BumpAllocator::BumpAllocator):
999         (bmalloc::BumpAllocator::init):
1000         (bmalloc::BumpAllocator::line):
1001         (bmalloc::BumpAllocator::validate):
1002         (bmalloc::BumpAllocator::allocate):
1003         (bmalloc::BumpAllocator::refill):
1004         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
1005         about the kinds of lines they allocate into. In some cases, the line
1006         type must be provided as a template parameter by the caller.
1007
1008         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1009         (bmalloc::SmallAllocator::line): Deleted.
1010         (bmalloc::SmallAllocator::allocate): Deleted.
1011         (bmalloc::SmallAllocator::objectCount): Deleted.
1012         (bmalloc::SmallAllocator::derefCount): Deleted.
1013         (bmalloc::SmallAllocator::refill): Deleted.
1014         (bmalloc::SmallAllocator::clear): Deleted.
1015
1016         * bmalloc/ObjectType.h:
1017         (bmalloc::isMedium):
1018
1019         * bmalloc/SmallAllocator.h:
1020         (bmalloc::SmallAllocator::isNull): Deleted.
1021         (bmalloc::SmallAllocator::canAllocate): Deleted.
1022         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
1023         (bmalloc::SmallAllocator::line): Deleted.
1024         (bmalloc::SmallAllocator::allocate): Deleted.
1025         (bmalloc::SmallAllocator::objectCount): Deleted.
1026         (bmalloc::SmallAllocator::derefCount): Deleted.
1027         (bmalloc::SmallAllocator::refill): Deleted.
1028         (bmalloc::SmallAllocator::clear): Deleted.
1029
1030 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
1031
1032         Fixed a goof in bmalloc Vector sizing
1033         https://bugs.webkit.org/show_bug.cgi?id=136795
1034
1035         Reviewed by Gavin Barraclough and Sam Weinig.
1036
1037         We want our minimum vector to be page-sized since the OS will give us
1038         a page no matter what -- but we want that many bytes, and not enough
1039         bytes to store that many elements.
1040
1041         * bmalloc/Vector.h: Math is hard.
1042
1043 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1044
1045         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
1046         https://bugs.webkit.org/show_bug.cgi?id=136693
1047
1048         Reviewed by Gavin Barraclough.
1049
1050         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
1051
1052         No throughput change.
1053
1054         We keep an array of medium allocators, just like our array of small
1055         allocators.
1056
1057         In future, we can simplify the allocation fast path by merging the small
1058         and medium allocator arrays. For now, this is the simplest change that
1059         gets the win.
1060
1061         * bmalloc/Allocator.cpp:
1062         (bmalloc::Allocator::Allocator):
1063         (bmalloc::Allocator::scavenge):
1064         (bmalloc::Allocator::allocateMedium):
1065         * bmalloc/Allocator.h:
1066         * bmalloc/Sizes.h:
1067         (bmalloc::Sizes::mediumSizeClassFor):
1068
1069 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1070
1071         Reviewed by Sam Weinig.
1072
1073         Renamed log => retire for clarity.
1074
1075         * bmalloc/Allocator.cpp:
1076         (bmalloc::Allocator::scavenge):
1077         (bmalloc::Allocator::retire):
1078         (bmalloc::Allocator::allocateMedium):
1079         (bmalloc::Allocator::allocateSlowCase):
1080         (bmalloc::Allocator::log): Deleted.
1081         * bmalloc/Allocator.h:
1082
1083 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1084
1085         bmalloc: eager scavenge leaves behind a bogus allocator
1086         https://bugs.webkit.org/show_bug.cgi?id=136743
1087
1088         Reviewed by Sam Weinig.
1089
1090         Be sure to clear the allocator after logging it in the eager scavenge
1091         case, so that we don't later try to allocate out of the lines that we
1092         have thrown away.
1093
1094         We didn't need to do this previously because scavenge would only happen
1095         at thread exit time, after which no further allocation from the per-thread
1096         cache would take place.
1097
1098         * bmalloc/Allocator.cpp:
1099         (bmalloc::Allocator::scavenge):
1100         * bmalloc/MediumAllocator.h:
1101         (bmalloc::MediumAllocator::clear):
1102         * bmalloc/SmallAllocator.h:
1103         (bmalloc::SmallAllocator::clear):
1104
1105 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1106
1107         bmalloc should honor the FastMalloc statistics API
1108         https://bugs.webkit.org/show_bug.cgi?id=136592
1109
1110         Reviewed by Gavin Barraclough.
1111
1112         We do this by tracking "size" and "capacity" in the VM heap.
1113
1114         The VM heap's "capacity" is all the VM we ever allocated.
1115
1116         The VM heap's "size" the subset of VM currently held onto by the
1117         VM heap (and therefore not in use by the regular heap).
1118
1119         Somewhat ironically, reducing the process's memory footprint, increases
1120         the size of the VM heap, since the VM heap holds the pages that are
1121         purely virtual and not physical.
1122
1123         * bmalloc/Heap.cpp:
1124         (bmalloc::Heap::size):
1125         (bmalloc::Heap::capacity):
1126         * bmalloc/Heap.h:
1127         * bmalloc/VMHeap.cpp:
1128         (bmalloc::VMHeap::VMHeap):
1129         (bmalloc::VMHeap::allocateSmallChunk):
1130         (bmalloc::VMHeap::allocateMediumChunk):
1131         (bmalloc::VMHeap::allocateLargeChunk):
1132         * bmalloc/VMHeap.h:
1133         (bmalloc::VMHeap::size):
1134         (bmalloc::VMHeap::capacity):
1135         (bmalloc::VMHeap::allocateSmallPage):
1136         (bmalloc::VMHeap::allocateMediumPage):
1137         (bmalloc::VMHeap::allocateLargeRange):
1138         (bmalloc::VMHeap::deallocateSmallPage):
1139         (bmalloc::VMHeap::deallocateMediumPage):
1140         (bmalloc::VMHeap::deallocateLargeRange):
1141         * bmalloc/bmalloc.h:
1142         (bmalloc::api::heapSize):
1143         (bmalloc::api::heapCapacity):
1144
1145 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1146
1147         bmalloc crashes on the EWS bots (due to bad large object allocation)
1148         https://bugs.webkit.org/show_bug.cgi?id=136469
1149
1150         Reviewed by Andreas Kling.
1151
1152         It's possible to convince bmalloc to perform a bad large object allocation,
1153         through these steps:
1154
1155         (1) Insert object A into freelist F0.
1156
1157         (2) Split, merge and split again A's neighbors such that object B is
1158         inserted into freelist F0, with boundary tag and size equal to object A,
1159         but pointer not completely equal to object A. Put object B at the head of F0.
1160
1161         (3) Allocate some other object from F0, swapping its position in the
1162         freelist with object B, such that object A is now ahead of object B.
1163
1164         --> Now, the next allocation for size A/B will allocate object A, which
1165         has a slightly wrong idea about where the object actually begins.
1166         Immediately, you'll corrupt a little memory, and over time, you'll also
1167         corrupt boundary tag metadata.
1168
1169         The solution is to store the begin pointer in the boundary tag. Luckily,
1170         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1171         on MallocBench.
1172
1173         * bmalloc/Algorithm.h:
1174         (bmalloc::rightShift):
1175         * bmalloc/BeginTag.h:
1176         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1177         validate the start pointer when popping off the free list. Through a
1178         very uncommon set of steps, it is possible to have an item in the free
1179         list that is valid by all accounts except for its start pointer.
1180
1181         * bmalloc/BoundaryTag.h:
1182         (bmalloc::BoundaryTag::compactBegin):
1183         (bmalloc::BoundaryTag::setRange):
1184         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1185         start pointer. We don't need the whole pointer -- just the offset, in
1186         largeAlignment increments, into the relevant boundary tag bucket.
1187
1188         * bmalloc/BoundaryTagInlines.h:
1189         (bmalloc::validateNext):
1190         (bmalloc::BoundaryTag::init):
1191         (bmalloc::BoundaryTag::mergeLarge):
1192         (bmalloc::BoundaryTag::splitLarge):
1193         * bmalloc/SegregatedFreeList.cpp:
1194         (bmalloc::SegregatedFreeList::insert):
1195         (bmalloc::SegregatedFreeList::takeGreedy):
1196         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1197         the size when establishing a boundary tag, as required by the new
1198         interface.
1199
1200         * bmalloc/Sizes.h:
1201
1202 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1203
1204         Fixed a bmalloc crash seen on the EWS bot
1205         https://bugs.webkit.org/show_bug.cgi?id=135955
1206
1207         Reviewed by Andreas Kling.
1208
1209         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1210         input buffer is a malloc region, that region will get marked Copy-On-Write
1211         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1212         on older OS X's. In 10.10, they still fail, but they do not return
1213         EINVAL.
1214
1215         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1216
1217 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1218
1219         Fixed the bmalloc build
1220         https://bugs.webkit.org/show_bug.cgi?id=135953
1221
1222         Reviewed by Andreas Kling.
1223
1224         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1225         These headers are used, so they must be available outside the project.
1226
1227 2014-08-13  Daniel Bates  <dabates@apple.com>
1228
1229         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1230         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1231
1232         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1233         implementation of PerThread<T>::getFastCase().
1234
1235         * bmalloc/PerThread.h:
1236         (bmalloc::PerThread<T>::getFastCase):
1237
1238 2014-08-13  Daniel Bates  <dabates@apple.com>
1239
1240         Make bmalloc::PerThread work without C++ thread local storage
1241         https://bugs.webkit.org/show_bug.cgi?id=135895
1242
1243         Reviewed by Geoffrey Garen.
1244
1245         Implement support for building bmalloc without C++ thread local storage.
1246
1247         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1248         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1249         to determine whether the compiler supports C++ thread local storage.
1250         * bmalloc/PerThread.h:
1251         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1252         without C++ thread local storage.
1253         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1254         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1255         PerThreadStorage::initSharedKeyIfNeeded().
1256         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1257         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1258         ensure that the shared key has been initialized.
1259         (_pthread_setspecific_direct): Deleted.
1260         (_pthread_getspecific_direct): Deleted.
1261
1262 2014-08-13  Daniel Bates  <dabates@apple.com>
1263
1264         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1265         https://bugs.webkit.org/show_bug.cgi?id=135848
1266
1267         Reviewed by Geoffrey Garen.
1268
1269         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1270         when building for the iOS Simulator.
1271         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1272         using the public SDK.
1273         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1274         using the public SDK.
1275         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1276         using the public SDK.
1277
1278 2014-08-12  Daniel Bates  <dabates@apple.com>
1279
1280         BPLATFORM(IOS) always evaluates to false
1281         https://bugs.webkit.org/show_bug.cgi?id=135843
1282
1283         Reviewed by Geoffrey Garen.
1284
1285         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1286         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1287         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1288         building for iOS.
1289
1290         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1291         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1292
1293         * bmalloc/BPlatform.h:
1294
1295 2014-08-11  Andy Estes  <aestes@apple.com>
1296
1297         [iOS] Get rid of iOS.xcconfig
1298         https://bugs.webkit.org/show_bug.cgi?id=135809
1299
1300         Reviewed by Joseph Pecoraro.
1301
1302         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1303
1304         * Configurations/Base.xcconfig:
1305         * Configurations/iOS.xcconfig: Removed.
1306         * bmalloc.xcodeproj/project.pbxproj:
1307
1308 2014-05-01  Dan Bernstein  <mitz@apple.com>
1309
1310         Fixed production builds for the iOS Simulator.
1311         <rdar://problem/16792221>
1312
1313         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1314         PRIVATE_HEADERS_FOLDER_PATH when installing.
1315
1316 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1317
1318         bmalloc: Segregate pages by objects size
1319         https://bugs.webkit.org/show_bug.cgi?id=131909
1320
1321         Reviewed by Andreas Kling.
1322
1323         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1324
1325         * bmalloc/Allocator.cpp:
1326         (bmalloc::Allocator::allocateSlowCase):
1327         * bmalloc/Allocator.h:
1328         (bmalloc::Allocator::allocateFastCase):
1329         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1330         function for size class calculation.
1331
1332         * bmalloc/Deallocator.cpp:
1333         (bmalloc::Deallocator::Deallocator):
1334         (bmalloc::Deallocator::scavenge):
1335         (bmalloc::Deallocator::deallocateSmallLine):
1336         (bmalloc::Deallocator::allocateSmallLine):
1337         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1338         cache can't be shared anymore.
1339
1340         * bmalloc/Heap.cpp:
1341         (bmalloc::Heap::allocateSmallLineSlowCase):
1342         * bmalloc/Heap.h:
1343         (bmalloc::Heap::deallocateSmallLine): Ditto.
1344
1345         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1346         page refcount when allocating a line because we might have deallocated
1347         the page and the recycled it for another size class.
1348
1349         (bmalloc::Heap::deallocateMediumLine):
1350         (bmalloc::Heap::allocateMediumLine):
1351         * bmalloc/Line.h:
1352         (bmalloc::Line::refCount):
1353         * bmalloc/Page.h:
1354         (bmalloc::Page::refCount):
1355         (bmalloc::Page::smallSizeClass):
1356         (bmalloc::Page::setSmallSizeClass):
1357         (bmalloc::Page<Traits>::refCount): Deleted.
1358         * bmalloc/Sizes.h:
1359         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1360         an index into an array from a size.
1361
1362 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1363
1364         bmalloc: Improved alignment in LargeChunk
1365         https://bugs.webkit.org/show_bug.cgi?id=131895
1366
1367         Reviewed by Andreas Kling.
1368
1369         * bmalloc/Chunk.h:
1370         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1371         Technically, the previous alignment was harmless, but I would prefer,
1372         dear reader, not to have to explain the interlocking set of
1373         circumstances that made it so.
1374
1375 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1376
1377         Rolled out r167502 because it caused a crash on the facebook benchmark.
1378
1379         Unreviewed.
1380
1381             bmalloc: Added an XSmall line size
1382             https://bugs.webkit.org/show_bug.cgi?id=131851
1383
1384             Reviewed by Sam Weinig.
1385
1386 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1387
1388         bmalloc: Mutex should be harder to use wrong
1389         https://bugs.webkit.org/show_bug.cgi?id=131879
1390
1391         Reviewed by Andreas Kling.
1392
1393         Mutex now has a proper constructor, so you can't deadlock by forgetting
1394         to initialize it.
1395
1396         * bmalloc.xcodeproj/project.pbxproj:
1397         * bmalloc/Allocator.cpp:
1398         (bmalloc::Allocator::processXSmallAllocatorLog):
1399         (bmalloc::Allocator::processSmallAllocatorLog):
1400         (bmalloc::Allocator::processMediumAllocatorLog):
1401         (bmalloc::Allocator::allocateLarge):
1402         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1403         since the Heap mutex is a static.
1404
1405         * bmalloc/AsyncTask.h:
1406         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1407         need for explicit initialization anymore.
1408
1409         * bmalloc/Deallocator.cpp:
1410         (bmalloc::Deallocator::scavenge):
1411         (bmalloc::Deallocator::deallocateLarge):
1412         (bmalloc::Deallocator::deallocateXLarge):
1413         (bmalloc::Deallocator::processObjectLog):
1414         (bmalloc::Deallocator::deallocateSmallLine):
1415         (bmalloc::Deallocator::deallocateXSmallLine):
1416         (bmalloc::Deallocator::allocateSmallLine):
1417         (bmalloc::Deallocator::allocateXSmallLine):
1418         (bmalloc::Deallocator::deallocateMediumLine):
1419         (bmalloc::Deallocator::allocateMediumLine):
1420         * bmalloc/Deallocator.h:
1421         * bmalloc/Heap.cpp:
1422         (bmalloc::sleep):
1423         (bmalloc::Heap::Heap):
1424         (bmalloc::Heap::concurrentScavenge):
1425         (bmalloc::Heap::scavenge):
1426         (bmalloc::Heap::scavengeSmallPages):
1427         (bmalloc::Heap::scavengeXSmallPages):
1428         (bmalloc::Heap::scavengeMediumPages):
1429         (bmalloc::Heap::scavengeLargeRanges):
1430         (bmalloc::Heap::allocateXSmallLineSlowCase):
1431         (bmalloc::Heap::allocateSmallLineSlowCase):
1432         (bmalloc::Heap::allocateMediumLineSlowCase):
1433         (bmalloc::Heap::allocateXLarge):
1434         (bmalloc::Heap::deallocateXLarge):
1435         (bmalloc::Heap::allocateLarge):
1436         (bmalloc::Heap::deallocateLarge):
1437         * bmalloc/Heap.h:
1438         (bmalloc::Heap::deallocateXSmallLine):
1439         (bmalloc::Heap::allocateXSmallLine):
1440         (bmalloc::Heap::deallocateSmallLine):
1441         (bmalloc::Heap::allocateSmallLine):
1442         (bmalloc::Heap::deallocateMediumLine):
1443         (bmalloc::Heap::allocateMediumLine):
1444         * bmalloc/Line.h:
1445         (bmalloc::Line<Traits>::deref):
1446         * bmalloc/Mutex.cpp: Removed.
1447         * bmalloc/Mutex.h:
1448         (bmalloc::Mutex::Mutex):
1449         (bmalloc::Mutex::init): Deleted.
1450         (bmalloc::Mutex::try_lock): Deleted.
1451         (bmalloc::Mutex::lock): Deleted.
1452         (bmalloc::Mutex::unlock): Deleted.
1453         * bmalloc/Page.h:
1454         (bmalloc::Page<Traits>::ref):
1455         (bmalloc::Page<Traits>::deref):
1456         (bmalloc::Page<Traits>::refCount):
1457         * bmalloc/PerProcess.h:
1458         (bmalloc::PerProcess::mutex):
1459         (bmalloc::PerProcess<T>::getSlowCase):
1460         * bmalloc/StaticMutex.cpp: Added.
1461         (bmalloc::StaticMutex::lockSlowCase):
1462         * bmalloc/StaticMutex.h: Added.
1463         (bmalloc::StaticMutex::init):
1464         (bmalloc::StaticMutex::try_lock):
1465         (bmalloc::StaticMutex::lock):
1466         (bmalloc::StaticMutex::unlock):
1467         * bmalloc/VMHeap.h:
1468         (bmalloc::VMHeap::deallocateXSmallPage):
1469         (bmalloc::VMHeap::deallocateSmallPage):
1470         (bmalloc::VMHeap::deallocateMediumPage):
1471         (bmalloc::VMHeap::deallocateLargeRange):
1472         * bmalloc/bmalloc.h:
1473         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1474         since the Heap mutex is a static.
1475
1476 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1477
1478         bmalloc: AsyncTask should use Mutex instead of std::mutex
1479         https://bugs.webkit.org/show_bug.cgi?id=131865
1480
1481         Reviewed by Gavin Barraclough.
1482
1483         std::mutex is so slow that it makes parallelizing simple tasks through
1484         AsyncTask a net regression. Mutex fixes this.
1485
1486         * bmalloc/AsyncTask.h:
1487         (bmalloc::Function>::AsyncTask):
1488         (bmalloc::Function>::join):
1489         (bmalloc::Function>::runSlowCase):
1490         (bmalloc::Function>::entryPoint):
1491         * bmalloc/Mutex.h:
1492         (bmalloc::Mutex::init):
1493
1494 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1495
1496         bmalloc: Added an XSmall line size
1497         https://bugs.webkit.org/show_bug.cgi?id=131851
1498
1499         Reviewed by Sam Weinig.
1500
1501         Reduces malloc footprint on Membuster recordings by 10%.
1502
1503         This is a throughput regression, but we're still way ahead of TCMalloc.
1504         I have some ideas for how to recover the regression -- but I wanted to
1505         get this win in first.
1506
1507         Full set of benchmark results:
1508
1509                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1510                                                                                                 
1511                                                                nopatch                      patch                                Δ
1512                 Peak Memory:
1513                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1514                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1515                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1516
1517                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1518                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1519                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1520
1521                 Memory at End:
1522                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1523                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1524                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1525
1526                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1527                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1528                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1529
1530                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1531                                                                                                 
1532                                                    nopatch                patch                         Δ
1533                 Execution Time:
1534                     churn                            127ms                151ms            ! 1.19x slower
1535                     list_allocate                    130ms                164ms            ! 1.26x slower
1536                     tree_allocate                    109ms                127ms            ! 1.17x slower
1537                     tree_churn                       115ms                120ms            ! 1.04x slower
1538                     facebook                         240ms                259ms            ! 1.08x slower
1539                     fragment                          91ms                131ms            ! 1.44x slower
1540                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1541                     message_one                      260ms                259ms             ^ 1.0x faster
1542                     message_many                     149ms                154ms            ! 1.03x slower
1543                     medium                           194ms                248ms            ! 1.28x slower
1544                     big                              157ms                160ms            ! 1.02x slower
1545
1546                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1547                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1548                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1549
1550                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1551                                                                                                 
1552                                                                        nopatch                          patch                                     Δ
1553                 Execution Time:
1554                     churn                                                126ms                          148ms                        ! 1.17x slower
1555                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1556                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1557                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1558                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1559                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1560                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1561                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1562                     facebook                                             240ms                          276ms                        ! 1.15x slower
1563                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1564                     fragment                                              92ms                          130ms                        ! 1.41x slower
1565                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1566                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1567                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1568                     message_one                                          260ms                          260ms                                      
1569                     message_many                                         170ms                          238ms                         ! 1.4x slower
1570                     medium                                               185ms                          250ms                        ! 1.35x slower
1571                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1572                     big                                                  150ms                          169ms                        ! 1.13x slower
1573                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1574
1575                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1576                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1577                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1578
1579                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1580
1581                                                                     TC                      patch                                Δ
1582                 Peak Memory:
1583                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1584                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1585                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1586
1587                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1588                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1589                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1590
1591                 Memory at End:
1592                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
1593                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
1594                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
1595
1596                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
1597                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
1598                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
1599
1600                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1601                                                                                                 
1602                                                         TC                patch                         Δ
1603                 Execution Time:
1604                     churn                            416ms                148ms            ^ 2.81x faster
1605                     list_allocate                    463ms                164ms            ^ 2.82x faster
1606                     tree_allocate                    292ms                127ms             ^ 2.3x faster
1607                     tree_churn                       157ms                120ms            ^ 1.31x faster
1608                     facebook                         327ms                276ms            ^ 1.18x faster
1609                     fragment                         335ms                129ms             ^ 2.6x faster
1610                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
1611                     message_one                      386ms                258ms             ^ 1.5x faster
1612                     message_many                     410ms                154ms            ^ 2.66x faster
1613                     medium                           391ms                245ms             ^ 1.6x faster
1614                     big                              261ms                167ms            ^ 1.56x faster
1615
1616                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
1617                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
1618                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
1619
1620         * bmalloc.xcodeproj/project.pbxproj:
1621         * bmalloc/Allocator.cpp:
1622         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
1623         index corresponds with its size. Instead, use the size selection function
1624         explicitly. Now that we have XSmall, some small allocator entries are
1625         unused.
1626
1627         (bmalloc::Allocator::scavenge):
1628         (bmalloc::Allocator::log):
1629         (bmalloc::Allocator::processXSmallAllocatorLog):
1630         (bmalloc::Allocator::allocateSlowCase):
1631         * bmalloc/Allocator.h:
1632         (bmalloc::Allocator::xSmallAllocatorFor):
1633         (bmalloc::Allocator::allocateFastCase):
1634         * bmalloc/Chunk.h:
1635         * bmalloc/Deallocator.cpp:
1636         (bmalloc::Deallocator::scavenge):
1637         (bmalloc::Deallocator::processObjectLog):
1638         (bmalloc::Deallocator::deallocateSlowCase):
1639         (bmalloc::Deallocator::deallocateXSmallLine):
1640         (bmalloc::Deallocator::allocateXSmallLine):
1641         * bmalloc/Deallocator.h:
1642         (bmalloc::Deallocator::deallocateFastCase):
1643         * bmalloc/Heap.cpp:
1644         (bmalloc::Heap::scavenge):
1645         (bmalloc::Heap::scavengeXSmallPages):
1646         (bmalloc::Heap::allocateXSmallLineSlowCase):
1647         * bmalloc/Heap.h:
1648         (bmalloc::Heap::deallocateXSmallLine):
1649         (bmalloc::Heap::allocateXSmallLine):
1650         * bmalloc/LargeChunk.h:
1651         (bmalloc::LargeChunk::get):
1652         (bmalloc::LargeChunk::endTag):
1653         * bmalloc/Line.h:
1654         * bmalloc/MediumAllocator.h:
1655         (bmalloc::MediumAllocator::allocate):
1656         (bmalloc::MediumAllocator::refill):
1657         * bmalloc/ObjectType.cpp:
1658         (bmalloc::objectType):
1659         * bmalloc/ObjectType.h:
1660         (bmalloc::isXSmall):
1661         (bmalloc::isSmall):
1662         (bmalloc::isMedium):
1663         (bmalloc::isLarge):
1664         (bmalloc::isSmallOrMedium): Deleted.
1665         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1666         handling small objects. There's probably a reasonable way to share this
1667         code in the future -- I'll look into that once it's stopped changing.
1668
1669         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1670         is the main reason things got slower.
1671
1672         * bmalloc/SmallAllocator.h:
1673         (bmalloc::SmallAllocator::allocate):
1674         * bmalloc/SmallTraits.h:
1675         * bmalloc/VMHeap.cpp:
1676         (bmalloc::VMHeap::allocateXSmallChunk):
1677         * bmalloc/VMHeap.h:
1678         (bmalloc::VMHeap::allocateXSmallPage):
1679         (bmalloc::VMHeap::deallocateXSmallPage):
1680         * bmalloc/XSmallAllocator.h: Added.
1681         (bmalloc::XSmallAllocator::isNull):
1682         (bmalloc::XSmallAllocator::canAllocate):
1683         (bmalloc::XSmallAllocator::XSmallAllocator):
1684         (bmalloc::XSmallAllocator::line):
1685         (bmalloc::XSmallAllocator::allocate):
1686         (bmalloc::XSmallAllocator::objectCount):
1687         (bmalloc::XSmallAllocator::derefCount):
1688         (bmalloc::XSmallAllocator::refill):
1689         * bmalloc/XSmallChunk.h: Added.
1690         * bmalloc/XSmallLine.h: Added.
1691         * bmalloc/XSmallPage.h: Added.
1692         * bmalloc/XSmallTraits.h: Added.
1693         * bmalloc/bmalloc.h:
1694         (bmalloc::api::realloc): Boiler-plate copy, as above.
1695
1696 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1697
1698         MallocBench should scavenge explicitly instead of waiting
1699         https://bugs.webkit.org/show_bug.cgi?id=131661
1700
1701         Reviewed by Andreas Kling.
1702
1703         Added explicit scavenge support to bmalloc. This isn't a memory win,
1704         since bmalloc's per-thread cache is so small. But it makes testing
1705         simpler.
1706
1707         * bmalloc/Allocator.cpp:
1708         (bmalloc::Allocator::~Allocator):
1709         (bmalloc::Allocator::scavenge):
1710         * bmalloc/Allocator.h:
1711         * bmalloc/Cache.cpp:
1712         (bmalloc::Cache::operator new):
1713         (bmalloc::Cache::operator delete):
1714         (bmalloc::Cache::Cache):
1715         (bmalloc::Cache::scavenge):
1716         * bmalloc/Cache.h:
1717         * bmalloc/Deallocator.cpp:
1718         (bmalloc::Deallocator::~Deallocator):
1719         (bmalloc::Deallocator::scavenge):
1720         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1721         functions, for reuse.
1722
1723         * bmalloc/Heap.cpp:
1724         (bmalloc::sleep):
1725         (bmalloc::Heap::concurrentScavenge):
1726         (bmalloc::Heap::scavenge):
1727         (bmalloc::Heap::scavengeSmallPages):
1728         (bmalloc::Heap::scavengeMediumPages):
1729         (bmalloc::Heap::scavengeLargeRanges):
1730         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1731         scavenging -- in response to a benchmark or a low memory warning --
1732         wants to complete as soon as possible, so its sleep duration is 0.
1733
1734         * bmalloc/bmalloc.h:
1735         (bmalloc::api::scavenge):
1736         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1737
1738 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1739
1740         Use 4kB pages on Mac
1741         https://bugs.webkit.org/show_bug.cgi?id=131658
1742
1743         Reviewed by Sam Weinig.
1744
1745         This reduces memory use a lot on Membuster:
1746
1747                                                                   base                      patch                                Δ
1748                 Execution Time:
1749                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1750                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1751                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1752
1753                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1754                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1755                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1756
1757                 Peak Memory:
1758                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1759                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1760                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1761
1762                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1763                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1764                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1765
1766                 Memory at End:
1767                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1768                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1769                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1770
1771                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1772                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1773                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1774
1775
1776         * bmalloc.xcodeproj/project.pbxproj:
1777         * bmalloc/BPlatform.h: Added.
1778         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1779         is 4kB on Mac.
1780
1781 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1782
1783         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1784
1785         * bmalloc.xcodeproj: Modified property svn:ignore.
1786
1787 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1788
1789         Fixed some mbmalloc exports
1790         https://bugs.webkit.org/show_bug.cgi?id=131599
1791
1792         Reviewed by Ryosuke Niwa.
1793
1794         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1795         of the project, so we can call them from API.
1796
1797         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1798         visibility, so they show up as exported in the .dylib.
1799
1800 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1801
1802         Put bmalloc headers in the right place
1803         https://bugs.webkit.org/show_bug.cgi?id=131464
1804
1805         Reviewed by Mark Rowe.
1806
1807         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1808         specify that we don't just want to dump all of our generically-named
1809         headers into /usr/local/include.
1810
1811 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1812
1813         Made bmalloc more #include friendly
1814         https://bugs.webkit.org/show_bug.cgi?id=131386
1815
1816         Reviewed by Andreas Kling.
1817
1818         Marked a bunch of headers private so they can be used from client code
1819         that #includes bmalloc.h.
1820
1821         Renamed ASSERT macros to BASSERT. This matches their header, which already
1822         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1823
1824         * bmalloc.xcodeproj/project.pbxproj:
1825         * bmalloc/Allocator.cpp:
1826         (bmalloc::Allocator::allocateSlowCase):
1827         * bmalloc/AsyncTask.h:
1828         (bmalloc::Function>::runSlowCase):
1829         * bmalloc/BAssert.h:
1830         * bmalloc/BoundaryTag.h:
1831         (bmalloc::BoundaryTag::setSize):
1832         * bmalloc/BoundaryTagInlines.h:
1833         (bmalloc::validate):
1834         (bmalloc::BoundaryTag::init):
1835         (bmalloc::BoundaryTag::deallocate):
1836         (bmalloc::BoundaryTag::splitLarge):
1837         (bmalloc::BoundaryTag::allocate):
1838         * bmalloc/Chunk.h:
1839         * bmalloc/Deallocator.cpp:
1840         (bmalloc::Deallocator::processObjectLog):
1841         (bmalloc::Deallocator::deallocateSlowCase):
1842         * bmalloc/Deallocator.h:
1843         (bmalloc::Deallocator::deallocateFastCase):
1844         * bmalloc/FixedVector.h:
1845         (bmalloc::Capacity>::operator):
1846         (bmalloc::Capacity>::push):
1847         (bmalloc::Capacity>::pop):
1848         (bmalloc::Capacity>::shrink):
1849         * bmalloc/Heap.cpp:
1850         (bmalloc::Heap::allocateLarge):
1851         * bmalloc/LargeChunk.h:
1852         (bmalloc::LargeChunk::get):
1853         (bmalloc::LargeChunk::endTag):
1854         * bmalloc/Line.h:
1855         (bmalloc::Line<Traits>::concurrentRef):
1856         (bmalloc::Line<Traits>::deref):
1857         * bmalloc/MediumAllocator.h:
1858         (bmalloc::MediumAllocator::allocate):
1859         * bmalloc/ObjectType.h:
1860         (bmalloc::isSmall):
1861         * bmalloc/Page.h:
1862         (bmalloc::Page<Traits>::ref):
1863         (bmalloc::Page<Traits>::deref):
1864         * bmalloc/PerThread.h:
1865         (bmalloc::PerThread<T>::getSlowCase):
1866         * bmalloc/SegregatedFreeList.cpp:
1867         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1868         (bmalloc::SegregatedFreeList::insert):
1869         * bmalloc/SmallAllocator.h:
1870         (bmalloc::SmallAllocator::allocate):
1871         (bmalloc::SmallAllocator::refill):
1872         * bmalloc/Syscall.h:
1873         * bmalloc/VMAllocate.h:
1874         (bmalloc::vmValidate):
1875         (bmalloc::vmAllocate):
1876         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1877         * bmalloc/Vector.h:
1878         (bmalloc::Vector<T>::operator):
1879         (bmalloc::Vector<T>::pop):
1880         (bmalloc::Vector<T>::shrink):
1881         * bmalloc/XLargeChunk.h:
1882         (bmalloc::XLargeChunk::range):
1883         (bmalloc::XLargeChunk::size):
1884
1885 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1886
1887         Removed an unused file.
1888
1889         Unreviewed.
1890
1891         * bmalloc/AsyncTask.cpp: Removed.
1892
1893 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1894
1895         Build bmalloc on Mac
1896         https://bugs.webkit.org/show_bug.cgi?id=131333
1897
1898         Reviewed by Mark Rowe.
1899
1900         * Makefile: Added. For make clients.
1901
1902         These files are required for building any project in WebKit. I copied
1903         them from WTF:
1904         * Configurations: Added.
1905         * Configurations/Base.xcconfig: Added.
1906         * Configurations/DebugRelease.xcconfig: Added.
1907         * Configurations/bmalloc.xcconfig: Added.
1908         * Configurations/iOS.xcconfig: Added.
1909         * Configurations/mbmalloc.xcconfig: Added.
1910
1911         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1912         from here because everything is in .xcconfig files now.
1913
1914         I had to fix a bunch of minor warnings, since they're enabled in our
1915         .xcconfig files:
1916
1917         * bmalloc/AsyncTask.h:
1918         (bmalloc::Function>::AsyncTask):
1919         * bmalloc/BAssert.h:
1920         * bmalloc/BoundaryTagInlines.h:
1921         (bmalloc::validate):
1922         * bmalloc/Heap.cpp:
1923         (bmalloc::Heap::Heap):
1924         (bmalloc::Heap::allocateLarge):
1925         (bmalloc::Heap::deallocateLarge):
1926         * bmalloc/Mutex.h:
1927         (bmalloc::Mutex::Mutex): Deleted.
1928         * bmalloc/VMAllocate.h:
1929         (bmalloc::vmValidate):
1930         * bmalloc/mbmalloc.cpp:
1931
1932 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1933
1934         bmalloc: Fixed a leak in the per-thread cache
1935         https://bugs.webkit.org/show_bug.cgi?id=131330
1936
1937         Reviewed by Andreas Kling.
1938
1939         Remember to deallocate our line caches upon thread exit.
1940
1941         * bmalloc/Deallocator.cpp:
1942         (bmalloc::Deallocator::~Deallocator):
1943
1944 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1945
1946         bmalloc: rolled out the tryLock experiment
1947         https://bugs.webkit.org/show_bug.cgi?id=131328
1948
1949         Reviewed by Andreas Kling.
1950
1951         It wasn't a speedup.
1952
1953         * bmalloc.xcodeproj/project.pbxproj:
1954         * bmalloc/Allocator.cpp:
1955         (bmalloc::Allocator::processSmallAllocatorLog):
1956         (bmalloc::Allocator::processMediumAllocatorLog):
1957         * bmalloc/Deallocator.cpp:
1958         (bmalloc::Deallocator::processObjectLog):
1959         (bmalloc::Deallocator::deallocateSlowCase):
1960         (bmalloc::Deallocator::deallocateSmallLine):
1961         (bmalloc::Deallocator::deallocateMediumLine):
1962         * bmalloc/Deallocator.h:
1963         (bmalloc::Deallocator::deallocateFastCase):
1964         * bmalloc/Heap.h:
1965         (bmalloc::Heap::deallocateSmallLine):
1966         (bmalloc::Heap::deallocateMediumLine):
1967         * bmalloc/Line.h:
1968         (bmalloc::Line<Traits>::deref):
1969         * bmalloc/Page.h:
1970         (bmalloc::Page<Traits>::deref):
1971
1972 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1973
1974         bmalloc
1975         https://bugs.webkit.org/show_bug.cgi?id=131170
1976
1977         Reviewed by Andreas Kling.
1978
1979         Initial commit.
1980
1981         * bmalloc: Added.
1982         * bmalloc.xcodeproj: Added.
1983         * bmalloc.xcodeproj/project.pbxproj: Added.
1984         * bmalloc/Algorithm.h: Added.
1985         (bmalloc::max):
1986         (bmalloc::min):
1987         (bmalloc::mask):
1988         (bmalloc::test):
1989         (bmalloc::roundUpToMultipleOf):
1990         (bmalloc::roundDownToMultipleOf):
1991         (bmalloc::sizeOf):
1992         (bmalloc::bitCount):
1993         (bmalloc::isPowerOfTwo):
1994         * bmalloc/Allocator.cpp: Added.
1995         (bmalloc::Allocator::Allocator):
1996         (bmalloc::Allocator::~Allocator):
1997         (bmalloc::Allocator::log):
1998         (bmalloc::Allocator::processSmallAllocatorLog):
1999         (bmalloc::Allocator::processMediumAllocatorLog):
2000         (bmalloc::Allocator::allocateLarge):
2001         (bmalloc::Allocator::allocateXLarge):
2002         (bmalloc::Allocator::allocateMedium):
2003         (bmalloc::Allocator::allocateSlowCase):
2004         * bmalloc/Allocator.h: Added.
2005         (bmalloc::Allocator::smallAllocatorFor):
2006         (bmalloc::Allocator::allocateFastCase):
2007         (bmalloc::Allocator::allocate):
2008         * bmalloc/AsyncTask.cpp: Added.
2009         (bmalloc::AsyncTask<Function>::runSlowCase):
2010         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
2011         (bmalloc::AsyncTask<Function>::entryPoint):
2012         * bmalloc/AsyncTask.h: Added.
2013         (bmalloc::Function>::AsyncTask):
2014         (bmalloc::Function>::join):
2015         (bmalloc::Function>::run):
2016         (bmalloc::Function>::runSlowCase):
2017         (bmalloc::Function>::pthreadEntryPoint):
2018         (bmalloc::Function>::entryPoint):
2019         * bmalloc/BAssert.h: Added.
2020         * bmalloc/BeginTag.h: Added.
2021         (bmalloc::BeginTag::isInFreeList):
2022         * bmalloc/BoundaryTag.h: Added.
2023         (bmalloc::BoundaryTag::isXLarge):
2024         (bmalloc::BoundaryTag::setXLarge):
2025         (bmalloc::BoundaryTag::isFree):
2026         (bmalloc::BoundaryTag::setFree):
2027         (bmalloc::BoundaryTag::isEnd):
2028         (bmalloc::BoundaryTag::setEnd):
2029         (bmalloc::BoundaryTag::hasPhysicalPages):
2030         (bmalloc::BoundaryTag::setHasPhysicalPages):
2031         (bmalloc::BoundaryTag::isNull):
2032         (bmalloc::BoundaryTag::clear):
2033         (bmalloc::BoundaryTag::size):
2034         (bmalloc::BoundaryTag::setSize):
2035         (bmalloc::BoundaryTag::prev):
2036         (bmalloc::BoundaryTag::next):
2037         * bmalloc/BoundaryTagInlines.h: Added.
2038         (bmalloc::validate):
2039         (bmalloc::validatePrev):
2040         (bmalloc::validateNext):
2041         (bmalloc::BoundaryTag::init):
2042         (bmalloc::BoundaryTag::mergeLargeLeft):
2043         (bmalloc::BoundaryTag::mergeLargeRight):
2044         (bmalloc::BoundaryTag::mergeLarge):
2045         (bmalloc::BoundaryTag::deallocate):
2046         (bmalloc::BoundaryTag::splitLarge):
2047         (bmalloc::BoundaryTag::allocate):
2048         * bmalloc/Cache.cpp: Added.
2049         (bmalloc::Cache::operator new):
2050         (bmalloc::Cache::operator delete):
2051         (bmalloc::Cache::Cache):
2052         (bmalloc::Cache::allocateSlowCase):
2053         (bmalloc::Cache::allocateSlowCaseNullCache):
2054         (bmalloc::Cache::deallocateSlowCase):
2055         (bmalloc::Cache::deallocateSlowCaseNullCache):
2056         * bmalloc/Cache.h: Added.
2057         (bmalloc::Cache::allocator):
2058         (bmalloc::Cache::deallocator):
2059         (bmalloc::Cache::allocateFastCase):
2060         (bmalloc::Cache::deallocateFastCase):
2061         (bmalloc::Cache::allocate):
2062         (bmalloc::Cache::deallocate):
2063         * bmalloc/Chunk.h: Added.
2064         (bmalloc::Chunk::begin):
2065         (bmalloc::Chunk::end):
2066         (bmalloc::Chunk::lines):
2067         (bmalloc::Chunk::pages):
2068         * bmalloc/Deallocator.cpp: Added.
2069         (bmalloc::Deallocator::Deallocator):
2070         (bmalloc::Deallocator::~Deallocator):
2071         (bmalloc::Deallocator::deallocateLarge):
2072         (bmalloc::Deallocator::deallocateXLarge):
2073         (bmalloc::Deallocator::processObjectLog):
2074         (bmalloc::Deallocator::deallocateSlowCase):
2075         (bmalloc::Deallocator::deallocateSmallLine):
2076         (bmalloc::Deallocator::allocateSmallLine):
2077         (bmalloc::Deallocator::deallocateMediumLine):
2078         (bmalloc::Deallocator::allocateMediumLine):
2079         * bmalloc/Deallocator.h: Added.
2080         (bmalloc::Deallocator::deallocateFastCase):
2081         (bmalloc::Deallocator::deallocate):
2082         * bmalloc/EndTag.h: Added.
2083         (bmalloc::EndTag::operator=):
2084         * bmalloc/FixedVector.h: Added.
2085         (bmalloc::FixedVector::begin):
2086         (bmalloc::FixedVector::end):
2087         (bmalloc::FixedVector::size):
2088         (bmalloc::FixedVector::capacity):
2089         (bmalloc::FixedVector::clear):
2090         (bmalloc::FixedVector::isEmpty):
2091         (bmalloc::Capacity>::FixedVector):
2092         (bmalloc::Capacity>::operator):
2093         (bmalloc::Capacity>::push):
2094         (bmalloc::Capacity>::pop):
2095         (bmalloc::Capacity>::shrink):
2096         * bmalloc/Heap.cpp: Added.
2097         (bmalloc::sleep):
2098         (bmalloc::Heap::Heap):
2099         (bmalloc::Heap::concurrentScavenge):
2100         (bmalloc::Heap::scavengeSmallPages):
2101         (bmalloc::Heap::scavengeMediumPages):
2102         (bmalloc::Heap::scavengeLargeRanges):
2103         (bmalloc::Heap::allocateSmallLineSlowCase):
2104         (bmalloc::Heap::allocateMediumLineSlowCase):
2105         (bmalloc::Heap::allocateXLarge):
2106         (bmalloc::Heap::deallocateXLarge):
2107         (bmalloc::Heap::allocateLarge):
2108         (bmalloc::Heap::deallocateLarge):
2109         * bmalloc/Heap.h: Added.
2110         (bmalloc::Heap::deallocateSmallLine):
2111         (bmalloc::Heap::allocateSmallLine):
2112         (bmalloc::Heap::deallocateMediumLine):
2113         (bmalloc::Heap::allocateMediumLine):
2114         * bmalloc/Inline.h: Added.
2115         * bmalloc/LargeChunk.h: Added.
2116         (bmalloc::LargeChunk::begin):
2117         (bmalloc::LargeChunk::end):
2118         (bmalloc::LargeChunk::create):
2119         (bmalloc::LargeChunk::get):
2120         (bmalloc::LargeChunk::beginTag):
2121         (bmalloc::LargeChunk::endTag):
2122         * bmalloc/Line.h: Added.
2123         (bmalloc::Line<Traits>::begin):
2124         (bmalloc::Line<Traits>::end):
2125         (bmalloc::Line<Traits>::concurrentRef):
2126         (bmalloc::Line<Traits>::deref):
2127         * bmalloc/MediumAllocator.h: Added.
2128         (bmalloc::MediumAllocator::isNull):
2129         (bmalloc::MediumAllocator::MediumAllocator):
2130         (bmalloc::MediumAllocator::line):
2131         (bmalloc::MediumAllocator::allocate):
2132         (bmalloc::MediumAllocator::derefCount):
2133         (bmalloc::MediumAllocator::refill):
2134         * bmalloc/MediumChunk.h: Added.
2135         * bmalloc/MediumLine.h: Added.
2136         * bmalloc/MediumPage.h: Added.
2137         * bmalloc/MediumTraits.h: Added.
2138         * bmalloc/Mutex.cpp: Added.
2139         (bmalloc::Mutex::lockSlowCase):
2140         * bmalloc/Mutex.h: Added.
2141         (bmalloc::Mutex::Mutex):
2142         (bmalloc::Mutex::try_lock):
2143         (bmalloc::Mutex::lock):
2144         (bmalloc::Mutex::unlock):
2145         * bmalloc/ObjectType.cpp: Added.
2146         (bmalloc::objectType):
2147         * bmalloc/ObjectType.h: Added.
2148         (bmalloc::isSmallOrMedium):
2149         (bmalloc::isSmall):
2150         * bmalloc/Page.h: Added.
2151         (bmalloc::Page<Traits>::ref):
2152         (bmalloc::Page<Traits>::deref):
2153         (bmalloc::Page<Traits>::refCount):
2154         * bmalloc/PerProcess.h: Added.
2155         (bmalloc::PerProcess::mutex):
2156         (bmalloc::PerProcess<T>::getFastCase):
2157         (bmalloc::PerProcess<T>::get):
2158         (bmalloc::PerProcess<T>::getSlowCase):
2159         * bmalloc/PerThread.h: Added.
2160         (bmalloc::PerThreadStorage<Cache>::get):
2161         (bmalloc::PerThreadStorage<Cache>::init):
2162         (bmalloc::PerThreadStorage::get):
2163         (bmalloc::PerThreadStorage::init):
2164         (bmalloc::PerThread<T>::getFastCase):
2165         (bmalloc::PerThread<T>::get):
2166         (bmalloc::PerThread<T>::destructor):
2167         (bmalloc::PerThread<T>::getSlowCase):
2168         * bmalloc/Range.h: Added.
2169         (bmalloc::Range::Range):
2170         (bmalloc::Range::begin):
2171         (bmalloc::Range::end):
2172         (bmalloc::Range::size):
2173         (bmalloc::Range::operator!):
2174         (bmalloc::Range::operator<):
2175         * bmalloc/SegregatedFreeList.cpp: Added.
2176         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2177         (bmalloc::SegregatedFreeList::insert):
2178         (bmalloc::SegregatedFreeList::takeGreedy):
2179         (bmalloc::SegregatedFreeList::take):
2180         * bmalloc/SegregatedFreeList.h: Added.
2181         * bmalloc/Sizes.h: Added.
2182         * bmalloc/SmallAllocator.h: Added.
2183         (bmalloc::SmallAllocator::isNull):
2184         (bmalloc::SmallAllocator::canAllocate):
2185         (bmalloc::SmallAllocator::SmallAllocator):
2186         (bmalloc::SmallAllocator::line):
2187         (bmalloc::SmallAllocator::allocate):
2188         (bmalloc::SmallAllocator::objectCount):
2189         (bmalloc::SmallAllocator::derefCount):
2190         (bmalloc::SmallAllocator::refill):
2191         * bmalloc/SmallChunk.h: Added.
2192         * bmalloc/SmallLine.h: Added.
2193         * bmalloc/SmallPage.h: Added.
2194         * bmalloc/SmallTraits.h: Added.
2195         * bmalloc/Syscall.h: Added.
2196         * bmalloc/VMAllocate.h: Added.
2197         (bmalloc::vmSize):
2198         (bmalloc::vmValidate):
2199         (bmalloc::vmAllocate):
2200         (bmalloc::vmDeallocate):
2201         (bmalloc::vmDeallocatePhysicalPages):
2202         (bmalloc::vmAllocatePhysicalPages):
2203         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2204         (bmalloc::vmAllocatePhysicalPagesSloppy):
2205         * bmalloc/VMHeap.cpp: Added.
2206         (bmalloc::VMHeap::VMHeap):
2207         (bmalloc::VMHeap::allocateSmallChunk):
2208         (bmalloc::VMHeap::allocateMediumChunk):
2209         (bmalloc::VMHeap::allocateLargeChunk):
2210         * bmalloc/VMHeap.h: Added.
2211         (bmalloc::VMHeap::allocateSmallPage):
2212         (bmalloc::VMHeap::allocateMediumPage):
2213         (bmalloc::VMHeap::allocateLargeRange):
2214         (bmalloc::VMHeap::deallocateSmallPage):
2215         (bmalloc::VMHeap::deallocateMediumPage):
2216         (bmalloc::VMHeap::deallocateLargeRange):
2217         * bmalloc/Vector.h: Added.
2218         (bmalloc::Vector::begin):
2219         (bmalloc::Vector::end):
2220         (bmalloc::Vector::size):
2221         (bmalloc::Vector::capacity):
2222         (bmalloc::Vector::last):
2223         (bmalloc::Vector::pop):
2224         (bmalloc::Vector<T>::Vector):
2225         (bmalloc::Vector<T>::~Vector):
2226         (bmalloc::Vector<T>::operator):
2227         (bmalloc::Vector<T>::push):
2228         (bmalloc::Vector<T>::pop):
2229         (bmalloc::Vector<T>::shrink):
2230         (bmalloc::Vector<T>::reallocateBuffer):
2231         (bmalloc::Vector<T>::shrinkCapacity):
2232         (bmalloc::Vector<T>::growCapacity):
2233         * bmalloc/XLargeChunk.h: Added.
2234         (bmalloc::XLargeChunk::get):
2235         (bmalloc::XLargeChunk::begin):
2236         (bmalloc::XLargeChunk::XLargeChunk):
2237         (bmalloc::XLargeChunk::create):
2238         (bmalloc::XLargeChunk::destroy):
2239         (bmalloc::XLargeChunk::range):
2240         (bmalloc::XLargeChunk::size):
2241         * bmalloc/bmalloc.h: Added.
2242         (bmalloc::api::malloc):
2243         (bmalloc::api::free):
2244         (bmalloc::api::realloc):
2245         * bmalloc/mbmalloc.cpp: Added.
2246