bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2015-02-18  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc: VMHeap should keep a record of all of its VM ranges (for malloc introspection)
4         https://bugs.webkit.org/show_bug.cgi?id=141759
5
6         Reviewed by Andreas Kling.
7
8         * bmalloc.xcodeproj/project.pbxproj:
9         * bmalloc/SuperChunk.h: Added.
10         (bmalloc::SuperChunk::create):
11         (bmalloc::SuperChunk::SuperChunk):
12         (bmalloc::SuperChunk::smallChunk):
13         (bmalloc::SuperChunk::mediumChunk):
14         (bmalloc::SuperChunk::largeChunk): Factored out super chunk creation
15         into a separate class, for clarity and type safety.
16
17         * bmalloc/VMHeap.cpp:
18         (bmalloc::VMHeap::grow):
19         (bmalloc::VMHeap::allocateSuperChunk): Renamed "allocateSuperChunk" to
20         "grow" because Andreas found "allocateSuperChunk" to be unclear.
21
22         * bmalloc/VMHeap.h: Track all our VM ranges. We will use this information
23         for malloc introspection.
24
25         (bmalloc::VMHeap::allocateSmallPage):
26         (bmalloc::VMHeap::allocateMediumPage):
27         (bmalloc::VMHeap::allocateLargeRange): Updated for renames.
28
29 2015-02-18  Zan Dobersek  <zdobersek@igalia.com>
30
31         Build bmalloc through CMake as a static library. It's then linked either
32         into the WTF library (if built as a shared library) or into the JSC and
33         WebKit2 libraries. There's no need to build it as a standalone shared library.
34
35         Rubber-stamped by Carlos Garcia Campos.
36
37         * CMakeLists.txt:
38
39 2015-02-13  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
40
41         [BMalloc] Add a FIXME comment for memory alignas
42         https://bugs.webkit.org/show_bug.cgi?id=141556
43
44         Reviewed by Csaba Osztrogonác.
45
46         * bmalloc/Chunk.h: Add a FIXME comment.
47         * bmalloc/LargeChunk.h: ditto.
48
49 2015-02-11  Csaba Osztrogonác  <ossy@webkit.org>
50
51         bmalloc buildfix on 32 bit Linux (x86/ARM)
52         https://bugs.webkit.org/show_bug.cgi?id=141472
53
54         Reviewed by Gyuyoung Kim.
55
56         * bmalloc/Algorithm.h:
57         (bmalloc::roundUpToMultipleOf):
58         * bmalloc/FixedVector.h:
59         (bmalloc::FixedVector::clear):
60         * bmalloc/Sizes.h:
61         (bmalloc::Sizes::sizeClass):
62
63 2015-02-11  Gyuyoung Kim  <gyuyoung.kim@samsung.com>
64
65         [EFL][GTK] Use bmalloc instead of tcmalloc
66         https://bugs.webkit.org/show_bug.cgi?id=140162
67
68         Reviewed by Carlos Garcia Campos.
69
70         Support to use bmalloc on EFL and GTK ports.
71
72         * CMakeLists.txt: Added.
73         * bmalloc/Allocator.cpp:
74         (bmalloc::Allocator::allocate):
75             Fix unused return value caused by posix_memalign().
76         * bmalloc/AsyncTask.h:
77         * bmalloc/BoundaryTag.h:
78         (bmalloc::BoundaryTag::clear):
79         * bmalloc/Chunk.h:
80             Change Traits::Page with Traits::PageType in order to fix
81             -fpermitive build error on EFL and GTK port. 
82         * bmalloc/EndTag.h:
83         (bmalloc::EndTag::operator=):
84         * bmalloc/Line.h: ditto.
85         * bmalloc/MediumTraits.h:
86         * bmalloc/Page.h: ditto.
87         * bmalloc/PerThread.h:
88             EFL port doesn't support __has_include definition yet.
89             Define HAVE_PTHREAD_MACHDEP_H according to check if __has_include is supported.
90         * bmalloc/SmallTraits.h: ditto.
91         * bmalloc/VMAllocate.h:
92         (bmalloc::vmDeallocatePhysicalPages):
93         (bmalloc::vmAllocatePhysicalPages):
94         * bmalloc/Vector.h:
95         (bmalloc::Vector<T>::push):
96         (bmalloc::Vector<T>::reallocateBuffer):
97
98 2015-01-31  Sam Weinig  <sam@webkit.org>
99
100         Remove even more Mountain Lion support
101         https://bugs.webkit.org/show_bug.cgi?id=141124
102
103         Reviewed by Alexey Proskuryakov.
104
105         * Configurations/Base.xcconfig:
106         * Configurations/DebugRelease.xcconfig:
107
108 2015-01-30  Geoffrey Garen  <ggaren@apple.com>
109
110         GC marking threads should clear malloc caches
111         https://bugs.webkit.org/show_bug.cgi?id=141097
112
113         Reviewed by Andreas Kling.
114
115         Split the scavenging API into per-thread vs global, so that you can
116         request to scavenge your own thread without scavenging the whole heap.
117
118         * bmalloc/Cache.cpp:
119         (bmalloc::Cache::scavenge):
120         * bmalloc/bmalloc.h:
121         (bmalloc::api::scavengeThisThread):
122         (bmalloc::api::scavenge):
123
124 2015-01-28  Dana Burkart  <dburkart@apple.com>
125
126         Move ASan flag settings from DebugRelease.xcconfig to Base.xcconfig
127         https://bugs.webkit.org/show_bug.cgi?id=136765
128
129         Reviewed by Alexey Proskuryakov.
130
131         * Configurations/Base.xcconfig:
132         * Configurations/DebugRelease.xcconfig:
133
134 2015-01-21  Geoffrey Garen  <ggaren@apple.com>
135
136         bmalloc: support aligned allocation
137         https://bugs.webkit.org/show_bug.cgi?id=140732
138
139         Reviewed by Andreas Kling.
140
141         * bmalloc/Allocator.cpp:
142         (bmalloc::Allocator::allocate): New function for aligned allocation.
143
144         Small and medium requests just allocate and free until they find an
145         aligned pointer. This is slightly inefficient in the worst case, but
146         still constant-time with little-to-no space overhead.
147
148         Large requests use a new API that requires the client to specify both
149         its ideal size and alignment, and the worst-case size you would have to
150         allocate in order to produce some interior pointer of the requested size
151         and alignment. We put the burden of this calculation on the client
152         because it simplifies things if we guarantee that allocation won't fail.
153
154         XLarge requests are easy: we just forward them to vmAllocate, which
155         already supported aligned requests.
156
157         * bmalloc/BoundaryTag.h:
158         * bmalloc/BoundaryTagInlines.h:
159         (bmalloc::BoundaryTag::mergeLeft):
160         (bmalloc::BoundaryTag::mergeRight):
161         (bmalloc::BoundaryTag::merge):
162         (bmalloc::BoundaryTag::deallocate):
163         (bmalloc::BoundaryTag::split):
164         (bmalloc::BoundaryTag::allocate): No behavior change here. I just
165         refactored the interface to remove some reference out parameters in
166         order to clarify what changes and what doesn't.
167
168         * bmalloc/Heap.cpp:
169         (bmalloc::Heap::allocateXLarge): Added an alignment API.
170
171         (bmalloc::Heap::allocateLarge):
172         * bmalloc/Heap.h: Added an alignment API. I split out allocateLarge into
173         a few variants, so aligned and unaligned allocation could share some code.
174
175         * bmalloc/SegregatedFreeList.cpp:
176         (bmalloc::SegregatedFreeList::take):
177         * bmalloc/SegregatedFreeList.h: Changed to use a separate, explicit API
178         for aligned allocation. It turns out that the aligned path is pretty
179         different, since it ends up searching for two potential ways to satisfy
180         an allocation: either large enough and aligned, or large enough to split
181         into something not aligned and something large enough and aligned.
182
183         * bmalloc/VMAllocate.h:
184         (bmalloc::vmAllocate): Switched alignment to come before size because
185         that's how the memalign API specifies it.
186
187         * bmalloc/VMHeap.h:
188         (bmalloc::VMHeap::allocateLargeRange): Added an alignment API.
189
190 2015-01-20  Geoffrey Garen  <ggaren@apple.com>
191
192         bmalloc: a little bit of cleanup
193         https://bugs.webkit.org/show_bug.cgi?id=140687
194
195         Reviewed by Anders Carlsson.
196
197         * bmalloc/Algorithm.h:
198         (bmalloc::isPowerOfTwo): Added a check for 0, since 0 would break a lot
199         of code.
200
201         * bmalloc/BoundaryTag.h:
202         * bmalloc/BoundaryTagInlines.h:
203         (bmalloc::BoundaryTag::mergeLeft):
204         (bmalloc::BoundaryTag::mergeRight):
205         (bmalloc::BoundaryTag::merge):
206         (bmalloc::BoundaryTag::deallocate):
207         (bmalloc::BoundaryTag::split):
208         (bmalloc::BoundaryTag::allocate):
209         (bmalloc::BoundaryTag::mergeLargeLeft): Deleted.
210         (bmalloc::BoundaryTag::mergeLargeRight): Deleted.
211         (bmalloc::BoundaryTag::mergeLarge): Deleted.
212         (bmalloc::BoundaryTag::splitLarge): Deleted. Removed the word "Large"
213         from all these functions, since boundary tags always pertain to large
214         objects, and putting the word "Large" everywhere wasn't helping to
215         explain that.
216
217         * bmalloc/Heap.cpp:
218         (bmalloc::Heap::allocateXLarge):
219         (bmalloc::Heap::findXLarge):
220         (bmalloc::Heap::deallocateXLarge):
221         * bmalloc/Heap.h:
222         * bmalloc/VMHeap.h:
223         (bmalloc::VMHeap::allocateXLarge): Deleted.
224         (bmalloc::VMHeap::findXLarge): Deleted.
225         (bmalloc::VMHeap::deallocateXLarge): Deleted. Moved XLarge allocation
226         from VMHeap to Heap. Since the purpose of the VMHeap is to cache VM
227         ranges, and the VMHeap never caches any XLarge ranges, it doesn't
228         really make sense for the VMHeap to be involved.
229
230 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
231
232         bmalloc: refactored XLarge allocation for better alignment
233         https://bugs.webkit.org/show_bug.cgi?id=140582
234
235         Reviewed by Andreas Kling.
236
237         XLarge objects used to be Large objects with an extra bit of metadata
238         that said "actually, I'm not large -- I'm extra large".
239
240         The metadata header in an XLarge allocation made it impossible for the
241         XLarge object to honor a very large alignment request.
242
243         The solution is to stop using a metadata header for XLarge objects, and
244         instead to store explicit metadata on the side.
245
246         This is a bit less astonishing, which is also nice.
247
248         Finding XLarge metadata is now a linear search. That's probably OK, since
249         it was always so in TCMalloc, and the usual number of XLarge allocations
250         in a process is 0.
251
252         This design makes it possible for the heap to cache XLarge allocations
253         with and/or without physical pages. I haven't actually done that yet
254         because the tradeoffs are subtle, so I don't want to do anything without
255         a motivating test case.
256
257         * bmalloc.xcodeproj/project.pbxproj:
258         * bmalloc/Allocator.cpp:
259         (bmalloc::Allocator::reallocate): Removed the concept of an XLargeChunk,
260         since an XLarge allocation is now just a naked buffer without a header.
261
262         (bmalloc::Allocator::allocateXLarge): Added an explicit qualifier for
263         XLarge alignment, since XLargeChunk won't give this to us implicitly
264         anymore.
265
266         * bmalloc/BoundaryTag.h:
267         (bmalloc::BoundaryTag::setRange):
268         (bmalloc::BoundaryTag::isXLarge): Deleted.
269         (bmalloc::BoundaryTag::setXLarge): Deleted.
270         * bmalloc/BoundaryTagInlines.h:
271         (bmalloc::validate):
272         (bmalloc::BoundaryTag::deallocate): Removed the XLarge hacks from Large allocations.
273
274         * bmalloc/Deallocator.cpp:
275         (bmalloc::Deallocator::deallocateXLarge):
276         (bmalloc::Deallocator::deallocateSlowCase):
277         * bmalloc/Heap.cpp:
278         (bmalloc::Heap::findXLarge):
279         (bmalloc::Heap::allocateXLarge):
280         (bmalloc::Heap::deallocateXLarge):
281         * bmalloc/Heap.h: Updated for interface changes.
282
283         * bmalloc/ObjectType.cpp:
284         (bmalloc::objectType):
285         * bmalloc/ObjectType.h:
286         (bmalloc::isXLarge): We can now tell if a pointer is XLarge just by
287         examining its bit pattern -- just like we do for other kinds of
288         allocations -- which is nice.
289
290         * bmalloc/Sizes.h:
291         * bmalloc/VMHeap.h:
292         (bmalloc::VMHeap::allocateXLarge):
293         (bmalloc::VMHeap::findXLarge):
294         (bmalloc::VMHeap::deallocateXLarge): Keep an explicit vector of metadata
295         for XLarge allocations.
296
297         * bmalloc/XLargeChunk.h: Removed.
298
299 2015-01-16  Geoffrey Garen  <ggaren@apple.com>
300
301         bmalloc: added some infrastructure for aligned allocation
302         https://bugs.webkit.org/show_bug.cgi?id=140572
303
304         Reviewed by Andreas Kling.
305
306         * bmalloc/Algorithm.h:
307         (bmalloc::isPowerOfTwo):
308         (bmalloc::roundUpToMultipleOf):
309         (bmalloc::roundDownToMultipleOf): Refactored some duplicate code to use our
310         isPowerOfTwo helper function.
311
312         * bmalloc/Allocator.cpp:
313         (bmalloc::Allocator::allocate):
314         * bmalloc/Allocator.h: Stubbed out an implementation of aligned allocation.
315         Doesn't do anything yet, but does correctly forward to system malloc
316         when bmalloc is disabled.
317
318         * bmalloc/Cache.cpp:
319         (bmalloc::Cache::allocateSlowCaseNullCache):
320         * bmalloc/Cache.h:
321         (bmalloc::Cache::allocate):
322         * bmalloc/bmalloc.h:
323         (bmalloc::api::memalign):
324         * bmalloc/mbmalloc.cpp: Stubbed out an API for aligned allocation.
325
326 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
327
328         Consider alignment when allocating from a SegregatedFreeList
329         https://bugs.webkit.org/show_bug.cgi?id=140408
330
331         Reviewed by Sam Weinig.
332
333         In preparation for supporting aligned allocation.
334
335         No performance change.
336
337         Since this is just one extra branch in an already expensive function,
338         I decided not to duplicate the function just to avoid the branch in
339         the un-aligned case.
340
341         * bmalloc/SegregatedFreeList.cpp:
342         (bmalloc::SegregatedFreeList::take):
343         * bmalloc/SegregatedFreeList.h:
344
345 2015-01-13  Geoffrey Garen  <ggaren@apple.com>
346
347         Renamed minimum to size in SegregatedFreeList
348         https://bugs.webkit.org/show_bug.cgi?id=140406
349
350         Reviewed by Sam Weinig.
351
352         In preparation for supporting aligned allocation.
353
354         * bmalloc/SegregatedFreeList.cpp:
355         (bmalloc::SegregatedFreeList::takeGreedy):
356         (bmalloc::SegregatedFreeList::take): Every size passed to malloc is
357         really just a minimum. Let's not imply that this value is special.
358
359 2015-01-11  Dan Bernstein  <mitz@apple.com>
360
361         Geoff is organized, but he is not an organization.
362
363         Rubber-stamped by Anders Carlsson.
364
365         * bmalloc.xcodeproj/project.pbxproj: Removed the ORGANIZATIONNAME project attribute.
366
367 2015-01-07  Geoffrey Garen  <ggaren@apple.com>
368
369         Make bmalloc work with ASan
370         https://bugs.webkit.org/show_bug.cgi?id=140194
371
372         Reviewed by Mark Lam.
373
374         * bmalloc/BPlatform.h: Added a way to detect Darwin OSes, since we need
375         an OS-specific API to test for loaded runtime libraries.
376
377         * bmalloc/Environment.cpp:
378         (bmalloc::isASanEnabled):
379         (bmalloc::Environment::computeIsBmallocEnabled): Disabled bmalloc if
380         ASan is enabled, since system malloc has the Asan hooks we need.
381
382         You could check for the ASan compile-time flag instead, but doing this
383         check at runtime prepares bmalloc for a world where it is a dynamic
384         library that might be loaded into projects it did not compile with.
385
386 2015-01-05  Geoffrey Garen  <ggaren@apple.com>
387
388         Fix up bmalloc's PerThread for use on Linux
389         https://bugs.webkit.org/show_bug.cgi?id=139804
390
391         Reviewed by Anders Carlsson.
392
393         The previous implementation was a bit slow.
394
395         * bmalloc/PerThread.h:
396         (bmalloc::PerThreadStorage<Cache>::get):
397         (bmalloc::PerThreadStorage::get):
398         (bmalloc::PerThreadStorage::init): Added a catch-all cross-platform Unix
399         way to do fast per-thread access without taking a lock every time. This
400         probably works on all the platforms we care about, and it matches other
401         techniques we use elsewhere in WebKit.
402
403         (bmalloc::PerThread<T>::getFastCase): Removed the conditional from
404         this class because PerThreadStorage now encapsulates everything that
405         needs to be conditional.
406
407         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Deleted.
408
409 2014-12-26  Dan Bernstein  <mitz@apple.com>
410
411         <rdar://problem/19348208> REGRESSION (r177027): iOS builds use the wrong toolchain
412         https://bugs.webkit.org/show_bug.cgi?id=139950
413
414         Reviewed by David Kilzer.
415
416         * Configurations/Base.xcconfig: Only define TOOLCHAINS when building for OS X, doing so
417         in a manner that works with Xcode 5.1.1.
418
419 2014-12-15  Geoffrey Garen  <ggaren@apple.com>
420
421         Safari crashes when you set Malloc environment variables
422         https://bugs.webkit.org/show_bug.cgi?id=139656
423
424         Reviewed by Michael Saboff.
425
426         I forgot to cover the realloc() case. Whoops. (OoPS?)
427
428         This time around, I ran the full MallocBench test suite in Malloc=1
429         mode, and it passed.
430
431         * bmalloc/Allocator.cpp:
432         (bmalloc::Allocator::reallocate):
433         * bmalloc/Allocator.h: Pushed realloc() logic down into the allocator.
434         It needs to be down there so that we can do the short-circuiting check
435         for whether bmalloc is enabled first.
436
437         Also added the check.
438
439         * bmalloc/Cache.cpp:
440         (bmalloc::Cache::scavenge):
441         (bmalloc::Cache::Cache):
442         (bmalloc::Cache::reallocateSlowCaseNullCache):
443         * bmalloc/Cache.h:
444         (bmalloc::Cache::deallocator):
445         (bmalloc::Cache::reallocate): Ditto.
446
447         * bmalloc/bmalloc.h:
448         (bmalloc::api::free):
449         (bmalloc::api::realloc): Ditto.
450
451         (bmalloc::api::scavenge): Pushed this down into Cache to match the
452         surrounding functions.
453
454 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
455
456         bmalloc should support system memory analysis tools (part 2)
457         https://bugs.webkit.org/show_bug.cgi?id=139565
458
459         Reviewed by Mark Lam.
460
461         This patch actually queries the environment to see if memory analysis
462         tools have been enabled.
463
464         * bmalloc/Deallocator.cpp:
465         (bmalloc::Deallocator::scavenge): Don't process the object log if
466         we've disabled bmalloc because it will be full of invalid nullptrs.
467
468         * bmalloc/Environment.cpp:
469         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
470         Malloc debugging flags. I also added a plain "Malloc" catch-all for
471         when you want to disable bmalloc without enabling any kind of funny
472         business.
473
474         It would be slightly nicer just to iterate the list of environment
475         variables and strstr them, but getenv is the more portable option,
476         and performance here doesn't really matter.
477
478         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
479         environment variable.
480
481         (bmalloc::Environment::computeIsBmallocEnabled):
482
483 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
484
485         Try to fix the iOS simulator build.
486
487         #include the declaration of malloc / free.
488
489         * bmalloc/Allocator.cpp:
490         * bmalloc/Deallocator.cpp:
491
492 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
493
494         Try to fix the build.
495
496         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
497
498 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
499
500         bmalloc should support system memory analysis tools (part 1)
501         https://bugs.webkit.org/show_bug.cgi?id=139559
502
503         Reviewed by Mark Lam.
504
505         This patch adds the hooks to disable bmalloc at runtime if certain
506         environment variables are set, but doesn't actually read from the
507         environment yet.
508
509         No performance change.
510
511         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
512         we'll use to read environment variables and see if memory analysis tools
513         have been enabled.
514
515         * bmalloc/Allocator.cpp:
516         (bmalloc::Allocator::Allocator):
517         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
518         on the allocation path. We cache the setting to make the check fast.
519
520         * bmalloc/Allocator.h: Interface changes.
521
522         * bmalloc/Cache.cpp:
523         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
524         and deallocator. This main purpose is to enable them to query the
525         environment for whether bmalloc is enabled; but this is also a slightly
526         cleaner way to guarantee to them that the Heap has been pre-initialized.
527
528         * bmalloc/Deallocator.cpp:
529         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
530         fill the object log to force us to take the slow path on all deallocations.
531
532         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
533
534         * bmalloc/Deallocator.h: Interface changes.
535
536         * bmalloc/Environment.cpp: Added.
537         (bmalloc::Environment::Environment):
538         (bmalloc::Environment::computeIsBmallocEnabled):
539         * bmalloc/Environment.h: Added.
540         (bmalloc::Environment::isBmallocEnabled): This is the class that will
541         encapsulate looking for environment variables that turn on heap
542         analysis tools.
543
544         * bmalloc/Heap.h:
545         (bmalloc::Heap::environment):
546
547         * bmalloc/Mutex.h:
548         (bmalloc::Mutex::Mutex):
549         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
550         since I got super confused about them while writing this patch.
551
552         * bmalloc/VMHeap.cpp: Fixed an #include.
553
554 2014-12-09  David Kilzer  <ddkilzer@apple.com>
555
556         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
557         <http://webkit.org/b/139212>
558
559         Reviewed by Joseph Pecoraro.
560
561         * Configurations/Base.xcconfig:
562         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
563           on OS X.
564         * Configurations/DebugRelease.xcconfig:
565         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
566
567 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
568
569         bmalloc uses 8X more virtual memory than necessary
570         https://bugs.webkit.org/show_bug.cgi?id=138495
571
572         Reviewed by Mark Lam.
573
574         iOS has a per-process virtual memory cap around 1GB, so there's some
575         value to not going totally ham with virtual memory.
576
577         We currently use about 8X the necessary amount:
578             - 2X to align our VM allocation
579             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
580
581         We can cut that down:
582             - Return the unaligned portion of our VM allocation (-2X)
583             - Use all the chunks in a superchunk, instead of allocating one
584               chunk per superchunk (-4X)
585
586         * bmalloc/Algorithm.h:
587         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
588         function so we can call it with getpagesize() at runtime.
589
590         * bmalloc/Chunk.h:
591         * bmalloc/LargeChunk.h:
592         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
593         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
594
595         * bmalloc/VMAllocate.h:
596         (bmalloc::vmValidate):
597         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
598         if it does not succeed. Allocate precisely, and give back the extra.
599
600         * bmalloc/VMHeap.cpp:
601         (bmalloc::VMHeap::allocateSuperChunk):
602         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
603         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
604         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
605         in a superchunk, instead of just one.
606
607         * bmalloc/VMHeap.h:
608         (bmalloc::VMHeap::allocateSmallPage):
609         (bmalloc::VMHeap::allocateMediumPage):
610         (bmalloc::VMHeap::allocateLargeRange):
611         * bmalloc/XLargeChunk.h:
612         (bmalloc::XLargeChunk::create): Updated to match changes above.
613
614 2014-11-01  David Kilzer  <ddkilzer@apple.com>
615
616         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
617         <https://webkit.org/b/138286>
618         <rdar://problem/18847087>
619
620         Reviewed by Dan Bernstein.
621
622         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
623         to NO for the target that produces libbmalloc.a so that the
624         debug symbols will be linked into JavaScriptCore and end up in
625         its dSYM file.
626
627 2014-10-30  Dana Burkart  <dburkart@apple.com>
628
629         <rdar://problem/18821260> Prepare for the mysterious future
630
631         Reviewed by Lucas Forschler.
632
633         * Configurations/Base.xcconfig:
634         * Configurations/DebugRelease.xcconfig:
635
636 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
637
638         bmalloc: cleaned up fast path vs slow path
639         https://bugs.webkit.org/show_bug.cgi?id=137081
640
641         Reviewed by Sam Weinig.
642
643         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
644
645         * bmalloc/Allocator.cpp:
646         (bmalloc::Allocator::Allocator): Merged the small and medium range
647         caches, just like the small and medium allocators. Ranges are abstract
648         objects that don't really care whether they hold small or medium objects,
649         so they don't need to be segregated.
650
651         (bmalloc::Allocator::scavenge): Ditto.
652
653         (bmalloc::Allocator::allocateBumpRangeSlowCase):
654         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
655         we do care a tiny bit, because we need to specify small vs medium when
656         allocating new ranges from the heap, to ensure that the heap allocates
657         from the right segment of VM.
658
659         (bmalloc::Allocator::allocateLarge):
660         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
661         up the fast path. Large allocation performance is dominated by allocation
662         logic and initialization, so inlining it doesn't help.
663
664         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
665         it doesn't need to distinguish small vs medium objects.
666
667         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
668         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
669
670         * bmalloc/Allocator.h:
671         * bmalloc/BumpRange.h:
672
673         * bmalloc/Cache.cpp:
674         (bmalloc::Cache::allocateSlowCase): Deleted.
675         (bmalloc::Cache::deallocateSlowCase): Deleted.
676         * bmalloc/Cache.h:
677         (bmalloc::Cache::allocate):
678         (bmalloc::Cache::deallocate):
679         (bmalloc::Cache::allocateFastCase): Deleted.
680         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
681         paths. The downside to this change is that the fast path branches to two
682         distinct failure cases instead of one. The upside is that the slow path
683         doesn't need to re-read the segment register, which is not as cheap as a
684         normal register, and it doesn't need to do an extra level of function 
685         call. Seems to be worth it.
686
687         * bmalloc/Deallocator.h:
688         * bmalloc/Heap.cpp:
689         (bmalloc::Heap::refillSmallBumpRangeCache):
690         (bmalloc::Heap::refillMediumBumpRangeCache):
691         * bmalloc/Heap.h: Updated for interface changes.
692
693         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
694         small lines in a page / 2, since any other free lines will coalesce
695         with their neighbors.
696
697 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
698
699         Rolled out r173346.
700
701             bmalloc should honor the FastMalloc statistics API
702             https://bugs.webkit.org/show_bug.cgi?id=136592
703
704         This didn't really work. Because we allow ranges with and without
705         physical pages to merge, and we allow double-committing and
706         double-decommitting, we can't rely on commit actions to track memory
707         footprint.
708
709         * bmalloc/Heap.cpp:
710         (bmalloc::Heap::size): Deleted.
711         (bmalloc::Heap::capacity): Deleted.
712         * bmalloc/Heap.h:
713         * bmalloc/VMHeap.cpp:
714         (bmalloc::VMHeap::VMHeap):
715         (bmalloc::VMHeap::allocateSmallChunk):
716         (bmalloc::VMHeap::allocateMediumChunk):
717         (bmalloc::VMHeap::allocateLargeChunk):
718         * bmalloc/VMHeap.h:
719         (bmalloc::VMHeap::allocateSmallPage):
720         (bmalloc::VMHeap::allocateMediumPage):
721         (bmalloc::VMHeap::allocateLargeRange):
722         (bmalloc::VMHeap::deallocateSmallPage):
723         (bmalloc::VMHeap::deallocateMediumPage):
724         (bmalloc::VMHeap::deallocateLargeRange):
725         (bmalloc::VMHeap::size): Deleted.
726         (bmalloc::VMHeap::capacity): Deleted.
727         * bmalloc/bmalloc.h:
728         (bmalloc::api::heapSize): Deleted.
729         (bmalloc::api::heapCapacity): Deleted.
730
731 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
732
733         bmalloc: Allocation should be more precise
734         https://bugs.webkit.org/show_bug.cgi?id=136993
735
736         Reviewed by Gavin Barraclough.
737
738         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
739
740         This patch teaches the allocator to merge adjacent free lines into a
741         single allocatable range. This allows us to shrink the size of an
742         individual line without increasing fragmentation or the rate of allocator
743         slow paths.
744
745         We'll only take more slow paths when available memory is sparse, which
746         is exactly when it's worth it. When available memory is dense, we'll
747         take fewer slow paths.
748
749         * bmalloc.xcodeproj/project.pbxproj:
750         * bmalloc/Algorithm.h:
751         (bmalloc::divideRoundingUp):
752
753         * bmalloc/Allocator.cpp:
754         (bmalloc::Allocator::Allocator): Updated for interface changes.
755
756         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
757         Now that we merge lines, it's not convenient to scavenge by line.
758
759         (bmalloc::Allocator::allocateSmallBumpRange):
760         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
761         instead of individual lines.
762
763         (bmalloc::Allocator::allocateSlowCase):
764         (bmalloc::Allocator::allocateSmallLine): Deleted.
765         (bmalloc::Allocator::allocateMediumLine): Deleted.
766         (bmalloc::Allocator::allocateMedium): Deleted.
767         * bmalloc/Allocator.h:
768         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
769         into the standard fast path with small allocations. Since a BumpAllocator
770         just allocates out of an arbitrary range, it doesn't need to distinguish
771         between small and medium lines.
772
773         * bmalloc/BumpAllocator.h:
774         (bmalloc::BumpAllocator::size):
775         (bmalloc::BumpAllocator::BumpAllocator):
776         (bmalloc::BumpAllocator::init):
777         (bmalloc::BumpAllocator::refill):
778         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
779         anymore: the heap just gives us a pointer and a pre-computed number of
780         objects, and we allocate them.
781
782         * bmalloc/Deallocator.cpp:
783         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
784
785         * bmalloc/Heap.cpp:
786         (bmalloc::Heap::Heap):
787         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
788         detailing where all objects will lie in memory. After we merge two lines,
789         we might allocate an object that spans from one line to the next. This
790         metadata details which bits of memory overlap in that way, and how they
791         overlap.
792
793         (bmalloc::Heap::refillSmallBumpRangeCache):
794         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
795         and merge adjacent free lines into BumpRanges.
796
797         (bmalloc::Heap::allocateSmallPage):
798         (bmalloc::Heap::allocateMediumPage):
799         (bmalloc::Heap::deallocateSmallLine):
800         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
801         since we scan for free memory a page at a time.
802
803         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
804         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
805         fast path.
806
807         * bmalloc/Heap.h:
808         (bmalloc::Heap::derefSmallLine):
809         (bmalloc::Heap::derefMediumLine):
810         (bmalloc::Heap::deallocateSmallLine): Deleted.
811         (bmalloc::Heap::allocateSmallLine): Deleted.
812         (bmalloc::Heap::deallocateMediumLine): Deleted.
813         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
814
815         * bmalloc/Line.h:
816         (bmalloc::Line<Traits>::ref):
817         (bmalloc::Line<Traits>::deref):
818         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
819         anymore, since we only ever deref by 1 now.
820
821         * bmalloc/MediumAllocator.h:
822         (bmalloc::MediumAllocator::isNull): Deleted.
823         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
824         (bmalloc::MediumAllocator::line): Deleted.
825         (bmalloc::MediumAllocator::allocate): Deleted.
826         (bmalloc::MediumAllocator::derefCount): Deleted.
827         (bmalloc::MediumAllocator::refill): Deleted.
828         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
829         been dead for a while, since it doesn't build anymore with this patch.
830
831         * bmalloc/Page.h:
832         (bmalloc::Page::sizeClass):
833         (bmalloc::Page::setSizeClass):
834         (bmalloc::Page::smallSizeClass): Deleted.
835         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
836         to sizeClass, since we use it for medium sizes too.
837
838         * bmalloc/Sizes.h:
839         (bmalloc::Sizes::sizeClass):
840         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
841
842         (bmalloc::Sizes::smallSizeClassFor): Deleted.
843         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
844
845         * bmalloc/bmalloc.h:
846         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
847         can be a bit more precise. It also has to be, since we can't guarantee
848         that an object ends at the end of a line anymore.
849
850 2014-09-19  Daniel Bates  <dabates@apple.com>
851
852         Always assume internal SDK when building configuration Production
853         https://bugs.webkit.org/show_bug.cgi?id=136925
854         <rdar://problem/18362399>
855
856         Reviewed by Dan Bernstein.
857
858         * Configurations/Base.xcconfig:
859
860 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
861
862         bmalloc: moved line caches from the deallocator to the allocator
863         https://bugs.webkit.org/show_bug.cgi?id=136868
864
865         Reviewed by Gavin Barraclough.
866
867         I did this mostly as a simplification, to make it easier to change the
868         allocation strategy.
869
870         No throughput change on MallocBench. Saves about 50kB.
871
872         Since the deallocator needs to lock the heap when freeing lines anyway,
873         there isn't much benefit to giving the deallocator a local cache of
874         deallocated lines.
875
876         We still give the allocator a local cache of lines because that does
877         reduce the frequency at which it needs to lock the heap in order to
878         acquire more lines.
879
880         * bmalloc/Allocator.cpp:
881         (bmalloc::Allocator::scavenge):
882         (bmalloc::Allocator::allocateSmallLine):
883         (bmalloc::Allocator::allocateMediumLine):
884         (bmalloc::Allocator::allocateMedium):
885         (bmalloc::Allocator::allocateSlowCase):
886         * bmalloc/Allocator.h:
887         * bmalloc/Deallocator.cpp:
888         (bmalloc::Deallocator::Deallocator):
889         (bmalloc::Deallocator::scavenge):
890         (bmalloc::Deallocator::processObjectLog):
891         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
892         (bmalloc::Deallocator::allocateSmallLine): Deleted.
893         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
894         (bmalloc::Deallocator::allocateMediumLine): Deleted.
895         * bmalloc/Deallocator.h:
896
897         * bmalloc/Sizes.h:
898         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
899         exactly one page in size. That's about what we were shooting for anyway,
900         and it may make it easier to switch to per-page allocation in future.
901
902 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
903
904         bmalloc: allocate small and medium objects using the same bump pointer class
905         https://bugs.webkit.org/show_bug.cgi?id=136843
906
907         Reviewed by Gavin Barraclough.
908
909         4% speedup on MallocBench.
910
911         Now that medium-sized objects have dedicated per-size allocators, they
912         don't need to use an arbitrary bump pointer allocator. This means that
913         every allocator knows how many objects it will allocate from the start,
914         and we don't need a post-processing step to adjust refcounts based on
915         real allocation count.
916
917         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
918         since it's used for small and medium objects now.
919
920         * bmalloc/Allocator.cpp:
921         (bmalloc::Allocator::Allocator): Updated to use new interface.
922         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
923         to make sure that we finish allocating all the objects in it.
924
925         (bmalloc::Allocator::allocateMedium):
926         (bmalloc::Allocator::allocateSlowCase):
927         (bmalloc::Allocator::retire): Deleted.
928         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
929         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
930         * bmalloc/Allocator.h:
931         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
932         used to post-process an allocator based on how many objects it allocated.
933
934         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
935         (bmalloc::BumpAllocator::BumpAllocator):
936         (bmalloc::BumpAllocator::init):
937         (bmalloc::BumpAllocator::line):
938         (bmalloc::BumpAllocator::validate):
939         (bmalloc::BumpAllocator::allocate):
940         (bmalloc::BumpAllocator::refill):
941         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
942         about the kinds of lines they allocate into. In some cases, the line
943         type must be provided as a template parameter by the caller.
944
945         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
946         (bmalloc::SmallAllocator::line): Deleted.
947         (bmalloc::SmallAllocator::allocate): Deleted.
948         (bmalloc::SmallAllocator::objectCount): Deleted.
949         (bmalloc::SmallAllocator::derefCount): Deleted.
950         (bmalloc::SmallAllocator::refill): Deleted.
951         (bmalloc::SmallAllocator::clear): Deleted.
952
953         * bmalloc/ObjectType.h:
954         (bmalloc::isMedium):
955
956         * bmalloc/SmallAllocator.h:
957         (bmalloc::SmallAllocator::isNull): Deleted.
958         (bmalloc::SmallAllocator::canAllocate): Deleted.
959         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
960         (bmalloc::SmallAllocator::line): Deleted.
961         (bmalloc::SmallAllocator::allocate): Deleted.
962         (bmalloc::SmallAllocator::objectCount): Deleted.
963         (bmalloc::SmallAllocator::derefCount): Deleted.
964         (bmalloc::SmallAllocator::refill): Deleted.
965         (bmalloc::SmallAllocator::clear): Deleted.
966
967 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
968
969         Fixed a goof in bmalloc Vector sizing
970         https://bugs.webkit.org/show_bug.cgi?id=136795
971
972         Reviewed by Gavin Barraclough and Sam Weinig.
973
974         We want our minimum vector to be page-sized since the OS will give us
975         a page no matter what -- but we want that many bytes, and not enough
976         bytes to store that many elements.
977
978         * bmalloc/Vector.h: Math is hard.
979
980 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
981
982         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
983         https://bugs.webkit.org/show_bug.cgi?id=136693
984
985         Reviewed by Gavin Barraclough.
986
987         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
988
989         No throughput change.
990
991         We keep an array of medium allocators, just like our array of small
992         allocators.
993
994         In future, we can simplify the allocation fast path by merging the small
995         and medium allocator arrays. For now, this is the simplest change that
996         gets the win.
997
998         * bmalloc/Allocator.cpp:
999         (bmalloc::Allocator::Allocator):
1000         (bmalloc::Allocator::scavenge):
1001         (bmalloc::Allocator::allocateMedium):
1002         * bmalloc/Allocator.h:
1003         * bmalloc/Sizes.h:
1004         (bmalloc::Sizes::mediumSizeClassFor):
1005
1006 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1007
1008         Reviewed by Sam Weinig.
1009
1010         Renamed log => retire for clarity.
1011
1012         * bmalloc/Allocator.cpp:
1013         (bmalloc::Allocator::scavenge):
1014         (bmalloc::Allocator::retire):
1015         (bmalloc::Allocator::allocateMedium):
1016         (bmalloc::Allocator::allocateSlowCase):
1017         (bmalloc::Allocator::log): Deleted.
1018         * bmalloc/Allocator.h:
1019
1020 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
1021
1022         bmalloc: eager scavenge leaves behind a bogus allocator
1023         https://bugs.webkit.org/show_bug.cgi?id=136743
1024
1025         Reviewed by Sam Weinig.
1026
1027         Be sure to clear the allocator after logging it in the eager scavenge
1028         case, so that we don't later try to allocate out of the lines that we
1029         have thrown away.
1030
1031         We didn't need to do this previously because scavenge would only happen
1032         at thread exit time, after which no further allocation from the per-thread
1033         cache would take place.
1034
1035         * bmalloc/Allocator.cpp:
1036         (bmalloc::Allocator::scavenge):
1037         * bmalloc/MediumAllocator.h:
1038         (bmalloc::MediumAllocator::clear):
1039         * bmalloc/SmallAllocator.h:
1040         (bmalloc::SmallAllocator::clear):
1041
1042 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
1043
1044         bmalloc should honor the FastMalloc statistics API
1045         https://bugs.webkit.org/show_bug.cgi?id=136592
1046
1047         Reviewed by Gavin Barraclough.
1048
1049         We do this by tracking "size" and "capacity" in the VM heap.
1050
1051         The VM heap's "capacity" is all the VM we ever allocated.
1052
1053         The VM heap's "size" the subset of VM currently held onto by the
1054         VM heap (and therefore not in use by the regular heap).
1055
1056         Somewhat ironically, reducing the process's memory footprint, increases
1057         the size of the VM heap, since the VM heap holds the pages that are
1058         purely virtual and not physical.
1059
1060         * bmalloc/Heap.cpp:
1061         (bmalloc::Heap::size):
1062         (bmalloc::Heap::capacity):
1063         * bmalloc/Heap.h:
1064         * bmalloc/VMHeap.cpp:
1065         (bmalloc::VMHeap::VMHeap):
1066         (bmalloc::VMHeap::allocateSmallChunk):
1067         (bmalloc::VMHeap::allocateMediumChunk):
1068         (bmalloc::VMHeap::allocateLargeChunk):
1069         * bmalloc/VMHeap.h:
1070         (bmalloc::VMHeap::size):
1071         (bmalloc::VMHeap::capacity):
1072         (bmalloc::VMHeap::allocateSmallPage):
1073         (bmalloc::VMHeap::allocateMediumPage):
1074         (bmalloc::VMHeap::allocateLargeRange):
1075         (bmalloc::VMHeap::deallocateSmallPage):
1076         (bmalloc::VMHeap::deallocateMediumPage):
1077         (bmalloc::VMHeap::deallocateLargeRange):
1078         * bmalloc/bmalloc.h:
1079         (bmalloc::api::heapSize):
1080         (bmalloc::api::heapCapacity):
1081
1082 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
1083
1084         bmalloc crashes on the EWS bots (due to bad large object allocation)
1085         https://bugs.webkit.org/show_bug.cgi?id=136469
1086
1087         Reviewed by Andreas Kling.
1088
1089         It's possible to convince bmalloc to perform a bad large object allocation,
1090         through these steps:
1091
1092         (1) Insert object A into freelist F0.
1093
1094         (2) Split, merge and split again A's neighbors such that object B is
1095         inserted into freelist F0, with boundary tag and size equal to object A,
1096         but pointer not completely equal to object A. Put object B at the head of F0.
1097
1098         (3) Allocate some other object from F0, swapping its position in the
1099         freelist with object B, such that object A is now ahead of object B.
1100
1101         --> Now, the next allocation for size A/B will allocate object A, which
1102         has a slightly wrong idea about where the object actually begins.
1103         Immediately, you'll corrupt a little memory, and over time, you'll also
1104         corrupt boundary tag metadata.
1105
1106         The solution is to store the begin pointer in the boundary tag. Luckily,
1107         this doesn't make the tag any bigger, and it's not a noticeable slowdown
1108         on MallocBench.
1109
1110         * bmalloc/Algorithm.h:
1111         (bmalloc::rightShift):
1112         * bmalloc/BeginTag.h:
1113         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
1114         validate the start pointer when popping off the free list. Through a
1115         very uncommon set of steps, it is possible to have an item in the free
1116         list that is valid by all accounts except for its start pointer.
1117
1118         * bmalloc/BoundaryTag.h:
1119         (bmalloc::BoundaryTag::compactBegin):
1120         (bmalloc::BoundaryTag::setRange):
1121         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
1122         start pointer. We don't need the whole pointer -- just the offset, in
1123         largeAlignment increments, into the relevant boundary tag bucket.
1124
1125         * bmalloc/BoundaryTagInlines.h:
1126         (bmalloc::validateNext):
1127         (bmalloc::BoundaryTag::init):
1128         (bmalloc::BoundaryTag::mergeLarge):
1129         (bmalloc::BoundaryTag::splitLarge):
1130         * bmalloc/SegregatedFreeList.cpp:
1131         (bmalloc::SegregatedFreeList::insert):
1132         (bmalloc::SegregatedFreeList::takeGreedy):
1133         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
1134         the size when establishing a boundary tag, as required by the new
1135         interface.
1136
1137         * bmalloc/Sizes.h:
1138
1139 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1140
1141         Fixed a bmalloc crash seen on the EWS bot
1142         https://bugs.webkit.org/show_bug.cgi?id=135955
1143
1144         Reviewed by Andreas Kling.
1145
1146         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
1147         input buffer is a malloc region, that region will get marked Copy-On-Write
1148         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
1149         on older OS X's. In 10.10, they still fail, but they do not return
1150         EINVAL.
1151
1152         So, we can only ASSERT that our syscalls succeed starting with 10.10.
1153
1154 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
1155
1156         Fixed the bmalloc build
1157         https://bugs.webkit.org/show_bug.cgi?id=135953
1158
1159         Reviewed by Andreas Kling.
1160
1161         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
1162         These headers are used, so they must be available outside the project.
1163
1164 2014-08-13  Daniel Bates  <dabates@apple.com>
1165
1166         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
1167         (https://bugs.webkit.org/show_bug.cgi?id=135895)
1168
1169         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
1170         implementation of PerThread<T>::getFastCase().
1171
1172         * bmalloc/PerThread.h:
1173         (bmalloc::PerThread<T>::getFastCase):
1174
1175 2014-08-13  Daniel Bates  <dabates@apple.com>
1176
1177         Make bmalloc::PerThread work without C++ thread local storage
1178         https://bugs.webkit.org/show_bug.cgi?id=135895
1179
1180         Reviewed by Geoffrey Garen.
1181
1182         Implement support for building bmalloc without C++ thread local storage.
1183
1184         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
1185         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
1186         to determine whether the compiler supports C++ thread local storage.
1187         * bmalloc/PerThread.h:
1188         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
1189         without C++ thread local storage.
1190         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
1191         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
1192         PerThreadStorage::initSharedKeyIfNeeded().
1193         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
1194         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
1195         ensure that the shared key has been initialized.
1196         (_pthread_setspecific_direct): Deleted.
1197         (_pthread_getspecific_direct): Deleted.
1198
1199 2014-08-13  Daniel Bates  <dabates@apple.com>
1200
1201         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
1202         https://bugs.webkit.org/show_bug.cgi?id=135848
1203
1204         Reviewed by Geoffrey Garen.
1205
1206         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
1207         when building for the iOS Simulator.
1208         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
1209         using the public SDK.
1210         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
1211         using the public SDK.
1212         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
1213         using the public SDK.
1214
1215 2014-08-12  Daniel Bates  <dabates@apple.com>
1216
1217         BPLATFORM(IOS) always evaluates to false
1218         https://bugs.webkit.org/show_bug.cgi?id=135843
1219
1220         Reviewed by Geoffrey Garen.
1221
1222         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
1223         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
1224         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
1225         building for iOS.
1226
1227         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
1228         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
1229
1230         * bmalloc/BPlatform.h:
1231
1232 2014-08-11  Andy Estes  <aestes@apple.com>
1233
1234         [iOS] Get rid of iOS.xcconfig
1235         https://bugs.webkit.org/show_bug.cgi?id=135809
1236
1237         Reviewed by Joseph Pecoraro.
1238
1239         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
1240
1241         * Configurations/Base.xcconfig:
1242         * Configurations/iOS.xcconfig: Removed.
1243         * bmalloc.xcodeproj/project.pbxproj:
1244
1245 2014-05-01  Dan Bernstein  <mitz@apple.com>
1246
1247         Fixed production builds for the iOS Simulator.
1248         <rdar://problem/16792221>
1249
1250         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
1251         PRIVATE_HEADERS_FOLDER_PATH when installing.
1252
1253 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
1254
1255         bmalloc: Segregate pages by objects size
1256         https://bugs.webkit.org/show_bug.cgi?id=131909
1257
1258         Reviewed by Andreas Kling.
1259
1260         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
1261
1262         * bmalloc/Allocator.cpp:
1263         (bmalloc::Allocator::allocateSlowCase):
1264         * bmalloc/Allocator.h:
1265         (bmalloc::Allocator::allocateFastCase):
1266         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
1267         function for size class calculation.
1268
1269         * bmalloc/Deallocator.cpp:
1270         (bmalloc::Deallocator::Deallocator):
1271         (bmalloc::Deallocator::scavenge):
1272         (bmalloc::Deallocator::deallocateSmallLine):
1273         (bmalloc::Deallocator::allocateSmallLine):
1274         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
1275         cache can't be shared anymore.
1276
1277         * bmalloc/Heap.cpp:
1278         (bmalloc::Heap::allocateSmallLineSlowCase):
1279         * bmalloc/Heap.h:
1280         (bmalloc::Heap::deallocateSmallLine): Ditto.
1281
1282         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
1283         page refcount when allocating a line because we might have deallocated
1284         the page and the recycled it for another size class.
1285
1286         (bmalloc::Heap::deallocateMediumLine):
1287         (bmalloc::Heap::allocateMediumLine):
1288         * bmalloc/Line.h:
1289         (bmalloc::Line::refCount):
1290         * bmalloc/Page.h:
1291         (bmalloc::Page::refCount):
1292         (bmalloc::Page::smallSizeClass):
1293         (bmalloc::Page::setSmallSizeClass):
1294         (bmalloc::Page<Traits>::refCount): Deleted.
1295         * bmalloc/Sizes.h:
1296         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
1297         an index into an array from a size.
1298
1299 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1300
1301         bmalloc: Improved alignment in LargeChunk
1302         https://bugs.webkit.org/show_bug.cgi?id=131895
1303
1304         Reviewed by Andreas Kling.
1305
1306         * bmalloc/Chunk.h:
1307         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
1308         Technically, the previous alignment was harmless, but I would prefer,
1309         dear reader, not to have to explain the interlocking set of
1310         circumstances that made it so.
1311
1312 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1313
1314         Rolled out r167502 because it caused a crash on the facebook benchmark.
1315
1316         Unreviewed.
1317
1318             bmalloc: Added an XSmall line size
1319             https://bugs.webkit.org/show_bug.cgi?id=131851
1320
1321             Reviewed by Sam Weinig.
1322
1323 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
1324
1325         bmalloc: Mutex should be harder to use wrong
1326         https://bugs.webkit.org/show_bug.cgi?id=131879
1327
1328         Reviewed by Andreas Kling.
1329
1330         Mutex now has a proper constructor, so you can't deadlock by forgetting
1331         to initialize it.
1332
1333         * bmalloc.xcodeproj/project.pbxproj:
1334         * bmalloc/Allocator.cpp:
1335         (bmalloc::Allocator::processXSmallAllocatorLog):
1336         (bmalloc::Allocator::processSmallAllocatorLog):
1337         (bmalloc::Allocator::processMediumAllocatorLog):
1338         (bmalloc::Allocator::allocateLarge):
1339         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
1340         since the Heap mutex is a static.
1341
1342         * bmalloc/AsyncTask.h:
1343         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
1344         need for explicit initialization anymore.
1345
1346         * bmalloc/Deallocator.cpp:
1347         (bmalloc::Deallocator::scavenge):
1348         (bmalloc::Deallocator::deallocateLarge):
1349         (bmalloc::Deallocator::deallocateXLarge):
1350         (bmalloc::Deallocator::processObjectLog):
1351         (bmalloc::Deallocator::deallocateSmallLine):
1352         (bmalloc::Deallocator::deallocateXSmallLine):
1353         (bmalloc::Deallocator::allocateSmallLine):
1354         (bmalloc::Deallocator::allocateXSmallLine):
1355         (bmalloc::Deallocator::deallocateMediumLine):
1356         (bmalloc::Deallocator::allocateMediumLine):
1357         * bmalloc/Deallocator.h:
1358         * bmalloc/Heap.cpp:
1359         (bmalloc::sleep):
1360         (bmalloc::Heap::Heap):
1361         (bmalloc::Heap::concurrentScavenge):
1362         (bmalloc::Heap::scavenge):
1363         (bmalloc::Heap::scavengeSmallPages):
1364         (bmalloc::Heap::scavengeXSmallPages):
1365         (bmalloc::Heap::scavengeMediumPages):
1366         (bmalloc::Heap::scavengeLargeRanges):
1367         (bmalloc::Heap::allocateXSmallLineSlowCase):
1368         (bmalloc::Heap::allocateSmallLineSlowCase):
1369         (bmalloc::Heap::allocateMediumLineSlowCase):
1370         (bmalloc::Heap::allocateXLarge):
1371         (bmalloc::Heap::deallocateXLarge):
1372         (bmalloc::Heap::allocateLarge):
1373         (bmalloc::Heap::deallocateLarge):
1374         * bmalloc/Heap.h:
1375         (bmalloc::Heap::deallocateXSmallLine):
1376         (bmalloc::Heap::allocateXSmallLine):
1377         (bmalloc::Heap::deallocateSmallLine):
1378         (bmalloc::Heap::allocateSmallLine):
1379         (bmalloc::Heap::deallocateMediumLine):
1380         (bmalloc::Heap::allocateMediumLine):
1381         * bmalloc/Line.h:
1382         (bmalloc::Line<Traits>::deref):
1383         * bmalloc/Mutex.cpp: Removed.
1384         * bmalloc/Mutex.h:
1385         (bmalloc::Mutex::Mutex):
1386         (bmalloc::Mutex::init): Deleted.
1387         (bmalloc::Mutex::try_lock): Deleted.
1388         (bmalloc::Mutex::lock): Deleted.
1389         (bmalloc::Mutex::unlock): Deleted.
1390         * bmalloc/Page.h:
1391         (bmalloc::Page<Traits>::ref):
1392         (bmalloc::Page<Traits>::deref):
1393         (bmalloc::Page<Traits>::refCount):
1394         * bmalloc/PerProcess.h:
1395         (bmalloc::PerProcess::mutex):
1396         (bmalloc::PerProcess<T>::getSlowCase):
1397         * bmalloc/StaticMutex.cpp: Added.
1398         (bmalloc::StaticMutex::lockSlowCase):
1399         * bmalloc/StaticMutex.h: Added.
1400         (bmalloc::StaticMutex::init):
1401         (bmalloc::StaticMutex::try_lock):
1402         (bmalloc::StaticMutex::lock):
1403         (bmalloc::StaticMutex::unlock):
1404         * bmalloc/VMHeap.h:
1405         (bmalloc::VMHeap::deallocateXSmallPage):
1406         (bmalloc::VMHeap::deallocateSmallPage):
1407         (bmalloc::VMHeap::deallocateMediumPage):
1408         (bmalloc::VMHeap::deallocateLargeRange):
1409         * bmalloc/bmalloc.h:
1410         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
1411         since the Heap mutex is a static.
1412
1413 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1414
1415         bmalloc: AsyncTask should use Mutex instead of std::mutex
1416         https://bugs.webkit.org/show_bug.cgi?id=131865
1417
1418         Reviewed by Gavin Barraclough.
1419
1420         std::mutex is so slow that it makes parallelizing simple tasks through
1421         AsyncTask a net regression. Mutex fixes this.
1422
1423         * bmalloc/AsyncTask.h:
1424         (bmalloc::Function>::AsyncTask):
1425         (bmalloc::Function>::join):
1426         (bmalloc::Function>::runSlowCase):
1427         (bmalloc::Function>::entryPoint):
1428         * bmalloc/Mutex.h:
1429         (bmalloc::Mutex::init):
1430
1431 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
1432
1433         bmalloc: Added an XSmall line size
1434         https://bugs.webkit.org/show_bug.cgi?id=131851
1435
1436         Reviewed by Sam Weinig.
1437
1438         Reduces malloc footprint on Membuster recordings by 10%.
1439
1440         This is a throughput regression, but we're still way ahead of TCMalloc.
1441         I have some ideas for how to recover the regression -- but I wanted to
1442         get this win in first.
1443
1444         Full set of benchmark results:
1445
1446                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1447                                                                                                 
1448                                                                nopatch                      patch                                Δ
1449                 Peak Memory:
1450                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
1451                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
1452                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1453
1454                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1455                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1456                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1457
1458                 Memory at End:
1459                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1460                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1461                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1462
1463                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1464                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1465                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1466
1467                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1468                                                                                                 
1469                                                    nopatch                patch                         Δ
1470                 Execution Time:
1471                     churn                            127ms                151ms            ! 1.19x slower
1472                     list_allocate                    130ms                164ms            ! 1.26x slower
1473                     tree_allocate                    109ms                127ms            ! 1.17x slower
1474                     tree_churn                       115ms                120ms            ! 1.04x slower
1475                     facebook                         240ms                259ms            ! 1.08x slower
1476                     fragment                          91ms                131ms            ! 1.44x slower
1477                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1478                     message_one                      260ms                259ms             ^ 1.0x faster
1479                     message_many                     149ms                154ms            ! 1.03x slower
1480                     medium                           194ms                248ms            ! 1.28x slower
1481                     big                              157ms                160ms            ! 1.02x slower
1482
1483                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1484                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1485                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1486
1487                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1488                                                                                                 
1489                                                                        nopatch                          patch                                     Δ
1490                 Execution Time:
1491                     churn                                                126ms                          148ms                        ! 1.17x slower
1492                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1493                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1494                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1495                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1496                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1497                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1498                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1499                     facebook                                             240ms                          276ms                        ! 1.15x slower
1500                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1501                     fragment                                              92ms                          130ms                        ! 1.41x slower
1502                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1503                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1504                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1505                     message_one                                          260ms                          260ms                                      
1506                     message_many                                         170ms                          238ms                         ! 1.4x slower
1507                     medium                                               185ms                          250ms                        ! 1.35x slower
1508                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1509                     big                                                  150ms                          169ms                        ! 1.13x slower
1510                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1511
1512                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1513                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1514                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1515
1516                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1517
1518                                                                     TC                      patch                                Δ
1519                 Peak Memory:
1520                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1521                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1522                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1523
1524                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1525                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1526                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1527
1528                 Memory at End:
1529                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
1530                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
1531                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
1532
1533                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
1534                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
1535                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
1536
1537                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1538                                                                                                 
1539                                                         TC                patch                         Δ
1540                 Execution Time:
1541                     churn                            416ms                148ms            ^ 2.81x faster
1542                     list_allocate                    463ms                164ms            ^ 2.82x faster
1543                     tree_allocate                    292ms                127ms             ^ 2.3x faster
1544                     tree_churn                       157ms                120ms            ^ 1.31x faster
1545                     facebook                         327ms                276ms            ^ 1.18x faster
1546                     fragment                         335ms                129ms             ^ 2.6x faster
1547                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
1548                     message_one                      386ms                258ms             ^ 1.5x faster
1549                     message_many                     410ms                154ms            ^ 2.66x faster
1550                     medium                           391ms                245ms             ^ 1.6x faster
1551                     big                              261ms                167ms            ^ 1.56x faster
1552
1553                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
1554                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
1555                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
1556
1557         * bmalloc.xcodeproj/project.pbxproj:
1558         * bmalloc/Allocator.cpp:
1559         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
1560         index corresponds with its size. Instead, use the size selection function
1561         explicitly. Now that we have XSmall, some small allocator entries are
1562         unused.
1563
1564         (bmalloc::Allocator::scavenge):
1565         (bmalloc::Allocator::log):
1566         (bmalloc::Allocator::processXSmallAllocatorLog):
1567         (bmalloc::Allocator::allocateSlowCase):
1568         * bmalloc/Allocator.h:
1569         (bmalloc::Allocator::xSmallAllocatorFor):
1570         (bmalloc::Allocator::allocateFastCase):
1571         * bmalloc/Chunk.h:
1572         * bmalloc/Deallocator.cpp:
1573         (bmalloc::Deallocator::scavenge):
1574         (bmalloc::Deallocator::processObjectLog):
1575         (bmalloc::Deallocator::deallocateSlowCase):
1576         (bmalloc::Deallocator::deallocateXSmallLine):
1577         (bmalloc::Deallocator::allocateXSmallLine):
1578         * bmalloc/Deallocator.h:
1579         (bmalloc::Deallocator::deallocateFastCase):
1580         * bmalloc/Heap.cpp:
1581         (bmalloc::Heap::scavenge):
1582         (bmalloc::Heap::scavengeXSmallPages):
1583         (bmalloc::Heap::allocateXSmallLineSlowCase):
1584         * bmalloc/Heap.h:
1585         (bmalloc::Heap::deallocateXSmallLine):
1586         (bmalloc::Heap::allocateXSmallLine):
1587         * bmalloc/LargeChunk.h:
1588         (bmalloc::LargeChunk::get):
1589         (bmalloc::LargeChunk::endTag):
1590         * bmalloc/Line.h:
1591         * bmalloc/MediumAllocator.h:
1592         (bmalloc::MediumAllocator::allocate):
1593         (bmalloc::MediumAllocator::refill):
1594         * bmalloc/ObjectType.cpp:
1595         (bmalloc::objectType):
1596         * bmalloc/ObjectType.h:
1597         (bmalloc::isXSmall):
1598         (bmalloc::isSmall):
1599         (bmalloc::isMedium):
1600         (bmalloc::isLarge):
1601         (bmalloc::isSmallOrMedium): Deleted.
1602         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1603         handling small objects. There's probably a reasonable way to share this
1604         code in the future -- I'll look into that once it's stopped changing.
1605
1606         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1607         is the main reason things got slower.
1608
1609         * bmalloc/SmallAllocator.h:
1610         (bmalloc::SmallAllocator::allocate):
1611         * bmalloc/SmallTraits.h:
1612         * bmalloc/VMHeap.cpp:
1613         (bmalloc::VMHeap::allocateXSmallChunk):
1614         * bmalloc/VMHeap.h:
1615         (bmalloc::VMHeap::allocateXSmallPage):
1616         (bmalloc::VMHeap::deallocateXSmallPage):
1617         * bmalloc/XSmallAllocator.h: Added.
1618         (bmalloc::XSmallAllocator::isNull):
1619         (bmalloc::XSmallAllocator::canAllocate):
1620         (bmalloc::XSmallAllocator::XSmallAllocator):
1621         (bmalloc::XSmallAllocator::line):
1622         (bmalloc::XSmallAllocator::allocate):
1623         (bmalloc::XSmallAllocator::objectCount):
1624         (bmalloc::XSmallAllocator::derefCount):
1625         (bmalloc::XSmallAllocator::refill):
1626         * bmalloc/XSmallChunk.h: Added.
1627         * bmalloc/XSmallLine.h: Added.
1628         * bmalloc/XSmallPage.h: Added.
1629         * bmalloc/XSmallTraits.h: Added.
1630         * bmalloc/bmalloc.h:
1631         (bmalloc::api::realloc): Boiler-plate copy, as above.
1632
1633 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1634
1635         MallocBench should scavenge explicitly instead of waiting
1636         https://bugs.webkit.org/show_bug.cgi?id=131661
1637
1638         Reviewed by Andreas Kling.
1639
1640         Added explicit scavenge support to bmalloc. This isn't a memory win,
1641         since bmalloc's per-thread cache is so small. But it makes testing
1642         simpler.
1643
1644         * bmalloc/Allocator.cpp:
1645         (bmalloc::Allocator::~Allocator):
1646         (bmalloc::Allocator::scavenge):
1647         * bmalloc/Allocator.h:
1648         * bmalloc/Cache.cpp:
1649         (bmalloc::Cache::operator new):
1650         (bmalloc::Cache::operator delete):
1651         (bmalloc::Cache::Cache):
1652         (bmalloc::Cache::scavenge):
1653         * bmalloc/Cache.h:
1654         * bmalloc/Deallocator.cpp:
1655         (bmalloc::Deallocator::~Deallocator):
1656         (bmalloc::Deallocator::scavenge):
1657         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1658         functions, for reuse.
1659
1660         * bmalloc/Heap.cpp:
1661         (bmalloc::sleep):
1662         (bmalloc::Heap::concurrentScavenge):
1663         (bmalloc::Heap::scavenge):
1664         (bmalloc::Heap::scavengeSmallPages):
1665         (bmalloc::Heap::scavengeMediumPages):
1666         (bmalloc::Heap::scavengeLargeRanges):
1667         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1668         scavenging -- in response to a benchmark or a low memory warning --
1669         wants to complete as soon as possible, so its sleep duration is 0.
1670
1671         * bmalloc/bmalloc.h:
1672         (bmalloc::api::scavenge):
1673         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1674
1675 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1676
1677         Use 4kB pages on Mac
1678         https://bugs.webkit.org/show_bug.cgi?id=131658
1679
1680         Reviewed by Sam Weinig.
1681
1682         This reduces memory use a lot on Membuster:
1683
1684                                                                   base                      patch                                Δ
1685                 Execution Time:
1686                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1687                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1688                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1689
1690                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1691                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1692                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1693
1694                 Peak Memory:
1695                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1696                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1697                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1698
1699                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1700                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1701                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1702
1703                 Memory at End:
1704                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1705                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1706                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1707
1708                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1709                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1710                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1711
1712
1713         * bmalloc.xcodeproj/project.pbxproj:
1714         * bmalloc/BPlatform.h: Added.
1715         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1716         is 4kB on Mac.
1717
1718 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1719
1720         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1721
1722         * bmalloc.xcodeproj: Modified property svn:ignore.
1723
1724 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1725
1726         Fixed some mbmalloc exports
1727         https://bugs.webkit.org/show_bug.cgi?id=131599
1728
1729         Reviewed by Ryosuke Niwa.
1730
1731         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1732         of the project, so we can call them from API.
1733
1734         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1735         visibility, so they show up as exported in the .dylib.
1736
1737 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1738
1739         Put bmalloc headers in the right place
1740         https://bugs.webkit.org/show_bug.cgi?id=131464
1741
1742         Reviewed by Mark Rowe.
1743
1744         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1745         specify that we don't just want to dump all of our generically-named
1746         headers into /usr/local/include.
1747
1748 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1749
1750         Made bmalloc more #include friendly
1751         https://bugs.webkit.org/show_bug.cgi?id=131386
1752
1753         Reviewed by Andreas Kling.
1754
1755         Marked a bunch of headers private so they can be used from client code
1756         that #includes bmalloc.h.
1757
1758         Renamed ASSERT macros to BASSERT. This matches their header, which already
1759         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1760
1761         * bmalloc.xcodeproj/project.pbxproj:
1762         * bmalloc/Allocator.cpp:
1763         (bmalloc::Allocator::allocateSlowCase):
1764         * bmalloc/AsyncTask.h:
1765         (bmalloc::Function>::runSlowCase):
1766         * bmalloc/BAssert.h:
1767         * bmalloc/BoundaryTag.h:
1768         (bmalloc::BoundaryTag::setSize):
1769         * bmalloc/BoundaryTagInlines.h:
1770         (bmalloc::validate):
1771         (bmalloc::BoundaryTag::init):
1772         (bmalloc::BoundaryTag::deallocate):
1773         (bmalloc::BoundaryTag::splitLarge):
1774         (bmalloc::BoundaryTag::allocate):
1775         * bmalloc/Chunk.h:
1776         * bmalloc/Deallocator.cpp:
1777         (bmalloc::Deallocator::processObjectLog):
1778         (bmalloc::Deallocator::deallocateSlowCase):
1779         * bmalloc/Deallocator.h:
1780         (bmalloc::Deallocator::deallocateFastCase):
1781         * bmalloc/FixedVector.h:
1782         (bmalloc::Capacity>::operator):
1783         (bmalloc::Capacity>::push):
1784         (bmalloc::Capacity>::pop):
1785         (bmalloc::Capacity>::shrink):
1786         * bmalloc/Heap.cpp:
1787         (bmalloc::Heap::allocateLarge):
1788         * bmalloc/LargeChunk.h:
1789         (bmalloc::LargeChunk::get):
1790         (bmalloc::LargeChunk::endTag):
1791         * bmalloc/Line.h:
1792         (bmalloc::Line<Traits>::concurrentRef):
1793         (bmalloc::Line<Traits>::deref):
1794         * bmalloc/MediumAllocator.h:
1795         (bmalloc::MediumAllocator::allocate):
1796         * bmalloc/ObjectType.h:
1797         (bmalloc::isSmall):
1798         * bmalloc/Page.h:
1799         (bmalloc::Page<Traits>::ref):
1800         (bmalloc::Page<Traits>::deref):
1801         * bmalloc/PerThread.h:
1802         (bmalloc::PerThread<T>::getSlowCase):
1803         * bmalloc/SegregatedFreeList.cpp:
1804         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1805         (bmalloc::SegregatedFreeList::insert):
1806         * bmalloc/SmallAllocator.h:
1807         (bmalloc::SmallAllocator::allocate):
1808         (bmalloc::SmallAllocator::refill):
1809         * bmalloc/Syscall.h:
1810         * bmalloc/VMAllocate.h:
1811         (bmalloc::vmValidate):
1812         (bmalloc::vmAllocate):
1813         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1814         * bmalloc/Vector.h:
1815         (bmalloc::Vector<T>::operator):
1816         (bmalloc::Vector<T>::pop):
1817         (bmalloc::Vector<T>::shrink):
1818         * bmalloc/XLargeChunk.h:
1819         (bmalloc::XLargeChunk::range):
1820         (bmalloc::XLargeChunk::size):
1821
1822 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1823
1824         Removed an unused file.
1825
1826         Unreviewed.
1827
1828         * bmalloc/AsyncTask.cpp: Removed.
1829
1830 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1831
1832         Build bmalloc on Mac
1833         https://bugs.webkit.org/show_bug.cgi?id=131333
1834
1835         Reviewed by Mark Rowe.
1836
1837         * Makefile: Added. For make clients.
1838
1839         These files are required for building any project in WebKit. I copied
1840         them from WTF:
1841         * Configurations: Added.
1842         * Configurations/Base.xcconfig: Added.
1843         * Configurations/DebugRelease.xcconfig: Added.
1844         * Configurations/bmalloc.xcconfig: Added.
1845         * Configurations/iOS.xcconfig: Added.
1846         * Configurations/mbmalloc.xcconfig: Added.
1847
1848         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1849         from here because everything is in .xcconfig files now.
1850
1851         I had to fix a bunch of minor warnings, since they're enabled in our
1852         .xcconfig files:
1853
1854         * bmalloc/AsyncTask.h:
1855         (bmalloc::Function>::AsyncTask):
1856         * bmalloc/BAssert.h:
1857         * bmalloc/BoundaryTagInlines.h:
1858         (bmalloc::validate):
1859         * bmalloc/Heap.cpp:
1860         (bmalloc::Heap::Heap):
1861         (bmalloc::Heap::allocateLarge):
1862         (bmalloc::Heap::deallocateLarge):
1863         * bmalloc/Mutex.h:
1864         (bmalloc::Mutex::Mutex): Deleted.
1865         * bmalloc/VMAllocate.h:
1866         (bmalloc::vmValidate):
1867         * bmalloc/mbmalloc.cpp:
1868
1869 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1870
1871         bmalloc: Fixed a leak in the per-thread cache
1872         https://bugs.webkit.org/show_bug.cgi?id=131330
1873
1874         Reviewed by Andreas Kling.
1875
1876         Remember to deallocate our line caches upon thread exit.
1877
1878         * bmalloc/Deallocator.cpp:
1879         (bmalloc::Deallocator::~Deallocator):
1880
1881 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1882
1883         bmalloc: rolled out the tryLock experiment
1884         https://bugs.webkit.org/show_bug.cgi?id=131328
1885
1886         Reviewed by Andreas Kling.
1887
1888         It wasn't a speedup.
1889
1890         * bmalloc.xcodeproj/project.pbxproj:
1891         * bmalloc/Allocator.cpp:
1892         (bmalloc::Allocator::processSmallAllocatorLog):
1893         (bmalloc::Allocator::processMediumAllocatorLog):
1894         * bmalloc/Deallocator.cpp:
1895         (bmalloc::Deallocator::processObjectLog):
1896         (bmalloc::Deallocator::deallocateSlowCase):
1897         (bmalloc::Deallocator::deallocateSmallLine):
1898         (bmalloc::Deallocator::deallocateMediumLine):
1899         * bmalloc/Deallocator.h:
1900         (bmalloc::Deallocator::deallocateFastCase):
1901         * bmalloc/Heap.h:
1902         (bmalloc::Heap::deallocateSmallLine):
1903         (bmalloc::Heap::deallocateMediumLine):
1904         * bmalloc/Line.h:
1905         (bmalloc::Line<Traits>::deref):
1906         * bmalloc/Page.h:
1907         (bmalloc::Page<Traits>::deref):
1908
1909 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1910
1911         bmalloc
1912         https://bugs.webkit.org/show_bug.cgi?id=131170
1913
1914         Reviewed by Andreas Kling.
1915
1916         Initial commit.
1917
1918         * bmalloc: Added.
1919         * bmalloc.xcodeproj: Added.
1920         * bmalloc.xcodeproj/project.pbxproj: Added.
1921         * bmalloc/Algorithm.h: Added.
1922         (bmalloc::max):
1923         (bmalloc::min):
1924         (bmalloc::mask):
1925         (bmalloc::test):
1926         (bmalloc::roundUpToMultipleOf):
1927         (bmalloc::roundDownToMultipleOf):
1928         (bmalloc::sizeOf):
1929         (bmalloc::bitCount):
1930         (bmalloc::isPowerOfTwo):
1931         * bmalloc/Allocator.cpp: Added.
1932         (bmalloc::Allocator::Allocator):
1933         (bmalloc::Allocator::~Allocator):
1934         (bmalloc::Allocator::log):
1935         (bmalloc::Allocator::processSmallAllocatorLog):
1936         (bmalloc::Allocator::processMediumAllocatorLog):
1937         (bmalloc::Allocator::allocateLarge):
1938         (bmalloc::Allocator::allocateXLarge):
1939         (bmalloc::Allocator::allocateMedium):
1940         (bmalloc::Allocator::allocateSlowCase):
1941         * bmalloc/Allocator.h: Added.
1942         (bmalloc::Allocator::smallAllocatorFor):
1943         (bmalloc::Allocator::allocateFastCase):
1944         (bmalloc::Allocator::allocate):
1945         * bmalloc/AsyncTask.cpp: Added.
1946         (bmalloc::AsyncTask<Function>::runSlowCase):
1947         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
1948         (bmalloc::AsyncTask<Function>::entryPoint):
1949         * bmalloc/AsyncTask.h: Added.
1950         (bmalloc::Function>::AsyncTask):
1951         (bmalloc::Function>::join):
1952         (bmalloc::Function>::run):
1953         (bmalloc::Function>::runSlowCase):
1954         (bmalloc::Function>::pthreadEntryPoint):
1955         (bmalloc::Function>::entryPoint):
1956         * bmalloc/BAssert.h: Added.
1957         * bmalloc/BeginTag.h: Added.
1958         (bmalloc::BeginTag::isInFreeList):
1959         * bmalloc/BoundaryTag.h: Added.
1960         (bmalloc::BoundaryTag::isXLarge):
1961         (bmalloc::BoundaryTag::setXLarge):
1962         (bmalloc::BoundaryTag::isFree):
1963         (bmalloc::BoundaryTag::setFree):
1964         (bmalloc::BoundaryTag::isEnd):
1965         (bmalloc::BoundaryTag::setEnd):
1966         (bmalloc::BoundaryTag::hasPhysicalPages):
1967         (bmalloc::BoundaryTag::setHasPhysicalPages):
1968         (bmalloc::BoundaryTag::isNull):
1969         (bmalloc::BoundaryTag::clear):
1970         (bmalloc::BoundaryTag::size):
1971         (bmalloc::BoundaryTag::setSize):
1972         (bmalloc::BoundaryTag::prev):
1973         (bmalloc::BoundaryTag::next):
1974         * bmalloc/BoundaryTagInlines.h: Added.
1975         (bmalloc::validate):
1976         (bmalloc::validatePrev):
1977         (bmalloc::validateNext):
1978         (bmalloc::BoundaryTag::init):
1979         (bmalloc::BoundaryTag::mergeLargeLeft):
1980         (bmalloc::BoundaryTag::mergeLargeRight):
1981         (bmalloc::BoundaryTag::mergeLarge):
1982         (bmalloc::BoundaryTag::deallocate):
1983         (bmalloc::BoundaryTag::splitLarge):
1984         (bmalloc::BoundaryTag::allocate):
1985         * bmalloc/Cache.cpp: Added.
1986         (bmalloc::Cache::operator new):
1987         (bmalloc::Cache::operator delete):
1988         (bmalloc::Cache::Cache):
1989         (bmalloc::Cache::allocateSlowCase):
1990         (bmalloc::Cache::allocateSlowCaseNullCache):
1991         (bmalloc::Cache::deallocateSlowCase):
1992         (bmalloc::Cache::deallocateSlowCaseNullCache):
1993         * bmalloc/Cache.h: Added.
1994         (bmalloc::Cache::allocator):
1995         (bmalloc::Cache::deallocator):
1996         (bmalloc::Cache::allocateFastCase):
1997         (bmalloc::Cache::deallocateFastCase):
1998         (bmalloc::Cache::allocate):
1999         (bmalloc::Cache::deallocate):
2000         * bmalloc/Chunk.h: Added.
2001         (bmalloc::Chunk::begin):
2002         (bmalloc::Chunk::end):
2003         (bmalloc::Chunk::lines):
2004         (bmalloc::Chunk::pages):
2005         * bmalloc/Deallocator.cpp: Added.
2006         (bmalloc::Deallocator::Deallocator):
2007         (bmalloc::Deallocator::~Deallocator):
2008         (bmalloc::Deallocator::deallocateLarge):
2009         (bmalloc::Deallocator::deallocateXLarge):
2010         (bmalloc::Deallocator::processObjectLog):
2011         (bmalloc::Deallocator::deallocateSlowCase):
2012         (bmalloc::Deallocator::deallocateSmallLine):
2013         (bmalloc::Deallocator::allocateSmallLine):
2014         (bmalloc::Deallocator::deallocateMediumLine):
2015         (bmalloc::Deallocator::allocateMediumLine):
2016         * bmalloc/Deallocator.h: Added.
2017         (bmalloc::Deallocator::deallocateFastCase):
2018         (bmalloc::Deallocator::deallocate):
2019         * bmalloc/EndTag.h: Added.
2020         (bmalloc::EndTag::operator=):
2021         * bmalloc/FixedVector.h: Added.
2022         (bmalloc::FixedVector::begin):
2023         (bmalloc::FixedVector::end):
2024         (bmalloc::FixedVector::size):
2025         (bmalloc::FixedVector::capacity):
2026         (bmalloc::FixedVector::clear):
2027         (bmalloc::FixedVector::isEmpty):
2028         (bmalloc::Capacity>::FixedVector):
2029         (bmalloc::Capacity>::operator):
2030         (bmalloc::Capacity>::push):
2031         (bmalloc::Capacity>::pop):
2032         (bmalloc::Capacity>::shrink):
2033         * bmalloc/Heap.cpp: Added.
2034         (bmalloc::sleep):
2035         (bmalloc::Heap::Heap):
2036         (bmalloc::Heap::concurrentScavenge):
2037         (bmalloc::Heap::scavengeSmallPages):
2038         (bmalloc::Heap::scavengeMediumPages):
2039         (bmalloc::Heap::scavengeLargeRanges):
2040         (bmalloc::Heap::allocateSmallLineSlowCase):
2041         (bmalloc::Heap::allocateMediumLineSlowCase):
2042         (bmalloc::Heap::allocateXLarge):
2043         (bmalloc::Heap::deallocateXLarge):
2044         (bmalloc::Heap::allocateLarge):
2045         (bmalloc::Heap::deallocateLarge):
2046         * bmalloc/Heap.h: Added.
2047         (bmalloc::Heap::deallocateSmallLine):
2048         (bmalloc::Heap::allocateSmallLine):
2049         (bmalloc::Heap::deallocateMediumLine):
2050         (bmalloc::Heap::allocateMediumLine):
2051         * bmalloc/Inline.h: Added.
2052         * bmalloc/LargeChunk.h: Added.
2053         (bmalloc::LargeChunk::begin):
2054         (bmalloc::LargeChunk::end):
2055         (bmalloc::LargeChunk::create):
2056         (bmalloc::LargeChunk::get):
2057         (bmalloc::LargeChunk::beginTag):
2058         (bmalloc::LargeChunk::endTag):
2059         * bmalloc/Line.h: Added.
2060         (bmalloc::Line<Traits>::begin):
2061         (bmalloc::Line<Traits>::end):
2062         (bmalloc::Line<Traits>::concurrentRef):
2063         (bmalloc::Line<Traits>::deref):
2064         * bmalloc/MediumAllocator.h: Added.
2065         (bmalloc::MediumAllocator::isNull):
2066         (bmalloc::MediumAllocator::MediumAllocator):
2067         (bmalloc::MediumAllocator::line):
2068         (bmalloc::MediumAllocator::allocate):
2069         (bmalloc::MediumAllocator::derefCount):
2070         (bmalloc::MediumAllocator::refill):
2071         * bmalloc/MediumChunk.h: Added.
2072         * bmalloc/MediumLine.h: Added.
2073         * bmalloc/MediumPage.h: Added.
2074         * bmalloc/MediumTraits.h: Added.
2075         * bmalloc/Mutex.cpp: Added.
2076         (bmalloc::Mutex::lockSlowCase):
2077         * bmalloc/Mutex.h: Added.
2078         (bmalloc::Mutex::Mutex):
2079         (bmalloc::Mutex::try_lock):
2080         (bmalloc::Mutex::lock):
2081         (bmalloc::Mutex::unlock):
2082         * bmalloc/ObjectType.cpp: Added.
2083         (bmalloc::objectType):
2084         * bmalloc/ObjectType.h: Added.
2085         (bmalloc::isSmallOrMedium):
2086         (bmalloc::isSmall):
2087         * bmalloc/Page.h: Added.
2088         (bmalloc::Page<Traits>::ref):
2089         (bmalloc::Page<Traits>::deref):
2090         (bmalloc::Page<Traits>::refCount):
2091         * bmalloc/PerProcess.h: Added.
2092         (bmalloc::PerProcess::mutex):
2093         (bmalloc::PerProcess<T>::getFastCase):
2094         (bmalloc::PerProcess<T>::get):
2095         (bmalloc::PerProcess<T>::getSlowCase):
2096         * bmalloc/PerThread.h: Added.
2097         (bmalloc::PerThreadStorage<Cache>::get):
2098         (bmalloc::PerThreadStorage<Cache>::init):
2099         (bmalloc::PerThreadStorage::get):
2100         (bmalloc::PerThreadStorage::init):
2101         (bmalloc::PerThread<T>::getFastCase):
2102         (bmalloc::PerThread<T>::get):
2103         (bmalloc::PerThread<T>::destructor):
2104         (bmalloc::PerThread<T>::getSlowCase):
2105         * bmalloc/Range.h: Added.
2106         (bmalloc::Range::Range):
2107         (bmalloc::Range::begin):
2108         (bmalloc::Range::end):
2109         (bmalloc::Range::size):
2110         (bmalloc::Range::operator!):
2111         (bmalloc::Range::operator<):
2112         * bmalloc/SegregatedFreeList.cpp: Added.
2113         (bmalloc::SegregatedFreeList::SegregatedFreeList):
2114         (bmalloc::SegregatedFreeList::insert):
2115         (bmalloc::SegregatedFreeList::takeGreedy):
2116         (bmalloc::SegregatedFreeList::take):
2117         * bmalloc/SegregatedFreeList.h: Added.
2118         * bmalloc/Sizes.h: Added.
2119         * bmalloc/SmallAllocator.h: Added.
2120         (bmalloc::SmallAllocator::isNull):
2121         (bmalloc::SmallAllocator::canAllocate):
2122         (bmalloc::SmallAllocator::SmallAllocator):
2123         (bmalloc::SmallAllocator::line):
2124         (bmalloc::SmallAllocator::allocate):
2125         (bmalloc::SmallAllocator::objectCount):
2126         (bmalloc::SmallAllocator::derefCount):
2127         (bmalloc::SmallAllocator::refill):
2128         * bmalloc/SmallChunk.h: Added.
2129         * bmalloc/SmallLine.h: Added.
2130         * bmalloc/SmallPage.h: Added.
2131         * bmalloc/SmallTraits.h: Added.
2132         * bmalloc/Syscall.h: Added.
2133         * bmalloc/VMAllocate.h: Added.
2134         (bmalloc::vmSize):
2135         (bmalloc::vmValidate):
2136         (bmalloc::vmAllocate):
2137         (bmalloc::vmDeallocate):
2138         (bmalloc::vmDeallocatePhysicalPages):
2139         (bmalloc::vmAllocatePhysicalPages):
2140         (bmalloc::vmDeallocatePhysicalPagesSloppy):
2141         (bmalloc::vmAllocatePhysicalPagesSloppy):
2142         * bmalloc/VMHeap.cpp: Added.
2143         (bmalloc::VMHeap::VMHeap):
2144         (bmalloc::VMHeap::allocateSmallChunk):
2145         (bmalloc::VMHeap::allocateMediumChunk):
2146         (bmalloc::VMHeap::allocateLargeChunk):
2147         * bmalloc/VMHeap.h: Added.
2148         (bmalloc::VMHeap::allocateSmallPage):
2149         (bmalloc::VMHeap::allocateMediumPage):
2150         (bmalloc::VMHeap::allocateLargeRange):
2151         (bmalloc::VMHeap::deallocateSmallPage):
2152         (bmalloc::VMHeap::deallocateMediumPage):
2153         (bmalloc::VMHeap::deallocateLargeRange):
2154         * bmalloc/Vector.h: Added.
2155         (bmalloc::Vector::begin):
2156         (bmalloc::Vector::end):
2157         (bmalloc::Vector::size):
2158         (bmalloc::Vector::capacity):
2159         (bmalloc::Vector::last):
2160         (bmalloc::Vector::pop):
2161         (bmalloc::Vector<T>::Vector):
2162         (bmalloc::Vector<T>::~Vector):
2163         (bmalloc::Vector<T>::operator):
2164         (bmalloc::Vector<T>::push):
2165         (bmalloc::Vector<T>::pop):
2166         (bmalloc::Vector<T>::shrink):
2167         (bmalloc::Vector<T>::reallocateBuffer):
2168         (bmalloc::Vector<T>::shrinkCapacity):
2169         (bmalloc::Vector<T>::growCapacity):
2170         * bmalloc/XLargeChunk.h: Added.
2171         (bmalloc::XLargeChunk::get):
2172         (bmalloc::XLargeChunk::begin):
2173         (bmalloc::XLargeChunk::XLargeChunk):
2174         (bmalloc::XLargeChunk::create):
2175         (bmalloc::XLargeChunk::destroy):
2176         (bmalloc::XLargeChunk::range):
2177         (bmalloc::XLargeChunk::size):
2178         * bmalloc/bmalloc.h: Added.
2179         (bmalloc::api::malloc):
2180         (bmalloc::api::free):
2181         (bmalloc::api::realloc):
2182         * bmalloc/mbmalloc.cpp: Added.
2183