bmalloc should support system memory analysis tools (part 2)
[WebKit-https.git] / Source / bmalloc / ChangeLog
1 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
2
3         bmalloc should support system memory analysis tools (part 2)
4         https://bugs.webkit.org/show_bug.cgi?id=139565
5
6         Reviewed by Mark Lam.
7
8         This patch actually queries the environment to see if memory analysis
9         tools have been enabled.
10
11         * bmalloc/Deallocator.cpp:
12         (bmalloc::Deallocator::scavenge): Don't process the object log if
13         we've disabled bmalloc because it will be full of invalid nullptrs.
14
15         * bmalloc/Environment.cpp:
16         (bmalloc::isMallocEnvironmentVariableSet): Test for the list of known
17         Malloc debugging flags. I also added a plain "Malloc" catch-all for
18         when you want to disable bmalloc without enabling any kind of funny
19         business.
20
21         It would be slightly nicer just to iterate the list of environment
22         variables and strstr them, but getenv is the more portable option,
23         and performance here doesn't really matter.
24
25         (bmalloc::isLibgmallocEnabled): Test for the libgmalloc insertion
26         environment variable.
27
28         (bmalloc::Environment::computeIsBmallocEnabled):
29
30 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
31
32         Try to fix the iOS simulator build.
33
34         #include the declaration of malloc / free.
35
36         * bmalloc/Allocator.cpp:
37         * bmalloc/Deallocator.cpp:
38
39 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
40
41         Try to fix the build.
42
43         * bmalloc.xcodeproj/project.pbxproj: Marked a header exported.
44
45 2014-12-11  Geoffrey Garen  <ggaren@apple.com>
46
47         bmalloc should support system memory analysis tools (part 1)
48         https://bugs.webkit.org/show_bug.cgi?id=139559
49
50         Reviewed by Mark Lam.
51
52         This patch adds the hooks to disable bmalloc at runtime if certain
53         environment variables are set, but doesn't actually read from the
54         environment yet.
55
56         No performance change.
57
58         * bmalloc.xcodeproj/project.pbxproj: Added the Environment class, which
59         we'll use to read environment variables and see if memory analysis tools
60         have been enabled.
61
62         * bmalloc/Allocator.cpp:
63         (bmalloc::Allocator::Allocator):
64         (bmalloc::Allocator::allocateSlowCase): Added a hook to disable bmalloc
65         on the allocation path. We cache the setting to make the check fast.
66
67         * bmalloc/Allocator.h: Interface changes.
68
69         * bmalloc/Cache.cpp:
70         (bmalloc::Cache::Cache): Pass a heap pointer through to our allocator
71         and deallocator. This main purpose is to enable them to query the
72         environment for whether bmalloc is enabled; but this is also a slightly
73         cleaner way to guarantee to them that the Heap has been pre-initialized.
74
75         * bmalloc/Deallocator.cpp:
76         (bmalloc::Deallocator::Deallocator): If bmalloc is disable, artificially
77         fill the object log to force us to take the slow path on all deallocations.
78
79         (bmalloc::Deallocator::deallocateSlowCase): Do the disabled check.
80
81         * bmalloc/Deallocator.h: Interface changes.
82
83         * bmalloc/Environment.cpp: Added.
84         (bmalloc::Environment::Environment):
85         (bmalloc::Environment::computeIsBmallocEnabled):
86         * bmalloc/Environment.h: Added.
87         (bmalloc::Environment::isBmallocEnabled): This is the class that will
88         encapsulate looking for environment variables that turn on heap
89         analysis tools.
90
91         * bmalloc/Heap.h:
92         (bmalloc::Heap::environment):
93
94         * bmalloc/Mutex.h:
95         (bmalloc::Mutex::Mutex):
96         * bmalloc/StaticMutex.h: A little refactoring to clarify these comments,
97         since I got super confused about them while writing this patch.
98
99         * bmalloc/VMHeap.cpp: Fixed an #include.
100
101 2014-12-09  David Kilzer  <ddkilzer@apple.com>
102
103         Switch from using PLATFORM_NAME to SDK selectors in ANGLE, bmalloc, gtest, JavaScriptCore, WTF
104         <http://webkit.org/b/139212>
105
106         Reviewed by Joseph Pecoraro.
107
108         * Configurations/Base.xcconfig:
109         - Only set GCC_ENABLE_OBJC_GC, GCC_MODEL_TUNING and TOOLCHAINS
110           on OS X.
111         * Configurations/DebugRelease.xcconfig:
112         - Only set MACOSX_DEPLOYMENT_TARGET and SDKROOT on OS X.
113
114 2014-11-07  Geoffrey Garen  <ggaren@apple.com>
115
116         bmalloc uses 8X more virtual memory than necessary
117         https://bugs.webkit.org/show_bug.cgi?id=138495
118
119         Reviewed by Mark Lam.
120
121         iOS has a per-process virtual memory cap around 1GB, so there's some
122         value to not going totally ham with virtual memory.
123
124         We currently use about 8X the necessary amount:
125             - 2X to align our VM allocation
126             - 4X to reserve small / medium / (2) large chunk VM ranges per superchunk
127
128         We can cut that down:
129             - Return the unaligned portion of our VM allocation (-2X)
130             - Use all the chunks in a superchunk, instead of allocating one
131               chunk per superchunk (-4X)
132
133         * bmalloc/Algorithm.h:
134         (bmalloc::roundUpToMultipleOf): Added a non-constant version of this
135         function so we can call it with getpagesize() at runtime.
136
137         * bmalloc/Chunk.h:
138         * bmalloc/LargeChunk.h:
139         (bmalloc::LargeChunk::create): Deleted. Instead of each chunk allocating
140         its own VM, VMHeap allocates the superchunk and all the chunks in it at a time.
141
142         * bmalloc/VMAllocate.h:
143         (bmalloc::vmValidate):
144         (bmalloc::vmAllocate): ASSERT that mmap succeeds to make crashes clearer
145         if it does not succeed. Allocate precisely, and give back the extra.
146
147         * bmalloc/VMHeap.cpp:
148         (bmalloc::VMHeap::allocateSuperChunk):
149         (bmalloc::VMHeap::allocateSmallChunk): Deleted.
150         (bmalloc::VMHeap::allocateMediumChunk): Deleted.
151         (bmalloc::VMHeap::allocateLargeChunk): Deleted. Use all the chunks
152         in a superchunk, instead of just one.
153
154         * bmalloc/VMHeap.h:
155         (bmalloc::VMHeap::allocateSmallPage):
156         (bmalloc::VMHeap::allocateMediumPage):
157         (bmalloc::VMHeap::allocateLargeRange):
158         * bmalloc/XLargeChunk.h:
159         (bmalloc::XLargeChunk::create): Updated to match changes above.
160
161 2014-11-01  David Kilzer  <ddkilzer@apple.com>
162
163         JavaScriptCore is missing debug info for bmalloc because libbmalloc.a is stripped
164         <https://webkit.org/b/138286>
165         <rdar://problem/18847087>
166
167         Reviewed by Dan Bernstein.
168
169         * Configurations/bmalloc.xcconfig: Set STRIP_INSTALLED_PRODUCT
170         to NO for the target that produces libbmalloc.a so that the
171         debug symbols will be linked into JavaScriptCore and end up in
172         its dSYM file.
173
174 2014-10-30  Dana Burkart  <dburkart@apple.com>
175
176         <rdar://problem/18821260> Prepare for the mysterious future
177
178         Reviewed by Lucas Forschler.
179
180         * Configurations/Base.xcconfig:
181         * Configurations/DebugRelease.xcconfig:
182
183 2014-09-24  Geoffrey Garen  <ggaren@apple.com>
184
185         bmalloc: cleaned up fast path vs slow path
186         https://bugs.webkit.org/show_bug.cgi?id=137081
187
188         Reviewed by Sam Weinig.
189
190         Might be a 1% speedup on MallocBench. Also cleans up the code a bit.
191
192         * bmalloc/Allocator.cpp:
193         (bmalloc::Allocator::Allocator): Merged the small and medium range
194         caches, just like the small and medium allocators. Ranges are abstract
195         objects that don't really care whether they hold small or medium objects,
196         so they don't need to be segregated.
197
198         (bmalloc::Allocator::scavenge): Ditto.
199
200         (bmalloc::Allocator::allocateBumpRangeSlowCase):
201         (bmalloc::Allocator::allocateBumpRange): Same thing here, except that
202         we do care a tiny bit, because we need to specify small vs medium when
203         allocating new ranges from the heap, to ensure that the heap allocates
204         from the right segment of VM.
205
206         (bmalloc::Allocator::allocateLarge):
207         (bmalloc::Allocator::allocateXLarge): NO_INLINE because this was clouding
208         up the fast path. Large allocation performance is dominated by allocation
209         logic and initialization, so inlining it doesn't help.
210
211         (bmalloc::Allocator::allocateSlowCase): Slow path got a bit cleaner since
212         it doesn't need to distinguish small vs medium objects.
213
214         (bmalloc::Allocator::allocateSmallBumpRange): Deleted.
215         (bmalloc::Allocator::allocateMediumBumpRange): Deleted.
216
217         * bmalloc/Allocator.h:
218         * bmalloc/BumpRange.h:
219
220         * bmalloc/Cache.cpp:
221         (bmalloc::Cache::allocateSlowCase): Deleted.
222         (bmalloc::Cache::deallocateSlowCase): Deleted.
223         * bmalloc/Cache.h:
224         (bmalloc::Cache::allocate):
225         (bmalloc::Cache::deallocate):
226         (bmalloc::Cache::allocateFastCase): Deleted.
227         (bmalloc::Cache::deallocateFastCase): Deleted. Removed the Cache slow
228         paths. The downside to this change is that the fast path branches to two
229         distinct failure cases instead of one. The upside is that the slow path
230         doesn't need to re-read the segment register, which is not as cheap as a
231         normal register, and it doesn't need to do an extra level of function 
232         call. Seems to be worth it.
233
234         * bmalloc/Deallocator.h:
235         * bmalloc/Heap.cpp:
236         (bmalloc::Heap::refillSmallBumpRangeCache):
237         (bmalloc::Heap::refillMediumBumpRangeCache):
238         * bmalloc/Heap.h: Updated for interface changes.
239
240         * bmalloc/Sizes.h: The most ranges a cache will hold is the number of
241         small lines in a page / 2, since any other free lines will coalesce
242         with their neighbors.
243
244 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
245
246         Rolled out r173346.
247
248             bmalloc should honor the FastMalloc statistics API
249             https://bugs.webkit.org/show_bug.cgi?id=136592
250
251         This didn't really work. Because we allow ranges with and without
252         physical pages to merge, and we allow double-committing and
253         double-decommitting, we can't rely on commit actions to track memory
254         footprint.
255
256         * bmalloc/Heap.cpp:
257         (bmalloc::Heap::size): Deleted.
258         (bmalloc::Heap::capacity): Deleted.
259         * bmalloc/Heap.h:
260         * bmalloc/VMHeap.cpp:
261         (bmalloc::VMHeap::VMHeap):
262         (bmalloc::VMHeap::allocateSmallChunk):
263         (bmalloc::VMHeap::allocateMediumChunk):
264         (bmalloc::VMHeap::allocateLargeChunk):
265         * bmalloc/VMHeap.h:
266         (bmalloc::VMHeap::allocateSmallPage):
267         (bmalloc::VMHeap::allocateMediumPage):
268         (bmalloc::VMHeap::allocateLargeRange):
269         (bmalloc::VMHeap::deallocateSmallPage):
270         (bmalloc::VMHeap::deallocateMediumPage):
271         (bmalloc::VMHeap::deallocateLargeRange):
272         (bmalloc::VMHeap::size): Deleted.
273         (bmalloc::VMHeap::capacity): Deleted.
274         * bmalloc/bmalloc.h:
275         (bmalloc::api::heapSize): Deleted.
276         (bmalloc::api::heapCapacity): Deleted.
277
278 2014-09-23  Geoffrey Garen  <ggaren@apple.com>
279
280         bmalloc: Allocation should be more precise
281         https://bugs.webkit.org/show_bug.cgi?id=136993
282
283         Reviewed by Gavin Barraclough.
284
285         13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
286
287         This patch teaches the allocator to merge adjacent free lines into a
288         single allocatable range. This allows us to shrink the size of an
289         individual line without increasing fragmentation or the rate of allocator
290         slow paths.
291
292         We'll only take more slow paths when available memory is sparse, which
293         is exactly when it's worth it. When available memory is dense, we'll
294         take fewer slow paths.
295
296         * bmalloc.xcodeproj/project.pbxproj:
297         * bmalloc/Algorithm.h:
298         (bmalloc::divideRoundingUp):
299
300         * bmalloc/Allocator.cpp:
301         (bmalloc::Allocator::Allocator): Updated for interface changes.
302
303         (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
304         Now that we merge lines, it's not convenient to scavenge by line.
305
306         (bmalloc::Allocator::allocateSmallBumpRange):
307         (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
308         instead of individual lines.
309
310         (bmalloc::Allocator::allocateSlowCase):
311         (bmalloc::Allocator::allocateSmallLine): Deleted.
312         (bmalloc::Allocator::allocateMediumLine): Deleted.
313         (bmalloc::Allocator::allocateMedium): Deleted.
314         * bmalloc/Allocator.h:
315         (bmalloc::Allocator::allocateFastCase): Folded medium allocations
316         into the standard fast path with small allocations. Since a BumpAllocator
317         just allocates out of an arbitrary range, it doesn't need to distinguish
318         between small and medium lines.
319
320         * bmalloc/BumpAllocator.h:
321         (bmalloc::BumpAllocator::size):
322         (bmalloc::BumpAllocator::BumpAllocator):
323         (bmalloc::BumpAllocator::init):
324         (bmalloc::BumpAllocator::refill):
325         (bmalloc::BumpAllocator::line): Deleted. No need to track line information
326         anymore: the heap just gives us a pointer and a pre-computed number of
327         objects, and we allocate them.
328
329         * bmalloc/Deallocator.cpp:
330         (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
331
332         * bmalloc/Heap.cpp:
333         (bmalloc::Heap::Heap):
334         (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
335         detailing where all objects will lie in memory. After we merge two lines,
336         we might allocate an object that spans from one line to the next. This
337         metadata details which bits of memory overlap in that way, and how they
338         overlap.
339
340         (bmalloc::Heap::refillSmallBumpRangeCache):
341         (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
342         and merge adjacent free lines into BumpRanges.
343
344         (bmalloc::Heap::allocateSmallPage):
345         (bmalloc::Heap::allocateMediumPage):
346         (bmalloc::Heap::deallocateSmallLine):
347         (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
348         since we scan for free memory a page at a time.
349
350         (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
351         (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
352         fast path.
353
354         * bmalloc/Heap.h:
355         (bmalloc::Heap::derefSmallLine):
356         (bmalloc::Heap::derefMediumLine):
357         (bmalloc::Heap::deallocateSmallLine): Deleted.
358         (bmalloc::Heap::allocateSmallLine): Deleted.
359         (bmalloc::Heap::deallocateMediumLine): Deleted.
360         (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
361
362         * bmalloc/Line.h:
363         (bmalloc::Line<Traits>::ref):
364         (bmalloc::Line<Traits>::deref):
365         (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
366         anymore, since we only ever deref by 1 now.
367
368         * bmalloc/MediumAllocator.h:
369         (bmalloc::MediumAllocator::isNull): Deleted.
370         (bmalloc::MediumAllocator::MediumAllocator): Deleted.
371         (bmalloc::MediumAllocator::line): Deleted.
372         (bmalloc::MediumAllocator::allocate): Deleted.
373         (bmalloc::MediumAllocator::derefCount): Deleted.
374         (bmalloc::MediumAllocator::refill): Deleted.
375         (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
376         been dead for a while, since it doesn't build anymore with this patch.
377
378         * bmalloc/Page.h:
379         (bmalloc::Page::sizeClass):
380         (bmalloc::Page::setSizeClass):
381         (bmalloc::Page::smallSizeClass): Deleted.
382         (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
383         to sizeClass, since we use it for medium sizes too.
384
385         * bmalloc/Sizes.h:
386         (bmalloc::Sizes::sizeClass):
387         (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
388
389         (bmalloc::Sizes::smallSizeClassFor): Deleted.
390         (bmalloc::Sizes::mediumSizeClassFor): Deleted.
391
392         * bmalloc/bmalloc.h:
393         (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
394         can be a bit more precise. It also has to be, since we can't guarantee
395         that an object ends at the end of a line anymore.
396
397 2014-09-19  Daniel Bates  <dabates@apple.com>
398
399         Always assume internal SDK when building configuration Production
400         https://bugs.webkit.org/show_bug.cgi?id=136925
401         <rdar://problem/18362399>
402
403         Reviewed by Dan Bernstein.
404
405         * Configurations/Base.xcconfig:
406
407 2014-09-16  Geoffrey Garen  <ggaren@apple.com>
408
409         bmalloc: moved line caches from the deallocator to the allocator
410         https://bugs.webkit.org/show_bug.cgi?id=136868
411
412         Reviewed by Gavin Barraclough.
413
414         I did this mostly as a simplification, to make it easier to change the
415         allocation strategy.
416
417         No throughput change on MallocBench. Saves about 50kB.
418
419         Since the deallocator needs to lock the heap when freeing lines anyway,
420         there isn't much benefit to giving the deallocator a local cache of
421         deallocated lines.
422
423         We still give the allocator a local cache of lines because that does
424         reduce the frequency at which it needs to lock the heap in order to
425         acquire more lines.
426
427         * bmalloc/Allocator.cpp:
428         (bmalloc::Allocator::scavenge):
429         (bmalloc::Allocator::allocateSmallLine):
430         (bmalloc::Allocator::allocateMediumLine):
431         (bmalloc::Allocator::allocateMedium):
432         (bmalloc::Allocator::allocateSlowCase):
433         * bmalloc/Allocator.h:
434         * bmalloc/Deallocator.cpp:
435         (bmalloc::Deallocator::Deallocator):
436         (bmalloc::Deallocator::scavenge):
437         (bmalloc::Deallocator::processObjectLog):
438         (bmalloc::Deallocator::deallocateSmallLine): Deleted.
439         (bmalloc::Deallocator::allocateSmallLine): Deleted.
440         (bmalloc::Deallocator::deallocateMediumLine): Deleted.
441         (bmalloc::Deallocator::allocateMediumLine): Deleted.
442         * bmalloc/Deallocator.h:
443
444         * bmalloc/Sizes.h:
445         * bmalloc/VMAllocate.h: Took the opportunity to make the line cache size
446         exactly one page in size. That's about what we were shooting for anyway,
447         and it may make it easier to switch to per-page allocation in future.
448
449 2014-09-15  Geoffrey Garen  <ggaren@apple.com>
450
451         bmalloc: allocate small and medium objects using the same bump pointer class
452         https://bugs.webkit.org/show_bug.cgi?id=136843
453
454         Reviewed by Gavin Barraclough.
455
456         4% speedup on MallocBench.
457
458         Now that medium-sized objects have dedicated per-size allocators, they
459         don't need to use an arbitrary bump pointer allocator. This means that
460         every allocator knows how many objects it will allocate from the start,
461         and we don't need a post-processing step to adjust refcounts based on
462         real allocation count.
463
464         * bmalloc.xcodeproj/project.pbxproj: Renamed SmallAllocator to BumpAllocator
465         since it's used for small and medium objects now.
466
467         * bmalloc/Allocator.cpp:
468         (bmalloc::Allocator::Allocator): Updated to use new interface.
469         (bmalloc::Allocator::scavenge): To "retire" an allocator, we just need
470         to make sure that we finish allocating all the objects in it.
471
472         (bmalloc::Allocator::allocateMedium):
473         (bmalloc::Allocator::allocateSlowCase):
474         (bmalloc::Allocator::retire): Deleted.
475         (bmalloc::Allocator::processSmallAllocatorLog): Deleted.
476         (bmalloc::Allocator::processMediumAllocatorLog): Deleted.
477         * bmalloc/Allocator.h:
478         (bmalloc::Allocator::allocateFastCase): Removed abstractions and data
479         used to post-process an allocator based on how many objects it allocated.
480
481         * bmalloc/BumpAllocator.h: Copied from Source/bmalloc/bmalloc/SmallAllocator.h.
482         (bmalloc::BumpAllocator::BumpAllocator):
483         (bmalloc::BumpAllocator::init):
484         (bmalloc::BumpAllocator::line):
485         (bmalloc::BumpAllocator::validate):
486         (bmalloc::BumpAllocator::allocate):
487         (bmalloc::BumpAllocator::refill):
488         (bmalloc::BumpAllocator::clear): Updated these functions to be agnostic
489         about the kinds of lines they allocate into. In some cases, the line
490         type must be provided as a template parameter by the caller.
491
492         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
493         (bmalloc::SmallAllocator::line): Deleted.
494         (bmalloc::SmallAllocator::allocate): Deleted.
495         (bmalloc::SmallAllocator::objectCount): Deleted.
496         (bmalloc::SmallAllocator::derefCount): Deleted.
497         (bmalloc::SmallAllocator::refill): Deleted.
498         (bmalloc::SmallAllocator::clear): Deleted.
499
500         * bmalloc/ObjectType.h:
501         (bmalloc::isMedium):
502
503         * bmalloc/SmallAllocator.h:
504         (bmalloc::SmallAllocator::isNull): Deleted.
505         (bmalloc::SmallAllocator::canAllocate): Deleted.
506         (bmalloc::SmallAllocator::SmallAllocator): Deleted.
507         (bmalloc::SmallAllocator::line): Deleted.
508         (bmalloc::SmallAllocator::allocate): Deleted.
509         (bmalloc::SmallAllocator::objectCount): Deleted.
510         (bmalloc::SmallAllocator::derefCount): Deleted.
511         (bmalloc::SmallAllocator::refill): Deleted.
512         (bmalloc::SmallAllocator::clear): Deleted.
513
514 2014-09-12  Geoffrey Garen  <ggaren@apple.com>
515
516         Fixed a goof in bmalloc Vector sizing
517         https://bugs.webkit.org/show_bug.cgi?id=136795
518
519         Reviewed by Gavin Barraclough and Sam Weinig.
520
521         We want our minimum vector to be page-sized since the OS will give us
522         a page no matter what -- but we want that many bytes, and not enough
523         bytes to store that many elements.
524
525         * bmalloc/Vector.h: Math is hard.
526
527 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
528
529         bmalloc should segregate medium-sized objects by line like it does for small-sized objects
530         https://bugs.webkit.org/show_bug.cgi?id=136693
531
532         Reviewed by Gavin Barraclough.
533
534         4% reduction in heap size on the MallocBench *_memory_warning benchmarks.
535
536         No throughput change.
537
538         We keep an array of medium allocators, just like our array of small
539         allocators.
540
541         In future, we can simplify the allocation fast path by merging the small
542         and medium allocator arrays. For now, this is the simplest change that
543         gets the win.
544
545         * bmalloc/Allocator.cpp:
546         (bmalloc::Allocator::Allocator):
547         (bmalloc::Allocator::scavenge):
548         (bmalloc::Allocator::allocateMedium):
549         * bmalloc/Allocator.h:
550         * bmalloc/Sizes.h:
551         (bmalloc::Sizes::mediumSizeClassFor):
552
553 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
554
555         Reviewed by Sam Weinig.
556
557         Renamed log => retire for clarity.
558
559         * bmalloc/Allocator.cpp:
560         (bmalloc::Allocator::scavenge):
561         (bmalloc::Allocator::retire):
562         (bmalloc::Allocator::allocateMedium):
563         (bmalloc::Allocator::allocateSlowCase):
564         (bmalloc::Allocator::log): Deleted.
565         * bmalloc/Allocator.h:
566
567 2014-09-11  Geoffrey Garen  <ggaren@apple.com>
568
569         bmalloc: eager scavenge leaves behind a bogus allocator
570         https://bugs.webkit.org/show_bug.cgi?id=136743
571
572         Reviewed by Sam Weinig.
573
574         Be sure to clear the allocator after logging it in the eager scavenge
575         case, so that we don't later try to allocate out of the lines that we
576         have thrown away.
577
578         We didn't need to do this previously because scavenge would only happen
579         at thread exit time, after which no further allocation from the per-thread
580         cache would take place.
581
582         * bmalloc/Allocator.cpp:
583         (bmalloc::Allocator::scavenge):
584         * bmalloc/MediumAllocator.h:
585         (bmalloc::MediumAllocator::clear):
586         * bmalloc/SmallAllocator.h:
587         (bmalloc::SmallAllocator::clear):
588
589 2014-09-05  Geoffrey Garen  <ggaren@apple.com>
590
591         bmalloc should honor the FastMalloc statistics API
592         https://bugs.webkit.org/show_bug.cgi?id=136592
593
594         Reviewed by Gavin Barraclough.
595
596         We do this by tracking "size" and "capacity" in the VM heap.
597
598         The VM heap's "capacity" is all the VM we ever allocated.
599
600         The VM heap's "size" the subset of VM currently held onto by the
601         VM heap (and therefore not in use by the regular heap).
602
603         Somewhat ironically, reducing the process's memory footprint, increases
604         the size of the VM heap, since the VM heap holds the pages that are
605         purely virtual and not physical.
606
607         * bmalloc/Heap.cpp:
608         (bmalloc::Heap::size):
609         (bmalloc::Heap::capacity):
610         * bmalloc/Heap.h:
611         * bmalloc/VMHeap.cpp:
612         (bmalloc::VMHeap::VMHeap):
613         (bmalloc::VMHeap::allocateSmallChunk):
614         (bmalloc::VMHeap::allocateMediumChunk):
615         (bmalloc::VMHeap::allocateLargeChunk):
616         * bmalloc/VMHeap.h:
617         (bmalloc::VMHeap::size):
618         (bmalloc::VMHeap::capacity):
619         (bmalloc::VMHeap::allocateSmallPage):
620         (bmalloc::VMHeap::allocateMediumPage):
621         (bmalloc::VMHeap::allocateLargeRange):
622         (bmalloc::VMHeap::deallocateSmallPage):
623         (bmalloc::VMHeap::deallocateMediumPage):
624         (bmalloc::VMHeap::deallocateLargeRange):
625         * bmalloc/bmalloc.h:
626         (bmalloc::api::heapSize):
627         (bmalloc::api::heapCapacity):
628
629 2014-09-02  Geoffrey Garen  <ggaren@apple.com>
630
631         bmalloc crashes on the EWS bots (due to bad large object allocation)
632         https://bugs.webkit.org/show_bug.cgi?id=136469
633
634         Reviewed by Andreas Kling.
635
636         It's possible to convince bmalloc to perform a bad large object allocation,
637         through these steps:
638
639         (1) Insert object A into freelist F0.
640
641         (2) Split, merge and split again A's neighbors such that object B is
642         inserted into freelist F0, with boundary tag and size equal to object A,
643         but pointer not completely equal to object A. Put object B at the head of F0.
644
645         (3) Allocate some other object from F0, swapping its position in the
646         freelist with object B, such that object A is now ahead of object B.
647
648         --> Now, the next allocation for size A/B will allocate object A, which
649         has a slightly wrong idea about where the object actually begins.
650         Immediately, you'll corrupt a little memory, and over time, you'll also
651         corrupt boundary tag metadata.
652
653         The solution is to store the begin pointer in the boundary tag. Luckily,
654         this doesn't make the tag any bigger, and it's not a noticeable slowdown
655         on MallocBench.
656
657         * bmalloc/Algorithm.h:
658         (bmalloc::rightShift):
659         * bmalloc/BeginTag.h:
660         (bmalloc::BeginTag::isInFreeList): This is the bug fix. Make sure to
661         validate the start pointer when popping off the free list. Through a
662         very uncommon set of steps, it is possible to have an item in the free
663         list that is valid by all accounts except for its start pointer.
664
665         * bmalloc/BoundaryTag.h:
666         (bmalloc::BoundaryTag::compactBegin):
667         (bmalloc::BoundaryTag::setRange):
668         (bmalloc::BoundaryTag::setSize): Deleted. Record a compact version of the
669         start pointer. We don't need the whole pointer -- just the offset, in
670         largeAlignment increments, into the relevant boundary tag bucket.
671
672         * bmalloc/BoundaryTagInlines.h:
673         (bmalloc::validateNext):
674         (bmalloc::BoundaryTag::init):
675         (bmalloc::BoundaryTag::mergeLarge):
676         (bmalloc::BoundaryTag::splitLarge):
677         * bmalloc/SegregatedFreeList.cpp:
678         (bmalloc::SegregatedFreeList::insert):
679         (bmalloc::SegregatedFreeList::takeGreedy):
680         (bmalloc::SegregatedFreeList::take): Provide the whole range instead of
681         the size when establishing a boundary tag, as required by the new
682         interface.
683
684         * bmalloc/Sizes.h:
685
686 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
687
688         Fixed a bmalloc crash seen on the EWS bot
689         https://bugs.webkit.org/show_bug.cgi?id=135955
690
691         Reviewed by Andreas Kling.
692
693         * bmalloc/Syscall.h: Some CG APIs vm_copy their input buffers. If the
694         input buffer is a malloc region, that region will get marked Copy-On-Write
695         by the kernel. Calls to madvise() for COW regions fail and return EINVAL
696         on older OS X's. In 10.10, they still fail, but they do not return
697         EINVAL.
698
699         So, we can only ASSERT that our syscalls succeed starting with 10.10.
700
701 2014-08-14  Geoffrey Garen  <ggaren@apple.com>
702
703         Fixed the bmalloc build
704         https://bugs.webkit.org/show_bug.cgi?id=135953
705
706         Reviewed by Andreas Kling.
707
708         * bmalloc.xcodeproj/project.pbxproj: Marked a few headers as private.
709         These headers are used, so they must be available outside the project.
710
711 2014-08-13  Daniel Bates  <dabates@apple.com>
712
713         Attempt to fix the build following <http://trac.webkit.org/changeset/172576>
714         (https://bugs.webkit.org/show_bug.cgi?id=135895)
715
716         Substitute PerThreadStorage<T>::initSharedKeyIfNeeded() for initSharedKeyIfNeeded() in
717         implementation of PerThread<T>::getFastCase().
718
719         * bmalloc/PerThread.h:
720         (bmalloc::PerThread<T>::getFastCase):
721
722 2014-08-13  Daniel Bates  <dabates@apple.com>
723
724         Make bmalloc::PerThread work without C++ thread local storage
725         https://bugs.webkit.org/show_bug.cgi?id=135895
726
727         Reviewed by Geoffrey Garen.
728
729         Implement support for building bmalloc without C++ thread local storage.
730
731         * bmalloc/BPlatform.h: Remove macro define BPLATFORM_IOS_SIMULATOR. Added macro function
732         BCOMPILER_SUPPORTS() and macro define BCOMPILER_SUPPORTS_CXX_THREAD_LOCAL that can be used
733         to determine whether the compiler supports C++ thread local storage.
734         * bmalloc/PerThread.h:
735         (bmalloc::PerThreadStorage::get): Modified to call pthread_getspecific() when building
736         without C++ thread local storage.
737         (bmalloc::PerThreadStorage::initSharedKeyIfNeeded): Added.
738         (bmalloc::PerThreadStorage::init): Moved logic to initialize shared Pthread key from here to
739         PerThreadStorage::initSharedKeyIfNeeded().
740         (bmalloc::PerThread<T>::getFastCase): Modified to call PerThreadStorage::initSharedKeyIfNeeded()
741         before querying PerThreadStorage::get() when building without C++ thread local storage so as to
742         ensure that the shared key has been initialized.
743         (_pthread_setspecific_direct): Deleted.
744         (_pthread_getspecific_direct): Deleted.
745
746 2014-08-13  Daniel Bates  <dabates@apple.com>
747
748         [iOS] Make JavaScriptCore and bmalloc build with the public SDK
749         https://bugs.webkit.org/show_bug.cgi?id=135848
750
751         Reviewed by Geoffrey Garen.
752
753         * bmalloc/BPlatform.h: Added macro BPLATFORM_IOS_SIMULATOR, which evaluates to true
754         when building for the iOS Simulator.
755         * bmalloc/PerThread.h: Use pthread_machdep.h code path when building for iOS Simulator
756         using the public SDK.
757         (_pthread_setspecific_direct): Added; only defined when building for the iOS Simulator
758         using the public SDK.
759         (_pthread_getspecific_direct): Added; only defined when building for the iOS Simulator
760         using the public SDK.
761
762 2014-08-12  Daniel Bates  <dabates@apple.com>
763
764         BPLATFORM(IOS) always evaluates to false
765         https://bugs.webkit.org/show_bug.cgi?id=135843
766
767         Reviewed by Geoffrey Garen.
768
769         Fix typo in definition of BPLATFORM() and include system header TargetConditionals.h
770         (when building on an Apple platform) so that BPLATFORM(X) evaluates to true when
771         building for platform X. In particular, so that BPLATFORM(IOS) evaluates to true when
772         building for iOS.
773
774         As a side effect of this change, the change made in <http://trac.webkit.org/changeset/167289>
775         will be honored and iOS will assume a VM page size of 16kB (again) instead of 4kB.
776
777         * bmalloc/BPlatform.h:
778
779 2014-08-11  Andy Estes  <aestes@apple.com>
780
781         [iOS] Get rid of iOS.xcconfig
782         https://bugs.webkit.org/show_bug.cgi?id=135809
783
784         Reviewed by Joseph Pecoraro.
785
786         All iOS.xcconfig did was include AspenFamily.xcconfig, so there's no need for the indirection.
787
788         * Configurations/Base.xcconfig:
789         * Configurations/iOS.xcconfig: Removed.
790         * bmalloc.xcodeproj/project.pbxproj:
791
792 2014-05-01  Dan Bernstein  <mitz@apple.com>
793
794         Fixed production builds for the iOS Simulator.
795         <rdar://problem/16792221>
796
797         * Configurations/bmalloc.xcconfig: Include INSTALL_PATH_PREFIX in
798         PRIVATE_HEADERS_FOLDER_PATH when installing.
799
800 2014-04-20  Geoffrey Garen  <ggaren@apple.com>
801
802         bmalloc: Segregate pages by objects size
803         https://bugs.webkit.org/show_bug.cgi?id=131909
804
805         Reviewed by Andreas Kling.
806
807         2% reduction in memory-at-end on the Membuster memory_warning benchmarks.
808
809         * bmalloc/Allocator.cpp:
810         (bmalloc::Allocator::allocateSlowCase):
811         * bmalloc/Allocator.h:
812         (bmalloc::Allocator::allocateFastCase):
813         (bmalloc::Allocator::smallAllocatorFor): Use the new shared helper
814         function for size class calculation.
815
816         * bmalloc/Deallocator.cpp:
817         (bmalloc::Deallocator::Deallocator):
818         (bmalloc::Deallocator::scavenge):
819         (bmalloc::Deallocator::deallocateSmallLine):
820         (bmalloc::Deallocator::allocateSmallLine):
821         * bmalloc/Deallocator.h: Keep a cache for every size class, since the
822         cache can't be shared anymore.
823
824         * bmalloc/Heap.cpp:
825         (bmalloc::Heap::allocateSmallLineSlowCase):
826         * bmalloc/Heap.h:
827         (bmalloc::Heap::deallocateSmallLine): Ditto.
828
829         (bmalloc::Heap::allocateSmallLine): Check size class in addition to
830         page refcount when allocating a line because we might have deallocated
831         the page and the recycled it for another size class.
832
833         (bmalloc::Heap::deallocateMediumLine):
834         (bmalloc::Heap::allocateMediumLine):
835         * bmalloc/Line.h:
836         (bmalloc::Line::refCount):
837         * bmalloc/Page.h:
838         (bmalloc::Page::refCount):
839         (bmalloc::Page::smallSizeClass):
840         (bmalloc::Page::setSmallSizeClass):
841         (bmalloc::Page<Traits>::refCount): Deleted.
842         * bmalloc/Sizes.h:
843         (bmalloc::Sizes::smallSizeClassFor): New shared API for computing
844         an index into an array from a size.
845
846 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
847
848         bmalloc: Improved alignment in LargeChunk
849         https://bugs.webkit.org/show_bug.cgi?id=131895
850
851         Reviewed by Andreas Kling.
852
853         * bmalloc/Chunk.h:
854         * bmalloc/LargeChunk.h: Align to vmPageSize just like Chunk does.
855         Technically, the previous alignment was harmless, but I would prefer,
856         dear reader, not to have to explain the interlocking set of
857         circumstances that made it so.
858
859 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
860
861         Rolled out r167502 because it caused a crash on the facebook benchmark.
862
863         Unreviewed.
864
865             bmalloc: Added an XSmall line size
866             https://bugs.webkit.org/show_bug.cgi?id=131851
867
868             Reviewed by Sam Weinig.
869
870 2014-04-19  Geoffrey Garen  <ggaren@apple.com>
871
872         bmalloc: Mutex should be harder to use wrong
873         https://bugs.webkit.org/show_bug.cgi?id=131879
874
875         Reviewed by Andreas Kling.
876
877         Mutex now has a proper constructor, so you can't deadlock by forgetting
878         to initialize it.
879
880         * bmalloc.xcodeproj/project.pbxproj:
881         * bmalloc/Allocator.cpp:
882         (bmalloc::Allocator::processXSmallAllocatorLog):
883         (bmalloc::Allocator::processSmallAllocatorLog):
884         (bmalloc::Allocator::processMediumAllocatorLog):
885         (bmalloc::Allocator::allocateLarge):
886         (bmalloc::Allocator::allocateXLarge): Global replace Mutex => StaticMutex,
887         since the Heap mutex is a static.
888
889         * bmalloc/AsyncTask.h:
890         (bmalloc::Function>::AsyncTask): Use Mutex, since we're not static. No
891         need for explicit initialization anymore.
892
893         * bmalloc/Deallocator.cpp:
894         (bmalloc::Deallocator::scavenge):
895         (bmalloc::Deallocator::deallocateLarge):
896         (bmalloc::Deallocator::deallocateXLarge):
897         (bmalloc::Deallocator::processObjectLog):
898         (bmalloc::Deallocator::deallocateSmallLine):
899         (bmalloc::Deallocator::deallocateXSmallLine):
900         (bmalloc::Deallocator::allocateSmallLine):
901         (bmalloc::Deallocator::allocateXSmallLine):
902         (bmalloc::Deallocator::deallocateMediumLine):
903         (bmalloc::Deallocator::allocateMediumLine):
904         * bmalloc/Deallocator.h:
905         * bmalloc/Heap.cpp:
906         (bmalloc::sleep):
907         (bmalloc::Heap::Heap):
908         (bmalloc::Heap::concurrentScavenge):
909         (bmalloc::Heap::scavenge):
910         (bmalloc::Heap::scavengeSmallPages):
911         (bmalloc::Heap::scavengeXSmallPages):
912         (bmalloc::Heap::scavengeMediumPages):
913         (bmalloc::Heap::scavengeLargeRanges):
914         (bmalloc::Heap::allocateXSmallLineSlowCase):
915         (bmalloc::Heap::allocateSmallLineSlowCase):
916         (bmalloc::Heap::allocateMediumLineSlowCase):
917         (bmalloc::Heap::allocateXLarge):
918         (bmalloc::Heap::deallocateXLarge):
919         (bmalloc::Heap::allocateLarge):
920         (bmalloc::Heap::deallocateLarge):
921         * bmalloc/Heap.h:
922         (bmalloc::Heap::deallocateXSmallLine):
923         (bmalloc::Heap::allocateXSmallLine):
924         (bmalloc::Heap::deallocateSmallLine):
925         (bmalloc::Heap::allocateSmallLine):
926         (bmalloc::Heap::deallocateMediumLine):
927         (bmalloc::Heap::allocateMediumLine):
928         * bmalloc/Line.h:
929         (bmalloc::Line<Traits>::deref):
930         * bmalloc/Mutex.cpp: Removed.
931         * bmalloc/Mutex.h:
932         (bmalloc::Mutex::Mutex):
933         (bmalloc::Mutex::init): Deleted.
934         (bmalloc::Mutex::try_lock): Deleted.
935         (bmalloc::Mutex::lock): Deleted.
936         (bmalloc::Mutex::unlock): Deleted.
937         * bmalloc/Page.h:
938         (bmalloc::Page<Traits>::ref):
939         (bmalloc::Page<Traits>::deref):
940         (bmalloc::Page<Traits>::refCount):
941         * bmalloc/PerProcess.h:
942         (bmalloc::PerProcess::mutex):
943         (bmalloc::PerProcess<T>::getSlowCase):
944         * bmalloc/StaticMutex.cpp: Added.
945         (bmalloc::StaticMutex::lockSlowCase):
946         * bmalloc/StaticMutex.h: Added.
947         (bmalloc::StaticMutex::init):
948         (bmalloc::StaticMutex::try_lock):
949         (bmalloc::StaticMutex::lock):
950         (bmalloc::StaticMutex::unlock):
951         * bmalloc/VMHeap.h:
952         (bmalloc::VMHeap::deallocateXSmallPage):
953         (bmalloc::VMHeap::deallocateSmallPage):
954         (bmalloc::VMHeap::deallocateMediumPage):
955         (bmalloc::VMHeap::deallocateLargeRange):
956         * bmalloc/bmalloc.h:
957         (bmalloc::api::scavenge): Global replace Mutex => StaticMutex,
958         since the Heap mutex is a static.
959
960 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
961
962         bmalloc: AsyncTask should use Mutex instead of std::mutex
963         https://bugs.webkit.org/show_bug.cgi?id=131865
964
965         Reviewed by Gavin Barraclough.
966
967         std::mutex is so slow that it makes parallelizing simple tasks through
968         AsyncTask a net regression. Mutex fixes this.
969
970         * bmalloc/AsyncTask.h:
971         (bmalloc::Function>::AsyncTask):
972         (bmalloc::Function>::join):
973         (bmalloc::Function>::runSlowCase):
974         (bmalloc::Function>::entryPoint):
975         * bmalloc/Mutex.h:
976         (bmalloc::Mutex::init):
977
978 2014-04-18  Geoffrey Garen  <ggaren@apple.com>
979
980         bmalloc: Added an XSmall line size
981         https://bugs.webkit.org/show_bug.cgi?id=131851
982
983         Reviewed by Sam Weinig.
984
985         Reduces malloc footprint on Membuster recordings by 10%.
986
987         This is a throughput regression, but we're still way ahead of TCMalloc.
988         I have some ideas for how to recover the regression -- but I wanted to
989         get this win in first.
990
991         Full set of benchmark results:
992
993                 bmalloc> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks --measure-heap nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
994                                                                                                 
995                                                                nopatch                      patch                                Δ
996                 Peak Memory:
997                     reddit_memory_warning                      7,896kB                    7,532kB                  ^ 1.05x smaller
998                     flickr_memory_warning                     12,968kB                   12,324kB                  ^ 1.05x smaller
999                     theverge_memory_warning                   16,672kB                   15,200kB                   ^ 1.1x smaller
1000
1001                     <geometric mean>                          11,952kB                   11,216kB                  ^ 1.07x smaller
1002                     <arithmetic mean>                         12,512kB                   11,685kB                  ^ 1.07x smaller
1003                     <harmonic mean>                           11,375kB                   10,726kB                  ^ 1.06x smaller
1004
1005                 Memory at End:
1006                     reddit_memory_warning                      7,320kB                    6,856kB                  ^ 1.07x smaller
1007                     flickr_memory_warning                     10,848kB                    9,692kB                  ^ 1.12x smaller
1008                     theverge_memory_warning                   16,380kB                   14,872kB                   ^ 1.1x smaller
1009
1010                     <geometric mean>                          10,916kB                    9,961kB                   ^ 1.1x smaller
1011                     <arithmetic mean>                         11,516kB                   10,473kB                   ^ 1.1x smaller
1012                     <harmonic mean>                           10,350kB                    9,485kB                  ^ 1.09x smaller
1013
1014                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1015                                                                                                 
1016                                                    nopatch                patch                         Δ
1017                 Execution Time:
1018                     churn                            127ms                151ms            ! 1.19x slower
1019                     list_allocate                    130ms                164ms            ! 1.26x slower
1020                     tree_allocate                    109ms                127ms            ! 1.17x slower
1021                     tree_churn                       115ms                120ms            ! 1.04x slower
1022                     facebook                         240ms                259ms            ! 1.08x slower
1023                     fragment                          91ms                131ms            ! 1.44x slower
1024                     fragment_iterate                 105ms                106ms            ! 1.01x slower
1025                     message_one                      260ms                259ms             ^ 1.0x faster
1026                     message_many                     149ms                154ms            ! 1.03x slower
1027                     medium                           194ms                248ms            ! 1.28x slower
1028                     big                              157ms                160ms            ! 1.02x slower
1029
1030                     <geometric mean>                 144ms                163ms            ! 1.13x slower
1031                     <arithmetic mean>                152ms                171ms            ! 1.12x slower
1032                     <harmonic mean>                  137ms                156ms            ! 1.14x slower
1033
1034                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks nopatch:~/scratch/Build-nopatch/Release/ patch:~/webkit/WebKitBuild/Release/
1035                                                                                                 
1036                                                                        nopatch                          patch                                     Δ
1037                 Execution Time:
1038                     churn                                                126ms                          148ms                        ! 1.17x slower
1039                     churn --parallel                                      62ms                           76ms                        ! 1.23x slower
1040                     list_allocate                                        130ms                          164ms                        ! 1.26x slower
1041                     list_allocate --parallel                             120ms                          175ms                        ! 1.46x slower
1042                     tree_allocate                                        111ms                          127ms                        ! 1.14x slower
1043                     tree_allocate --parallel                              95ms                          135ms                        ! 1.42x slower
1044                     tree_churn                                           115ms                          124ms                        ! 1.08x slower
1045                     tree_churn --parallel                                107ms                          126ms                        ! 1.18x slower
1046                     facebook                                             240ms                          276ms                        ! 1.15x slower
1047                     facebook --parallel                                  802ms                        1,088ms                        ! 1.36x slower
1048                     fragment                                              92ms                          130ms                        ! 1.41x slower
1049                     fragment --parallel                                   66ms                          124ms                        ! 1.88x slower
1050                     fragment_iterate                                     109ms                          127ms                        ! 1.17x slower
1051                     fragment_iterate --parallel                           55ms                           64ms                        ! 1.16x slower
1052                     message_one                                          260ms                          260ms                                      
1053                     message_many                                         170ms                          238ms                         ! 1.4x slower
1054                     medium                                               185ms                          250ms                        ! 1.35x slower
1055                     medium --parallel                                    210ms                          334ms                        ! 1.59x slower
1056                     big                                                  150ms                          169ms                        ! 1.13x slower
1057                     big --parallel                                       138ms                          144ms                        ! 1.04x slower
1058
1059                     <geometric mean>                                     135ms                          170ms                        ! 1.26x slower
1060                     <arithmetic mean>                                    167ms                          214ms                        ! 1.28x slower
1061                     <harmonic mean>                                      117ms                          148ms                        ! 1.26x slower
1062
1063                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1064
1065                                                                     TC                      patch                                Δ
1066                 Peak Memory:
1067                     reddit_memory_warning                     13,836kB                   13,436kB                  ^ 1.03x smaller
1068                     flickr_memory_warning                     24,868kB                   25,188kB                   ! 1.01x bigger
1069                     theverge_memory_warning                   24,504kB                   26,636kB                   ! 1.09x bigger
1070
1071                     <geometric mean>                          20,353kB                   20,812kB                   ! 1.02x bigger
1072                     <arithmetic mean>                         21,069kB                   21,753kB                   ! 1.03x bigger
1073                     <harmonic mean>                           19,570kB                   19,780kB                   ! 1.01x bigger
1074
1075                 Memory at End:
1076                     reddit_memory_warning                      8,656kB                   10,016kB                   ! 1.16x bigger
1077                     flickr_memory_warning                     11,844kB                   13,784kB                   ! 1.16x bigger
1078                     theverge_memory_warning                   18,516kB                   22,748kB                   ! 1.23x bigger
1079
1080                     <geometric mean>                          12,382kB                   14,644kB                   ! 1.18x bigger
1081                     <arithmetic mean>                         13,005kB                   15,516kB                   ! 1.19x bigger
1082                     <harmonic mean>                           11,813kB                   13,867kB                   ! 1.17x bigger
1083
1084                 MallocBench> ~/webkit/PerformanceTests/MallocBench/run-malloc-benchmarks TC:~/scratch/Build-TCMalloc/Release/ patch:~/webkit/WebKitBuild/Release/
1085                                                                                                 
1086                                                         TC                patch                         Δ
1087                 Execution Time:
1088                     churn                            416ms                148ms            ^ 2.81x faster
1089                     list_allocate                    463ms                164ms            ^ 2.82x faster
1090                     tree_allocate                    292ms                127ms             ^ 2.3x faster
1091                     tree_churn                       157ms                120ms            ^ 1.31x faster
1092                     facebook                         327ms                276ms            ^ 1.18x faster
1093                     fragment                         335ms                129ms             ^ 2.6x faster
1094                     fragment_iterate                 344ms                108ms            ^ 3.19x faster
1095                     message_one                      386ms                258ms             ^ 1.5x faster
1096                     message_many                     410ms                154ms            ^ 2.66x faster
1097                     medium                           391ms                245ms             ^ 1.6x faster
1098                     big                              261ms                167ms            ^ 1.56x faster
1099
1100                     <geometric mean>                 332ms                164ms            ^ 2.02x faster
1101                     <arithmetic mean>                344ms                172ms            ^ 1.99x faster
1102                     <harmonic mean>                  317ms                157ms            ^ 2.02x faster
1103
1104         * bmalloc.xcodeproj/project.pbxproj:
1105         * bmalloc/Allocator.cpp:
1106         (bmalloc::Allocator::Allocator): Don't assume that each allocator's
1107         index corresponds with its size. Instead, use the size selection function
1108         explicitly. Now that we have XSmall, some small allocator entries are
1109         unused.
1110
1111         (bmalloc::Allocator::scavenge):
1112         (bmalloc::Allocator::log):
1113         (bmalloc::Allocator::processXSmallAllocatorLog):
1114         (bmalloc::Allocator::allocateSlowCase):
1115         * bmalloc/Allocator.h:
1116         (bmalloc::Allocator::xSmallAllocatorFor):
1117         (bmalloc::Allocator::allocateFastCase):
1118         * bmalloc/Chunk.h:
1119         * bmalloc/Deallocator.cpp:
1120         (bmalloc::Deallocator::scavenge):
1121         (bmalloc::Deallocator::processObjectLog):
1122         (bmalloc::Deallocator::deallocateSlowCase):
1123         (bmalloc::Deallocator::deallocateXSmallLine):
1124         (bmalloc::Deallocator::allocateXSmallLine):
1125         * bmalloc/Deallocator.h:
1126         (bmalloc::Deallocator::deallocateFastCase):
1127         * bmalloc/Heap.cpp:
1128         (bmalloc::Heap::scavenge):
1129         (bmalloc::Heap::scavengeXSmallPages):
1130         (bmalloc::Heap::allocateXSmallLineSlowCase):
1131         * bmalloc/Heap.h:
1132         (bmalloc::Heap::deallocateXSmallLine):
1133         (bmalloc::Heap::allocateXSmallLine):
1134         * bmalloc/LargeChunk.h:
1135         (bmalloc::LargeChunk::get):
1136         (bmalloc::LargeChunk::endTag):
1137         * bmalloc/Line.h:
1138         * bmalloc/MediumAllocator.h:
1139         (bmalloc::MediumAllocator::allocate):
1140         (bmalloc::MediumAllocator::refill):
1141         * bmalloc/ObjectType.cpp:
1142         (bmalloc::objectType):
1143         * bmalloc/ObjectType.h:
1144         (bmalloc::isXSmall):
1145         (bmalloc::isSmall):
1146         (bmalloc::isMedium):
1147         (bmalloc::isLarge):
1148         (bmalloc::isSmallOrMedium): Deleted.
1149         * bmalloc/SegregatedFreeList.h: I boiler-plate copied existing code for
1150         handling small objects. There's probably a reasonable way to share this
1151         code in the future -- I'll look into that once it's stopped changing.
1152
1153         * bmalloc/Sizes.h: Tweaked size classes to make Membuster happy. This
1154         is the main reason things got slower.
1155
1156         * bmalloc/SmallAllocator.h:
1157         (bmalloc::SmallAllocator::allocate):
1158         * bmalloc/SmallTraits.h:
1159         * bmalloc/VMHeap.cpp:
1160         (bmalloc::VMHeap::allocateXSmallChunk):
1161         * bmalloc/VMHeap.h:
1162         (bmalloc::VMHeap::allocateXSmallPage):
1163         (bmalloc::VMHeap::deallocateXSmallPage):
1164         * bmalloc/XSmallAllocator.h: Added.
1165         (bmalloc::XSmallAllocator::isNull):
1166         (bmalloc::XSmallAllocator::canAllocate):
1167         (bmalloc::XSmallAllocator::XSmallAllocator):
1168         (bmalloc::XSmallAllocator::line):
1169         (bmalloc::XSmallAllocator::allocate):
1170         (bmalloc::XSmallAllocator::objectCount):
1171         (bmalloc::XSmallAllocator::derefCount):
1172         (bmalloc::XSmallAllocator::refill):
1173         * bmalloc/XSmallChunk.h: Added.
1174         * bmalloc/XSmallLine.h: Added.
1175         * bmalloc/XSmallPage.h: Added.
1176         * bmalloc/XSmallTraits.h: Added.
1177         * bmalloc/bmalloc.h:
1178         (bmalloc::api::realloc): Boiler-plate copy, as above.
1179
1180 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1181
1182         MallocBench should scavenge explicitly instead of waiting
1183         https://bugs.webkit.org/show_bug.cgi?id=131661
1184
1185         Reviewed by Andreas Kling.
1186
1187         Added explicit scavenge support to bmalloc. This isn't a memory win,
1188         since bmalloc's per-thread cache is so small. But it makes testing
1189         simpler.
1190
1191         * bmalloc/Allocator.cpp:
1192         (bmalloc::Allocator::~Allocator):
1193         (bmalloc::Allocator::scavenge):
1194         * bmalloc/Allocator.h:
1195         * bmalloc/Cache.cpp:
1196         (bmalloc::Cache::operator new):
1197         (bmalloc::Cache::operator delete):
1198         (bmalloc::Cache::Cache):
1199         (bmalloc::Cache::scavenge):
1200         * bmalloc/Cache.h:
1201         * bmalloc/Deallocator.cpp:
1202         (bmalloc::Deallocator::~Deallocator):
1203         (bmalloc::Deallocator::scavenge):
1204         * bmalloc/Deallocator.h: Factored existing scavenging code into helper
1205         functions, for reuse.
1206
1207         * bmalloc/Heap.cpp:
1208         (bmalloc::sleep):
1209         (bmalloc::Heap::concurrentScavenge):
1210         (bmalloc::Heap::scavenge):
1211         (bmalloc::Heap::scavengeSmallPages):
1212         (bmalloc::Heap::scavengeMediumPages):
1213         (bmalloc::Heap::scavengeLargeRanges):
1214         * bmalloc/Heap.h: Made scavenge sleep duration a parameter. Forced
1215         scavenging -- in response to a benchmark or a low memory warning --
1216         wants to complete as soon as possible, so its sleep duration is 0.
1217
1218         * bmalloc/bmalloc.h:
1219         (bmalloc::api::scavenge):
1220         * bmalloc/mbmalloc.cpp: Exported the scavenge API for MallocBench's use.
1221
1222 2014-04-14  Geoffrey Garen  <ggaren@apple.com>
1223
1224         Use 4kB pages on Mac
1225         https://bugs.webkit.org/show_bug.cgi?id=131658
1226
1227         Reviewed by Sam Weinig.
1228
1229         This reduces memory use a lot on Membuster:
1230
1231                                                                   base                      patch                                Δ
1232                 Execution Time:
1233                     reddit_memory_warning                         18ms                       17ms                   ^ 1.06x faster
1234                     flickr_memory_warning                         34ms                       36ms                   ! 1.06x slower
1235                     theverge_memory_warning                       39ms                       41ms                   ! 1.05x slower
1236
1237                     <geometric mean>                              29ms                       29ms                   ! 1.02x slower
1238                     <arithmetic mean>                             30ms                       31ms                   ! 1.03x slower
1239                     <harmonic mean>                               27ms                       27ms                    ^ 1.0x faster
1240
1241                 Peak Memory:
1242                     reddit_memory_warning                     16,412kB                   16,436kB                    ! 1.0x bigger
1243                     flickr_memory_warning                     30,120kB                   30,184kB                    ! 1.0x bigger
1244                     theverge_memory_warning                   33,408kB                   33,420kB                    ! 1.0x bigger
1245
1246                     <geometric mean>                          25,466kB                   25,499kB                    ! 1.0x bigger
1247                     <arithmetic mean>                         26,647kB                   26,680kB                    ! 1.0x bigger
1248                     <harmonic mean>                           24,181kB                   24,214kB                    ! 1.0x bigger
1249
1250                 Memory at End:
1251                     reddit_memory_warning                      2,404kB                    1,920kB                  ^ 1.25x smaller
1252                     flickr_memory_warning                      3,764kB                    3,072kB                  ^ 1.23x smaller
1253                     theverge_memory_warning                    3,648kB                    3,132kB                  ^ 1.16x smaller
1254
1255                     <geometric mean>                           3,208kB                    2,644kB                  ^ 1.21x smaller
1256                     <arithmetic mean>                          3,272kB                    2,708kB                  ^ 1.21x smaller
1257                     <harmonic mean>                            3,139kB                    2,574kB                  ^ 1.22x smaller
1258
1259
1260         * bmalloc.xcodeproj/project.pbxproj:
1261         * bmalloc/BPlatform.h: Added.
1262         * bmalloc/VMAllocate.h: Only use 16kB pages on iOS because the page size
1263         is 4kB on Mac.
1264
1265 2014-04-14  Alexey Proskuryakov  <ap@apple.com>
1266
1267         Fixed svn:ignore on bmalloc.xcodeproj, it had erroneous leading spaces.
1268
1269         * bmalloc.xcodeproj: Modified property svn:ignore.
1270
1271 2014-04-13  Geoffrey Garen  <ggaren@apple.com>
1272
1273         Fixed some mbmalloc exports
1274         https://bugs.webkit.org/show_bug.cgi?id=131599
1275
1276         Reviewed by Ryosuke Niwa.
1277
1278         * bmalloc.xcodeproj/project.pbxproj: Made some headers a private part
1279         of the project, so we can call them from API.
1280
1281         * bmalloc/mbmalloc.cpp: Marked the mbmalloc functions with default
1282         visibility, so they show up as exported in the .dylib.
1283
1284 2014-04-09  Geoffrey Garen  <ggaren@apple.com>
1285
1286         Put bmalloc headers in the right place
1287         https://bugs.webkit.org/show_bug.cgi?id=131464
1288
1289         Reviewed by Mark Rowe.
1290
1291         * Configurations/bmalloc.xcconfig: Set PRIVATE_HEADERS_FOLDER_PATH to
1292         specify that we don't just want to dump all of our generically-named
1293         headers into /usr/local/include.
1294
1295 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1296
1297         Made bmalloc more #include friendly
1298         https://bugs.webkit.org/show_bug.cgi?id=131386
1299
1300         Reviewed by Andreas Kling.
1301
1302         Marked a bunch of headers private so they can be used from client code
1303         that #includes bmalloc.h.
1304
1305         Renamed ASSERT macros to BASSERT. This matches their header, which already
1306         had to be renamed, and fixes conflicts with WTF's ASSERT macros.
1307
1308         * bmalloc.xcodeproj/project.pbxproj:
1309         * bmalloc/Allocator.cpp:
1310         (bmalloc::Allocator::allocateSlowCase):
1311         * bmalloc/AsyncTask.h:
1312         (bmalloc::Function>::runSlowCase):
1313         * bmalloc/BAssert.h:
1314         * bmalloc/BoundaryTag.h:
1315         (bmalloc::BoundaryTag::setSize):
1316         * bmalloc/BoundaryTagInlines.h:
1317         (bmalloc::validate):
1318         (bmalloc::BoundaryTag::init):
1319         (bmalloc::BoundaryTag::deallocate):
1320         (bmalloc::BoundaryTag::splitLarge):
1321         (bmalloc::BoundaryTag::allocate):
1322         * bmalloc/Chunk.h:
1323         * bmalloc/Deallocator.cpp:
1324         (bmalloc::Deallocator::processObjectLog):
1325         (bmalloc::Deallocator::deallocateSlowCase):
1326         * bmalloc/Deallocator.h:
1327         (bmalloc::Deallocator::deallocateFastCase):
1328         * bmalloc/FixedVector.h:
1329         (bmalloc::Capacity>::operator):
1330         (bmalloc::Capacity>::push):
1331         (bmalloc::Capacity>::pop):
1332         (bmalloc::Capacity>::shrink):
1333         * bmalloc/Heap.cpp:
1334         (bmalloc::Heap::allocateLarge):
1335         * bmalloc/LargeChunk.h:
1336         (bmalloc::LargeChunk::get):
1337         (bmalloc::LargeChunk::endTag):
1338         * bmalloc/Line.h:
1339         (bmalloc::Line<Traits>::concurrentRef):
1340         (bmalloc::Line<Traits>::deref):
1341         * bmalloc/MediumAllocator.h:
1342         (bmalloc::MediumAllocator::allocate):
1343         * bmalloc/ObjectType.h:
1344         (bmalloc::isSmall):
1345         * bmalloc/Page.h:
1346         (bmalloc::Page<Traits>::ref):
1347         (bmalloc::Page<Traits>::deref):
1348         * bmalloc/PerThread.h:
1349         (bmalloc::PerThread<T>::getSlowCase):
1350         * bmalloc/SegregatedFreeList.cpp:
1351         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1352         (bmalloc::SegregatedFreeList::insert):
1353         * bmalloc/SmallAllocator.h:
1354         (bmalloc::SmallAllocator::allocate):
1355         (bmalloc::SmallAllocator::refill):
1356         * bmalloc/Syscall.h:
1357         * bmalloc/VMAllocate.h:
1358         (bmalloc::vmValidate):
1359         (bmalloc::vmAllocate):
1360         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1361         * bmalloc/Vector.h:
1362         (bmalloc::Vector<T>::operator):
1363         (bmalloc::Vector<T>::pop):
1364         (bmalloc::Vector<T>::shrink):
1365         * bmalloc/XLargeChunk.h:
1366         (bmalloc::XLargeChunk::range):
1367         (bmalloc::XLargeChunk::size):
1368
1369 2014-04-08  Geoffrey Garen  <ggaren@apple.com>
1370
1371         Removed an unused file.
1372
1373         Unreviewed.
1374
1375         * bmalloc/AsyncTask.cpp: Removed.
1376
1377 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1378
1379         Build bmalloc on Mac
1380         https://bugs.webkit.org/show_bug.cgi?id=131333
1381
1382         Reviewed by Mark Rowe.
1383
1384         * Makefile: Added. For make clients.
1385
1386         These files are required for building any project in WebKit. I copied
1387         them from WTF:
1388         * Configurations: Added.
1389         * Configurations/Base.xcconfig: Added.
1390         * Configurations/DebugRelease.xcconfig: Added.
1391         * Configurations/bmalloc.xcconfig: Added.
1392         * Configurations/iOS.xcconfig: Added.
1393         * Configurations/mbmalloc.xcconfig: Added.
1394
1395         * bmalloc.xcodeproj/project.pbxproj: I removed per-project-file stuff
1396         from here because everything is in .xcconfig files now.
1397
1398         I had to fix a bunch of minor warnings, since they're enabled in our
1399         .xcconfig files:
1400
1401         * bmalloc/AsyncTask.h:
1402         (bmalloc::Function>::AsyncTask):
1403         * bmalloc/BAssert.h:
1404         * bmalloc/BoundaryTagInlines.h:
1405         (bmalloc::validate):
1406         * bmalloc/Heap.cpp:
1407         (bmalloc::Heap::Heap):
1408         (bmalloc::Heap::allocateLarge):
1409         (bmalloc::Heap::deallocateLarge):
1410         * bmalloc/Mutex.h:
1411         (bmalloc::Mutex::Mutex): Deleted.
1412         * bmalloc/VMAllocate.h:
1413         (bmalloc::vmValidate):
1414         * bmalloc/mbmalloc.cpp:
1415
1416 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1417
1418         bmalloc: Fixed a leak in the per-thread cache
1419         https://bugs.webkit.org/show_bug.cgi?id=131330
1420
1421         Reviewed by Andreas Kling.
1422
1423         Remember to deallocate our line caches upon thread exit.
1424
1425         * bmalloc/Deallocator.cpp:
1426         (bmalloc::Deallocator::~Deallocator):
1427
1428 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1429
1430         bmalloc: rolled out the tryLock experiment
1431         https://bugs.webkit.org/show_bug.cgi?id=131328
1432
1433         Reviewed by Andreas Kling.
1434
1435         It wasn't a speedup.
1436
1437         * bmalloc.xcodeproj/project.pbxproj:
1438         * bmalloc/Allocator.cpp:
1439         (bmalloc::Allocator::processSmallAllocatorLog):
1440         (bmalloc::Allocator::processMediumAllocatorLog):
1441         * bmalloc/Deallocator.cpp:
1442         (bmalloc::Deallocator::processObjectLog):
1443         (bmalloc::Deallocator::deallocateSlowCase):
1444         (bmalloc::Deallocator::deallocateSmallLine):
1445         (bmalloc::Deallocator::deallocateMediumLine):
1446         * bmalloc/Deallocator.h:
1447         (bmalloc::Deallocator::deallocateFastCase):
1448         * bmalloc/Heap.h:
1449         (bmalloc::Heap::deallocateSmallLine):
1450         (bmalloc::Heap::deallocateMediumLine):
1451         * bmalloc/Line.h:
1452         (bmalloc::Line<Traits>::deref):
1453         * bmalloc/Page.h:
1454         (bmalloc::Page<Traits>::deref):
1455
1456 2014-04-07  Geoffrey Garen  <ggaren@apple.com>
1457
1458         bmalloc
1459         https://bugs.webkit.org/show_bug.cgi?id=131170
1460
1461         Reviewed by Andreas Kling.
1462
1463         Initial commit.
1464
1465         * bmalloc: Added.
1466         * bmalloc.xcodeproj: Added.
1467         * bmalloc.xcodeproj/project.pbxproj: Added.
1468         * bmalloc/Algorithm.h: Added.
1469         (bmalloc::max):
1470         (bmalloc::min):
1471         (bmalloc::mask):
1472         (bmalloc::test):
1473         (bmalloc::roundUpToMultipleOf):
1474         (bmalloc::roundDownToMultipleOf):
1475         (bmalloc::sizeOf):
1476         (bmalloc::bitCount):
1477         (bmalloc::isPowerOfTwo):
1478         * bmalloc/Allocator.cpp: Added.
1479         (bmalloc::Allocator::Allocator):
1480         (bmalloc::Allocator::~Allocator):
1481         (bmalloc::Allocator::log):
1482         (bmalloc::Allocator::processSmallAllocatorLog):
1483         (bmalloc::Allocator::processMediumAllocatorLog):
1484         (bmalloc::Allocator::allocateLarge):
1485         (bmalloc::Allocator::allocateXLarge):
1486         (bmalloc::Allocator::allocateMedium):
1487         (bmalloc::Allocator::allocateSlowCase):
1488         * bmalloc/Allocator.h: Added.
1489         (bmalloc::Allocator::smallAllocatorFor):
1490         (bmalloc::Allocator::allocateFastCase):
1491         (bmalloc::Allocator::allocate):
1492         * bmalloc/AsyncTask.cpp: Added.
1493         (bmalloc::AsyncTask<Function>::runSlowCase):
1494         (bmalloc::AsyncTask<Function>::pthreadEntryPoint):
1495         (bmalloc::AsyncTask<Function>::entryPoint):
1496         * bmalloc/AsyncTask.h: Added.
1497         (bmalloc::Function>::AsyncTask):
1498         (bmalloc::Function>::join):
1499         (bmalloc::Function>::run):
1500         (bmalloc::Function>::runSlowCase):
1501         (bmalloc::Function>::pthreadEntryPoint):
1502         (bmalloc::Function>::entryPoint):
1503         * bmalloc/BAssert.h: Added.
1504         * bmalloc/BeginTag.h: Added.
1505         (bmalloc::BeginTag::isInFreeList):
1506         * bmalloc/BoundaryTag.h: Added.
1507         (bmalloc::BoundaryTag::isXLarge):
1508         (bmalloc::BoundaryTag::setXLarge):
1509         (bmalloc::BoundaryTag::isFree):
1510         (bmalloc::BoundaryTag::setFree):
1511         (bmalloc::BoundaryTag::isEnd):
1512         (bmalloc::BoundaryTag::setEnd):
1513         (bmalloc::BoundaryTag::hasPhysicalPages):
1514         (bmalloc::BoundaryTag::setHasPhysicalPages):
1515         (bmalloc::BoundaryTag::isNull):
1516         (bmalloc::BoundaryTag::clear):
1517         (bmalloc::BoundaryTag::size):
1518         (bmalloc::BoundaryTag::setSize):
1519         (bmalloc::BoundaryTag::prev):
1520         (bmalloc::BoundaryTag::next):
1521         * bmalloc/BoundaryTagInlines.h: Added.
1522         (bmalloc::validate):
1523         (bmalloc::validatePrev):
1524         (bmalloc::validateNext):
1525         (bmalloc::BoundaryTag::init):
1526         (bmalloc::BoundaryTag::mergeLargeLeft):
1527         (bmalloc::BoundaryTag::mergeLargeRight):
1528         (bmalloc::BoundaryTag::mergeLarge):
1529         (bmalloc::BoundaryTag::deallocate):
1530         (bmalloc::BoundaryTag::splitLarge):
1531         (bmalloc::BoundaryTag::allocate):
1532         * bmalloc/Cache.cpp: Added.
1533         (bmalloc::Cache::operator new):
1534         (bmalloc::Cache::operator delete):
1535         (bmalloc::Cache::Cache):
1536         (bmalloc::Cache::allocateSlowCase):
1537         (bmalloc::Cache::allocateSlowCaseNullCache):
1538         (bmalloc::Cache::deallocateSlowCase):
1539         (bmalloc::Cache::deallocateSlowCaseNullCache):
1540         * bmalloc/Cache.h: Added.
1541         (bmalloc::Cache::allocator):
1542         (bmalloc::Cache::deallocator):
1543         (bmalloc::Cache::allocateFastCase):
1544         (bmalloc::Cache::deallocateFastCase):
1545         (bmalloc::Cache::allocate):
1546         (bmalloc::Cache::deallocate):
1547         * bmalloc/Chunk.h: Added.
1548         (bmalloc::Chunk::begin):
1549         (bmalloc::Chunk::end):
1550         (bmalloc::Chunk::lines):
1551         (bmalloc::Chunk::pages):
1552         * bmalloc/Deallocator.cpp: Added.
1553         (bmalloc::Deallocator::Deallocator):
1554         (bmalloc::Deallocator::~Deallocator):
1555         (bmalloc::Deallocator::deallocateLarge):
1556         (bmalloc::Deallocator::deallocateXLarge):
1557         (bmalloc::Deallocator::processObjectLog):
1558         (bmalloc::Deallocator::deallocateSlowCase):
1559         (bmalloc::Deallocator::deallocateSmallLine):
1560         (bmalloc::Deallocator::allocateSmallLine):
1561         (bmalloc::Deallocator::deallocateMediumLine):
1562         (bmalloc::Deallocator::allocateMediumLine):
1563         * bmalloc/Deallocator.h: Added.
1564         (bmalloc::Deallocator::deallocateFastCase):
1565         (bmalloc::Deallocator::deallocate):
1566         * bmalloc/EndTag.h: Added.
1567         (bmalloc::EndTag::operator=):
1568         * bmalloc/FixedVector.h: Added.
1569         (bmalloc::FixedVector::begin):
1570         (bmalloc::FixedVector::end):
1571         (bmalloc::FixedVector::size):
1572         (bmalloc::FixedVector::capacity):
1573         (bmalloc::FixedVector::clear):
1574         (bmalloc::FixedVector::isEmpty):
1575         (bmalloc::Capacity>::FixedVector):
1576         (bmalloc::Capacity>::operator):
1577         (bmalloc::Capacity>::push):
1578         (bmalloc::Capacity>::pop):
1579         (bmalloc::Capacity>::shrink):
1580         * bmalloc/Heap.cpp: Added.
1581         (bmalloc::sleep):
1582         (bmalloc::Heap::Heap):
1583         (bmalloc::Heap::concurrentScavenge):
1584         (bmalloc::Heap::scavengeSmallPages):
1585         (bmalloc::Heap::scavengeMediumPages):
1586         (bmalloc::Heap::scavengeLargeRanges):
1587         (bmalloc::Heap::allocateSmallLineSlowCase):
1588         (bmalloc::Heap::allocateMediumLineSlowCase):
1589         (bmalloc::Heap::allocateXLarge):
1590         (bmalloc::Heap::deallocateXLarge):
1591         (bmalloc::Heap::allocateLarge):
1592         (bmalloc::Heap::deallocateLarge):
1593         * bmalloc/Heap.h: Added.
1594         (bmalloc::Heap::deallocateSmallLine):
1595         (bmalloc::Heap::allocateSmallLine):
1596         (bmalloc::Heap::deallocateMediumLine):
1597         (bmalloc::Heap::allocateMediumLine):
1598         * bmalloc/Inline.h: Added.
1599         * bmalloc/LargeChunk.h: Added.
1600         (bmalloc::LargeChunk::begin):
1601         (bmalloc::LargeChunk::end):
1602         (bmalloc::LargeChunk::create):
1603         (bmalloc::LargeChunk::get):
1604         (bmalloc::LargeChunk::beginTag):
1605         (bmalloc::LargeChunk::endTag):
1606         * bmalloc/Line.h: Added.
1607         (bmalloc::Line<Traits>::begin):
1608         (bmalloc::Line<Traits>::end):
1609         (bmalloc::Line<Traits>::concurrentRef):
1610         (bmalloc::Line<Traits>::deref):
1611         * bmalloc/MediumAllocator.h: Added.
1612         (bmalloc::MediumAllocator::isNull):
1613         (bmalloc::MediumAllocator::MediumAllocator):
1614         (bmalloc::MediumAllocator::line):
1615         (bmalloc::MediumAllocator::allocate):
1616         (bmalloc::MediumAllocator::derefCount):
1617         (bmalloc::MediumAllocator::refill):
1618         * bmalloc/MediumChunk.h: Added.
1619         * bmalloc/MediumLine.h: Added.
1620         * bmalloc/MediumPage.h: Added.
1621         * bmalloc/MediumTraits.h: Added.
1622         * bmalloc/Mutex.cpp: Added.
1623         (bmalloc::Mutex::lockSlowCase):
1624         * bmalloc/Mutex.h: Added.
1625         (bmalloc::Mutex::Mutex):
1626         (bmalloc::Mutex::try_lock):
1627         (bmalloc::Mutex::lock):
1628         (bmalloc::Mutex::unlock):
1629         * bmalloc/ObjectType.cpp: Added.
1630         (bmalloc::objectType):
1631         * bmalloc/ObjectType.h: Added.
1632         (bmalloc::isSmallOrMedium):
1633         (bmalloc::isSmall):
1634         * bmalloc/Page.h: Added.
1635         (bmalloc::Page<Traits>::ref):
1636         (bmalloc::Page<Traits>::deref):
1637         (bmalloc::Page<Traits>::refCount):
1638         * bmalloc/PerProcess.h: Added.
1639         (bmalloc::PerProcess::mutex):
1640         (bmalloc::PerProcess<T>::getFastCase):
1641         (bmalloc::PerProcess<T>::get):
1642         (bmalloc::PerProcess<T>::getSlowCase):
1643         * bmalloc/PerThread.h: Added.
1644         (bmalloc::PerThreadStorage<Cache>::get):
1645         (bmalloc::PerThreadStorage<Cache>::init):
1646         (bmalloc::PerThreadStorage::get):
1647         (bmalloc::PerThreadStorage::init):
1648         (bmalloc::PerThread<T>::getFastCase):
1649         (bmalloc::PerThread<T>::get):
1650         (bmalloc::PerThread<T>::destructor):
1651         (bmalloc::PerThread<T>::getSlowCase):
1652         * bmalloc/Range.h: Added.
1653         (bmalloc::Range::Range):
1654         (bmalloc::Range::begin):
1655         (bmalloc::Range::end):
1656         (bmalloc::Range::size):
1657         (bmalloc::Range::operator!):
1658         (bmalloc::Range::operator<):
1659         * bmalloc/SegregatedFreeList.cpp: Added.
1660         (bmalloc::SegregatedFreeList::SegregatedFreeList):
1661         (bmalloc::SegregatedFreeList::insert):
1662         (bmalloc::SegregatedFreeList::takeGreedy):
1663         (bmalloc::SegregatedFreeList::take):
1664         * bmalloc/SegregatedFreeList.h: Added.
1665         * bmalloc/Sizes.h: Added.
1666         * bmalloc/SmallAllocator.h: Added.
1667         (bmalloc::SmallAllocator::isNull):
1668         (bmalloc::SmallAllocator::canAllocate):
1669         (bmalloc::SmallAllocator::SmallAllocator):
1670         (bmalloc::SmallAllocator::line):
1671         (bmalloc::SmallAllocator::allocate):
1672         (bmalloc::SmallAllocator::objectCount):
1673         (bmalloc::SmallAllocator::derefCount):
1674         (bmalloc::SmallAllocator::refill):
1675         * bmalloc/SmallChunk.h: Added.
1676         * bmalloc/SmallLine.h: Added.
1677         * bmalloc/SmallPage.h: Added.
1678         * bmalloc/SmallTraits.h: Added.
1679         * bmalloc/Syscall.h: Added.
1680         * bmalloc/VMAllocate.h: Added.
1681         (bmalloc::vmSize):
1682         (bmalloc::vmValidate):
1683         (bmalloc::vmAllocate):
1684         (bmalloc::vmDeallocate):
1685         (bmalloc::vmDeallocatePhysicalPages):
1686         (bmalloc::vmAllocatePhysicalPages):
1687         (bmalloc::vmDeallocatePhysicalPagesSloppy):
1688         (bmalloc::vmAllocatePhysicalPagesSloppy):
1689         * bmalloc/VMHeap.cpp: Added.
1690         (bmalloc::VMHeap::VMHeap):
1691         (bmalloc::VMHeap::allocateSmallChunk):
1692         (bmalloc::VMHeap::allocateMediumChunk):
1693         (bmalloc::VMHeap::allocateLargeChunk):
1694         * bmalloc/VMHeap.h: Added.
1695         (bmalloc::VMHeap::allocateSmallPage):
1696         (bmalloc::VMHeap::allocateMediumPage):
1697         (bmalloc::VMHeap::allocateLargeRange):
1698         (bmalloc::VMHeap::deallocateSmallPage):
1699         (bmalloc::VMHeap::deallocateMediumPage):
1700         (bmalloc::VMHeap::deallocateLargeRange):
1701         * bmalloc/Vector.h: Added.
1702         (bmalloc::Vector::begin):
1703         (bmalloc::Vector::end):
1704         (bmalloc::Vector::size):
1705         (bmalloc::Vector::capacity):
1706         (bmalloc::Vector::last):
1707         (bmalloc::Vector::pop):
1708         (bmalloc::Vector<T>::Vector):
1709         (bmalloc::Vector<T>::~Vector):
1710         (bmalloc::Vector<T>::operator):
1711         (bmalloc::Vector<T>::push):
1712         (bmalloc::Vector<T>::pop):
1713         (bmalloc::Vector<T>::shrink):
1714         (bmalloc::Vector<T>::reallocateBuffer):
1715         (bmalloc::Vector<T>::shrinkCapacity):
1716         (bmalloc::Vector<T>::growCapacity):
1717         * bmalloc/XLargeChunk.h: Added.
1718         (bmalloc::XLargeChunk::get):
1719         (bmalloc::XLargeChunk::begin):
1720         (bmalloc::XLargeChunk::XLargeChunk):
1721         (bmalloc::XLargeChunk::create):
1722         (bmalloc::XLargeChunk::destroy):
1723         (bmalloc::XLargeChunk::range):
1724         (bmalloc::XLargeChunk::size):
1725         * bmalloc/bmalloc.h: Added.
1726         (bmalloc::api::malloc):
1727         (bmalloc::api::free):
1728         (bmalloc::api::realloc):
1729         * bmalloc/mbmalloc.cpp: Added.
1730