[bmalloc] Add StaticPerProcess for known types to save pages
authorysuzuki@apple.com <ysuzuki@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 14 Mar 2019 08:02:00 +0000 (08:02 +0000)
committerysuzuki@apple.com <ysuzuki@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>
Thu, 14 Mar 2019 08:02:00 +0000 (08:02 +0000)
https://bugs.webkit.org/show_bug.cgi?id=195691

Reviewed by Mark Lam.

As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.

This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.

* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):

git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc

30 files changed:
Source/bmalloc/ChangeLog
Source/bmalloc/bmalloc.xcodeproj/project.pbxproj
Source/bmalloc/bmalloc/AllIsoHeaps.cpp
Source/bmalloc/bmalloc/AllIsoHeaps.h
Source/bmalloc/bmalloc/Allocator.cpp
Source/bmalloc/bmalloc/Cache.cpp
Source/bmalloc/bmalloc/CryptoRandom.cpp
Source/bmalloc/bmalloc/Deallocator.cpp
Source/bmalloc/bmalloc/DebugHeap.cpp
Source/bmalloc/bmalloc/DebugHeap.h
Source/bmalloc/bmalloc/Environment.cpp
Source/bmalloc/bmalloc/Environment.h
Source/bmalloc/bmalloc/Gigacage.cpp
Source/bmalloc/bmalloc/Heap.cpp
Source/bmalloc/bmalloc/IsoDirectoryInlines.h
Source/bmalloc/bmalloc/IsoHeapImpl.cpp
Source/bmalloc/bmalloc/IsoPage.cpp
Source/bmalloc/bmalloc/IsoTLS.cpp
Source/bmalloc/bmalloc/IsoTLSEntry.cpp
Source/bmalloc/bmalloc/IsoTLSInlines.h
Source/bmalloc/bmalloc/IsoTLSLayout.cpp
Source/bmalloc/bmalloc/IsoTLSLayout.h
Source/bmalloc/bmalloc/Scavenger.cpp
Source/bmalloc/bmalloc/Scavenger.h
Source/bmalloc/bmalloc/StaticPerProcess.h [new file with mode: 0644]
Source/bmalloc/bmalloc/VMHeap.cpp
Source/bmalloc/bmalloc/VMHeap.h
Source/bmalloc/bmalloc/Zone.h
Source/bmalloc/bmalloc/bmalloc.cpp
Source/bmalloc/test/testbmalloc.cpp

index 9e32bb2..86886c5 100644 (file)
@@ -1,3 +1,90 @@
+2019-03-14  Yusuke Suzuki  <ysuzuki@apple.com>
+
+        [bmalloc] Add StaticPerProcess for known types to save pages
+        https://bugs.webkit.org/show_bug.cgi?id=195691
+
+        Reviewed by Mark Lam.
+
+        As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
+        For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
+        sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
+        size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
+        this page.
+
+        This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
+        allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
+
+        * bmalloc.xcodeproj/project.pbxproj:
+        * bmalloc/AllIsoHeaps.cpp:
+        * bmalloc/AllIsoHeaps.h:
+        * bmalloc/Allocator.cpp:
+        (bmalloc::Allocator::Allocator):
+        * bmalloc/Cache.cpp:
+        (bmalloc::Cache::Cache):
+        * bmalloc/CryptoRandom.cpp:
+        (bmalloc::cryptoRandom):
+        * bmalloc/Deallocator.cpp:
+        (bmalloc::Deallocator::Deallocator):
+        * bmalloc/DebugHeap.cpp:
+        * bmalloc/DebugHeap.h:
+        (bmalloc::DebugHeap::tryGet):
+        * bmalloc/Environment.cpp:
+        * bmalloc/Environment.h:
+        * bmalloc/Gigacage.cpp:
+        (Gigacage::Callback::Callback):
+        (Gigacage::Callback::function):
+        (bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
+        (Gigacage::disablePrimitiveGigacage):
+        (Gigacage::addPrimitiveDisableCallback):
+        (Gigacage::removePrimitiveDisableCallback):
+        (Gigacage::shouldBeEnabled):
+        (Gigacage::bmalloc::Callback::Callback): Deleted.
+        (Gigacage::bmalloc::Callback::function): Deleted.
+        (Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
+        * bmalloc/Heap.cpp:
+        (bmalloc::Heap::Heap):
+        (bmalloc::Heap::tryAllocateLarge):
+        * bmalloc/IsoDirectoryInlines.h:
+        (bmalloc::passedNumPages>::takeFirstEligible):
+        (bmalloc::passedNumPages>::didBecome):
+        * bmalloc/IsoHeapImpl.cpp:
+        (bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
+        * bmalloc/IsoPage.cpp:
+        (bmalloc::IsoPageBase::allocatePageMemory):
+        * bmalloc/IsoTLS.cpp:
+        (bmalloc::IsoTLS::IsoTLS):
+        (bmalloc::IsoTLS::ensureEntries):
+        (bmalloc::IsoTLS::forEachEntry):
+        * bmalloc/IsoTLSEntry.cpp:
+        (bmalloc::IsoTLSEntry::IsoTLSEntry):
+        * bmalloc/IsoTLSInlines.h:
+        (bmalloc::IsoTLS::allocateSlow):
+        (bmalloc::IsoTLS::deallocateSlow):
+        * bmalloc/IsoTLSLayout.cpp:
+        * bmalloc/IsoTLSLayout.h:
+        * bmalloc/Scavenger.cpp:
+        (bmalloc::Scavenger::Scavenger):
+        (bmalloc::dumpStats):
+        (bmalloc::Scavenger::scavenge):
+        (bmalloc::Scavenger::partialScavenge):
+        (bmalloc::Scavenger::freeableMemory):
+        (bmalloc::Scavenger::footprint):
+        * bmalloc/Scavenger.h:
+        * bmalloc/StaticPerProcess.h: Added.
+        * bmalloc/VMHeap.cpp:
+        * bmalloc/VMHeap.h:
+        * bmalloc/Zone.h:
+        * bmalloc/bmalloc.cpp:
+        (bmalloc::api::scavenge):
+        (bmalloc::api::isEnabled):
+        (bmalloc::api::setScavengerThreadQOSClass):
+        (bmalloc::api::enableMiniMode):
+        * test/testbmalloc.cpp:
+        (assertEmptyPointerSet):
+        (assertHasObjects):
+        (assertHasOnlyObjects):
+        (assertClean):
+
 2019-03-13  Yoshiaki Jitsukawa  <yoshiaki.jitsukawa@sony.com>
 
         [bmalloc] Use MADV_FREE on FreeBSD
index 14c593c..5840f53 100644 (file)
                AD14AD29202529C400890E3B /* ProcessCheck.h in Headers */ = {isa = PBXBuildFile; fileRef = AD14AD27202529A600890E3B /* ProcessCheck.h */; };
                AD14AD2A202529C700890E3B /* ProcessCheck.mm in Sources */ = {isa = PBXBuildFile; fileRef = AD14AD28202529B000890E3B /* ProcessCheck.mm */; };
                DE8B13B321CC5D9F00A63FCD /* BVMTags.h in Headers */ = {isa = PBXBuildFile; fileRef = DE8B13B221CC5D9F00A63FCD /* BVMTags.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               E31E74802238CA5C005D084A /* StaticPerProcess.h in Headers */ = {isa = PBXBuildFile; fileRef = E31E747F2238CA5B005D084A /* StaticPerProcess.h */; settings = {ATTRIBUTES = (Private, ); }; };
 /* End PBXBuildFile section */
 
 /* Begin PBXContainerItemProxy section */
                AD14AD27202529A600890E3B /* ProcessCheck.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = ProcessCheck.h; path = bmalloc/ProcessCheck.h; sourceTree = "<group>"; };
                AD14AD28202529B000890E3B /* ProcessCheck.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = ProcessCheck.mm; path = bmalloc/ProcessCheck.mm; sourceTree = "<group>"; };
                DE8B13B221CC5D9F00A63FCD /* BVMTags.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = BVMTags.h; path = bmalloc/BVMTags.h; sourceTree = "<group>"; };
+               E31E747F2238CA5B005D084A /* StaticPerProcess.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = StaticPerProcess.h; path = bmalloc/StaticPerProcess.h; sourceTree = "<group>"; };
 /* End PBXFileReference section */
 
 /* Begin PBXFrameworksBuildPhase section */
                                144469FD17A61F1F00F9EA1D /* PerThread.h */,
                                145F6878179E3A4400D65598 /* Range.h */,
                                148EFAE61D6B953B008E721E /* ScopeExit.h */,
+                               E31E747F2238CA5B005D084A /* StaticPerProcess.h */,
                                7C571F0022388B840077A3C7 /* StdLibExtras.h */,
                                1417F64F18B7280C0076FA3F /* Syscall.h */,
                                1479E21217A1A255006D4E9D /* Vector.h */,
                                14DD789018F48CEB00950702 /* Sizes.h in Headers */,
                                14DD78BC18F48D6B00950702 /* SmallLine.h in Headers */,
                                14DD78BD18F48D6B00950702 /* SmallPage.h in Headers */,
+                               E31E74802238CA5C005D084A /* StaticPerProcess.h in Headers */,
                                7C571F0122388B840077A3C7 /* StdLibExtras.h in Headers */,
                                14DD78CE18F48D7500950702 /* Syscall.h in Headers */,
                                14DD78CF18F48D7500950702 /* Vector.h in Headers */,
index a0709c8..fff0e40 100644 (file)
@@ -27,6 +27,8 @@
 
 namespace bmalloc {
 
+DEFINE_STATIC_PER_PROCESS_STORAGE(AllIsoHeaps);
+
 AllIsoHeaps::AllIsoHeaps(const std::lock_guard<Mutex>&)
 {
 }
index 78eae34..a0fceeb 100644 (file)
 #pragma once
 
 #include "IsoHeapImpl.h"
+#include "StaticPerProcess.h"
 #include "Vector.h"
 
 namespace bmalloc {
 
-class AllIsoHeaps {
+class AllIsoHeaps : public StaticPerProcess<AllIsoHeaps> {
 public:
     AllIsoHeaps(const std::lock_guard<Mutex>&);
     
@@ -44,6 +45,7 @@ private:
     Mutex m_lock;
     IsoHeapImplBase* m_head { nullptr };
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(AllIsoHeaps);
 
 } // namespace bmalloc
 
index 409140e..d086d22 100644 (file)
@@ -40,7 +40,7 @@ Allocator::Allocator(Heap& heap, Deallocator& deallocator)
     : m_heap(heap)
     , m_deallocator(deallocator)
 {
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
     for (size_t sizeClass = 0; sizeClass < sizeClassCount; ++sizeClass)
         m_bumpAllocators[sizeClass].init(objectSize(sizeClass));
 }
index f97d45e..0d1fa9d 100644 (file)
@@ -48,7 +48,7 @@ Cache::Cache(HeapKind heapKind)
     : m_deallocator(PerProcess<PerHeapKind<Heap>>::get()->at(heapKind))
     , m_allocator(PerProcess<PerHeapKind<Heap>>::get()->at(heapKind), m_deallocator)
 {
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
 }
 
 BNO_INLINE void* Cache::tryAllocateSlowCaseNullCache(HeapKind heapKind, size_t size)
index 8c415ee..63dd9d4 100644 (file)
@@ -33,7 +33,7 @@
 #include "BAssert.h"
 #include "BPlatform.h"
 #include "Mutex.h"
-#include "PerProcess.h"
+#include "StaticPerProcess.h"
 #include "VMAllocate.h"
 #include <mutex>
 
@@ -59,7 +59,7 @@ public:
     uint8_t s[256];
 };
 
-class ARC4RandomNumberGenerator {
+class ARC4RandomNumberGenerator : public StaticPerProcess<ARC4RandomNumberGenerator> {
 public:
     ARC4RandomNumberGenerator(const std::lock_guard<Mutex>&);
 
@@ -76,6 +76,8 @@ private:
     int m_count;
     Mutex m_mutex;
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(ARC4RandomNumberGenerator);
+DEFINE_STATIC_PER_PROCESS_STORAGE(ARC4RandomNumberGenerator);
 
 ARC4Stream::ARC4Stream()
 {
@@ -176,7 +178,7 @@ void ARC4RandomNumberGenerator::randomValues(void* buffer, size_t length)
 
 void cryptoRandom(void* buffer, size_t length)
 {
-    PerProcess<ARC4RandomNumberGenerator>::get()->randomValues(buffer, length);
+    ARC4RandomNumberGenerator::get()->randomValues(buffer, length);
 }
 
 } // namespace bmalloc
index 58ce3a6..3772270 100644 (file)
@@ -40,7 +40,7 @@ namespace bmalloc {
 Deallocator::Deallocator(Heap& heap)
     : m_heap(heap)
 {
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
 }
 
 Deallocator::~Deallocator()
index cd5609e..0b0fdb5 100644 (file)
@@ -36,6 +36,8 @@ namespace bmalloc {
 
 DebugHeap* debugHeapCache { nullptr };
     
+DEFINE_STATIC_PER_PROCESS_STORAGE(DebugHeap);
+
 #if BOS(DARWIN)
 
 DebugHeap::DebugHeap(std::lock_guard<Mutex>&)
index 77a2658..7032b02 100644 (file)
@@ -27,7 +27,7 @@
 
 #include "Environment.h"
 #include "Mutex.h"
-#include "PerProcess.h"
+#include "StaticPerProcess.h"
 #include <mutex>
 #include <unordered_map>
 
@@ -37,7 +37,7 @@
 
 namespace bmalloc {
     
-class DebugHeap {
+class DebugHeap : private StaticPerProcess<DebugHeap> {
 public:
     DebugHeap(std::lock_guard<Mutex>&);
     
@@ -64,14 +64,15 @@ private:
     std::mutex m_lock;
     std::unordered_map<void*, size_t> m_sizeMap;
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(DebugHeap);
 
 extern BEXPORT DebugHeap* debugHeapCache;
 BINLINE DebugHeap* DebugHeap::tryGet()
 {
     if (debugHeapCache)
         return debugHeapCache;
-    if (PerProcess<Environment>::get()->isDebugHeapEnabled()) {
-        debugHeapCache = PerProcess<DebugHeap>::get();
+    if (Environment::get()->isDebugHeapEnabled()) {
+        debugHeapCache = DebugHeap::get();
         return debugHeapCache;
     }
     return nullptr;
index 81f7c71..87c0050 100644 (file)
@@ -125,6 +125,8 @@ static bool isNanoMallocEnabled()
 }
 #endif
 
+DEFINE_STATIC_PER_PROCESS_STORAGE(Environment);
+
 Environment::Environment(std::lock_guard<Mutex>&)
     : m_isDebugHeapEnabled(computeIsDebugHeapEnabled())
 {
index 917c6f3..a56256e 100644 (file)
 #define Environment_h
 
 #include "Mutex.h"
+#include "StaticPerProcess.h"
 
 namespace bmalloc {
 
-class Environment {
+class Environment : public StaticPerProcess<Environment> {
 public:
     BEXPORT Environment(std::lock_guard<Mutex>&);
     
@@ -41,6 +42,7 @@ private:
 
     bool m_isDebugHeapEnabled;
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(Environment);
 
 } // namespace bmalloc
 
index 3a9c7d2..8209831 100644 (file)
@@ -27,8 +27,8 @@
 
 #include "CryptoRandom.h"
 #include "Environment.h"
-#include "PerProcess.h"
 #include "ProcessCheck.h"
+#include "StaticPerProcess.h"
 #include "VMAllocate.h"
 #include "Vector.h"
 #include "bmalloc.h"
 
 namespace Gigacage {
 
+struct Callback {
+    Callback() { }
+    
+    Callback(void (*function)(void*), void *argument)
+        : function(function)
+        , argument(argument)
+    {
+    }
+    
+    void (*function)(void*) { nullptr };
+    void* argument { nullptr };
+};
+
+}
+
+namespace bmalloc {
+
+struct PrimitiveDisableCallbacks : public StaticPerProcess<PrimitiveDisableCallbacks> {
+    PrimitiveDisableCallbacks(std::lock_guard<Mutex>&) { }
+    
+    Vector<Gigacage::Callback> callbacks;
+};
+DECLARE_STATIC_PER_PROCESS_STORAGE(PrimitiveDisableCallbacks);
+DEFINE_STATIC_PER_PROCESS_STORAGE(PrimitiveDisableCallbacks);
+
+} // namespace bmalloc
+
+namespace Gigacage {
+
 // This is exactly 32GB because inside JSC, indexed accesses for arrays, typed arrays, etc,
 // use unsigned 32-bit ints as indices. The items those indices access are 8 bytes or less
 // in size. 2^32 * 8 = 32GB. This means if an access on a caged type happens to go out of
@@ -84,25 +113,6 @@ public:
     }
 };
 
-struct Callback {
-    Callback() { }
-    
-    Callback(void (*function)(void*), void *argument)
-        : function(function)
-        , argument(argument)
-    {
-    }
-    
-    void (*function)(void*) { nullptr };
-    void* argument { nullptr };
-};
-
-struct PrimitiveDisableCallbacks {
-    PrimitiveDisableCallbacks(std::lock_guard<Mutex>&) { }
-    
-    Vector<Callback> callbacks;
-};
-
 size_t runwaySize(Kind kind)
 {
     switch (kind) {
@@ -199,8 +209,8 @@ void disablePrimitiveGigacage()
         return;
     }
     
-    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
-    std::unique_lock<Mutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
+    PrimitiveDisableCallbacks& callbacks = *PrimitiveDisableCallbacks::get();
+    std::unique_lock<Mutex> lock(PrimitiveDisableCallbacks::mutex());
     for (Callback& callback : callbacks.callbacks)
         callback.function(callback.argument);
     callbacks.callbacks.shrink(0);
@@ -217,15 +227,15 @@ void addPrimitiveDisableCallback(void (*function)(void*), void* argument)
         return;
     }
     
-    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
-    std::unique_lock<Mutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
+    PrimitiveDisableCallbacks& callbacks = *PrimitiveDisableCallbacks::get();
+    std::unique_lock<Mutex> lock(PrimitiveDisableCallbacks::mutex());
     callbacks.callbacks.push(Callback(function, argument));
 }
 
 void removePrimitiveDisableCallback(void (*function)(void*), void* argument)
 {
-    PrimitiveDisableCallbacks& callbacks = *PerProcess<PrimitiveDisableCallbacks>::get();
-    std::unique_lock<Mutex> lock(PerProcess<PrimitiveDisableCallbacks>::mutex());
+    PrimitiveDisableCallbacks& callbacks = *PrimitiveDisableCallbacks::get();
+    std::unique_lock<Mutex> lock(PrimitiveDisableCallbacks::mutex());
     for (size_t i = 0; i < callbacks.callbacks.size(); ++i) {
         if (callbacks.callbacks[i].function == function
             && callbacks.callbacks[i].argument == argument) {
@@ -267,7 +277,7 @@ bool shouldBeEnabled()
     std::call_once(
         onceFlag,
         [] {
-            bool debugHeapEnabled = PerProcess<Environment>::get()->isDebugHeapEnabled();
+            bool debugHeapEnabled = Environment::get()->isDebugHeapEnabled();
             if (debugHeapEnabled)
                 return;
 
index 25a29ab..ef03fd4 100644 (file)
@@ -54,7 +54,7 @@ Heap::Heap(HeapKind kind, std::lock_guard<Mutex>&)
     initializeLineMetadata();
     initializePageMetadata();
     
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
 
     Gigacage::ensureGigacage();
 #if GIGACAGE_ENABLED
@@ -69,7 +69,7 @@ Heap::Heap(HeapKind kind, std::lock_guard<Mutex>&)
     }
 #endif
     
-    m_scavenger = PerProcess<Scavenger>::get();
+    m_scavenger = Scavenger::get();
 }
 
 bool Heap::usingGigacage()
@@ -574,7 +574,7 @@ void* Heap::tryAllocateLarge(std::unique_lock<Mutex>& lock, size_t alignment, si
         if (usingGigacage())
             return nullptr;
 
-        range = PerProcess<VMHeap>::get()->tryAllocateLargeChunk(alignment, size);
+        range = VMHeap::get()->tryAllocateLargeChunk(alignment, size);
         if (!range)
             return nullptr;
         
index 12a48e5..f388ded 100644 (file)
@@ -53,7 +53,7 @@ EligibilityResult<Config> IsoDirectory<Config, passedNumPages>::takeFirstEligibl
 
     m_highWatermark = std::max(pageIndex, m_highWatermark);
     
-    Scavenger& scavenger = *PerProcess<Scavenger>::get();
+    Scavenger& scavenger = *Scavenger::get();
     scavenger.didStartGrowing();
     
     IsoPage<Config>* page = m_pages[pageIndex];
@@ -108,7 +108,7 @@ void IsoDirectory<Config, passedNumPages>::didBecome(IsoPage<Config>* page, IsoP
         BASSERT(!!m_committed[pageIndex]);
         this->m_heap.isNowFreeable(page, IsoPageBase::pageSize);
         m_empty[pageIndex] = true;
-        PerProcess<Scavenger>::get()->schedule(IsoPageBase::pageSize);
+        Scavenger::get()->schedule(IsoPageBase::pageSize);
         return;
     }
     BCRASH();
index 9f39a75..696d22e 100644 (file)
@@ -41,7 +41,7 @@ IsoHeapImplBase::~IsoHeapImplBase()
 
 void IsoHeapImplBase::addToAllIsoHeaps()
 {
-    PerProcess<AllIsoHeaps>::get()->add(this);
+    AllIsoHeaps::get()->add(this);
 }
 
 void IsoHeapImplBase::scavengeNow()
index 2560c1d..2dac0f4 100644 (file)
@@ -32,7 +32,7 @@ namespace bmalloc {
 
 void* IsoPageBase::allocatePageMemory()
 {
-    return PerProcess<VMHeap>::get()->tryAllocateLargeChunk(pageSize, pageSize).begin();
+    return VMHeap::get()->tryAllocateLargeChunk(pageSize, pageSize).begin();
 }
 
 } // namespace bmalloc
index 28e005a..dbc0e7e 100644 (file)
@@ -54,7 +54,7 @@ void IsoTLS::scavenge()
 
 IsoTLS::IsoTLS()
 {
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
 }
 
 IsoTLS* IsoTLS::ensureEntries(unsigned offset)
@@ -77,7 +77,7 @@ IsoTLS* IsoTLS::ensureEntries(unsigned offset)
         });
     
     IsoTLS* tls = get();
-    IsoTLSLayout& layout = *PerProcess<IsoTLSLayout>::get();
+    IsoTLSLayout& layout = *IsoTLSLayout::get();
 
     IsoTLSEntry* oldLastEntry = tls ? tls->m_lastEntry : nullptr;
     RELEASE_BASSERT(!oldLastEntry || oldLastEntry->offset() < offset);
@@ -167,7 +167,7 @@ void IsoTLS::forEachEntry(const Func& func)
 {
     if (!m_lastEntry)
         return;
-    PerProcess<IsoTLSLayout>::get()->head()->walkUpToInclusive(
+    IsoTLSLayout::get()->head()->walkUpToInclusive(
         m_lastEntry,
         [&] (IsoTLSEntry* entry) {
             func(entry, m_data + entry->offset());
index c76962a..7f6d777 100644 (file)
@@ -37,7 +37,7 @@ IsoTLSEntry::IsoTLSEntry(size_t alignment, size_t size)
     , m_alignment(alignment)
     , m_size(size)
 {
-    PerProcess<IsoTLSLayout>::get()->add(this);
+    IsoTLSLayout::get()->add(this);
     RELEASE_BASSERT(m_offset != UINT_MAX);
 }
 
index 7d866a3..7ec4f7f 100644 (file)
@@ -96,7 +96,7 @@ BNO_INLINE void* IsoTLS::allocateSlow(api::IsoHeap<Type>& handle, bool abortOnFa
     }
     
     // If debug heap is enabled, s_mallocFallbackState becomes MallocFallbackState::FallBackToMalloc.
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
     
     IsoTLS* tls = ensureHeapAndEntries(handle);
     
@@ -139,7 +139,7 @@ BNO_INLINE void IsoTLS::deallocateSlow(api::IsoHeap<Type>& handle, void* p)
     }
     
     // If debug heap is enabled, s_mallocFallbackState becomes MallocFallbackState::FallBackToMalloc.
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
     
     RELEASE_BASSERT(handle.isInitialized());
     
index fdd9a17..76c2dd1 100644 (file)
@@ -29,6 +29,8 @@
 
 namespace bmalloc {
 
+DEFINE_STATIC_PER_PROCESS_STORAGE(IsoTLSLayout);
+
 IsoTLSLayout::IsoTLSLayout(const std::lock_guard<Mutex>&)
 {
 }
index cebaa29..70465ba 100644 (file)
 #pragma once
 
 #include "Mutex.h"
+#include "StaticPerProcess.h"
 #include <mutex>
 
 namespace bmalloc {
 
 class IsoTLSEntry;
 
-class IsoTLSLayout {
+class IsoTLSLayout : public StaticPerProcess<IsoTLSLayout> {
 public:
     IsoTLSLayout(const std::lock_guard<Mutex>&);
     
@@ -44,6 +45,7 @@ private:
     IsoTLSEntry* m_head { nullptr };
     IsoTLSEntry* m_tail { nullptr };
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(IsoTLSLayout);
 
 } // namespace bmalloc
 
index f9ee8c0..d668fe0 100644 (file)
@@ -65,9 +65,11 @@ struct PrintTime {
     bool printed { false };
 };
 
+DEFINE_STATIC_PER_PROCESS_STORAGE(Scavenger);
+
 Scavenger::Scavenger(std::lock_guard<Mutex>&)
 {
-    BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    BASSERT(!Environment::get()->isDebugHeapEnabled());
 
 #if BOS(DARWIN)
     auto queue = dispatch_queue_create("WebKit Malloc Memory Pressure Handler", DISPATCH_QUEUE_SERIAL);
@@ -165,8 +167,8 @@ inline void dumpStats()
     }
 #endif
 
-    dump("bmalloc-freeable", PerProcess<Scavenger>::get()->freeableMemory());
-    dump("bmalloc-footprint", PerProcess<Scavenger>::get()->footprint());
+    dump("bmalloc-freeable", Scavenger::get()->freeableMemory());
+    dump("bmalloc-footprint", Scavenger::get()->footprint());
 }
 
 std::chrono::milliseconds Scavenger::timeSinceLastFullScavenge()
@@ -230,7 +232,7 @@ void Scavenger::scavenge()
 
     {
         RELEASE_BASSERT(!m_deferredDecommits.size());
-        PerProcess<AllIsoHeaps>::get()->forEach(
+        AllIsoHeaps::get()->forEach(
             [&] (IsoHeapImplBase& heap) {
                 heap.scavenge(m_deferredDecommits);
             });
@@ -297,7 +299,7 @@ void Scavenger::partialScavenge()
 
     {
         RELEASE_BASSERT(!m_deferredDecommits.size());
-        PerProcess<AllIsoHeaps>::get()->forEach(
+        AllIsoHeaps::get()->forEach(
             [&] (IsoHeapImplBase& heap) {
                 heap.scavengeToHighWatermark(m_deferredDecommits);
             });
@@ -329,7 +331,7 @@ size_t Scavenger::freeableMemory()
         }
     }
 
-    PerProcess<AllIsoHeaps>::get()->forEach(
+    AllIsoHeaps::get()->forEach(
         [&] (IsoHeapImplBase& heap) {
             result += heap.freeableMemory();
         });
@@ -339,7 +341,7 @@ size_t Scavenger::freeableMemory()
 
 size_t Scavenger::footprint()
 {
-    RELEASE_BASSERT(!PerProcess<Environment>::get()->isDebugHeapEnabled());
+    RELEASE_BASSERT(!Environment::get()->isDebugHeapEnabled());
 
     size_t result = 0;
     for (unsigned i = numHeaps; i--;) {
@@ -348,7 +350,7 @@ size_t Scavenger::footprint()
         result += PerProcess<PerHeapKind<Heap>>::get()->at(i).footprint();
     }
 
-    PerProcess<AllIsoHeaps>::get()->forEach(
+    AllIsoHeaps::get()->forEach(
         [&] (IsoHeapImplBase& heap) {
             result += heap.footprint();
         });
index 5962fc7..d52a0de 100644 (file)
@@ -28,7 +28,7 @@
 #include "BPlatform.h"
 #include "DeferredDecommit.h"
 #include "Mutex.h"
-#include "PerProcess.h"
+#include "StaticPerProcess.h"
 #include "Vector.h"
 #include <chrono>
 #include <condition_variable>
@@ -40,7 +40,7 @@
 
 namespace bmalloc {
 
-class Scavenger {
+class Scavenger : public StaticPerProcess<Scavenger> {
 public:
     BEXPORT Scavenger(std::lock_guard<Mutex>&);
     
@@ -112,6 +112,7 @@ private:
     
     Vector<DeferredDecommit> m_deferredDecommits;
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(Scavenger);
 
 } // namespace bmalloc
 
diff --git a/Source/bmalloc/bmalloc/StaticPerProcess.h b/Source/bmalloc/bmalloc/StaticPerProcess.h
new file mode 100644 (file)
index 0000000..794b069
--- /dev/null
@@ -0,0 +1,106 @@
+/*
+ * Copyright (C) 2014-2019 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "BInline.h"
+#include "Mutex.h"
+#include "Sizes.h"
+
+namespace bmalloc {
+
+// StaticPerProcess<T> behaves like PerProcess<T>, but we need to explicitly define storage for T with EXTERN.
+// In this way, we allocate storage for a per-process object statically instead of allocating memory at runtime.
+// To enforce this, we have DECLARE and DEFINE macros. If you do not know about T of StaticPerProcess<T>, you should use PerProcess<T> instead.
+//
+// Usage:
+//     In Object.h
+//         class Object : public StaticPerProcess<Object> {
+//             ...
+//         };
+//         DECLARE_STATIC_PER_PROCESS_STORAGE(Object);
+//
+//     In Object.cpp
+//         DEFINE_STATIC_PER_PROCESS_STORAGE(Object);
+//
+//     Object* object = Object::get();
+//     x = object->field->field;
+//
+// Object will be instantiated only once, even in the presence of concurrency.
+//
+template<typename T> struct StaticPerProcessStorageTraits;
+
+template<typename T>
+class BEXPORT StaticPerProcess {
+public:
+    static T* get()
+    {
+        T* object = getFastCase();
+        if (!object)
+            return getSlowCase();
+        return object;
+    }
+
+    static T* getFastCase()
+    {
+        using Storage = typename StaticPerProcessStorageTraits<T>::Storage;
+        return (Storage::s_object).load(std::memory_order_relaxed);
+    }
+
+    static Mutex& mutex()
+    {
+        using Storage = typename StaticPerProcessStorageTraits<T>::Storage;
+        return Storage::s_mutex;
+    }
+
+private:
+    BNO_INLINE static T* getSlowCase()
+    {
+        using Storage = typename StaticPerProcessStorageTraits<T>::Storage;
+        std::lock_guard<Mutex> lock(Storage::s_mutex);
+        if (!Storage::s_object.load(std::memory_order_consume)) {
+            T* t = new (&Storage::s_memory) T(lock);
+            Storage::s_object.store(t, std::memory_order_release);
+        }
+        return Storage::s_object.load(std::memory_order_consume);
+    }
+};
+
+#define DECLARE_STATIC_PER_PROCESS_STORAGE(Type) \
+template<> struct StaticPerProcessStorageTraits<Type> { \
+    using Memory = typename std::aligned_storage<sizeof(Type), std::alignment_of<Type>::value>::type; \
+    struct BEXPORT Storage { \
+        BEXPORT static std::atomic<Type*> s_object; \
+        BEXPORT static Mutex s_mutex; \
+        BEXPORT static Memory s_memory; \
+    }; \
+};
+
+#define DEFINE_STATIC_PER_PROCESS_STORAGE(Type) \
+    std::atomic<Type*> StaticPerProcessStorageTraits<Type>::Storage::s_object { nullptr }; \
+    Mutex StaticPerProcessStorageTraits<Type>::Storage::s_mutex { }; \
+    StaticPerProcessStorageTraits<Type>::Memory StaticPerProcessStorageTraits<Type>::Storage::s_memory { };
+
+} // namespace bmalloc
index af966a6..5c8e28a 100644 (file)
@@ -29,6 +29,8 @@
 
 namespace bmalloc {
 
+DEFINE_STATIC_PER_PROCESS_STORAGE(VMHeap);
+
 VMHeap::VMHeap(std::lock_guard<Mutex>&)
 {
 }
index 41a173b..47a113d 100644 (file)
@@ -31,6 +31,7 @@
 #include "HeapKind.h"
 #include "LargeRange.h"
 #include "Map.h"
+#include "StaticPerProcess.h"
 #include "Vector.h"
 #if BOS(DARWIN)
 #include "Zone.h"
@@ -44,12 +45,13 @@ class Heap;
 
 typedef enum { Sync, Async } ScavengeMode;
 
-class VMHeap {
+class VMHeap : public StaticPerProcess<VMHeap> {
 public:
     VMHeap(std::lock_guard<Mutex>&);
     
     LargeRange tryAllocateLargeChunk(size_t alignment, size_t);
 };
+DECLARE_STATIC_PER_PROCESS_STORAGE(VMHeap);
 
 } // namespace bmalloc
 
index b2f0fa1..e541e0f 100644 (file)
@@ -29,6 +29,7 @@
 #include "FixedVector.h"
 #include "Mutex.h"
 #include "Range.h"
+#include "StaticPerProcess.h"
 #include <malloc/malloc.h>
 #include <mutex>
 
index cccb594..59b6642 100644 (file)
@@ -95,12 +95,12 @@ void scavenge()
     if (DebugHeap* debugHeap = DebugHeap::tryGet())
         debugHeap->scavenge();
     else
-        PerProcess<Scavenger>::get()->scavenge();
+        Scavenger::get()->scavenge();
 }
 
 bool isEnabled(HeapKind)
 {
-    return !PerProcess<Environment>::get()->isDebugHeapEnabled();
+    return !Environment::get()->isDebugHeapEnabled();
 }
 
 #if BOS(DARWIN)
@@ -109,7 +109,7 @@ void setScavengerThreadQOSClass(qos_class_t overrideClass)
     if (DebugHeap::tryGet())
         return;
     std::unique_lock<Mutex> lock(Heap::mutex());
-    PerProcess<Scavenger>::get()->setScavengerThreadQOSClass(overrideClass);
+    Scavenger::get()->setScavengerThreadQOSClass(overrideClass);
 }
 #endif
 
@@ -132,7 +132,7 @@ void decommitAlignedPhysical(void* object, size_t size, HeapKind kind)
 void enableMiniMode()
 {
     if (!DebugHeap::tryGet())
-        PerProcess<Scavenger>::get()->enableMiniMode();
+        Scavenger::get()->enableMiniMode();
 }
 
 } } // namespace bmalloc::api
index d71dc1b..0b7c742 100644 (file)
@@ -73,7 +73,7 @@ static std::set<void*> toptrset(const std::vector<void*>& ptrs)
 
 static void assertEmptyPointerSet(const std::set<void*>& pointers)
 {
-    if (PerProcess<Environment>::get()->isDebugHeapEnabled()) {
+    if (Environment::get()->isDebugHeapEnabled()) {
         printf("    skipping checks because DebugHeap.\n");
         return;
     }
@@ -90,7 +90,7 @@ static void assertEmptyPointerSet(const std::set<void*>& pointers)
 template<typename heapType>
 static void assertHasObjects(IsoHeap<heapType>& heap, std::set<void*> pointers)
 {
-    if (PerProcess<Environment>::get()->isDebugHeapEnabled()) {
+    if (Environment::get()->isDebugHeapEnabled()) {
         printf("    skipping checks because DebugHeap.\n");
         return;
     }
@@ -106,7 +106,7 @@ static void assertHasObjects(IsoHeap<heapType>& heap, std::set<void*> pointers)
 template<typename heapType>
 static void assertHasOnlyObjects(IsoHeap<heapType>& heap, std::set<void*> pointers)
 {
-    if (PerProcess<Environment>::get()->isDebugHeapEnabled()) {
+    if (Environment::get()->isDebugHeapEnabled()) {
         printf("    skipping checks because DebugHeap.\n");
         return;
     }
@@ -123,7 +123,7 @@ template<typename heapType>
 static void assertClean(IsoHeap<heapType>& heap)
 {
     scavengeThisThread();
-    if (!PerProcess<Environment>::get()->isDebugHeapEnabled()) {
+    if (!Environment::get()->isDebugHeapEnabled()) {
         auto& impl = heap.impl();
         {
             std::lock_guard<Mutex> locker(impl.lock);
@@ -131,7 +131,7 @@ static void assertClean(IsoHeap<heapType>& heap)
         }
     }
     heap.scavenge();
-    if (!PerProcess<Environment>::get()->isDebugHeapEnabled()) {
+    if (!Environment::get()->isDebugHeapEnabled()) {
         auto& impl = heap.impl();
         std::lock_guard<Mutex> locker(impl.lock);
         CHECK(!impl.numCommittedPages());