WeakPtrFactory should populate m_ref lazily.