SWServerToContextConnection / SWServerWorker do not need to be ThreadSafeRefCounted