Optimize the panel attach method to no append if there is a