Avoid doing an extra layout in some cases while doing scale-to-fit