Don't subtract too much from nonSilentFramesToProcess