Place it at the exact moment of the bounce or collision in your edit, not before the setup. Follow it with a short bonk or a fail sting when the gag turns into a mistake, and keep it as the only effect on that beat.