Put it at the end of a setup, right after the question is asked, then follow it with a dramatic sting to underline the rejection, or open with a surprised reaction before the refusal for a two-beat joke.