Does there exist a lone word for "replying in an annoyed manner", or something to that effect?

A curt reply.?

"Whatever," he said annoyed.

"said" is pretty much invisible to the reader, so use it as much as possible when it is necessary to indicate who is speaking. If the content of the speech doesn't make it clear the speaker is annoyed, then just add that. Don't try to get clever, or you pull the reader's attention from the character to the author.