I sent in a payment to a collection agency and they sent it back. Can they do this

I am not sure they can or not but Keep it and the envelope & letter as proof that you attempted to pay.

If you are in default of the agreement and if you and the collection agency does not have a written agreement to make payments, they may refuse any payment that is not payment in full.