I just recently installed a shower from scratch running new hot and cold copper. The shower is not used often and when you first turn on the shower a stream of rust water comes out for less than a minute.

Any ideas?

Shower heads do not rust nor do your copper pipes. I'd venture to guess you have some older galavized pipes prior to the new work you did. Did you see any galvanized pipe in the wall when you did the job or elsewhere in the house?

Somewhere in the lines before you went to copper there must be some galvanized pipe that is rusty inside, if you can trace these pipes and see what you can find. If you find something just replace it with copper like you use in the rest of the job.