When the car is cold and you go to start it, turn the key to start and it doesn't do anything or it will click once then nothing. But, if you turn the key off and on a couple times it will finally start. When the car is warm it doesn't do the same thing.
Is it the stater or is there a relay that is going bad.

Clean connections to battery and starter motor/solenoid. The link below should help:

https://www.askmehelpdesk.com/cars-t...tml#post235560

It's the starter. We change 4Runner starters (and Tacoma's) by the bucket loads.
What you're hearing is the solenoid clicking. At that point, it normaly make a contact and spins the starter. "The contacts" is the problem. They wear out.
A Toyota reman starter is your best bet in replacment. Trust me on this one.

These battery connections are very susceptible to corrosion and looseness. And the starters fail a somewhat higher rate than some others.