My existing doorbell was wired with 5-colored telephone wire. There is a doorbell button in both the front and back of the house. How do I know which color wire to use on the doorbell? The instructions don't tell me. There are three screws; one for the transformer, one for the front door and one for the back door. Which color attaches to which screw?

If it is a new install, as far as colors I don't see where it would matter. Just as long as they are connected to the correct terminals.

How many pairs of wires are in the sheathing? Ive never seen 5 conductor in a phone line. Its usually in pairs of 2 and is usually 4 cond, 6 cond, etc..

What you are calling telephone wire is probably 5 conductor bell or thermostat wire.
Here is the most common way of wiring a door bell. You pick the colors.