When one of us showers in the upstairs master shower, if the next person wait about 3-5 minute to get in the shower, the water temperature will begin at the original cold temperature and take just as long to warm up as the first shower of the day?
I have had shower take a little time to warm back up, but I have never seen it take the original amount of time as if it were the first shower of the day!

How long has this been going on? Sounds like you have a cross connection or cross flow from cold to hot somewhere. This can happen in any single lever faucet or valve, but is rare. My favorite place to look is in a clothes washer. If a the mixing valve is defective, cold water to the washer can be drawn into the hot water pipe through the defective mixing valve when hot water is used at another faucet, say a kitchen sink.

To test this hypothesis, close one of the stop valves at the washer, either hot or cold. If the mixing valve is defective the closed stop valve will prevent the cross flow.