Wow! That's something new, OK well talk to him! Be like woah I bearly get to see and talk to you during the day and I can't even go with you to the gym? Well why can't I go with both of you? I want to work out with you and talk to you during the time we are doing the workout together... talk to him about it, if cares for what you have to say, hey he will listen!