This seems to be an on-going issue with the two of you:
https://www.askmehelpdesk.com/relati...nd-399792.html
He gets to play games with his co-worker and act up and you have to toe the line when your friends are playing around.
I think he has a bit of guilty conscience and the banter poked it. Are you getting better at communicating with each other?