All it means is you were down stairs and he was up stairs,
I guess instead of getting mad why you did not just join him at the time. You are making way to much out of this,