I would have told him to get lost when he fed me the that's the way I am crap. You take way too much crap from him, and you are better than that.
Sorry, but half your battle has been trying to please him because you have high hopes for his potential. Instead what you have is slow torture.