Do you honestly believe there is a future with him, and the way you are treated by his family why would you want to stay in that mess?
You saw him and are missing being with him, but not him. Brush this off and keep going. This guy is not good for you.