I did leave my ex but I didn't go out with this guy for 3 years after the separation. He didn't go out with anyone too for 2 years after she left him. WE both was alone for years before we started going out.
Ah, good clarification, thank you for that.
... was wondering if it was the whole commitment thing that freaked him out.. or is it that he can't let go of her controlling him.
Well, in either case there's not much you can do, at least nothing to change him. Can you?
In your shoes, I'd put more credence on what he actually does, not what he says he wants to do. If he wants to make you primary in his life, he does that. If he doesn't, then you have to decide YOURSELF how long you'll take second/third string.
You may have the time to wait for him to complete all his other obligations and you move to first slot just because everything else just "finished"... but itf you don't have that time/patience, then waste as little of it as possible wishing things were different and start the moving on process.