My boyfriend and I have been together for 4 years and he has never put his hands on me. But one day we were really arguing and it was getting worse, he followed me from room to room. He wouldn't let me leave out of the bathroom and he was blocking all the doors to every room. And he's a very large man so its not easy to move him.

Well while we were arguing I was pointing in his face but I never touched him and I was back at least 3ft away from him, when out of nowhere he pushes me and held me down on the bed by my wrist with all his weight on. And later that night when I was telling how pissed I was and that he will never put his hands on me again, he tells me that it was my fault because I was pointing at him that's why he pushed me.

Is that a sign that I should be worried about the future? But he hasn't done anything like that since that day!

Well its hard to say really if he'll ever lay his hands on you again because that's usually how it all starts .
I'm not saying your partner will continue doing it but what normally happens is when you do something once like that , you'll do it again.
Another thing to be concerned about is that he's saying you caused him to do that. He's putting the blame on you for his actions. Which is totally wrong. That way you may start to think to yourself that you're the one to blame. That you deserved it because you were pointing your finger in his face.
Discussing this with him may cause another argument , maybe speak to a close friend who know the both of you well .

Yes you should be concerned. You should always be concerned, when there is an argument that leads to physical violence of any kind. If it happens once, it can happen again, and it can get worse.

The red flag though is he wouldn't let you get away. Pay attention, if you cannot escape, then things could get really ugly and I think you better run while you can.