My girlfriend and I were having unprotected sex and I started feeling my semen build up and I immediately pulled out, moved away and started masturbating and ejaculated no where close to her. I nowhere felt like ejaculating inside of her and I had to masturbate for a quite few seconds or so to even ejaculate outside.
We're kind of worried although I am sure I didn't ejaculate inside of her. Should we be worried?

The time to worry is the time when you say no to Mr. Condom.

Sperm doesn't just come out when you ejaculate. It can happen well before that.

Pre-ejaculate can contain some sperm, although a small amount or none. You don't know when that is even happening. It can be a few drops dribbling into her, and neither of you feels it.
The trouble is, you could be one of the men who does have a little sperm in his pre-ejaculate, OR you could be mistaken about building up inside of her and pulling out in time. There's just no way to study this question, which comes up all the time. The amount of sperm in many thousands of men's pre- and regular ejaculate has been studied, of course, but pregnancy rates are calculated by what pregnant women report as their method of birth control. All that statisticians know is that there is a fairly high rate of pregnancy using this withdrawal 'method' of birth control. Whether that includes men who lie to women, only the men know, so it can't be studied easily.

There is a chance that any act of intercourse can lead to pregnancy. Protected or not.

Yes, you should be worried, there is chance of pregnancy. Even with a condom, there is a 3 to 18 percent chance of failure, no condom increases that greatly.

Plenty of sperm in the early fluids.

"Pre-ejaculate (also known as pre-ejaculatory fluid, preseminal fluid, or Cowper's fluid, and colloquially as pre-) is the clear, colorless, viscous fluid that emits from the urethra of a man's penis when he is sexually aroused. It is similar in composition to semen, but has some significant chemical differences. The presence of sperm in the fluid is debated. Research has found no or low levels of sperm in pre-ejaculate, though these existing studies are non-generalizable due to examining small numbers of men. A contrary, yet non-generalizable study that found mixed evidence, including individual cases of a high sperm concentration, has been recently published. Pre-ejaculate is believed to function as a lubricant and an acid neutralizer. The amount of pre-ejaculate emitted varies widely between individual men. There are some cases where men do not pre-ejaculate."
- wikipedia