Can my girlfriends ex husband try an take her son away if we move in with each other and we aren't married

Where do all of you live (country, state)?
Is there any unusual reason, (not being unmarried), why your ex would try to claim custody (your boyfriend has a criminal background, etc.)?

We live in the united states in Georgia.. but my boyfriend does have felonies for marijuana that are 6 years in his past now.. this is the only reason other then not being married

The only other reason is my boyfriend has felonies for marijauna that is 6 years in his past.. we live in the united states in Georgia

Not unless he can prove current drug use, or a threat to the child. Sadly ( and I really mean that) there are most likely in Georgia more children living in homes with unmarried split families than with both parents

There is with out a shadow of a doubt that there is no threat to the child or any drug use at all he is disabled from a car accident years ago and does not even take pain meds because of it being frowned upon as drug use

What is the official status of the custody/visitation? How old is the son, and how long have you had him?

A lot may depend on how much he spends to try to show that you and your situation are unfit.
So keep your finances in order and documented, your child in good health, your home safe, and don't do anything remotely negligent, from letting him ride a bike without a helmet to leaving him in the car while you dash in to get a coffee. Make copies of any papers concerning prison, disability, and any doctor's statements.

Joypulv-ive had him since birth he is three years old an have joint phisical custody with his father getting him every other weekend and I also let him get him to go to family outtings an when he is rained out frm work so he can spend time with his father as much as he can... so basically you are telling me to walk a staight line and really put a strong point of focus in my boyfriend not doing anything that could give a court bad judgement on his part ever again