My son has lived with me his whole life. Would go visit with his father when his father had the time. There would be months he wouldn't see him. He never calls. We have had our fights about this. Him not calling to check on his son or just to tell him "I love you"

Well we came to an understanding that we do the every other weekend deal. Which is fine, I'm glad he is trying to build that father son bond. I went through the state and he is ordered to pay me child support.

My son went there for christmas and the family vacation and I was suppose to get him back today but they are telling me NO...
They are saying I am a bad mother because I work at a liq store, smoke and currently living with my parents because I am re-doing the whole inside of our house. I only have so much money to do what I am doing so it's taking me a little time getting our house ready. My son goes to a private school, has Toy-R-Us in his bedroom, he wants for nothing. Since he was born I have kept journals of when his father called, kept him for the weekend, stuff like that.
What should I do??

You need to go to court. I don't know what to do to get him back asap but they can not judge that you are an unfit mother with the things you have said. I have seen drug addict mothers that the court does not take the kids off as being unfit so I don't think their interpretation of unfit will work in court.
You need to go to court and get a primary custody order.

I'm going talk to a lawyer first thing in the morning. They said they are enrolling him in school tomorrow but they can't because they have none of his legal documents