I would venture to say that you can't get arrested for kidnapping your children. Since there is no custody order in place - you can leave with your children. Just as he could right now if he wanted. He throws that out there to scare you and to keep you where he thinks you should be. It is an empty threat.

You would be surprised what you can do when you are against the wall. You have to decide what is important - the safety and well being for your children or money. Yes he may provide a roof over your head, but you can do that on your own. Seek out an attorney. Most will give free consults, so you can get some questions answered.
Then check with legal aid and see if you qualify since you are unemployed. You may have to wait - because they are typically backed up - but you could qualify for free legal services. Check it out, get informed on what you can do next.

Don't be a victim - take control of your life and do what is best for you and your kids.