What if you haven't paid on your car loan because you couldn't afford it and you haven't really received calls from the bank and they still haven't repossed your car? Could you go to jail?

No, there is no debtor's prison. You should be trying to sell the car yourself if you can't afford it. Otherwise they will, eventually, repo the car, sell it at auction (for less than the balnace) and come after you for the balance due.