I have sole parental rights and resposibilities. My sons father has rights to visit when I see fit. He does not pay me any child support and sometimes goes months without calling. I want to move out of Maine. Do I need his permission?

Yep, you sure do. Since he has visitation anything that may affect it in the long run will need his and the courts permission.

Yes you do need it.

How exactly is the court order that says "when you see fit"
I have never heard of one that gives the other parent control.

As long as the court order says he is entitled to visitation, then you need to get the court's permission to do anything that will affect that court order.