I have joint custody and physical custody of my daughter and want to move away.. Can I just pick up and go the mom agrees that if that's what I want I can go.But am I walking into a wall

If you want to cross state lines you need to get approval at court. In the meantime try to get her to notarize that she agrees to show the Judge and have something in your favor if something should happen before you can get to court.
The court will work out a custody order either way.

If the non custodial parent has court ordered visitation and your moving will materially affect that visitation, then you need to get court approval of the move. If the non custodial parent is agreeable to the move then it shouldn't be a problem.

But if you move without getting such approval the non custodial parent could get you in trouble for defying a court order.

If the non custodial parent is in agreement with you moving with the child... just go about it the legal way so you don't get in trouble down the road.