Sounds like I hit a pot hole in the road when I disengage it out of 4 wheel drive. I would say about 5 seconds after.

I would say you have a problem with your transmission.

Tick

But that is the only time I hear the noise. But that was my first thought.

I think if you are hearing that clunk it is probably beyond transmission fluid, but a major overhaul.

My guess is that your 4X doesn't disengage "immediately" but actually disengages about 5 seconds after you've opted out or 4X.
I've seen bulletins that address driveline noise.
What year is your truck and what is your current mileage?