When I removed on of the nipples on the of the water heater, the was a black rubber gasket in the hole. Doe it have to be there? The other side does not have one,
I am replacing the nipples with dielectric nipples.

Sounds like the old nipple was a heat trap nipple and gasket was part of that. Discard.