I have a hard lump at the side of my vagina, it went away then came back again, should I be worried?

Whether to worry is up to you. There are many possibilities when there's a lump. It could be an ingrown hair, a cyst, a pimple, cancer, an STD, etc. etc.

We can't diagnose online. If you're concerned about this lump, see your doctor for a diagnosis.

Good luck.