Is this an old wives tale or something I should actually worry about?

I've never tested it, but it's not that they are allergic to them, it's the skin they can't digest and can actually become entangled in their bowels. I know my vet advises against it.

It's that grapes can be toxic to dogs.

This article explains it better than most I have read: Danger for Dogs - Grapes, Raisins: Dog Foods That are Toxic to Your Pets.

It's one of those things that it is safer to keep away from the animal than to be sorry later.

Yep, best not to chance it.

If your dog has eaten a grape though I wouldn't panic. Some dogs are fine with them, others have problems.
It's the same with chocolate, some dogs can tolerate chocolate and be perfectly fine, others get very sick.

Just watch out for signs such as vomiting, lethargy, diarrhea, excessive drooling.