I see. Yuck. You've got quite a chore ahead of you.
I find a couple suggestions:
Here a torch and razor are suggested, but mention is also made of a stripper made by Jasco.
Here's a good link, too:
http://www.concretenetwork.com/concr...e_staining.htm
There we read
"Bob Harris, president of the Decorative Concrete Institute and author of Bob Harris’ Guide to Stained Concrete Interior Floors, says caulking compound and mastic are two of the most difficult substances to remove. He scrapes off as much material as possible using a putty knife or floor scrapper and then applies a poultice to remove the remainder. For a poultice, he recommends mixing an inert fine powder, such as fly ash or hydrated lime, with denatured alcohol to make a smooth paste. Once the poultice dries, the caulk or mastic residue usually is brittle enough to remove with a stiff-bristle brush."
...Grinding is the "last resort" optioned there.