The mortar is well intact but looks real grungy/dirty

Get some TSP and mix it per the package. Get some heavy duty neoprene gloves and a very stiff bristle brush Start scrubbing. This stuff will eat up your hands so be sure to wear gloves.

ma0641 gives good advice but test a small spot out of the way. Once dried the cleaned area may not long significantly better.