I had the same problem and I just tapped all sides of the handle to loosen a bit. It came off.

I once used an angle grinder (power tool) with a cut-off wheel to remove a badly stuck handle without damaging the spindle or its splines.