So I had a retaining wall that fell down onto my neighbors yard due to loose soil. I don't have the money to have it fixed right now but I keep getting harassed by the neighbor so now I'm wondering if since it fell on their side, do I even have to fix it or should it b their problem now? Should I get a lawyer?

Not a lawyer, but if it was your fence and it fell on their property, I would believe the most you could be responsible for would be, is to remove any debris?
Good Luck.

You have money to hire an attorney but not fix the wall?

I will assume you went out there that day and cleaned up the wall, took all the debri off their property??

I will assume you have put some type of temp plastic net or something to keep dirt from washing.

But it was your wall, so you need to clean it up and fix it up. They can not make you put it back like it was, but you will be liable to clean up any and all damage to their property

What is the neighbor harassing you about? As noted you have a responsiblilty to clean up the debris and fix any damage caused by the fall.

If they take you to Small Claims they will most likely win, and then you will owe court costs as well. Hire two strong guys with a truck if you can't do it yourself.
If my rusted out junker rolls onto your property, is it your problem now? Of course not. The wall was a man made addition to the land, not soil and stone working it's way down a slope.