Can the landlord keep some of the renter's personal property for back rent that equals more than what the back rent is?

No. I'm not even going to ask which State because it's illegal everywhere.

Well, almost everywhere.
Landlord's lien - Wikipedia, the free encyclopedia

A lien is an encumbrance against property, not ownership of it.

Yes its illegal to keep their property without a court order giving permission to do so. If the tenant owes you back rent you sue them as part of an eviction order, and ask the court for a writ of execution to seize personal property. It may or may not be granted depending on local laws.

And, if the property seized is worth more then the amount owed, you can sell the property and then must return any difference (less costs).