I had to evict a friend who lived with my ill mother, my son and myself for about 1 year due to dishonesty and drug use. Although I told her she could get her belongings as long as I was home, she showed up with the police to get a "change of clothes". At that time she gave me her house key, and a key to our p.o. box. Over the following week, I neatly boxed her things and stacked them downstairs for her to pick-up, although there were still a few clothing items yet to pack in her closet. She never called me to make arrangements, but showed up with her sister and some large woman I didn't know and used a key she had retained without my knowledge, to barge in my house and physically pushed me aside, as did the stranger, to go to her old room. I had a 13 yr. old girl staying in that room and they still barged in and began tearing through everything, looking for her stuff which was mostly downstairs, packed. They both verbally and physically threatened me and caused great distress to my mother who has heart disease. I called the police, but they did not arrive before they grabbed all they could and left in a u-haul.
We had never charged this woman rent as she was a friend and helped care for my mother, but our landlord knew about her living here. Do I have any legal recourse for the key retained without my knowledge and used without my permission, and her physical and verbal assault on my household?

You can call the police and file a breaking and entering charge against her and her friends. She did, unlawfully, enter the premises.

But I don't see any other legal recourse. Frankly, I wouldn't have trusted her to have handed over all the keys. I would have changed the locks as soon as she was legally removed.

In my opinion... I doubt it. Unfortunately verbal abuse won't go far in court... even physical abuse... you could get a restraining order, but really it's a big pain for no real protection. Its too much of a hassle.
I would make sure nothing was stolen. Then CHANGE THE LOCKS ASAP. If you don't have deadbolts... put some in. be sure to get the kind that use a key on BOTH sides, and keep the key far enough away from the door that it can't be reached through a broken window.

Be ready to call the police if she shows again... but other than that. Just hope that she doesn't come back.

Double deadbolts are illegal in many places, because they can trap people in a burning building unless you leave the inside key in them. And if you are going to leave the inside key in it, then there's no real reason not to get the one that just has the handle on the inside.