When I was 17 a 9 month old child was left with me to "babysit" for two weeks. The mother, my mothers cousin by marriage never asked me to keep the child he was brought to me by one of my aunts. There has never been a formal talk about the future i.e.. Her coming back and trying to take him or leaving him with me. However she did sign power of attorney papers so that I could make decisions for him. He is now 4 and a half years old. She sees him about 3 times a year and has sent me a total of $200. I have been the sole caregiver and provider of him since the day he was left with me. I want to adopt him but I am not sure what to do. I'm hoping there is a way that she can simply sign over her rights. And if not if I should file abandonment or file for custody? Any advice on what steps I need to take would be greatly appreciated. It has been almost 4 years and I am tired of dealing with the stress/threat of knowing she can come take a child I have raised away form me.

No, you will have to get the mother and father to agree to sign over their rights to allow you to adopt.
This really needs an attorney.

At this point you should at least get court appointed guardianship over the child. I doubt the POA will allow you to enroll the child in grade school in a couple years.