She is telling me she is going to call the police on me if I don't give her back. Will I get in trouble?

So your mom has legal guardianship and you now have the child. If mom says to bring her to mom's house and you do not you will get in a great deal of trouble. Tell us how and why guardianship was given to grandama in the first place

Yes, if she has guardianship though the courts, you need to go gain custody of the child if not, give the child back or you will be in trouble