If She is there for more then a period with consent with me is there any way he can try to file for abandment in the state of New York

Do you already have a visitation agreement in place with the Courts? If not, then anything can happen.

Tick

No. See this sticky for what constitutes abandonment. https://www.askmehelpdesk.com/family-law/child-abandonment-read-first-364259.html

However, if you have no court ordered custody/visitation then he has joint custody and doesn't have to return her without a court battle. So you need to formalize things in the courts.

The person who HAS the child controls the child unless there is a Court Order. "He" takes her, "he" controls the situation.

Get a Court Order.

There is a court order for visitation I was trying to let him spend extra time with his daughter and wasn't sure if I should because of any laws that he could try to keep her

No, You can always allow him more time if you wish, just not less. If there is a visitation order that means you have primary physical custody. So if he refused to return her, you go to the police with a copy of your custody order and he will be forced to return her.

... if he refused to return her, you go to the police ...

The police have a stock answer for that kind of situation:

"It's a civil matter."

The police have a stock answer for that kind of situation:

"It's a civil matter."

I cannot agree more.

The police have a stock answer for that kind of situation:

"It's a civil matter."

Sometimes, not always. A lot depends on how much documentation the person has.