I need the co signer remove from the financial part of the mortgage and be solely responsible. How can I do that without refinancing

Unless your lender agrees to the removal of the consigner... which I doubt they would, you only other option is to do a refi.