What would happen if a person transferred a deed for 12acres of land; but had signed over rights to 1acre of said land that has not been recorded at the registry of deeds as of yet which would make this persons land only 11acres?

Both parties would have to sue it out in court, and depending on the type of deed they used, there can be serious civil court issues.
And depnding on why they did it, there could even be criminal issues if there was fraud involved.

No, there deed is still for the 12 acres, but they will have to have a court decide the ownership of the disputed one acre

Transfer of title to real property takes place upon delivery of the deed to the grantee. While recordation is always advised, its absence will not nullify the transfer. If a person gave a quit claim deed for 12 acres, even though he only owned 11 of them, it might be no big deal, as I could give you a quit claim deed to the Golden Gate Bridge, provided that I didn't charge you for it. If I charged you for it then I would be committing fraud. Same thing with selling 12 acres even though only 11 were still owned by the grantor.