Please clarify something - when you filed your NY State income tax returns did you report that both NY State and NY City tax had been withheld? Both the NY State and NY City withheld amounts are shown on your W-2 form each year, so hopefully you reported both numbers on your return. See the attached instructions - I'm asking about the entry you made (or should have made) on line 47 (see page 28 of the instructions). Now, since both the NYS and NYC taxes are calculated on a single tax return, if you properly reported the amounts of both state and city taxes that were withheld, then you're OK - the amounts withheld for both NYC and NYS withholding would have gone to pay just the NY State tax you actually owed. If this is what you did then all is well and there is no need to file amended returns.
However, if you did not report any NY City tax withheld on line 47, then you should file amended returns as ATE suggests. You'll have to include copies of your old W-2's to document that there was NYC withholding that you did not include on your original returns.
http://www.tax.state.ny.us/pdf/2008/inc/it150_201i_2008.pdf