If I get a state tax refund, it may be subject to federal tax the next year. If instead I ended up paying additional state tax when I file my state return, would I be able to enter that amount as a negative on line 10 of Fed 1040 for the following year? If not, it seems that you would always want to make sure you withhold more state tax than you will actually owe.

No - line 10 is for the refund only, and should match the information on the 1099-G form you received. What you can do is enter the state tax you've paid in 2011 on line 5 of schedule A, which ultimately carries to line 40 of Form 1040 and hence reduces your taxable income. If you don't itemize this doesn't help, but in that case you shouldn't be reporting the state tax refund as income in the first place. Instructions for line 10 state: "None of your refund is taxable if, in the year you paid the tax, you either (a) did not itemize deductions, or (b) elected to deduct state and local general sales taxes instead of state and local income taxes."