I have 2-8ft 2000 watt (8.3 amps each, total 16.6 amps) heaters wired to a 20 amp breaker with no problems.
But since each of the heaters are in different rooms, and one room is bigger (where the thermastat is) I need to change this one baseboard to a 2500 watt 8 ft 10.1 amp unit to equal out the temp in the two rooms.

My question is will my 20 amp breaker, and 12 ga wiring handle this load. I believe it will, but with winter coming methinks with a continuous load, this may be a problem, what do you think??

Electric heat is considered a continuous load regardless. So the circuit MUST be figures at 125%. Even your 16.6A is not compliant.
The .6a is not anything I'd loose sleep over, but 18.4 on a 20A circuit pretty much is.

You need two circuits.