I know it's for numbers, but I can't seem to remember the correct name for it.

The "pound" sign... Or the "Hash" Sign. :)

Number sign - Wikipedia, the free encyclopedia (http://en.wikipedia.org/wiki/Number_sign)

Beginning in the 1960s, telephone engineers have attempted to coin a special name for this symbol, with variant spellings including octothorp, octothorpe, octathorp, octotherp, octathorpe, and octatherp;[2] none of these have become widely accepted.

In addition to what solidzane said.

The official name is the octothorpe. The name was assigned by a Bell labs technician working on the first touchpad phones.

However, this name has never taken hold with the general public. It is usually called the pound sign or hash sign, sometime the tictactoe symbol.