I have hot water baseboard heat with an oil burning system. The pressure on the system had dropped to about 2 PSI. I added water to the system from the filler valve and got the pressure back up to 12 PSI. I still do not have heat to the second floor. What do I do to get heat up there in the bedrooms?

Now you will need to bleed the upper section and keep an eye on the water pressure so it stays at 12.

How do I bleed the upper section? Is this done at the heater or is it done at the baseboard unit? I've never done this before so I have no idea what I am doing.

First you have to locate the bleeders. They can be located anywhere but many times they are located near or on the baseboards. Also you will need to keep an eye on the auto fill valve if so equipped and if not the manual fill valve.

I suggest you do a Google search for bleeding a boiler etc since any and all info is on the net.