I just moved in to a house that has oil heat. We haven't turned on the heat yet but randomly throughout the day a smell emanates from the vents. It makes my whole house smell like a sewer or port-a-potty. At the same time I am having trouble with the downstairs toilet not flushing correctly. Could it be that there is a back up in the pipes and the smell is coming through the vents?

Please describe how the toilet flushes. Does it back up? Does it still upset just swirl around and then go down? Let me know your concern.
I'm going to put up some vent caps and back draft preventer.(See image) these will prevent back drafts from entering your vents and smelly up your house. Good luck, Tom

Whenever the toilet upstairs is flushed the downstairs one bubbles. When the downstairs is flushed it swirls some but then stops. Whatever is in the toilet doesn't go down. That was happening first then a couple days later the smell started. The smell is sporadic.

You have more than one issues. Luckily, there's a salacious above.
Bubbles indicate a partial blockage downstream from the downstairs toilet that must be snaked. Unless you have a basement with a cleanout you will have to pull the toilet and snake from there. That should take care of the bubble.
Now for the "swirly" toilet. I have one that most repair plumbers miss.. Look down at the bottom of the bowl. If there is a small hole, then that is a jet that starts the syphon action. If it's clogged the water will just swirl around and slowly go down leaving solids behind. Take your finger,(UGH! ) and run it around the inside of the opening. Over the years minerals build up and cut down on the syphon (flush) action. If it is rough or you feel build up, take a table knife and put a bend in it to get around the curve in the bowl and chip and scrap it clear. Next take a coathanger and clear out the holes around the rim. They start the swirling action. And last, check the water level in the tank. It should be 1/2" below the top of the over flow tube. And speaking of the overflow tube, Make sure the small 1/8" tube from the ballcock to the white overflow tube is connected so it discharges in it and that it's flowing when the ballcock fills. This is what raises the water level in the bowl. For a good solid flush they all have to work together. Hope this helps. Good luck, Tom