Does turning your furnace on and off via the emergency switch save you any money?

My roommate and I have an ongoing argument in which she insists we need to turn the furnace off via the emergency switch in order to save money (and oil). The furnace is old and doesn't register when the house is warm enough, so it runs pretty consistently. We rent and our landlord (nor the previous tenants) notified us of this prior to move in (why would they, I guess). Essentially, I pay for half the heat and I have no desire to be freezing. Nor do I care to pay my landlord for burst pipes or a broken furnace. Last winter, we left the heat at 55F pretty much all the time and when she wasn't looking I would move it to 60F at most. Again, she insists that we turn the entire furnace off via the emergency switch and only turn it on when we're desperate. I don't understand where this logic is coming from (need to save money, I get it) and need to have a concrete answer(s) to combat her above logic. Thanks so much. -Don't Want to Burst Pipes