I have ice build up around the inside of my windows, does this mean that outside air is getting in?

More than likely this is the air that is trapped inside your house is condensing on the windows and freezing. You could have some cold air leakage around the windows as well. You need to recaulk around your windows for any air leaks.