When picking at it or pressing it with your tongue you cause the pus to ooze and so you notice the odour. Definitely need to see a dentist. Antibiotics are probably in order.
Using hydrogen peroxide as a rinse for extended periods has been linked to increasing the risk of oral cancer especially if you are a smoker and/or drinker. You are best to use the peroxide on a Qtip and gently dab it over the area as a TEMPORARY measure.