It's possible, yes.

Another possibility is an allergic reaction to latex. Did this start happening after sex with a condom?

Either way, you NEED to get to a doctor, ASAP. Swelling labia is NOT normal, and should be looked at by a physician.