I am applying for a job with a company I do not work for.

I get a phone call from a colleague who is interviewing me threatening to 'go to HR' because evidently someone who will be working with me I slept with four years ago and she would feel uncomfortable.

Should I withdraw my candidacy, per her threat? The person who would be working from me is interviewing for this position as well evidently.

I do not work for the company and did not work for the company when this one night stand happened.

Who is 'a colleague who is interviewing me?' How can you have a colleague if you don't work there AND this person is interviewing you?

Anyway, I would give up, if the woman of the one night stand actually does work there already. She can so easily disrupt any chances of you being hired. It is wrong of her, and you could fight it, but is it worth it? (I assume you 'never called her again' and she never forgot.. )