I don't think people are judging you but only judging your actions. Is anyone perfect no, do anyone do wrong, yes.
The only problem is when you do wrong, you try to correct it by doing right. Getting involve with a married man is wrong but I am assuming he did not tell you this, but his wife did. As soon as you found this out you should have been out the door.
Today is a new day and you still can make this wrong a right by leaving him alone, like you stated. When you do, stay strong and stick to your guns.