Should I take this to the authorities or not?
It is all out in the open and my brother, mother and father all knew about the allegations back then. It is just that my children never heard me discuss it until my one granddaughter was 3, the same age as my daughter was when I caught him... they were going to let him babysit and I just blurted it out. I thought my children would open up and want their children to be safe, but no... he still gets to babysit and I am excommunicated.

DO something about it now, why post here asking if you should, if that is your question. Get you facts straight and go to Child Protrection Services in your area.