I would certainly argue that, since the town failed to properly enforce the permit requirement against the previous owner, it is estopped from pursuing you. After all, you did not know of any violation and, apparently, there is no inquiry notice requirement. Take that argument to an attorney in your jurisdiction and see what he/she thinks.