I don't want to keep lending money to my irresponsible family members, how can I tell them no!

I suppose we can say something like form your lips into an O and say NOOOOO.
But I know it's not easy with relatives sometimes.
I would make a list of what they have borrowed and not paid back and send it to each one, and add a note saying 'no more; I am no longer willing to be taken advantage of.'
You could lie if it helps. Say you had a medical scare recently and decided to sock away most of your funds into extra insurance, long term care, and long term investments that you can't cash out for 10 years.
Say whatever you feel most comfortable saying. I personally would go the 'I am tired of being taken advantage of' route - the truth, right?

Tell them no more money till you are paid back all that they owe you. Add up the money they owe you and give them a bill.

Tell them the bank of Jack is now closed with no future opening date. Full stop