It could be lots of things. Maybe your 5yr old doesn't want to have the little one tagging after him and while he is there, its move likely worse that normal, that would cause me to want to stay home no matter how wonderful the place was. Maybe it just being there and you aren't. When my oldest son was 5 he wouldn't stay anywhere unless I was with him.
My gut says to tell you, you should just go for the day or stay one night, but stay with them. If that's not something you can/want to do then maybe leave one this month and the other next month. That might be best for all. Granma gets one on one with each child and so do you.
I hate to ask but if the woman is sick should she be taking care of two very small kids all weekend? Its not a questions of if she wants to, but should she. Is she on meds, can she still drive, there are lots of questions but I'm sure that you already asked them.
I wouldn't force my child to do anything like this. Only forcing them makes things ten times worse for everyone and with Granma being sick I'm sure that the last thing she needs is a tantrum from a 5yr old and the stress of him not wanting to be there.