My brother in law loaned his dad money to pay off the mortagage 40 years ago. According to him he never got the money back. His father died 4 years ago. Does he have any recourse against his brother and sister to pay him back?

It's out of Statute - WAY out of Statute.

Was the loan filed against the real property in any way - a lien, a mortgage, anything?

Why does he think the brother and sister are obligated to pay off the debts of the late father?

IF the estate has not been completed, he might put in a claim against the estate if he has some documentation that the money was a loan.