I want to loan my son more than the annual gift exclusion to buy his estranged wife's share of their house. If I file a form"709" will it be taxed as a gift or will I not be taxed until the amount is over $1000000?

Yes, you can give a loan of up to $100,000 interest free or at below market interest rate. However the loan must not be used for investment for generating income.
If your son invests the money for generating income, you have to report market rate interest income up to the maximum of his investment income. You son can take income deduction.

Thank you. I read the IRS may consider this a gift subject to gift tax, is that true? Also he, and the house he would use the money for are in Canada. Although he has dual citizenship,he has not filed US taxes- does that matter? An alternative I thought of is gifting the money in his two children's namesfor their education. (Within ten years my son plans to sell the house to fund the kids' college education).

If the loan is properly drawn up with a written contract that specifies terms for repayment and a "reasonable" interest rate then it is a loan (not a gift) and no need to file form 709. However, if the terms of the loan are too generous or vague it may be considered a gift, in which case you would have to file form 709. No actual tax would be due on this gift however - you don't pay gift taxes until your total gifts in excess of the annual exclusion gifts given over your lifetime exceeds $5M.

The fact that your son lives in Canada does not matter, at least with respect to any US taxes. I can't speak to any Canadian tax implications he may face.

I don't see how you providing money for an education account on behalf of his children would help here - your son needs cash, and he cannot use his children's money for his own purposes.