My wife and I made an offer on a condo which we wanted to buy as a 2nd home. The property has been forclosed on and is owned by a bank.
We put down 1% ernest money and the contract says that this is non-refundable (with the standard expections like can't obtain mortgage etc which does not apply to us)

So what will happen to that money if we walk away and do not close? We are having second thought due to numerous reasons.

Does the escrow agent get to keep it?

Yes, the earnest money will be retained.

The bank gets it, to compensate them for the loss ( the other party in the real estate contract)

The bank gets to keep the money if you back out, it's their "insurance" and to compensate for the time that the house was "not sellable" because you were under contract with it. However if you decide to continue and close, it will be applied toward YOUR closing costs.