How do I end the trim on an open corner? Straight cut or miter cut?

There a couple ways you can end it. I like to use a return.
That is when you cut a 45 on the wall piece, cut the opposite 45 on a scrap piece. The same 45's you would use for an outside corner.
On the scrap piece very carefully cut the 45 miter off in the miter box.
Take this little piece with the 45 cut and put it against the 45 on the wall piece.

You can also end the run with a 22.5 miter.