Your cat's behavior is telling you that he doesn't want to be put outside!!!
Cats naturally use the litter box. It's a life skill they learn from their mothers as kittens. With a litter box, the cat can go to the bathroom when it needs to, in private, and can burry the evidence following natural instincts.
Even if you taught the cat that it was okay to use some part of your back yard as the litter box, I don't think you could control when it needs to go. If the cat needs to go, and can't get outside or get to a litter box, your going to find more of those little gifts and stains around the house, and that will really smell bad.
Can you put the litter box in your basement? That way the cat has the litter box and you don't have to smell it. One cat won't produce too much waste, so it would be minimal effort on your part to scoop it every night or two.