My door hinge that attached to the jamb will take a screw, because the wood has
Rotted in the jamb. I know I could relocate the hinge to another part of the door.
I would have to mortise. Another option is to fill in the decaying wood with wood
Dough so the screw would have a better surface.

Any tips??

Any short cut here will not give you much more serviceable time on a shot door frame. If part of the frame is too rotten to hold a screw I'd recommend pulling the whole thing and start new. If you just need a temporary fix try a much longer screw like 3 1/2"

There is another fix. A 1/4 or 1/2" wood screw extractor is rally a hole saw. You drill where the rotten area is an glue in a dowel. Worked for me when a screw in a hinge broke off.

There is an epoxy for just that application, Repairing decaying wood.

I can't give any better advice without knowing the extent of the problem.