So I asked a guy how are u and he answered not to bad yourself.. Any explanation? And what could be my reply?

How ARE you? Tell him.

It is a way of saying that he is doing okay/well/fine and asking how you are.

Your reply would depend on how you are doing and if you want to start a conversation. It can also depend on how well you know the person and how personal you want to be.

Does it help if it is written out like this: Not too bad, yourself? or Not too bad. Yourself?

Not to bad. Yourself ?

That is what he said.