What's the age for a mans sperm count to go lower or having a slimmer chance of having kids

Men are not the same as women in that aspect. A man can easily have a child in his 60s or 70s, whereas a woman is most of the time out of luck.

See, women are born with a specific number of eggs (different for every woman) and men are not. Men regenerate their sperm continuously throughout their lives.

So there is no specific age for men as there is for women.