I have a 125 amp panel w/ 1- 50 amp, 1- 30 amp and 1- 20 amp breaker
Can I add a 40 amp breaker to feed a sub panel , and also can I use 8/3 cable to feed sub
Panel

Yes you can. 8/3 would be fine.

Can your service handle the extra load? Regardless of what breakers are in the panel, what is the load like on the house? Can you do a load calculation?