Yes... but that has nothign to do with the wall breaking. You hit the board with a force greater than the materials strength. Thus it breaks.
The material exerts the same but opposite force on your hand, but your hand doesn't break because it's stronger..
I don't understand what you're confused about
