I have two large sink holes in my backyard from where the builders put all their debris
While building the house. We have covered these holes many times through the years but they keep coming back. Do I have to dig down & get all the debris out to keep them from sinking again?

Yep. You don't know how extensive they are and they can be pricey to get rid of. There may be some relief by the county in which you live.

Most builders use dummy corps and dissolve it when the development is finished, hence no recourse.