I'm pretty sure that the dirt serves to fill in the air spaces of the old concrete rubble.
How well it's tamped down might be the important part.
Also how solid your ground is under all this in the...