At school in the 1950s a teacher read a book to the class about some children who discovered a priests hole in the house they were staying in. This priests hole lead down to a beach that smugglers used.

Possibly one of The Famous Five series. Try Five Go to Smuggler's Top.