I paid up front to a dentist but I do not continue the service, can he hold my money, or release money only under a condition?

It depends on what you paid for and why you discontinued. If you paid money so he could make crowns or plates, then you can't get it back, even if you decided not to get the crowns etc.

If it was payment in advance for something like dental check-ups that you decided against, you have a good case for getting it back.

Did you sign any form of contract or form when you paid the money?