You can’t keep him out of her life unless he may be a danger to her. He is her father, and a rubbish father is better than no father. Try to get across to him how important it is that his visits are regular. Tell him to think about his daughter, and if he cares about her he should try not to upset her.
Be open and honest with your young daughter about her dad, and never put him down to her. You must protect your daughter but if you have no worries in that area, you are merely a bystander.
Over the next few years your daughter will come to know her dad and will or will not accept him for who he is. Trust her to make her own decision on the matter when the time comes. You cannot deny her the chance to do that. But you can do everything in your power to convince him to be more reliable for his daughter’s sake.