I read a book about a lady prisoner who was shipped from home to an isolated island. She got pregnant in the prison and finally lost the child. I think she has a younger sister or a child who was a girl but was left back home. A widower came to the prison to get her to breastfeed the son.

In fact he owns her as a slave. Eventually, he slept with her and she became pregnant. The man later married her but it was not too accepted by the society because she was just a slave. The man later sent for her child or sister to be shipped down.

In that novel he has other slaves that worked for him and I think the island was where they sent rogues or criminal to as servant or prisoners.