I watched this movie as a child with my brother. It was supposed to be scary but it was apparently not so much so that we couldn't watch it as children. I remember they (2 men) went by boat or canoe to an island (possibly a lady swam there at the same time?) and there were a few inhabitants, that were zombies? There was a large mansion with a portrait painting of a lady on the stairwell. I do remember it being funny, with jokes often. All characters were possibly on a cruise to begin the story, not sure how the island/mansion came into play. If anyone has a link to a scene I will know immediately if that is it because the title and actor/s are all just on the tip of my tongue!

I only just came across your post but I don't know if you're still looking for the answer to your question after all this time. The movie is called "The Ghost Breakers" (1940) starring Bob Hope and Paulette Goddard. This is not to be confused with "The Cat and the Canary" (1939) also starring Bob Hope and Paulette Goddard with a similar inheritance of old mansion premise. As you can imagine, with Bob Hope they were both comedies, but at the same time quite scary when I was watching them as a kid - LOL. In the Ghost Breakers, Paulette Goddard goes by ship to inherit her house in Cuba and does indeed swim to the island, while Bob Hope and his black valet go by boat, where they are followed by the zombie. Both great old B&W movies worth watching again, with excellent quality downloads easy to find online.