It didn't get stuck inside completely, but it came off inside me and I'm worried some sperm might have gotten in, should I take plan b?

First of course I am sure you know that even if it did not come off, condoms are not nearly 100 percent effective, so you could be pregnant even if the condom did not come off.

But yes it coming off greatly increases your risk. And the longer from the time of sex you wait, the less effective the morning after pill is