I know all parts of oleander plants are poisonous. Would lots of fallen oleander flowers ruin a garden plot for growing food?

I wouldn't have anything to do with this plant near a food lot. It is highly toxic.

Oleander ranks with yew in toxicity. Oleander contains a poisonous principle similar to digitalis in its effect on the heart, causing arrhythmia and cardiac arrest. The effects are reversible and the horse may recover if he ingests a less-than-lethal dose (a single ounce of oleander leaves can kill a 1,000 lb. horse)