I'm putting tile in my master bath so I pulled the toilet off... so my wife took a shower in the other bathroom and the next thing I know there is a steady stream of water dripping from my ceiling in the basement... so I ripped it down... I did not plug the toilet I removed... could the vent in the bathtub overflowed?. or do I need to open up walls because there is a bigger problem?

The vent of the bathtub is not a vent. It is a tub overflow. It is there just in case you forget to turn off the water when filling the bath tub. It is just a pipe that that connect to the tub drain. So no, the vent of the bathtub cannot overflow.

You should plug the toilet pipe to keep construction debris out and to prevent sewer gasses from entering the house.

Use of the shower and especially use of a shower in another bathroom should not cause water to come out the open toilet pipe unless you have a blockage in the main sewer line. If the water did come out the toilet pipe it should be immediately obvious by the water on the floor surrounding the toilet flange.

If you have already pulled down the ceiling below you should see signs of where the water came from.

If you can't see where the water came from, I suggest that you turn the shower on again and watch for water in the basement ceiling.