How to prevant oil sucked from oil chamber of vacuum booster during operation?

Boosters in a vacuum system can refer to mechanical (Roots) blower or an oil diffusion pump and the source of oil back-streaming is different for the two cases.

If the oil is coming from a blower, it is likely a seal failure. The gear housing of these pumps are held under a vacuum so you don't have a (large) differential across the rotating shaft seal. When an external seal fails (usually on the drive side of the pump) air leaks into the oil-filled gear box and pushes oil through the shaft seal into the vacuum chamber.

If the oil is coming from a diffusion pump, you may also have a leak in the pump foreline. Leak check the system while it is hot since it may seal itself when it cools. If it isn't an air leak, check that the oil level is correct; that the water cooling lines in the condenser and chamber are not clogged, and the water is flowing properly; and finally make sure the internal baffles in the pump are properly installed--they make a cascade of oil vapor jets all pointing downward to the sump.

Make sure that your process vacuum pressure is not lower than the vapor pressure of the vacuum pump oils. In some metal processes, like titanium melting, the melt absorbs oxygen and nitrogen. With a mechanical pump system oil vapor wants to flow toward the melt rather than the mechanical pumps.

If all else fails install a refrigerated cold plate between the pump and the vacuum chamber.

Good luck!