Bought some OSB to put under a new countertop. The fumes are strong. Should I use plywood to fill the 3/4 " space instead?

OSB does have a larger amount of binder adhesive than plywood. Usually the smell goes away. If you are concerned, go with the plywood. The extra $30 may be worth the worry.