I am thinking about installing a tankless hot water heater and was wondering if I could use a "Y" connector to vent the hot water heater with the gas furnace vent?

It is not suggested to do so. About all tankless water heaters need stainless steel venting. Condensation can form from tankless water heaters and rust out your current flue pipe.