You can get your sea legs with a 16 ft albacore day tripper without spending a lot. Will get you used to dealing with a jib, tacking, sailboat protocol and safety.
Yes I guess some of the issues are , budget ? I mean a yacht with a crew of 10 is different than one with no one but you and/or family.
Are you going to take classes?
Do you know anything about boating?
Are you buying ?
Have you considered renting and trying different types