I know what avagadro's number is, but don't know its physical significance

Avagodro's number is the number of atoms or molecules in a mole of a substance. Hope that is the answer for which you were looking.

That just begs the question - why is a mole set at 6.022x10^(23) units? Here's the answer - Avogadro' s constant was initially defined to be the number of atoms in 1 gram of hydrogen.

One gram of hydrogen would indeed be 6.023x10^23 atoms but would only be 1/2 mole since hydrogen doesn't exist as H but H2. Is that not correct?

You are right that one mole of H atoms makes up 1/2 mole of H2. That's why I specified one mole of H atoms is 1 gram, which is the same as saying 1/2 mole of hydrogen molecules is 1 gram.

Actually the more modern definition has set the mole to equal the number of atoms of Carbon-12 that makes up 12 grams.