How newton used to calculate the force of gravitation between two objects before value of G was known?

The history of astronomy is filled with attempts to get better and beter estimates of the value of G. You can make a pretty reasonable estimate by studying the motion of the moon around the earth. Taking Newton's Law of Gavitation, and setting it equal to the centripetal force on the moon as it orbits the earth (assuming a circular orbit), you get:


\frac { G M_e M_m} {R^2} = M_m \omega ^2 R


where M_e is the mass of the earth M_m is the mass of the moon, R is the distance from the center of the earth to the center of the moon, and \omega is the angular evlocity of the moon around the earth. Rearrange:


G = \frac { \omega ^2 R^3} {M_e}


So if you know the orbital period of the moon, have an estimate of the distance to the moon, and can estimate the mass of the earth, you can determine G. The trick is coming up with these estimates. One way to find the distance to the moon in Newton's day is to use basic trigonometry and compare the angle of the moon above the horizon from different points of the earth at the same time (for example, when a lunar eclipse starts). The mass of the earth can be estimated (roughly) by assuming an average composition of iron and rock.

You can also do the actual experiment of measuring the attractive force between two large masses using a torson balance, as Cavendish did. See: Cavendish experiment - Wikipedia, the free encyclopedia (http://en.wikipedia.org/wiki/Cavendish_experiment)