Hi
I got a letter today from my employer (supposedly an investment firm!! ) stating that they had neglected to withhold social security and medicare tax starting from 2005 till end of 2007. They are now asking me to pay them back a considerable amount of money.

This is my first job, so I didn't really know about different types of taxes . I am on H1-B (started as OPT).

My question is: who is responsible to pay all these taxes - who owes the taxes to IRS at this point? Can a company ask for such a large amount of money due to its own negligence?

I am not an independent contractor. I am a full time employee from the beginning. I have always received W2.

It appears that you were employed as independent contractor. You must have received 1099-Misc instead of W2.
Your 1099-Misc income during OPT period is exempt from SE taxes. But you must pay SE taxes on 1099-Misc income during H1-B.

How you will treat the 1099-Misc income, read: Your U.S. Tax Return: W2 or 1099: Employee or Independent Contractor.