I overstayed my vacation is the United states by 6 months,and working their while in colledge for my friends business,I was made to sign paper work stating all of this and was told I would be given a copy,I did not receive any paperwork when I boarded my plain baack to Toronto.The only item that was in the envelope they gave me was my passport.What kind of paperwork do I need to file to get a visa to be able to visit the united states or do I have to wait the full 5 years?

There are serious consequences to overstaying your visa AND working in the US when you don't have the status to do so. I am reading that you can be barred for up to 10 years. Do You Know the Consequences of Overstaying a Visa in the US? - VisaPro Immigration Articles

Yes, you have to wait.

So will I have to file paperwork and pay fees to be to enter the United states when my 5 years is up

Yes, that's the procedure but there is no guarantee it's 5 years and there's no guarantee it will be granted.

Did you read the info I posted?