I've been spending the last 3 or so hours on this problem and I cannot figure it out!

5x^3=5x^2+12x

I subtract the 5x^3 to get it on one side so that it equals to zero. Now when I take out the GCF of x it leaves me with x(-5x^2+5x+12) and from there I have no idea... You can't multiply 12 and negative 5 and find somethat when when multipied =-60 but when added get 5. When I do the quadratic Formula I end up getting something like 2i square root 43 and I'm not sure that sounds right since I need 3 answers seeing as how the highest exponet is 3

You are on the right track. So far you have:

0 = x(-5x^2 + 5x +12)

From this you already have one of your solutions - namely x=0. To get the other two solutions apply the quadratic formula (since you can't just factor -5x^2 + 5x +12). You will end up with two more solutions, but be careful of signs - you should find the solutions involve the square root of a positive number, not negative.