I noticed that my track lighting was starting to detach from the ceiling so I tried to tighten it. Unfortunately the lights were still on and when I tried to tighten the ballast (middle piece... is that what it's called?? ) there was a pop and a little smoke. The lights remained on but now the dimmer switch is not working. I can only turn on and off the lights through the fuse box.

Any ideas?

You will need to replace the dimmer. Was a short, and now cleared.
If you remove wire from dimmer and light goes out, Definitely dimmer.