Yesterday, my landline phone suddenly has no dial tone (all the phonejacks in the house not working, and I tested them with 2 different phones), and at the same time, my home security alarm began flashing a "phone off hook" message. My DSL connection was fine the entire time. I checked and didn't find any phone off hook. When I plugged a phone into the junction box outside my house, the phone was working fine. So I know it's wiring issue inside my house. Also, when I called my home number with a cell phone, there was no ringing, just a busy tone. What can I do next to help figure out where the problem is located? And how do I rule out that it's not a problem with the security alarm itself? (is the alarm not working because phoneline is messed up, or are the phones not working because the alarm is messed up?) Any help would be greatly appreciated!

You get a busy signal if the phone is off the hook. From this standpoint, let's assume a phone off the hook.

Now, a nest of insects like spiders can cause a phone line to do weird things.

"Phone off hook" is not necessarily a security system problem. It may be able to seize the line. The security system should plug into a special jack called a RJ31x. When you unplug it, the phone line enters and exits the RJ31x. In esscence, it removes the security alarm from the equation.

I don't know if you own a voltmeter. Do you?

First, eliminate the security system by unplugging it.

I'd start by unplugging ALL of your phones/answering machines/modems/sattelite box etc. then checking the jacks one at a time. Inspect the jacks after unplugging the phone.

Wait about 5 minutes for the line to reset and take a known good phone and test the jacks one at a time.

So, initially I'd be looking for a shorted jack or broken device connected to the phone line.

If you had a voltmeter, the voltage across the line measured outside would be fine.

There is a very remote possibility that the interface module has a defective 1/2 ringer. Look at the 1/2 ringer pages here: TCI Library - General Information

The diode and the capacitor would have to short though.

Try unplugging at the alarm panel as suggested by KISS.
If that doesn't work, the pins inside that 31X may not be reconnecting to premise line.
Try plugging phone into the 31X at the alarm panel(remove the 8 conductor cord first)
Also, Most Alarm systems will tell if line trouble, but don't tell when a phone is off hook.
In an Alarm condition, Panel doesn't care, and will cut off phones weather you are using the phone or not.