What is the penalty or how the IRS will track those who are many filling the 1040 form instead of the 1040 NR.

If you go online and use any of the software for taxes they don't option for NR so you end up using 1040 as resident.

How this will affect these people?

It is cumbersome to figure out if you are a resident or not. The last thing I read is that Student are considere nonresident for 5 years. Read publication 519 from the IRS.gov

F1 students are non-resident for 5 years and must file non-resident tax return.

If you filed resident tax return, you should file Form 1040X (amended tax return) with properly done Form 1040NR or 1040NR-EZ.

Most of the softwares do not do non-resident tax return. You must do paper return.

The IRS has neither the time nor the resources to properly track tax returns filed by foreign students, so it is unlikely they will determine on their own whether you filed correctly or not.

That said, if you apply for a green card, you are specifically asked if you filed the proper tax returns and paid taxes while in the U.S. under all visas. If you answer incorrectly and the USCIS determines this fact, it can be grounds to deny your green card application. The USCIS does request the IRS verify your filing status during all previous visits to the U.S. and that is when the IRS will decide that you filed incorrectly.

If additional taxes are due, the penalties are 25% of the taxes owed, plus interest compounded annually. Over ten years, that can easily DOUBLE or TRIPLE the taxes owed.