If my brothers and I sell the house we inherited in my mother's estate, and the selling price was less than what the FMV was at the time of her death, there is no capital gain... Does it still have to be reported on the Schedule D?

Yes, you should report it, especially if you have received a 1099-S from the settling agent. You report the sale on schedule D and form 8949. List the property on form 8949 with the correct selling price and cost basis, and in column B indicate code "L" - which means it's a non-deductible loss. In column G enter an amount equal to the adjustment as a positive number necessary to make the loss = $0. You can read about Schedule D and Form 8949: http://www.irs.gov/pub/irs-pdf/i1040sd.pdf

Remember that if a 1099-S form was issued to each of the siblings, the settling agent should have split the amount of proceeds between all of you. For cost basis each of you then uses his or her share of the FMV as of date of death.