Will colleges care that my math grades weren't so great (average I guess) if I'm not looking to go into a field with much math?

Colleges look at your entire transcript, so to some extent - yes, they care. I assume you will be senior in HS next year. It really helps to show general improvement in your GPA year over year - colleges view that as showing maturity and dedication to academics. So for example if you got a D in math as a freshman, and a C as a sophomore, and a B as a junior that is viewed much more positively than if your grades were the other way around. And of couse your math score on the SAT is important as well.