Can I downsize from a 6 inch hose to a 4 inch hose for my Portable ac unit?

It depends on the portable unit you have as while there are many ways to easily downsize the size of the vent hosing, it will no doubt effect the efficiency of the unit. Consider the effects of a clogged dryer vent (Or lint screen) and how well it dries cloths. If your unit comes with a 6 inch vent hose design then highly recommended you stay with that.

Curious as to why this modification is being considered. Maybe you have a 4 inch wall vent hole already? Does your unit have a humidifier mode, or do you a dehumidifier? More info on your unit is needed or simply consult with the manufacturer or seller of the unit.