Why is it that water regulating control valves are always installed on the return side of a coil regardless of whether it is chilled water or hot water for an air handling unit? What would happen if it were installed on the supply side?

If we talk about 3-port control valves than the reason for installing them into return side is because those valves are physically designed to have two inputs and one output. If you install them in supply side than they will have one input and two outputs which is not the way those valves are meant to be used. You can produce sound by blowing in to the wrong side of a trombone :) but it is much easier if you blow where you should. Putting 3-port valve in supply side of a circuit will cause problems with noise and controllability.

2 port valves can be installed in supply side but if you do that they will see higher pressure (higher by the pressure drop through the coil). It is better to install them in return side of the circuit to minimise possibility of noise issues.

Wouldn't it also have something to do with starving the coil thereby creating temperature control issues? Do higher pressure drops mean less flow and subsequently flow problems which would lead to temperature fluctuations or am I reading into this too far?

Nothing to do with starving the coil. Pressure drop through the circuit (coil + valve) will be the same wherever you install the valve. Because valve and coil are in series total pressure drop = sum of individual pressure drops.

Another issue is cavitation but that is more theoretical than practical. You will never get cavitation (at valves) in chilled water systems if you have well designed pressurisation unit

Lets not talk about 3-way systems since it is stone age technology. If used, it will drain your bank balance with the electricity bills.

2-way systems : Chilled water system is a closed circuit. Hence, in a unit comprising of 1 coil, 1 strainer, 2 or 3isolation valves, 1 Balancing valves (automatic or manual) and 1 control valve, the complete pressue drop across the unit i.e. measured at the inlet and outlet of the unit remains same regardless of in which order you have installed the various elements. It does not matter if you place the control valve on the supply or the return line ( technically speaking).
However, practically speaking, the control valve and the balancing valve require certain amout of straight line length at their respective inlets and outlets so that there will be laminar flow through them. The balancing valve and the control valve will be able to perform its functions only when the flow is laminar ( including measurements during commissioing). But in supply line you place a strainer and an isolation valve leaving very little space for the installation of the control valve and balancing valve. You will not have enough space to provide the required straight line length before and after the valve. Hence it is very convenient to place the balancing valve anc control valve on the retun line.

Regards,

Anoop