Is there an insulating mastic that can be used to build up an insulating layer with or without some sort of additional material that can be put on in layers to both seal and insulate an air duct from sweating?

Yes, I once used servicemaster to clean the ducts by vacuuming through hepa filters. When they were done they blew a sealant through the duct.
Additionally, insulation professionals can blow expanding foam on hard duct.
For flex duct, we have been pulling down and replacing with 3 inch insulated flexduct which is now the national code.