I've thye Gen/Trac 30 amps TRANSFER SWITCH. It will be installed in the basement with the main power breaker. The outside inlet will be at the out side wall, about 40 feet to the transfer switch. I will then use the 25 feet extension cord to run from the generator to the inlet box.
Can I use 8-3 instead of 10-3 wire?

Yes, if you want to.