If you let go of handle bars and pulls to one side

Possibly the wheel isn't properly lined up in the fork, bad bearings in the wheel or steering, caliper hanging up.