Try feeling around that one side that is bad and try to see if he is sore if so then get tour vet to look at the sore places
Get him physically checked, and your saddle checked. This is a very common avoidance technique. Also, get yourself checked. If you have a physical problem that is restricting you in some way, you could be unconsciously affecting the horse who may have to compensate for you.
After that, there are several exercises you can do to teach a horse to go straight, however since the one-sidedness is consistent, I'd get any physical issues checked first.