I would take one rein and pull him into a circle (not harshly) just slow and gradule but hold on because its a bumpy ride. And you may be having some trust or respect issues so maybe ask a trainer or someone around you who might know, what they think.
My horse does that...keep him slow and controlled with pressure on the reins if he's trying to be faster until he slows, do half halts, do circles when he tries to focus faster than asked, use the outside rein and pull his head to the rail...hope this helped
my horse does the same thing!normally i make him do a lot of halts and backing up to begin with to help him not get too forward . then trotting a few steps then making him immediately halt helps him realize who's in control. but when he starts to take of turn him in a small circle and if he try's to get strong and push through it just try and slow him down and then do circles.