You could try hand walking him by the pasture until he calms down and knows that he doesn't get to go into the pasture every time he passes it.
And remember to use positive rewards❤
Tell him no or what ever your "naughty word" is as soon as he gets quick you tell him no and slow down to your pace or make him wait till your satisfaction is up! It worked for my mare too.
maybe while going by foot to the pasture maybe start with a stud chain (if you don't agree with those that's ok) and just make sure that your horse knows your the boss and when he/she stands quietly or walks quietly give them a treat to show that they're doing the right thing
Maybe on foot go over to the pasture and make him stand quietly when he does make sure to reward him. Do this quite a bit till he learns to be quite then move into doing this while on him.