When ever he is pushing you in to the fence turn his head towards it, it will naturally push his body away from the fence. And like Alannah said, make sure he listens to your leg so you can get him to yield away from the fence and not falling towards it
A small pair of spurs might help so he will listen better to your leg. Start with a very soft pair, and then if it isn’t working move up little by little. When you feel him going towards the fence give him a light tap with the spur so he gets off your leg. If he doesn’t listen try it a little bit harder, but make sure not to kick him with it, just a tap.