Lots of good advice here; start slow and allow him to build confidence, when he starts to take you to the fence, let him. Be sure to give him plenty release (use a neck strap or grab mane) so if he gives you a really good effort you don't accidentally punish him, keep your seat out of the saddle without any contact in the saddle for at least a stride away from the fence. Also build inviting jumps, even cross rails, give him a ground line too a little in front of the cross rail, make it really wide, use the widest poles you have. Do everything you can to make the experience as easy and forgiving as possible.
I ride a horse on a regular basis that makes really dirty refusals. get and keep him straight to the "jump", have lots of leg and a pony pat for him afterwards :) (I would start off at a walk and then trot etc)
Exactly as Emily said, you should take things slow. Start with just one pole on the ground, then slowly add more. After a few days of poles, add a cavaletti in, that is only raised on one side. If this is going well you can raise it on both sides. Eventually, you can move onto multiple cavalettis, but space them out well. Once he's fine with the cavalettis, you can move on to crossrails, and then move forward at you own discretion.
Be careful not to over face him and further put him off jumping. It may help his confidence to jump a small crossrail through a shoot, where he can assess the situation, and take his time to think about where he's placing his feet. Make sure that under saddle, you are giving him all the reinforcements he needs from your leg, hand, seat, and voice. Good luck :)