Slow and steady. Grow the horses confidence as it's probably scared of them
If he is nice and gentle towards you. Start bringing people out to the barn with you. Be sure you are there beside him. If he is not aggressive towards other people and is generally disinterested in them. Try to get other people to work with him so he can get used to lots of different people being around him.