First you need to know that this is a fear reaction. Horses are used to be together, this is a sign of strength for them in order to fight predators if they are coming. This is why when they are alone they feel more insecure and may not want to work or even leave the barn.
You need to make him understand that he can be relaxed with you, there is no danger and you can protect him, you won't ask him something he can't do or that will hurt him.
You need to make him focus on you in order to forget his fears a little bit, this way he will see you as a landmark of safety.
To do so, make him do gait transitions and maneuver (at the beginning close to his friends or the barn), congrats him a lot and then do it repeatedly a little more far every day. Once he'll see there's no danger, he will follow you without reluctance.
Always reward him a lot, even to the slightest positive movement and never use the whip, it will scare him more and won't give you anything because of his emotions (he's mostly scared so you need to reassure him and not scare him more).
However, you can take a long whip with you and wave it to his hindquarters (never hit him of course) in order to make move in the right direction (go forward).
Finally, you can make a lot of ground work to increase his confidence and respect in you.
I hope i could help you, good luck :)