I think that he doesn't trust people so he will not follow you into a scary looking place (for him). Fear of people can also manifest as lack of respect for people, which can be resolved with time and work.
When loading a horse in a trailer, the calm is extremely important ! Maybe, when your horse see many people around him: he hesitates. I think, he has this reaction because he doesn't know who he must listen and who to trust.
Sometimes, it's the commonest sign of mistrust of people, he shows up as a simple lack of willingness to go forward.