You need to give time to the relation to be built.
Your horse needs to trust you and vice versa.
For that, don't come only to ride him, do a lot of ground work, play with balloons, go on long walks juste the two of you, come just to groom him and let him eat grass.
Or you can just come to his field to see him, cuddle him, he will get used to you and won't associate you with only work. He will be pleased and more likely to give you what you ask him.
I hope i could help you :)