Ask @cedrictriolet : How to improve bad ground manners with young horses? My 5 years old warmblood can be very rough, pushy and steps on my feet. Trainers at my barn suggest smacking him but I disagree.
You know, a horse have to respect you on the ground as much as on top of them, by instinct a horse is either a leader or a follower, when they are followers it's easy to deal with them as they won't challenge you, but when they have leader character, you need to prove them that you are the leader so they can accept to follow you, those attitudes he have with you is a proof of he's challenging you, as much as I understand that you don't want to be ruff with him, you do need to be strong enough so he stops challenging you.
What I generally do with horses like that on the ground is to have a dressage whip on my hand and move it toward them when they come to me without hitting them, that's generally good enough to keep them away from pushing me and make them respect me and my space.