He could just be being obnoxious, although he could be sensitive to his ears being touched. If he is just being obnoxious, you could try double haltering him so when you take off the first one he might hit the lead rope of the second one. Or you could just halter and unhalter him multiple times a day until he gets over it.
is he ear shy? desensitize ears, face & above the head. dont take the halter off and let him run off. make him stand and wait for your cue to go. practice removing the haler and putting it back on in the pasture