Hello Lian,
Maybe you use your leg too strongly?
I would suggest you to first teach you horse respond a soft pressure on ground work, ask him to move his shoulders and hip when you ask him. This way he will understand that a pressure on his body won't hurt him.
Then when you ride him, be careful to push him first with a calf pressure than directly with your heel.
Be patient, ask few at the begining and congratulate him when he understands.
Good luck! :)