I agree with Mark.
Also, the issue is also how long he stays there. He probably will stay still for a certain time and then try to move.
This problem is solved by you asking him to walk before he decides to.
First you need to know how long he stays still for. Let's say 10 seconds.
So you ask him to stay stop. You don't pull on the reins at all when he's stopped. Then you count five seconds and take a few steps forward. Then repeat. This time wait 8 seconds. And walk. And again 10 seconds and walk. And then 15, 20. Until he understands.
There is a reason for which he wants to walk so you reassure him that if he stays still until you tell him otherwise, you will let him do what he wants , in this case to walk.
It's important that once he's stopped you don't hold him there. If he walks you let him take a step or two and stop him again.
I agree with everyone else, though what I think helps my new horse a ton is if they don't stand still right after being asked to halt you make their feet move, as in tightening the lead rope toward their but and doing a couple small circle like that. They learn that if they want to move their feet when you ask them to stand, that they're gonna have to move their feet a bunch more which creates unessesary work for them.
Teach them to back up when you ask for a stop, and when they stepp forward without permission take a step or two back, hold them there until they settle a bit and their reward should be the pressure off their mouth.
I started with this "Step forward - no - step back" and it does really work! I do this on the ground, it's amazing how well it works.
To teach my horse to stand still I go to put my foot in the stirrup and he starts to move off. Ask him to stop (take your foot out of the stirrup). If he doesn't then, I walk him around me. I do this again, ask him to stop, if he even shifts his weight, I walk him around me.
I use a dressage whip to just tap him on the hip lightly to keep him moving until I ask to stop again. Again, you don't have to run the horse around you, I simply do all this at the walk.
I think you can fix it quite simple by simply putting him back. Every time he takes a step forward, immediately step him back to where he was. You shouldn't need to be rough about it, but you need to be quick about it, not move him back a while later, because he won't get it if time passes before you react. Just be really patient and matter of fact.