Close your hip angle. Some other thing to keep in mind: Put your weight in your heels, balance yourself, and use your upper calf to steady yourself and keep your leg on the horse. It's all about balance, and having a closed hip angle is important. Hope that helps!
When you lean back too much your abs disengage. So try thinking about tucking your pelvis underneath you and contract your ab muscles. This should help stabilize your seat and give you more balance as well. You can also try and imagine someone pulling a string attached to the top of your head upwards.