I’m 5’6 and I ride a 13.2 pony. My Grandfather always rode small horses and never found any issues. I don’t see myself riding much bigger any time soon. I mean it’s just further to fall 😂. However if there is visible signs that your pony is uncomfortable try a bigger horse and ride your current one every now and then if you don’t want to give them up
the general rule is 20% of their body weight is max, but i rode a horse that size (a lesson mix up with a rider of the same name as me haha) and i am 5’4 so i don’t think it should be a problem. i’d start worrying if the pony shows signs of discomfort, pain, or distress. sometimes these little things can be eased with stretches and massages and they won’t continue.