I started since I was two:)))
My sister is having her first lesson on Wednesday and she is 5 .
We got my bush pony x welsh from the wild and were told he was 2 when we got him and broke him in when he supposedly three, turns out he would've been only two when we broke him but it was all done very slowly and gently and he turned out fine.
Kinsey I agree with much of what you are saying in that although you shouldn't ride them when they are extremely young, they should be vigorously worked so that when the time comes they are capable of carrying weight. I still think it is better however to wait until they are over two to start actually riding to avoid hindering their confirmation ( ex. causing a swayed back), possibly hindering their fullest potential athleticism in the future.
I think horses shouldn't be backed before they are 2 1/2, and shouldn't really be ridden before they are 3 at the earliest. Some breeds take longer to grow fully so you'd want to wait for them to do a bit more growing before backing them
I started backing my horse when he was 2 years, 8 months, basic walk-trot, with an emphasis on steering moving off the leg, etc. until he was around 3 years, 3 months. From there, we did basic dressage and went to our first competition (actually on his 4th birthday!).
When a horse is almost or done growing, 4 or 5 for most horses..
I would wait until at least 2 and a half just to be safe, however many people do start riding at 2. Remember that horse's joints don't fully develop until 5 or 6.