As a rule of thumb, I usually wear one when riding horses under the age of 5, as they can still be extremely unpredictable.
I do it at my own discretion though. For example my horse is 4 but I only wear the vest when we are jumping big scary fillers.
I do wear it sometimes but I don't need to (I don't unless my trainer tells me to or I'm doing cross country) and don't like to in the summer because it's so hot. You should wear it if you feel you need to for safety reasons though.