If the horse backs up in the competition, you have a second chance to try the jump, however it will be considered a fault of 4 points! If she backs up again you are disqualified! You are also disqualified if you whip her more than once! however it varies from competition, some of them you are disqualified for only one whip!
If my mare back off a jump, I use firm legs to make my mare know that she needs to jump. Also, I make my mare go in front of my leg towards the next jump. If my mare stops at a jump, I give her a strong correction using my legs. When I go back towards the jump, I ride like nothing happened to give her confidence. Hope this helps answer your question!