Refusal, jumping the wrong jump or the wrong way, circling, or knocking a pole. Time faults as well if timed.
You have two differents types of penalties, the one due to time and the other due to poles down.
When you have a pole down you have 4 points of penalty, when two down you have 8 points, and then 12, etc...
Also, there is often a time allowed and if you exceed the time, you have axtra penalties.
A refusal is also 4 points of penalty, the second 8 points and the third is the elimination.
Hope this helps :)