Trial and error for the shape of the rings. For the size, it depends on the horse's head. You can choose the number and shape of the mouthpieces depending on your horse's mouth morphology: a small mouth might be embarrassed with a very thick mouthpiece, and a low palate will be hurt by one-break snaffles, double-break or straight bits are better in this case.
I agree with trial and error. Just start out very light and move up. Also look at what your horse needs. If he's very soft a responsive try just a loose ring if he needs more steering try a full check or d-ring.