The more he is exposed to windy conditions, the more comfortable he will learn to be in weather which he currently finds uncomforting. I turn my mare out in all sorts of weather so that she can get used to being exposed to different conditions. If your horse is only out on dry days, then he will find things such as wind and rain unnerving because he has not experienced it often enough. A short 10-minute ride in different weather conditions will build his confidence and yours. Keep him occupied by asking for different things, but not so many things that he gets frustrated and confused.
If you're nervous then your horse will be too. You can relax by taking a deep breath in and allowing your shoulders to relax as you breathe out.
A quick fix like others have said is to use sound resistant ear bonnets. However, this is not a long-term fix since it will simply dull the sounds which he should be desensitized to. This will not prepare him if he ever comes across loud noises or heavy wind in the future and will most likely make him more spooky instead. It's best to find a solution to the cause of the problem rather than a solution to the problem which will set both you and your horse up for success.