I wouldn't suggest walking him out on a hack, it can be dangerous for both you and your horse. I wouldn't also start small, a few short trail rides and then back to the barn. I'd also see if there's a more specific problem, such a small being barn sour or herd bound, and then work on it from there.
My horse had an accident when we were out once and he was always crazy when we were out after that. I had to walk him on foot for a while maybe that will help you.