First of all you need to think about your horse, giving him vacations should not be stressful but more relaxing for your horse. Don't change all of his habits all at ones. You can't abandon him neither, you need to stay his landmark because he trustes you.
Don't change his environment, it will be too stressful but you can put him in the field with his stable friends.
Stop working gradually, you can ride him for walk in the woods or just make groundwork or even games together, it will improve your complicity and the trust he puts in you.