The Prison Was Unguarded When the Allies Arrived
With the end of the war on the horizon, the guards at Schloss Itter saw little reason to remain at their post and wait for the Allies to show up. Sebastian Wimmer was the head guard of the prison, and he knew that it was highly dependent on who reached the prison first whether or not he would live. He abandoned his post and most of the other German guards in the castle followed suit.
The prisoners were then able to take over the castle and arm themselves, but they had nowhere to go. They were outnumbered by the SS that were wandering the forest and they did not know which way was safe for them to travel.
With the guards gone it seemed like a rescue would be an easy enough operation but even with the German forces completely disorganized, they had a substantial force mobilized around the castle. Getting to the castle was no easy feat, and escaping from the castle would be even less so. The prisoners had very little in terms of weapons that would offer up much defense against an SS battalion. The SS soldiers were also setting up road blocks to prevent the Allies from getting to the castle and to prevent the prisoners from escaping.
It would be found out that later that an SS troop had been dispatched to the castle with the purpose of executing all the prisoners that were held there. Time was of the essence for the Allies to come to the rescue. When Lee and Gangl finally arrived with their small group of men, the prisoners were relieved and yet saddened by the size of the force. Instead of hiding in the castle as was commanded by Lee and Gangl (they were not soldiers after all but women and dignitaries), the prisoners refused and instead fought side by side with their rescuers.