Months-Old Bug Triggered Microsoft 365's Global Outage
A software bug introduced in July lay dormant in Microsoft’s networking gateway servers for four months before triggering a nearly 24-hour global Microsoft 365 outage on November 12. Testing laboratories failed to detect the latent defect because lab equipment lacked the processing power of production servers. When Teams automatically switched to Exchange Online’s infrastructure roughly 15 hours before the outage, the increased traffic collided with already-reduced gateway capacity—14% was offline due to unrelated issues. Within two hours, over 80% of gateway servers became unresponsive.
Microsoft’s regional safeguards failed to contain the cascade, spreading outages globally. Exchange Online couldn’t failover to alternative infrastructure as other services did, leaving users unable to access email and causing significant delivery backlogs. Restarting crashed servers proved counterproductive, as backed-up traffic immediately overwhelmed them again. After engineers identified and deployed a fix, services stabilized by November 13. Microsoft has since increased gateway capacity, improved monitoring systems, and added safeguards specifically designed to prevent traffic redirection from causing cascading failures.
Read the full article
Continue reading at BleepingComputer →This is an AI-generated summary. Read the original for the full story.