Microsoft traces last week's day-long Outlook outage to a hidden bug from a July fix
Microsoft's preliminary review says a months-old defect in its email gateway servers made more than 80% of them unresponsive. Scammers are already sending fake outage notices.
Microsoft has explained what caused last week's Microsoft 365 email outage, which left many Outlook users unable to send or receive mail for about a day. Its preliminary post-incident review, published November 15, blames a software defect introduced in July that sat unnoticed for months before it took down most of the company's email gateway servers.
What went wrong
Microsoft 365 runs traffic through two independent network gateway systems, each meant to be able to carry the full load alone, 9to5Mac reports. According to The Register's account of the review, two things happened before the outage:
- At 01:07 UTC on November 12, Teams failed over to the gateway that also serves Exchange Online, adding load.
- About 14% of gateway server capacity was already out of service.
Microsoft says it would normally have had enough headroom for that. But a change made in July, part of a fix for its networking gateway servers, contained a latent defect. Under certain load conditions it caused lock contention inside an internal Windows process, which exhausted CPU and memory and made servers stop responding. Microsoft's platform head, Charles Lamanna, said the defect had been previously unknown, 9to5Mac reports.
When a server froze, its traffic moved to healthy servers, which then hit the same bug. Within two hours the problem had spread across the fleet, and at the low point more than 80% of servers were unresponsive.
Why it took a day to fix
Microsoft's monitoring flagged connection problems at 16:03 UTC on November 12, and the first support case came in half an hour later, The Register reports. Restarting servers didn't help because they were swamped again immediately, and automatic recovery wasn't designed for that many unresponsive servers at once. A simple rollback wasn't possible either, because the change was months old.
Engineers instead built an update that returned the code to a previous known-good state and applied it to groups of servers starting at 07:20 UTC on November 13. Service was confirmed healthy at 16:10 UTC that day. Teams, OneDrive, SharePoint and Office apps could fail over away from the bad gateways, but Exchange Online couldn't because of its scale, so email was hit hardest. A final report is due November 20.
How the bug got through testing
The change was rolled out gradually, with Microsoft's normal safe-deployment process, and ran worldwide for more than three months without visible trouble. Microsoft says its load tests missed the defect because they used a different traffic and hardware mix. The contention gets worse on newer machines with more cores, while the test labs focused on weaker hardware with fewer cores.
Microsoft also says the outage had nothing to do with the thousands of layoffs it announced the previous month. 9to5Mac's writer is openly skeptical of that claim, though that is opinion, not a finding.
What to do: watch for fake outage emails
Microsoft says the outage was "a technical issue not related to cyber or AI / agents," according to SecurityWeek. Attackers are using it anyway. Microsoft Threat Intelligence has seen phishing emails posing as Microsoft 365 service health notices that cite the real incident ID, EX1550543. Some falsely say that emails held during the outage will be deleted within 24 hours unless the reader clicks a link. Microsoft says the activity looks like credential phishing.
Signs to look for, according to SecurityWeek:
- subject lines starting with "Outage -" followed by a six-character code;
- sender names that combine your organization's name with "Outage Notification" or "Service Health";
- sending domains including moments-notice[.]com, planetmaison[.]com and newbooker[.]com.
The deletion threat in these emails is false. Microsoft's advice is to check service incidents in the Microsoft 365 admin center rather than through links in an email, to investigate any clicks on these links, and to review Safe Links policies. Microsoft says its security products are now blocking the campaign.
Sources
This story is based on reporting from these outlets. Read them for the full picture: