Microsoft’s outage last week saw disruption across the board starting last Thursday, hitting multiple different sectors across the whole world. Some services recovered as soon as that day, while others took longer to recover and some minor aftershocks are still ongoing.

What happened?
The problem was caused by several interconnected issues, starting with what Microsoft describes as “a latent defect [which] caused Windows servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal Windows process which caused the machine’s resources (CPU & memory) to be exhausted.”
Due to reduced server capacity on the networking gateway (due to an unrelated error from the previous day) and increased traffic load, servers began to stop responding during the peak window of traffic in the United States. This turned into a cascading system failure as failed servers would dump load onto the remainder of the network. At the peak, 80% of servers were unresponsive.
Recovery took longer than anticipated because automated recovery systems were not designed to handle this failure state – as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again.
Teams, OneDrive, SharePoint, and Office apps generally recovered faster. Exchange Online and Outlook took longest, and are still reportedly suffering some aftereffects.
What now?
The ball is in Microsoft’s court with regards to preventative measures; a cascade failure is not an unknown issue, but few recovery systems are designed to account for a “start from zero” or “bootstrap from off” case.
For Microsoft’s clients, the take-away should be a lesson on critical infrastructure and how your operations can be affected by issues downstream of your own business. Ensuring offline off-system access to critical data (via backups or archives) is a good start, though of course communications (Outlook and Teams) will be harder.
Your Data in Your hands – With TECH-ARROW
by Matúš Koronthály
Image generated by Canva