Inside Microsoft’s Rush to Get Back Online After a Global Outage

Late into the night Thursday, hundreds of Microsoft engineers and top executives were sitting in their darkened offices or home basements, logged into a video conference triaging a nightmare scenario for a technology company.

The world still runs on email, and much of that is processed through Microsoft Outlook. For hours, a company outage affected hospitals, schools, businesses and Congressional offices in Washington. Even Microsoft’s own staff couldn’t use email.

On the videoconference, Scott Guthrie, the senior most executive overseeing the company’s computing infrastructure, was proposing possible fixes, according to a person familiar with the call who was not authorized to speak publicly. People chewed on fingernails, rubbed their foreheads and debated how to turn off other services to prevent a “stampeding herd” so that as systems come back online, they would not be overwhelmed with new demands and shut back down.

By Friday morning, the company had apologized at least twice. It said it determined that the problem stemmed from a recent upgrade to its network, which shut down some servers, and that the systems were steadily improving. “Given the scale of demand, we restored services gradually, validating stability at each stage,” the company said in a statement. “While this took longer, it helped ensure a stable restoration.”

Finally, they had kept the herd at bay.

The prolonged and widespread outage was just the most recent reminder that the technological underpinnings of modern life and business are run through just a handful of major companies. And while cyberattacks and malicious actors can target their systems, sometimes the companies learn the hard way that they can introduce devastating flaws when updating or maintaining their own global systems.

Hospitals, law enforcement and government agencies and other vital services are often left vulnerable — unable to function for hours or days at a time.

---

Thank you for your patience while we verify access. If you are in Reader mode please exit and log into your Times account, or subscribe for all of The Times.

---

Thank you for your patience while we verify access.

Already a subscriber? Log in.

Want all of The Times? Subscribe.