Microsoft outage traced to network upgrade, report pending

âš¡ TL;DR
Microsoft says its Nov. 12-14 email blackout began with an issue during an earlier network capacity upgrade that surfaced after a later network change, not a cyberattack. The full post-incident report it promised has not been published, business customers must request SLA credits themselves, and Microsoft said it had no indication of any link to Friday’s FAA ground stop.

Microsoft has blamed its worldwide email blackout of November 12-14 on a flawed network capacity upgrade, but days after the Microsoft outage began, the company has not yet published the full post-incident report it promised. Outlook, Teams and Copilot failed for users in more than 70 countries, and Microsoft did not say service was fully restored until early Saturday, November 14.

Microsoft outage

The disruption started with Exchange Online around midday Thursday in the U.S. and lasted more than 24 hours, CBS News reported, after NarwhalTV reported that repeated fixes faltered past the 27-hour mark. The open questions now are why it happened and what customers can claim.

Microsoft outage cause, in the company’s own words

Microsoft gave its fullest explanation so far in a statement on X shared at 8:40 a.m. Pacific on Friday, according to Variety. It opened with an apology: “We are deeply sorry for the disruption this incident has caused our customers and partners.” It then described the cause:

“This was a technical issue not related to cyber or AI / agents. The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.”

The company also defended the slow recovery: “Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration.”

Earlier updates described the mechanics. Servers were failing automated health checks and dropping out of service, which overloaded the ones left running, GeekWire reported. Microsoft paused all deployments while it worked on a fix, and some repairs on Thursday evening were “not taking as expected.”

Thundering herds and a promised post-mortem

Charles Lamanna, Microsoft’s executive vice president of Copilot, Agents and Platform, wrote on X on Friday afternoon that the investigation points to “cascading failures + thundering herds at our network perimeter.” A thundering herd happens when huge numbers of clients try to reconnect at once and swamp systems as they come back online.

“I’m working closely with the engineering team to complete the full post-mortem,” Lamanna said, as quoted by CBS News. “We will publish the post-mortem shortly covering the step-by-step timeline and our many repair items to avoid similar outages in the future.”

Microsoft’s own statement said: “We will provide a full post-incident report once our review is complete.” As of November 15, no such report had appeared in public coverage of the incident, so the account above remains Microsoft’s preliminary version.

One more specific claim is still unconfirmed by the company. The Verge reported that senior correspondent Tom Warren’s sources linked the outage to Azure VNet configuration changes. Microsoft has not publicly named Azure Virtual Network as the trigger. Australia’s Home Affairs Minister Tony Burke told reporters in Sydney that Microsoft likened the event to the 2024 CrowdStrike outage, “where there is an update that has gone awry.”

How far the email blackout reached

Downdetector logged more than 130,000 user reports on Thursday, according to CBS News, and the Associated Press reported that complaints arrived from 74 countries and territories within an hour of the first problems. Microsoft said in January that it had more than 450 million paid commercial subscriptions to its 365 services, though the number of people affected has not been disclosed.

  • A trial in Manchester, England, stopped when Judge Maurice Greene sent jurors home, the AP reported.
  • British police could not receive crime reports online, and the National Pharmacy Association warned of prescription delays.
  • Workhorse Group delayed its quarterly earnings call, and OpenAI and Anthropic reported problems connecting to Microsoft 365 tools.
  • Morningstar DBRS estimated insured losses of $300 million to $500 million, GeekWire reported.

Closer to Microsoft’s home, a spokesperson for the Seattle mayor’s office confirmed to GeekWire that he had called the outage “a huge pain in the ass.”

SLA credits must be requested, not granted

Microsoft has not announced automatic compensation. Service level agreement credits are not applied automatically, Variety reported; affected business customers must submit a verified request. Microsoft’s partner center guidance says applicants must “provide evidence that the outage affected the customer,” and that evidence email must come from the affected tenant’s domain, not a personal address.

Credits are prorated based on the service affected and the duration of the outage, according to Variety. They arrive as credit toward future service rather than cash refunds, Cybernews reported. Microsoft 365 Personal and Family subscribers are generally not covered by the business service level agreement, the outlet added.

No Microsoft outage link to the FAA ground stop

On Friday afternoon the Federal Aviation Administration halted departures nationwide, prompting online speculation about a connection to the email failure. Microsoft rejected that in a post on X: “We have no indication that the FAA’s nationwide ground stop is related to any Microsoft services.”

Federal officials agreed. “I want to make sure everybody understands that there is no connection between this and Microsoft,” Homeland Security Secretary Markwayne Mullin said at an evening news conference, GeekWire reported. Transportation Secretary Sean Duffy blamed aging FAA switches and routers that “got overloaded with data,” saying the agency still relies on “1980s little copper lines.” More than 3,500 planes were in the air when the FAA ground stop began, and FlightAware counted nearly 7,000 flights delayed or canceled.

Questions waiting on the report

“There will be major questions for MSFT in this global outage,” Yorkville Ives and Co. tech analyst Dan Ives wrote on X, according to CBS News. Jeremy Singer, a professor at the University of Glasgow’s School of Computing Science, said the episode, according to the AP, “serves to emphasize the dependence we have built on one company’s digital infrastructure.”

Until the post-incident report is published, key details remain unanswered in public: which network change exposed the flaw from the earlier network capacity upgrade, why health checks pulled so many servers at once, and which repairs Lamanna’s promised list will include.

0
Show Comments (0) Hide Comments (0)
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x