Friday’s Microsoft problems are already shaping up to be one of the biggest IT outages ever, affecting countless businesses and individuals all over the world. It is another example of how a minor technical change, made by a company that is unknown to most outside the IT industry, can wreak widespread havoc.
What happened?
Companies are grappling with problems affecting PCs, servers and other IT equipment running Microsoft Windows. Affected PC users are seeing a “blue screen of death”, indicating that Windows has failed to load.
Microsoft has blamed a flawed software update from CrowdStrike, a security technology vendor. CrowdStrike’s Falcon software is designed to stop cyber attacks and includes a suite of products running on individual devices and delivered via the cloud.
In a post on X, CrowdStrike’s chief executive George Kurtz said the cause of the problems was a “defect found in a single content update for Windows”. PCs and servers running Apple’s MacOS and the open-source Linux operating system, which is widely used in internet infrastructure, were “not impacted”, he said.
“This is not a security incident or cyber attack,” Kurtz said. “The issue has been identified, isolated and a fix has been deployed.”
How widespread is it?
CrowdStrike is one of the largest providers of “endpoint” security software, which protects connections between computer networks and remote devices — from laptops, phones and servers to retail payment terminals and cash machines. Any of those devices that run Windows might be affected by the bug.
The IT failure has affected airlines, banks, broadcasters and healthcare providers from the US and Europe to Australia, Japan and India.
“The worldwide IT outage experienced this morning is unprecedented in the range and scale of systems it has impacted,” said Harjinder Lallie, a cyber security expert at the University of Warwick.
Ian Batten, a lecturer for the School of Computer Science at the University of Birmingham, said that in order to run effectively, cyber security and virus-scanning software such as CrowdStrike’s needed to have “deep and profound privileges” across a computer’s system and be “injected deep in the operating system”. But that meant that if something went wrong, the system would “stop dead” in order to protect itself, he said.
Customers of Microsoft’s Azure cloud computing platform, much of which runs on Windows, have also reported problems. However, the issue has been complicated by an unrelated earlier Azure outage, primarily affecting the US, on Thursday evening.
Microsoft said on Friday that the earlier issue had been resolved, bringing services such as its online Office software and Teams collaboration tools back online. Even so, several hours after the CrowdStrike problems began, Microsoft’s Azure status page was showing ongoing problems related to the Falcon update around the world.
What is CrowdStrike?
CrowdStrike is a cyber security company that was founded in 2011 and headquartered in Austin, Texas. It says it is the “cloud security provider of choice for 62 of the Fortune 100”, with more than 29,000 companies using its products.
Analysts at Gartner say CloudStrike is the second-largest company in the global enterprise endpoint security market, behind Microsoft itself. Its market share is more than double that of its three closest rivals.
Its software is widely deployed to protect critical business infrastructure at some of the world’s largest companies because of its “reputation for technical excellence, which is why this particular issue is so surprising”, said Gartner analyst Neil MacDonald.
CrowdStrike is well known for investigating Russian hackers. It helped to investigate the cyber attacks on the US Democratic National Committee in 2015-16 and its connection to Russian intelligence services. The same Russian group then attempted unsuccessfully to hack into CrowdStrike in 2020.
Nasdaq-listed CrowdStrike has been growing quickly in the past few years and joined the S&P 500 last month. Its revenue rose by a third to $3.1bn in the most recent fiscal year, ending in January, while net income swung to $90.6mn, from a loss of $183.2mn in the previous year.
Prior to Friday’s outage, shares in CrowdStrike had more than doubled over the past year, giving the company a market capitalisation of $83.5bn. However, its stock opened about 15 per cent lower when trading began on Friday morning in New York, knocking almost $12bn off its market value.
How long will the problems take to fix?
While CrowdStrike said a “fix has been deployed”, it is unclear how long that may take to distribute to the very large number of affected customers and all their employees’ devices.
Kurtz told NBC that “many” of CrowdStrike’s customers were “rebooting and it’s coming up operational because we’ve fixed it on our end”. However, the chief executive noted that it could still “be some time” for some companies’ systems to recover.
The issues could “take days to resolve — if not weeks”, said Vasileios Karagiannopoulos, a cyber security researcher at the University of Portsmouth. He added that the problems were “so global and extensive across systems that IT support might be sparse due to the demand”.
Kevin Beaumont, a cyber security researcher, said in social media posts that CrowdStrike customers were in for an “incredibly painful” process to remedy the problem.
“Recovery is only possible manually,” he said. “You have to go to a server or PC, boot it in safe mode at the console, log in as admin, then basically hack the system to get it back online.”
