AWS Wide Spread Outage
Learn how an AWS outage halted services for individual users and businesses across the globe.
We'll cover the following...
Introduction
Several Amazon services and other services that depend on AWS were disrupted by an outage incident that spanned more than eight hours on Tuesday, December 7, 2021, at approximately 7:35 a.m. PST. The incident impacted everything from home consumer products to numerous commercial services.
This hours-long outage made headlines in the popular media, such as this one from the Financial Times: “From angry Adele fans to broken robot vacuums: AWS outage ripples through the US.” The outage affected millions of users worldwide, including individuals who were using the AWS online stores and other businesses that relied heavily on AWS for providing their services.
The disruption caused by AWS emphasized the need for a decentralized Internet where services don’t rely on a small number of giant companies. According to Gartner, 80% of the cloud market is handled by just five companies. Amazon, with a 41% share of the cloud computing market, is the largest.
Outages like the one above remind us of Lamport’s famous quip: “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.”
Sequence of events
An automated action to expand the capacity of one of the AWS services near the main AWS network elicited unusual behavior from a significant number of customers within the internal network.
As a result, there was a significant increase in connection activity, which swamped the networking equipment that connected the internal network to the main AWS network.
Communication between these networks got delayed. These delays enhanced latency and failures for services interacting between these networks, leading to a rise in retries and ping requests.
As a result, the devices connecting the two networks experienced constant overload and performance difficulties.
This overload instantly affected the availability of real-time monitoring data for AWS internal operations teams, hampering their ability to identify and remedy the cause of the congestion.
Operators relied on logs to figure out what was going on and initially observed heightened internal DNS failures.
The following slides show the series of events that led to the outage.