Facebook, WhatsApp, Instagram, Oculus Outage

On October 4, 2021, at 15:39 UTC, the social network Facebook and its subsidiaries (Messenger, Instagram, WhatsApp, Mapillary, Oculus) experienced a global outage for about six hours. The popular media reported the impact of this failure prominently.

NYT reported: “Gone in Minutes, Out for Hours: Outage Shakes Facebook”.

According to one estimate, this outage cost Facebook about $100 million in revenue losses and many billions due to the declining stock of the company.

We now see the sequence of events that caused this global problem.

Sequence of events

The following sequence of events led to the outage of Facebook and its accompanied services:

• A routine maintenance system was needed to find out the spare capacity on Facebook’s backbone network.
• Due to a configuration error, the maintenance system disconnected all the datacenters from each other on the backbone network. Earlier an automated configuration review tool was used to look for any issues in the configuration, but such tools are not perfect. In this specific case, the review tool missed the problems present in a configuration.
• The authoritative Domain Name Systems (DNS) of Facebook had a health-check rule that if it can not reach Facebook’s internal data centers, it stops replying to client DNS queries by withdrawing the routes.
• When networks routes (where Facebook’s authoritative DNS was hosted) were withdrawn, soon all cached mapping of human-readable names to IPs timed out at all public DNS resolvers. (When a client resolves www.facebook.com, the DNS resolver first goes to one of the root DNS servers, which provides a list of authoritative DNS servers for .com. Resolver connects to one of them, and then they provide IPs for the authoritative DNS servers for Facebook. But now, after route withdrawal, it is impossible to reach them.)
• Now, no one is able to reach Facebook and its subsidiaries.

The slides below depict the events in a pictorial form.