Outage Amplification
Explore outage amplification in distributed systems by examining conflicts between automation and manual interventions, such as the Reddit outage. Understand how failures in service discovery and control plane software can multiply issues, and learn strategies to prevent these cascading failures in real-world production environments.
We'll cover the following...
Force Multiplier
Like a lever, automation allows administrators to make large movements with less effort. It’s a force multiplier.
Reddit outage
On August 11, 2016, link aggregator Reddit.com suffered an outage. It was unavailable for approximately 90 minutes and had degraded service for about another 90 minutes . In their postmortem, Reddit admins described a conflict between deliberate, manual changes and their automation platform:
-
First, the ...