Summary

Need for vantage points

Most modern services are designed in such a way that failures are contained, and others might be localized for some users. Something is always failing for large services, and graceful degradation is used so that only a small portion of users are impacted for short time. Therefore we need globally dispersed vantage points to independently see the service status.

Many independent services, for example, Downdetector, exist for crowd-sourced problem reporting. An interesting point is that if you visit such a service and see the status of your favorite services, you will see that is always someone in the world facing some service problem.

Importance of independent service providers

One of the design goals of the original Internet was to provide resilience such that if one part fails, the rest could operate.

With the emergence of a handful of service providers over the last decade, critics have raised concernsSee “It’s time to decentralize the internet, again: What was distributed is now centralized by Google, Facebook, etc” by Bruce Davie. about such centralization and the potential impacts of failures. Most companies provide some kind of dashboard to enable users to see the service status. Some failures might knock out such dashboards, and other independent services such as Twitter have proven this point many times that independent, third-party services are valuable.

We will discuss the root cause of many failures to understand why they occurred and what to be cautious about in our designs.

The broader concept here is to use independent failure domains. A failure domain is a concept where anything failing inside a domain/network should not affect other components/services in other domains. At times we say that two domains are independent if they are outside the blast radius of each other.

Outage of Internet or service

As we discussed earlier that a distributed system can be divided into two components—compute part and the network. An outage can be over the network or inside the service.