System Design: Metrics Monitoring and Alerting System
A metrics monitoring and alerting system is essential for managing large-scale services, enabling operators to detect issues before they impact customers. It involves collecting numerical telemetry, storing it as time series data, and evaluating alert rules. Key requirements include metrics collection, time-series storage, alert management, and efficient querying. The system must handle high ingestion rates, ensure low-latency reads, and maintain reliability and cost efficiency. Resource estimation for storage, bandwidth, and server needs is crucial, alongside the integration of AI-assisted features for incident analysis, all of which shape the overall architecture of the system.
Introduction
A large service can have hundreds of thousands of machines, and each one can emit health signals every few seconds. If a bad deploy pushes latency up in one region, operators need the monitoring stack to surface it before customers open incident tickets.
From a System Design perspective, a metrics monitoring and alerting system handles that flow by collecting numerical telemetry, storing it as time series, serving dashboards, and continuously evaluating alert rules.
The following illustration shows the high-level workflow of a metrics monitoring and alerting system, from metric collection to alert notification:
A metric is a numeric measurement recorded at a specific point in time. Common examples include CPU utilization, request latency, error rate, queue depth, and throughput. Each measurement is called a sample, and it usually carries labels such as service, region, environment, and instance ID so the system can group and filter later.
Note: A single sample is small, but the system never stops receiving new ones.
Time-series data behaves differently from transactional data. Metric data is ingested continuously, making data freshness essential for timely and accurate alerting. To optimize storage efficiency and query performance, historical data is typically
In this chapter, we will focus on designing a metrics monitoring and alerting system. This lesson covers its requirements, scale, and capacity. Let’s start by identifying the requirements for such a system.
Requirements
The design starts with the data path. Hosts, containers, and services expose or emit measurements, and the platform ingests them through either push or pull collection. The incoming samples must be normalized into a common schema of metric name, timestamp, value, and labels before storage. This design focuses on metrics and the alerts built on them. ...