The EC2 Monitoring Loop

An EC2 instance can show 2/2 checks passed in the console while the workload is still failing. The instance is reachable, the hypervisor can talk to it, but the process serving requests is throwing 500s, stuck on disk I/O, or blocked on an upstream dependency that only application logs reveal.

The opposite also happens. A status check can fail because the platform cannot reach the instance’s network path, even if the last SSH session was fine and the app looked healthy minutes ago. Those two scenarios are why two kinds of evidence should stay in view at all times: AWS-native health and host-level telemetry.

By the end of this course, the steady state is an evidence-driven loop for operating AWS services under exam-style and production-style constraints. In this chapter, that loop is applied to EC2 hosts and fleets so that every alarm points to a next inspection step, and every fix has a clear recovery signal.

The minimum topology and signal boundaries

The basic topology is an EC2 instance with an instance profile that can publish telemetry, CloudWatch receiving metrics, and CloudWatch Logs receiving log streams. If an Auto Scaling group is added, the replacement behavior matters because telemetry breaks in new ways when instances churn, such as new instance IDs producing new log streams and dashboards going quiet if they filter on old dimensions.

What makes the topology workable is knowing what AWS can observe from the hypervisor and platform layer alone, without any access to the guest OS itself. EC2 status checks come from the platform and include system reachability and instance reachability. Basic CloudWatch metrics like CPUUtilization are also native. They exist because AWS can measure them at the host and hypervisor layers, even when the filesystem is full or the process table is thrashing.

The diagram below shows signals when native and CloudWatch Agent signals are enabled.

When the native path is all we have, a 2/2 checks passed status tells us the platform can run and reach the instance, not that our workload is healthy. When the agent is enabled, we add OS and application evidence that can explain why latency and errors rise even though the platform still reports healthy infrastructure.

Safe access and verification checklist

Evidence collection starts with what can be gathered without changing anything on the instance. Because a recent reboot can otherwise be mistaken for a long-running, stable host, the console State, InstanceId, and LaunchTime are confirmed first. Since assuming reachability leads to chasing the wrong layer, the intended access path, such as SSH on 22 from a bastion or Systems Manager, is confirmed next, and the security group rules are checked against that path rather than assumed.

In the AWS CLI, commands stay read-only, and hardcoded credentials are avoided by relying on the configured environment and role chaining.

The following query gathers evidence rather than confirming health on its own.

This returns State.Name, PrivateIpAddress, and SecurityGroups, which map network symptoms to the actual attachment points.

The next query retrieves the status check results directly.

This returns InstanceStatus.Status and SystemStatus.Status together with the timestamps showing when checks last changed. A common mistake is reading an old failing check as current, so recency fields such as events and check transition times should always be checked.

The following query pulls metric data points with timestamps for a requested window.

An empty array here should be treated as missing evidence until the metric name, namespace, dimensions like InstanceId, and whether detailed monitoring is enabled are all confirmed.

Evidence rule: A signal is usable only if it can be shown to be present and fresh. Green checks without recent timestamps are not enough.

What the CloudWatch agent changes

The CloudWatch agent turns OS state and application logs into first-class signals that alarms can act on. Instead of guessing whether a slow instance is out of CPU or out of memory, disk-used percent, swap, and filesystem metrics can be published directly. Instead of logging into the host to read /var/log/..., those log lines can be routed into CloudWatch Logs, where filters and alarms can watch for error signatures.

Missing telemetry has a distinct shape. There might be no log group, or a log group with no streams for the current instance ID, or streams that stop updating. Custom metrics might never appear in the expected namespace, or they might flatline because the agent is not running, is misconfigured, or cannot reach CloudWatch endpoints due to VPC routing.

This comparison table helps decide which signal family can answer a given question and what risk to accept if it is absent.

Monitoring verifies that metrics and logs arrive. We check freshness by looking at the newest timestamp, and we write down the exact namespace and log group patterns to set an alarm later. A common reason alarms never fire is naming drift.

The monitoring loop

The monitoring loop is to observe signals, decide based on evidence, change one thing, and then confirm recovery in the same signals that triggered the action. Proof means that signals are present and fresh, alarms transition state when a condition is induced or fixed, remediation changes the instance life cycle state when appropriate, and metrics and logs show that the system has returned to expected levels.

Procedures that vary by AMI, OS, or agent version stay verification-oriented. The stance throughout is to confirm what is running, what it is emitting, and where it lands in CloudWatch and CloudWatch Logs. This chapter builds reliable alerts and remediation steps on top of that instrumentation.