Expand your Kubernetes knowledge with techniques to make your clusters dynamic and resilient.

DevOps for Developers

The world of DevOps has exploded with the help of technologies like Kubernetes and Docker.
In this course, you will learn about more advanced Kubernetes features such as monitoring, alerting, logging, and scaling - crucial concepts for any DevOps professional. You will learn how to monitor clusters, how to send alerts when potential issues arise, and how to query the whole system’s metrics and logs.
By the end of this course, you will have the required knowledge to make your clusters and applications truly dynamic and resilient, so that they require minimal manual involvement and are self-adaptive.

Kubernetes Monitoring, Logging and Auto-Scaling

The title might be confusing by itself, so let us briefly describe each of the elements mentioned in it.

# `Prometheus` Alerts vs `Grafana` notifications

`Prometheus`'s Alerts and `Grafana` notifications serve the same purpose, even though we did not explore the latter. I'll let you learn how `Grafana` notifications work on your own. Who knows? After the discussion that follows you might not even want to spend time with them.

`Grafana` notifications can be forwarded to different recipients in a similar manner as how `Prometheus`'s alerts are forwarded with `Alertmanager`. However, there are a few things that make `Grafana` notifications less appealing.

If we can accomplish the same result with `Prometheus`'s alerts as with `Grafana` alerts, there is a clear advantage with the former. If an alert is fired from `Prometheus`, that means that the rules that caused the alert to fire are also defined in `Prometheus`. As a result, evaluations are happening at the data source, and we are avoiding unnecessary latency between `Grafana` and `Prometheus`. The closer we are to the data source, the better. In case of alerts/notifications, closer means inside `Prometheus`.

Another advantage of defining alerts in `Prometheus` is the fact that it allows us to do more. For example, there is no equivalent to Prometheus' `for` statement in `Grafana`. We cannot define a notification that will fire only if the conditions persist for a while. We'd need to resort to non-trivial additions to the queries to accomplish the same. `Alertmanager`, on the other hand, provides more sophisticated ways to filter the alerts, to group them, and to forward only those that match certain criteria. There are many other advantages to defining alerts in `Prometheus` and `Alertmanager` instead of notifications in `Grafana`. But, we won't go into all of them. I'll leave it to you to find all the differences unless you are already convinced to ditch `Grafana` notifications in favor of `Prometheus` alerts and `Alertmanager`.

There is one important reason why you shouldn't dismiss `Grafana` notifications completely. The data source you're using might not have an alert/notifications mechanism, or it might be part of an enterprise license you do not possess. Since `Grafana` supports many different data sources, with `Prometheus` being only one of them, `Grafana` notifications allow us to use any of those data sources, or even to combine them.

> 🔍 Stick with Prometheus for alerts/notifications based on metrics stored there. For other data sources, Grafana alerts might be better or even the only option.
Now that we briefly explored the differences between `Prometheus` Alerts and `Grafana` notifications, we'll move into `semaphores`.

# `Semaphores` provide awareness of the System

`Semaphores` (`Grafana` dashboards based on single stat panels) do not replace `Prometheus` alerts. First of all, it is hard, or even impossible, to create semaphores that turn red only if a value reaches a threshold for some time (e.g., like the `for` statement in `Prometheus`'s alerts). That means that a semaphore might turn red, only to go back to green a few moments later. That is not a cause for action since the problem was resolved automatically short while later. If we would jump every time something turns red in `Grafana`, we'd be in excellent physical shape, but we wouldn't do much useful work.

`Semaphores` are an indication of a possible problem that might not require any intervention. While such false positives should be avoided, it's almost impossible to get rid of them altogether. That means that we should stare at the screen to see whether a red box continues being red for at least a few minutes before we act. The primary purpose of `semaphores` is not to provide a notification to a person or a team that should fix the issue. Notifications to Slack, email, and other destinations do that. Semaphores provide awareness of the state of the system.

Finally, we explored alerts defined on graphs. Those are the red lines and zones in the graphs. They are not good indicators that there is something wrong. They are not that easy to spot so they cannot raise awareness, and they definitely do not replace notifications. Instead, they help us after we discover that there is an issue. If a notification or a `semaphore` alerts us that there is a problem that might need to be fixed, graph alerts help us identify the culprit. Which Pod is in the red zone? Which ingress received more requests than expected? Those are only a few questions that we can answer through graph alerts.


In this lesson, we will see the difference between Prometheus  Alerts and Grafana notifications.

Prometheus and Graph Alerts, Grafana Notifications, Semaphores