Expand your Kubernetes knowledge with techniques to make your clusters dynamic and resilient.

DevOps for Developers

The world of DevOps has exploded with the help of technologies like Kubernetes and Docker.
In this course, you will learn about more advanced Kubernetes features such as monitoring, alerting, logging, and scaling - crucial concepts for any DevOps professional. You will learn how to monitor clusters, how to send alerts when potential issues arise, and how to query the whole system’s metrics and logs.
By the end of this course, you will have the required knowledge to make your clusters and applications truly dynamic and resilient, so that they require minimal manual involvement and are self-adaptive.

Kubernetes Monitoring, Logging and Auto-Scaling

## Logstash & Fluentd

For a long time now, there have been two major contestants for the "logs collection and shipping" throne. Those are [Logstash](https://www.elastic.co/products/logstash) and [Fluentd](https://www.fluentd.org/). Both are open-source, and both are widely accepted and actively maintained. While both have their pros and cons, **Fluentd** turned up to have an edge with cloud-native distributed systems. It consumes fewer resources and, more importantly, it is not tied to a single destination (**Elasticsearch**). While **Logstash** can push logs to many different targets, it is primarily designed to work with **Elasticsearch**. For that reason, other logging solutions adopted **Fluentd**. As of today, no matter which logging product you embrace, the chances are that it will support **Fluentd**. The culmination of that adoption can be seen by **Fluentd's** entry into the list of [Cloud Native Computing Foundation](https://www.cncf.io/) projects. Even **Elasticsearch** users are adopting **Fluentd** over **Logstash**. What was previously commonly referred to as ELK (*Elasticsearch*, **Logstash**, **Kibana**) stack, is now called EFK (**Elasticsearch**, **Fluentd**, **Kibana**).

We'll follow the trend and adopt **Fluentd** as the solution for collecting and shipping logs, no matter whether the destination is **Papertrail**, **Elasticsearch**, or something else.

We'll install **Fluentd** soon. But, since **Papertrail** is our first target, we need to create and set up an account. For now, remember that we need to collect logs from all the nodes of the cluster and, as you already know, Kubernetes' DaemonSet will ensure that a **Fluentd** Pod will run in each of our servers.


# Logstash & Fluentd

For a long time now, there have been two major contestants for the "logs collection and shipping" throne. Those are [Logstash](https://www.elastic.co/products/logstash) and [Fluentd](https://www.fluentd.org/). Both are open-source, and both are widely accepted and actively maintained. While both have their pros and cons, **Fluentd** turned up to have an edge with cloud-native distributed systems. It consumes fewer resources and, more importantly, it is not tied to a single destination (**Elasticsearch**). While **Logstash** can push logs to many different targets, it is primarily designed to work with **Elasticsearch**. For that reason, other logging solutions adopted **Fluentd**. As of today, no matter which logging product you embrace, the chances are that it will support **Fluentd**. The culmination of that adoption can be seen by **Fluentd's** entry into the list of [Cloud Native Computing Foundation](https://www.cncf.io/) projects. Even **Elasticsearch** users are adopting **Fluentd** over **Logstash**. What was previously commonly referred to as ELK (*Elasticsearch*, **Logstash**, **Kibana**) stack, is now called EFK (**Elasticsearch**, **Fluentd**, **Kibana**).

We'll follow the trend and adopt **Fluentd** as the solution for collecting and shipping logs, no matter whether the destination is **Papertrail**, **Elasticsearch**, or something else.

We'll install **Fluentd** soon. But, since **Papertrail** is our first target, we need to create and set up an account. For now, remember that we need to collect logs from all the nodes of the cluster and, as you already know, Kubernetes' DaemonSet will ensure that a **Fluentd** Pod will run in each of our servers.


In this lesson, we will explore two different contestants which we can use to explore logs collection and shipping.

Before We Begin

Before Getting Started

Autoscaling Deployments and StatefulSets

Auto-Scaling Nodes Of A Kubernetes Cluster

Collecting and Querying Metrics and Sending Alerts

Debugging Issues Discovered Through Metrics and Alerts

Extending HorizontalPodAutoscaler With Custom Metrics

Visualizing Metrics And Alerts

Collecting And Querying Logs

Conclusion

Exploring Logs Collection and Shipping

Logstash & Fluentd #