Expand your Kubernetes knowledge with techniques to make your clusters dynamic and resilient.

DevOps for Developers

The world of DevOps has exploded with the help of technologies like Kubernetes and Docker.
In this course, you will learn about more advanced Kubernetes features such as monitoring, alerting, logging, and scaling - crucial concepts for any DevOps professional. You will learn how to monitor clusters, how to send alerts when potential issues arise, and how to query the whole system’s metrics and logs.
By the end of this course, you will have the required knowledge to make your clusters and applications truly dynamic and resilient, so that they require minimal manual involvement and are self-adaptive.

Kubernetes Monitoring, Logging and Auto-Scaling

# Database for logs

The first thing we need to do is to find a place where we'll store logs. Given that we want to have the ability to filter log entries, storing them in files should be discarded from the start. What we need is a database, of sorts. It is more important that it is fast than transactional, so we are most likely looking into a solution that is an in-memory database. But, before we take a look at the choices, we should discuss the location of our database. Should we run it inside our cluster, or should we use a service? Instead of making that decision right away, we'll explore both options, before we make a choice.

# Logging-as-a-service solutions

There are two major groups of logging-as-a-service types. If we are running our cluster with one of the Cloud providers, an obvious choice might be to use a logging solution they provide. EKS has **AWS CloudWatch**, GKE has **GCP Stackdriver**, and AKS has **Azure Log Analytics**. If you choose to use one of the Cloud vendors, that might make a lot of sense. Why bother with setting up your own solution of looking for a third-party service, if everything is already set up and waiting for you? We'll explore them soon.

## Outside hosting vendors

Since my mission is to provide instructions that work for (almost) anyone, we'll also explore one of the logging-as-a-service solutions found outside hosting vendors. But, which one should we select? There are too many solutions in the market. We could, for example, choose [Splunk](https://www.splunk.com/) or [DataDog](https://www.datadoghq.com/). Both are excellent choices, and both are much more than only logging solutions. We can use them to collect metrics (like with `Prometheus`). They provide dashboards (like `Grafana`), and a few other things. Later on, we'll discuss whether we should combine logs and metrics in a single tool. For now, our focus is only on logging, and that's the main reason we'll skip Splunk, DataDog, and similar comprehensive tools that offer much more than what we need. That does not mean that you should discard them, but rather this chapter tries to maintain the focus on logging.

There are many logging services available, with [Scalyr](https://www.scalyr.com/pricing), [logdna](https://logdna.com/), and [sumo logic](https://www.sumologic.com/) being only a few. We won't go through all of them since it would take much more time and space than I feel is useful. Given that most services are very similar when logging is involved, I'll skip a detailed comparison and jump straight into [Papertrail](https://papertrailapp.com/), my favorite logging service. Please bear in mind that we'll use it only as an example. I will assume that you'll check at least a few others and make your own choice based on your needs.

# Self-hosted logging solution

Logging-as-a-service might not be a good fit for all. Some might prefer a self-hosted solution, while others might not even be allowed to send data outside their clusters. In those cases, a self-hosted logging solution is likely the only choice. Even if you are not restrained to your own cluster, there might be other reasons to keep it inside, latency being only one of many. We'll explore a self-hosted solution as well, so let us pick one. Which one will it be?

## Traditional databases

Given that we need a place to store our logs, we might look into traditional databases. However, most of them would not fit our needs. Transactional databases like **MySQL** require fixed schemas so we can discard them immediately. **NoSQL** is a better fit so we might choose something like **MongoDB**. But, that would be a poor choice since we require the ability to perform very fast free-text searches. For that, we probably need an in-memory database. **MongoDB** is not one of those. We could use Splunk Enterprise, but this course is dedicated to free (mostly open-source) solutions. The only exception we made so far is with Cloud providers, and I intend to keep it that way.

The few requirements we mentioned (fast, free-text, in-memory, and a free solution) limit our potential candidates to only a few. [Solr](http://lucene.apache.org/solr/) is one of those, but its usage has been dropping, and it is rarely used today (for a good reason). The solution that sticks out from a tiny crowd is [Elasticsearch](https://www.elastic.co/products/elasticsearch). If you have a preference for a different on-prem solution, consider the examples we'll go through as a set of practices that you should be able to apply to other centralized logging solutions. 

All in all, we'll explore an example of independent logging-as-a-service offering (**Papertrail**), we'll explore the solutions provided with Cloud hosting vendors (**AWS CloudWatch**, **GCP Stackdriver**, and **Azure Log Analytics**), and we'll try to set up **ElasticSearch** with a few friends. Those should provide enough examples for you to choose which type of solution fits your use case the best.


In this lesson, we will see which solution to choose as a centralized logging solution.

Choosing a Centralized Logging Solution

Database for logs