Jump right into Kubernetes development with career-focused lessons and hands-on practice.

DevOps for Developers

Kubernetes is a powerful container management tool that's taking the world by storm. This detailed course will help you master it.

In this course, you'll start with the fundamentals of Kubernetes and what the main components of a cluster look like. You'll then learn how to use those components to build, test, deploy, and upgrade applications, as well as how to achieve state persistence once your application is deployed. You'll also learn how to secure your deployments and manage resources, which are crucial DevOps skills.

By the time you're done, you'll have a firm grasp on what Kubernetes is, as well as the skills to deploy your own clusters and applications with confidence.

A Practical Guide to Kubernetes

# Verifying the state persistence
Now that Jenkins is up and running, we’ll execute a similar set of steps as before and validate that the state has persisted across failures.


We have opened the Jenkins home screen.

## Creating a new job
If you are not authenticated, please click the “Log in” link and type “jdoe” as the User and “incognito” as the Password. Click the “log in” button.

You’ll see the “Create new jobs” link. Click it. Type “my-job” as the item name, select “Pipeline” as the job type, and click the “OK” button. Once inside the job configuration screen, all we have to do is click the “Save” button. An empty job will be enough to test persistence.
## Getting the Pod
Now we need to find out the name of the Pod created through the `jenkins` Deployment.

POD_NAME=$(kubectl \
    --namespace jenkins \
    get pod \
    --selector=app=jenkins \
    -o jsonpath="{.items[*].metadata.name}")

## Terminating a process
With the name of the Pod stored in the environment variable `POD_NAME`, we can proceed and terminate the `java` process that’s running Jenkins.

kubectl --namespace jenkins \
    exec -it $POD_NAME pkill java

We teminated the Jenkins process and therefore simulated failure of the container. As a result, Kubernetes detected the failure and recreated the container.

## Verification
A minute later, we can open the Jenkins home screen again and check whether the state (the job we created) is preserved.

As you can see, the job is still available, therefore proving that we successfully mounted the EBS volume as the directory where Jenkins preserves its state.

# Exploring the failures
## What if a node gets terminated?
If instead of destroying the container, we terminate the server where the Pod is running, the result would be the same from the functional perspective. The Pod would be rescheduled to a healthy node. Jenkins would start again and restore its state from the EBS volume. However, such behavior is not guaranteed to happen in our cluster.

We have only two worker nodes distributed in two (out of three) availability zones. If the node that hosted Jenkins failed, we’d be left with only one node. To be more precise, we’d have only one worker node running in the cluster until the Auto Scaling Group detects that an EC2 instance is missing and recreates it. During those few minutes, the single node we’re left with is not in the same zone. As we already mentioned, each EBS instance is tied to a zone, and the one we mounted to the Jenkins Pod would not be associated with the zone where the other EC2 instance is running.

As a result, the PersistentVolume could not rebound the EBS volume and, therefore, the failed container could not be recreated until the failed EC2 instance is recreated.

The chances are that the new EC2 instance would not be in the same zone as the one where the failed server was running. Since we’re using three availability zones, and one of them already has an EC2 instance, AWS would recreate the failed server in one of the other two zones. We have a fifty percent chance that the new EC2 would be in the same zone as the one where the failed server was running. Those are not good odds.


## More servers, more tolerance
In the real world scenario, we’d probably have more than two worker nodes. Even a slight increase to three nodes would give us a very good chance that the failed server would be recreated in the same zone. Auto-Scaling Groups are trying to distribute EC2 instances more or less equally across all the zones. However, that is not guaranteed to happen. A good minimum number of worker nodes would be six.

The more servers we have, the higher the chances that the cluster is fault-tolerant. That is especially true if we are hosting stateful applications. As it goes, we almost certainly have those. There’s hardly any system that does not have a state in some form or another.

If it’s better to have more servers than fewer, we might be in a complicated position if our system is small and needs, let’s say fewer than six servers. In such cases, we’d recommend running smaller VMs. If, for example, you planned to use three t2.xlarge EC2 instances for worker nodes, we might reconsider that and switch to six t2.large servers. More nodes mean more resource overhead spent on operating systems, Kubernetes system Pods, and few other things. However, we believe that is compensated with a more stable cluster.


## What if the availability zone fails?

There is still one more situation we might encounter. A whole availability zone (data center) might fail. Kubernetes will continue operating correctly. It’ll have two instead of three primary nodes, and the failed worker nodes will be recreated in healthy zones. However, we’d run into trouble with our stateful services. Kubernetes would be unable to reschedule those that were mounted to EBS volumes from the failed zone. We’d need to wait for the availability zone to come back online, or we’d need to move the EBS volume to a healthy zone manually. The chances are that, in such a case, the EBS would not be available and, therefore, could not be moved.
## Exploring the solutions
We could create a process that would replicate data in (near) real-time between EBS volumes spread across multiple availability zones, but that also comes with a downside. Such an operation would be expensive and likely slow down state retrieval while everything is fully operational. Should we choose lower performance over high availability? Is the increased operational overhead worth the trouble? The answer to those questions will differ from one use case to another.

There is yet another option. We could use the [Elastic File System (EFS)](https://aws.amazon.com/efs/) instead of EBS. But that would also impact performance since EFS tends to be slower than EBS. On top of that, there is no production-ready EFS support in Kubernetes. At the time of this writing, the [efs provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs) is still in the beta phase. By the time you read this, things might have changed. Or maybe they didn't. Even when the efs provisioner becomes stable, it will still be a slower and more expensive solution than EBS.

Maybe you'll decide to ditch EBS (and EFS) in favor of some other type of persistent storage. There are many different options you can choose. We won't explore them since an in-depth comparison of all the popular solutions would require much more space than what we have left.

>**Note:** All in all, every solution has pros and cons, and none would fit all use cases. For good or bad, we’ll stick with EBS for the remainder of this course.




Now that we’ve explored how to manage persistent static volumes, we’ll try to accomplish the same results using a dynamic approach. But before we do that, we’ll see what happens when some of the resources we created are removed.

---
In the next lesson, we will play around with the created deployment by removing the resources and exploring the effects.


Verify the state persistence of our Jenkins deployment and explore different failures that can occur.


Verifying State Persistence and Exploring Failures

Verify the state persistence of our Jenkins deployment and explore different failures that can occur.

Verifying the State Persistence and Exploring the Failures

Verifying the state persistence

Creating a new job