Level-up your testing with Chaos Engineering. Learn how pushing your programs to failure can help you plan your improvements.

DevOps for Developers

Let’s face it, bad things happen in software engineering, often when you least expect it. Servers go down, there are spikes in traffic, you receive networking errors. Luckily, you can avoid these problems. Enter chaos engineering.
Chaos engineering is where you try to find the limits of a system and simulate adverse effects that might happen. These simulations then help you to determine the best way to improve your system and build up its resiliency.
This course is divided into various sections to get you used to the different types of experiments you can run in chaos engineering. You will start by learning how to destroy application instances in order to show you the importance of scaling, fault tolerance, and high availability.
You will then dive deep into other types of experiments such as destroying a network, draining and deleting nodes, and testing availability. Towards the end of the course, you will learn how to run experiments in a Kubernetes cluster.

Kubernetes Chaos Engineering

We established that the requirements for a tool we'll pick are:

* To be open-source
* To work both inside Kubernetes and outside Kubernetes

What are the options? What are the tools we can choose from?

# Manual executions or automation by scripting

To begin, we could **do things manually**. We could modify and tweak Kubernetes resources ourselves, we could do some changes to service mesh, and so on and so forth. However, manual execution of experiments is not what we want. As I mentioned before, I believe that the execution of chaos experiments should be automated. It should be executable periodically, or through continuous delivery pipelines. So, manual chaos is not an option.

Of course, we could automate things by writing our own scripts. But, why would we do that? There are tools that can help us and get us from nothing to something very fast. That does not exclude writing your own custom scripts. You’re almost certainly going to end up creating your scripts sooner or later. However, picking a tool that already does at least some of the things we need will get us to a certain level much faster.

Now that we know that we’re not going to do manual chaos engineering or write all the scripts from scratch ourselves, let’s see which tools we have at our disposal.


# Chaos Monkey or Simian Army

There are [Chaos Monkey](https://github.com/netflix/chaosmonkey), [Simian Army](https://github.com/Netflix/SimianArmy), and other Netflix tools aimed at chaos engineering. What Netflix did with Chaos Monkey and the other tools is excellent. They were pioneers, at least among those that made their tools public. However, Chaos Monkey does not work well in Kubernetes. On top of that, it requires Spinnaker and MySQL. I don’t believe that everybody should adopt Spinnaker only for chaos engineering. Not working well in Kubernetes is a huge negative point and Spinnaker is another one. I’m not saying it’s a bad tool. Quite the contrary. Spinnaker is very useful for certain things. However, the question is whether you should adopt it only because you want to do some chaos engineering? Most likely not. If you are already using it, Chaos Monkey might be the right choice, but I cannot make that assumption.

# Gremlin

Next, we have [Gremlin](https://www.gremlin.com/). It might be one of the best tools we have on the market. While I encourage you to try it out and see whether it fits your needs, it's a service (you cannot run it yourself), and it's not open-source. Since open-source is one of the requirements, Gremlin is out as well.

# PowerfulSeal

Further on, we have [PowerfulSeal](https://github.com/bloomberg/powerfulseal/blob/master/README.md), which is immature and poorly documented. Besides that, it works only in Kubernetes and that fails one of our requirements. 

# kube-monkey

We also have [kube-monkey](https://github.com/asobti/kube-monkey), which is inspired by Chaos Monkey but is designed for Kubernetes. Just like PowerfulSeal, it is immature and poorly documented, and I do not recommend it.


# LitmusChaos

Then, we have [LitmusChaos](https://litmuschaos.io/), which suffers from similar problems. It is better documented than other tools I mentioned, but it is still not there. It is green, and it is Kubernetes only.

# Gloo Shot

We also have [Gloo Shot](https://glooshot.solo.io). I love the tools coming from [solo.io](https://www.solo.io/). I like Gloo, and I like their service mesh. However, at least at the time of this writing (March 2020), Gloo Shot is relatively new, and it works only on the service mesh level. So, it's also not a good choice.

# Chaos Toolkit

Finally, we have [Chaos Toolkit](https://chaostoolkit.org/). It is very well documented, and you should have no problems finding all the information you need. It has quite a few modules that significantly extend its capabilities. We can use it with or without Kubernetes. We can run experiments against GCP, AWS, Azure, service mesh (Istio in particular), etc. It has a very active community. If you go to their [Slack workspace](https://join.chaostoolkit.org/), you will see that there are quite a few folks who will be happy to help you out. Even though the project is not fully there, I think it's the best choice we have today (March 2020), at least among those I mentioned. 

# Our choice of tool

We are going to choose Chaos Toolkit because it’s open-source and works both inside and outside Kubernetes. It has decent documentation, and its community is always willing to help. That does not mean that other tools are bad. They’re not. However, we have to make a choice and that choice is Chaos Toolkit.

Remember, the goal of this course, and of almost everything I do, is to teach you how to think and the principles behind something, rather than how to use a specific tool. Tools are a means to an end. The goal should rarely be to master a tool but to understand the processes and the principles behind it.

By the end of this course, you will, hopefully, be a chaos engineering ninja, and you will know how to use Chaos Toolkit. If you do choose to use some other tool, you will be able to translate the knowledge of this course as the principles behind chaos engineering are the same no matter which tool you choose. You should be able to adapt to any tool easily. Nevertheless, I have a suspicion that you will like Chaos Toolkit and will find it very useful.


---

In the next lesson, we will define the requirements of this course.

Different options for tools are discussed in this lesson and the choice for the appropriate one is made.

Which Tool Should We Choose?