Deploying Dashboard Applications

Improving chaos experiments

We’re going to take chaos experiments to the next level. We’ll randomize the targets by making them independent of specific applications. To do that, we need to rethink how we validate the results of such experiments. Steady-state hypotheses in Chaos Toolkit will not get us there. They assume that we know in advance what to measure and what the desired state is, both before and after the experiments. If, for example, we go crazy and start removing random nodes, we won’t be able to define the state of the whole cluster or the entire Namespace in advance. Or, at least, we won’t be able to do that in Chaos Toolkit format. That would be impractical, if not impossible.

What do we need?

What we need is a proper monitoring and alerting system that will allow us to observe the cluster as a whole and (potentially) get alerts when something goes wrong in any part of the system. With proper monitoring, dashboards, and alerting in place, we can start doing random destruction and catch many (if not all) anomalies in the system. So, our first action will be to introduce a few additional tools.

1. We’re going to use Prometheus for gathering and querying metrics. It’s already running, so there’s no need to deploy it.
2. We’re going to install Grafana for visualizing those metrics.
3. We’re going to use Kiali for visualizing service mesh with Istio.

But before we do any of those things, we need to generate some traffic in our cluster that will create observable metrics. Otherwise, if our cluster is dormant and nothing is happening inside, we would not be able to see those metrics in action and explore monitoring and alerting.

IP address to generate traffic

So, the first thing we are going to do is to generate some traffic. For that, we need to know the IP address through which we can send requests to the cluster in general, and the demo application in particular. We already have Istio Gateway, and you should have defined the environment variable INGRESS_HOST. At least, you should have those if you run the commands from the Gists with the instructions on how to create a cluster. To be on the safe side, we’re going to output the value of the INGRESS_HOST variable and confirm that it looks like a valid address or an IP.