Search⌘ K

Terminating Random Nodes

Explore how to perform random node termination experiments in a Kubernetes cluster using CronJobs. Understand how to monitor the effects on cluster health and application stability through Grafana and Kiali dashboards. Gain insight into system resiliency and identify weak points in a controlled chaos environment.

Applying the CronJob

Let’s apply the new definition of the CronJob.

Shell
kubectl --namespace chaos apply --filename k8s/chaos/periodic-node.yaml

Assuming that you left the loop that sends requests running in the second terminal, we should be able to observe that the demo application keeps responding with 200. At the moment, the demo application seems to be working correctly.

As you already know, we’ll need to wait for a while until the first Job is created, and the experiment is executed.

Retrieving all CronJobs from the chaos Namespace

We’ll retrieve CronJobs to make it more interesting than staring at the blank screen.

Shell
kubectl --namespace chaos get cronjobs

After a while, and a few repetitions of the previous command, the output should be similar to the one that follows.

NAME        SCHEDULE    SUSPEND ACTIVE LAST SCHEDULE AGE
nodes-chaos */5 * * * * False   0      1m01s         5m2s

Once there is something other than <none> in the LAST ...