Probing Phases and Conditions
Explore how to validate the state and health of Kubernetes Pods using probes that check for phases like Running and readiness status. Understand the importance of these checks in chaos engineering to ensure experiments accurately reflect system resilience and reveal underlying issues such as missing dependencies or faulty application states.
We'll cover the following...
We are most likely not sure whether Pods exist. That does not really serve much of a purpose. Instead, we should validate whether the Pods are healthy. Knowing that a Pod exists does not really do us much good if it is unhealthy or if the application inside it is hanging. What we really want to validate is whether the Pods exist and whether they are in a certain phase.
Describing the pod and inspecting the conditions
Before we go through that, let’s describe the Pod that we created and see the types, phases, and conditions it’s in.
The output, limited to the relevant parts, is as follows.
...
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
...
If we take a closer look at the conditions section, we can see that there are two columns: the type and the status. We can see that this Pod is initialized, but it is not yet Ready. We can see that there are different conditions of that Pod that we might want to take into account when constructing our experiments. What is even more important is that we have not really checked the status of our Pod, neither through experiments nor with kubectl. Maybe that Pod was never really running. Perhaps it was always failing, no matter the experiments that we were running, which is entirely possible because our experiments were not validating the readiness and the state of our Pod. They were just checking whether it exists. Maybe it did exist, but it was never functional.
Let’s introduce some additional arguments to our experiment to validate this.
Inspecting the definition of terminate-pod-phase.yaml
We’ll take a look at a new YAML definition.
What matters more is the diff.
The output is as follows.
> - name: pod-in-phase
> type: probe
> tolerance: true
> provider:
> type: python
> func: pods_in_phase
> module: chaosk8s.pod.probes
> arguments:
> label_selector: app=go-demo-8
> ns: go-demo-8
> phase: Running
> - name: pod-in-conditions
> type: probe
> tolerance: true
> provider:
> type: python
>