Deploying Dashboard Applications
Explore how to deploy and configure Grafana and Kiali dashboards within a Kubernetes cluster to support chaos engineering experiments. Understand how to generate traffic, monitor system metrics, and validate chaotic tests using these monitoring tools to improve cluster observability and resilience.
Improving chaos experiments
We’re going to take chaos experiments to the next level. We’ll randomize the targets by making them independent of specific applications. To do that, we need to rethink how we validate the results of such experiments. Steady-state hypotheses in Chaos Toolkit will not get us there. They assume that we know in advance what to measure and what the desired state is, both before and after the experiments. If, for example, we go crazy and start removing random nodes, we won’t be able to define the state of the whole cluster or the entire Namespace in advance. Or, at least, we won’t be able to do that in Chaos Toolkit format. That would be impractical, if not impossible.
What do we need?
What we need is a proper monitoring and alerting system that will allow us to observe the cluster as a whole and (potentially) get alerts when something goes wrong in any part of the system. With proper monitoring, dashboards, and alerting in place, we can start doing random ...