Running Scheduled Experiments
Explore how to run scheduled chaos engineering experiments in Kubernetes clusters with CronJobs. Understand how to automate and randomize experiment timing to objectively test system resilience, while persisting experiment results for analysis.
We'll cover the following...
In some cases, one-shot experiments are useful. You might want to trigger an experiment based on specific events (e.g., deployment of a new release), or as a scheduled exercise with “all hands on deck.” However, there are situations when you might want to run chaos experiments periodically. You might, for example, decide to execute them once a day at a specific hour, or you might even choose to randomize that and run them periodically at a random time of a day.
Testing for objectiveness
We want to test the system and be objective. This might sound strange, but being objective with chaos engineering often means being, more or less, random. If we know when something potentially disrupting might happen, we might react differently than when in unexpected situations. We might be biased and schedule the execution of experiments at the time when we know that there will be no adverse effect on the system. ...