Sending Selective Notifications

We are now receiving notifications when an experiment starts and ends, no matter if it is successful or not. However, this might be too much. We can easily be overwhelmed with too many notifications, and, as a result, we will likely start ignoring notifications and miss those that do matter. Instead, I prefer to receive notifications only when something terrible happens. I prefer to receive notifications from the system only when they are critical. More often than not, that means that we might want to receive only notifications that should result in some actions. What’s the point of knowing that an experiment was successful just as many others were successful before. No news can be interpreted as good news. Instead, I prefer to receive notifications that indicate that there is an action that I should perform. A failed experiment should be a clear indication that the system does not behave as it should, and that we should do our best to improve it so that the same experiment is successful the next time it runs.

We’re going to modify the notifications section in Chaos Toolkit settings so that we are notified only when experiments fail. That way, we’ll be ignorant when they’re successful, and we’ll be ready to act when they fail.

Inspecting the settings-failure.yaml file

Let’s take a look at yet another YAML.