Designing Safe, Concurrent Change Automations
Understand how to design and implement concurrent change automation in systems through Go.
We'll cover the following...
So far, we have shown how to execute commands locally or remotely. In the modern day, we often need to run sets of commands across multiple systems to achieve some end state. Depending on our scale, we may want to run a system such as Ansible or Jenkins to attempt to automate these processes.
For some work, it is simpler to use Go directly to execute changes across a set of systems. This allows the DevOps group to simply understand the Go language and a small bit of code versus understanding the complexities of a workflow system such as Ansible, which requires its own skillset, system updates, and so on.
In this lesson, we are going to talk about the components of changing a set of systems, a framework for achieving this, and an example application to apply a set of changes.
Components of a change
When writing a system that makes a change, there are several types of actions that must be dealt with. In broad terms, we define these as the following:
Global preconditions: Global preconditions are a set of conditions that need to be true to move forward. When doing network automation, this would be things such as the packet loss on the network being under a certain threshold. For devices, this might be that our services are in a green state before proceeding. No one wants to push changes during a problem.
Local preconditions: Local preconditions are the state of the individual work unit (say, a server) that needs to be in a certain state to proceed.
Actions: Actions are operations that will mutate the state of a work unit.
Action validations: Checks that are done to validate an action was successful.
Local postconditions: Local postconditions are checks that the work unit is both in the configuration state we want and meets some state. This might be that it is still reachable, possibly serving traffic or not serving traffic, whatever the end state should be.
Global postconditions: Global postconditions are the state of conditions after execution, usually similar to global preconditions.
Not every set of changes on multiple systems requires all these, but they will at least need a subset of them.
Let's take a look at doing a rollout of jobs on a set of virtual machines (VMs) in a single data center. For small shops that have a limited number of machines, a setup such as this can be sufficient when we aren't large enough to use something such as Kubernetes but can't fit ...