Understanding Remote Configuration

In the age of the cloud and distributed systems, applications often run in multiple instances or containers. Pushing a configuration change to all the instances simultaneously is often not trivial. With remote configuration, all the instances can read their configuration from a single source of truth.

Overview

Viper can work with multiple remote key-value stores as configuration sources. Remote configuration has the lowest priority, besides defaults. Configuration files, environment variables, and command-line arguments can all override it. Let’s discuss the key-value stores Viper supports as remote configuration sources.

Understanding remote key-value stores

Remote key-value stores can store and retrieve configuration data as key-value pairs. The data is stored in a central location and serves as a one-stop shop for the clients. This enables dynamic configuration because there is no need to re-deploy or even re-run the program with a new configuration file, new environment, or new command-line arguments.

You can also do dynamic configuration with configuration files by watching the file for changes, but this that still requires updating the configuration file. In many cloud-based systems, the configuration files are packaged with the code (e.g. in containers), and changes to a configuration file require a whole new deployment. When you have multiple instances of the same program or service, it becomes even more complicated and might require rolling updates, blue-green, or canary deployments.

Remote configuration stores can be updated once, and all the programs or services will be notified that the configuration changed without requiring re-deployment or even restart.

Viper stores its configuration in remote key-value stores in one of the configuration file formats it supports, like JSON, TOML, YAML, HCL or envfile.

Viper supports the following remote configuration stores:

• Etcd (V2)
• Consul
• Firestore

Working with remote configuration

Let’s start a local etcd V2 key-value store

docker run -d -p 4001:4001 quay.io/coreos/etcd:v2.0.11 --advertise-client-urls http://0.0.0.0:4002 --listen-client-urls http://0.0.0.0:4001

Note that the Viper doesn’t support the latest version of etcd V3. I spent some time before I realized it.

Viper uses a command-line tool called crypt to manage information in etcd, or other supported key-value stores. With crypt, you can store the data encrypted or as plain text. We will use plaintext for simplicity.

Let’s get crypt:

$ go get github.com/bketelsen/crypt/bin/crypt
go: downloading github.com/bketelsen/crypt v0.0.2
go: found github.com/bketelsen/crypt/bin/crypt in github.com/bketelsen/crypt v0.0.2

With crypt installed, we can put some content into our etcd key-value store:

$ crypt set -plaintext /config/config.toml ./config.toml
$ crypt get -plaintext /config/config.toml
a = 1
b = 2
c = 3