We’ll start the lesson at the cluster level, switch to the runtime level, and then look outside of the cluster at supporting infrastructure, such as network firewalls.

Cluster-level workload isolation

Cutting straight to the chase, Kubernetes does not support secure multi-tenant clusters. The only cluster-level security boundary in Kubernetes is the cluster itself.

Let’s look a bit closer.

The only way to divide a Kubernetes cluster is by creating namespaces. A Kubernetes namespace is not the same as a Linux kernel namespace, it is a logical partition of a single Kubernetes cluster. In fact, it’s little more than a way of grouping resources and applying things like:

• Limits
• Quotas
• RBAC rules

The take-home point is that Kubernetes namespaces cannot guarantee a Pod in one namespace will not impact a Pod in another namespace. As a result, you should not run potentially hostile production workloads on the same physical cluster. The only way to run potentially hostile workloads, and guarantee true isolation, is to run them on separate clusters.

Despite this, Kubernetes namespaces are useful, and you should use them – just don’t use them as security boundaries.

Let’s look at how namespaces relate to soft multi-tenancy and hard multi-tenancy.

Namespaces and soft multi-tenancy

For our purposes, we’ll define soft multi-tenancy as hosting multiple trusted workloads on shared infrastructure. By trusted, we mean workloads that do not require absolute guarantees that one Pod/container cannot impact another.

An example of two trusted workloads might be an e-commerce application with a web front-end service and a back end recommendation service. Both services are part of the same e-commerce application, so they are not hostile. However, they might benefit from:

• Isolating the teams responsible for the different services
• Having different resource limits and quotas for each service

In this situation, a single cluster with one namespace for the front-end service and another for the back-end service might be a good solution. However, exploiting a vulnerability in one service might give the attacker access to Pods in the other service.

Namespaces and hard multi-tenancy

Let’s define hard multi-tenancy as hosting untrusted and potentially hostile workloads on shared infrastructure. Only, as we said before, this isn’t currently possible with Kubernetes.

This means that truly hostile workloads – workloads that require a strong security boundary – need to run on separate Kubernetes clusters! Examples include:

• Isolating production and non-production workloads on dedicated clusters
• Isolating different customers on dedicated clusters
• Isolating sensitive projects and business functions on separate clusters

Other examples exist, but you get the picture. If you have workloads that require strong separation, put them on their own clusters.

Note: The Kubernetes project has a dedicated Multitenancy Working Group that is actively working on the multitenancy models that Kubernetes supports. This means that future releases of Kubernetes might support hard multitenancy.