In my work I have to connect to 60+ clusters and maintaining the kubeconfigs of these clusters has become a task in itself. In this post I describe how I made this process easy without the use of external tools.

The problem

Just some context of this problem for those of you who are not familiar with kubeconfigs. If you are just looking for the solution scroll down to the solution.

Out of the box kubectl looks in the ~/.kube/config file for the clusters it can to connect to (the contexts). A simple example of such a file looks like this:

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <snip>
    server: https://kubernetes.docker.internal:6443
  name: docker-desktop
users:
- name: docker-desktop
  user:
    client-certificate-data: <snip>
    client-key-data: <snip>
contexts:
  - context:
      cluster: docker-desktop
      namespace: playground
      user: docker-desktop
  name: docker-desktop

Where you can see that there is a cluster key holding the cluster address and CA certificate. Then there is a user which has the client certificates. Then at last there is a context which relates those to be able to connect to the cluster. The name of the context is what you use with kubectl to switch clusters, docker-desktop in the above kubeconfig:

# kubectl config use-context docker-desktop

But since you now have discovered how awesome Kubernetes is, more clusters will follow ;) These clusters will have a similar context which needs to be merged into this ~/.kube/config file. Oke, easy. Just add the cluster, user and context by hand into the file. This works and that is how I started, but when you add a cluster you need to add the references in three places and that becomes annoying pretty quick.

The research

The first and most logic steps is to automate the merging and for a while I settled on this script to do that for me:

cp "$HOME/.kube/config" "$HOME/.kube/config.bak"; KUBECONFIG="$HOME/.kube/config:$HOME/Downloads/new.yaml"; \
kubectl config view --flatten > "$HOME/.kube/config"

And that works fine for adding clusters but after a while I needed to remove a cluster from my kubeconfig.

You can’t reverse the previous command so, I was back to editing the file by hand to remove the cluster, user and context for this cluster.

This is still manageable when you have just a couple clusters in the file, but as said in the intro I have more then a couple clusters to connect to (60+). When you have that many clusters in a single file these three items are miles apart and an error is made easily.

My current solution

At some point I was reading the Kubernetes documentation and I stumbled upon this:

The KUBECONFIG environment variable holds a list of kubeconfig files. For Linux and Mac, the list is colon-delimited.

If the KUBECONFIG environment variable does exist, kubectl uses an effective configuration that is the result of merging the files listed in the KUBECONFIG environment variable.

link to the kubernetes.io docs

So the merging by hand or automated isn’t even needed! You can create separate files per context, fill the KUBECONFIG environment variable with a comma-delimited list of these files and kubectl will handle the merging.

That sounds great and would simplify everything a lot! I created a $HOME/.kube/clusters/ directory in which I place all my contexts as separate files:

# ls -1 $HOME/.kube/clusters
docker-desktop.yaml
cluster-1.yaml
cluster-2.yaml

Then in my ~/.zshrc I fill the variable with all the files in this directory, de-limited by a colon (kubectl doesn’t care about the trailing colon):

export KUBECONFIG=$(find ~/.kube/clusters -type f | tr '\n' ':')

And after a reload of the shell, voila!

# echo KUBECONFIG
/home/wilmardo/.kube/clusters/docker-desktop.yaml:/home/wilmardo/.kube/clusters/cluster-1.yaml:/home/wilmardo/.kube/clusters/cluster-2.yaml:

# kubectl config get-clusters
docker-desktop
cluster-1
cluster-2

kubectl is correctly picking up the contexts from separate files in the $HOME/.kube/clusters directory. So now each clusters kubeconfig can exist as a separate file and removing the cluster is as easy as removing said file.

Honorable mentions

My solution is very simple, portable and completely independent of external tooling and will just work with kubectl. Of course everyone working with Kubernetes has this problem so there are several open-source tools to help you work efficiently with kubeconfigs:

  • sbstp/kubie: Has multiple extra features like a context per shell and executing commands against multiple clusters.
  • danielfoehrKn/kubeswitch: Overlapping features with kubie and adds the possibility to get kubecontexts from the cloud providers directly, so no local files anymore!