Once you start managing more than one Kubernetes cluster, you'll start to demand more from your
$KUBECONFIG. Early on in my career spinning pods, I kept a separate
.kubeconfig file for each cluster, and wrote a small shell function to set the requisite environment variables. To switch between clusters was as simple as:
→ k is buffalo-lab kubernetes: buffalo-lab (config): /Users/jhunt/.k/buffalo-lab
I did this because the "correct" command was too much to remember:
kubectl config use-context buffalo-lab
See what I mean?
Then I met kubectx and kubens, and they quickly became invaluable.
Now I've got this directory full of all these disparate, discrete configuration files, and I don't want to hand-edit them back together. Thankfully,
kubectl itself can merge them for us!
kubectl will only merge configuration files if you specify them via the
$KUBECONFIG environment variable. You cannot use the
--kubeconfig command-line flag, it won't work.
To demonstrate this, I'm going to show you my
$ ls -l ~/.k total 96 -rw-r--r-- 1 jhunt staff 7050 Jan 6 09:56 buffalo-lab -rw-r--r-- 1 jhunt staff 5641 Jan 1 11:30 jhunt-vcp-prodernetes.yml -rw-r--r-- 1 jhunt staff 5633 Jan 1 11:30 jhunt-vcp-tinynetes -rw-rw-rw- 1 jhunt staff 5481 Jan 3 12:32 lke-eval1 -rw-r--r-- 1 jhunt staff 23633 Jan 1 11:56 merged -rw------- 1 jhunt staff 422 Nov 17 14:25 minikube -rw-r--r-- 1 jhunt staff 7023 Jan 1 11:30 oss -rw-r--r-- 1 jhunt staff 25706 Jan 3 12:40 tmp
kubectl config get-contexts commands, we can see that each of them is different:
$ KUBECONFIG=~/.k/buffalo-lab \ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * buffalo-lab-k8s buffalo-lab-k8s admin default $ KUBECONFIG=~/.k/lke-eval1 \ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kubernetes-admin@kubernetes kubernetes kubernetes-admin k8s-cp-ubuntu
If you put two (or more) files, separated by colons, in
kubectl will merge them together semantically, and operate on the composite configuration. Any changes you make, like changing namespaces or setting contexts, will update the appropriate member file(s).
$ KUBECONFIG=~/.k/buffalo-lab:~/.k/lke-eval1 \ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * buffalo-lab-k8s buffalo-lab-k8s admin default kubernetes-admin@kubernetes kubernetes kubernetes-admin k8s-cp-ubuntu
To write a new, aggregate configuration that contains the results of the merge, we can use
kubectl config view --raw. The
--raw flag prevents
kubectl from redacting user secrets, certificates, and keys. To save this combined file, we can employ some shell redirection, along with a bit of caution.
Every UNIX shell I've ever used truncates files that are the target of
>-style output redirection, before the command producing the output gets executed. It's the nature of UNIX pipelines. We have to be careful with the order of our operations.
This will fail spectacularly (and most assuredly destroy data in the process!):
# DON'T RUN THIS! $ KUBECONFIG=a:b:c kubectl config view --raw > a
a configuration file will be empty by the time
kubectl gets around to trying to merge it in with the other two. Instead, use a temporary file and a rename operation:
$ KUBECONFIG=tmp:a:b:c kubectl config view --raw > tmp \ && mv tmp a
Pro Tip: While you are merging all of these configurations together, you can also embed any referenced, external files (like certificates and private keys) by swapping out the
--raw flag for
--flatten. This produces "portable" kubeconfigs, which do not have any dependencies on other parts of your filesystem. They can be moved, or copied, without losing their utility.