TauCeti.blog
Posts Tags
TauCeti.blog

Kubernetes the Not So Hard Way With Ansible - Upgrading Kubernetes

     2348 words   11 minutes 

2024-04-10

  • Use etcd version 3.5.13 as example
  • Use Kubernetes >= 1.29 as example

2023-07-20

  • Use etcd version 3.5.8 as example
  • Use Kubernetes >= 1.27 as example

2022-01-09

  • Use etcd version 3.5.1 as example
  • Use Kubernetes >= 1.21 as example
  • Mention that Docker/dockershim is deprecated since K8s 1.20 and will be removed with K8s 1.24.

2020-11-03

  • etcd: cert-etcd.pem was renamed to cert-etcd-server.pem
  • etcd: cert-etcd-key.pem was renamed to cert-etcd-server-key.pem
  • updated text

2020-04-04

  • Use etcd version 3.4.7 as example
  • Use ansible command to figure out etcd status and version

2019-08-18

  • Heptio Ark is now called Velero

2018-09-30

  • added hint about error message that could occur during upgrade of master modes while not all master/controller nodes using the same Kubernetes version

2018-09-26

  • added text about upgrading etcd cluster

If you followed my Kubernetes the Not So Hard Way With Ansible blog posts so far and have a Kubernetes cluster up and running you’ll sooner or later want to upgrade to the next version. With this setup it’s pretty easy.

My experience so far with upgrading to a new major Kubernetes release is that it is a good idea to wait for at least the .2 release. E.g. if K8s 1.29 is the freshest major release and you run 1.28.5 at the moment I would strongly recommend to wait at least for 1.29.2 before upgrading to 1.29 major release. The .0 are often contains bugs that are fixed in later minor releases and that are sometimes really hurts in production. But even minor releases sometimes contain changes that you wouldn’t expect. Having a development K8s cluster which is pretty close to the production cluster is very helpful to find issues before they hit you in production…

Of course if everyone waits for the .2 release nobody would test the releases before 😉 So if you’ve a development or staging environment to test new Kubernetes releases please test as early as possible and open bug tickets if you found any issues! If you are even able to test Alpha or Beta versions or release candidates of Kubernetes then even better. I’m pretty sure developers and all users will be very graceful to fix bugs before the final release.

BEFORE upgrading to a new major release (e.g. 1.28.x -> 1.29.x) make sure that you upgraded to the latest minor release of your currently running major release! E.g. if you currently run 1.28.5 and want to upgrade to 1.29.2 make sure you upgrade 1.28.5 to the latest 1.28.x release like 1.28.7 if that’s the latest 1.28.x release! Afterwards you can do the major release upgrade. That’s important! Otherwise you may risk a corrupt cluster state in etcd or other strange behaviors.

The first thing you should do is to read the CHANGELOG of the version you want to upgrade. E.g. if you upgrade from 1.29.1 to 1.29.2 (which in this case is only a bugfix release) you only need to read CHANGELOG-1.26. Watch out for Urgent Upgrade Notes headlines. This could even happen for patch releases. That shouldn’t happen for patch releases but sometimes it can’t be avoided (Kubernetes version schema doesn’t follow SemVer btw.).

If you want to upgrade the major version e.g. from 1.28.x to 1.29.x read the CHANGELOG-1.29. The same advice as above applies of course.

As the whole Kubernetes cluster state is stored in etcd you should definitely consider creating a backup of the etcd data. Have a look at the etcd Admin Guide how to do this. This is especially true if you upgrading to a new major release. Also Velero (formerly Heptio Ark) could be a option. Velero is a utility for managing disaster recovery, specifically for your Kubernetes cluster resources and persistent volumes.

If you considered above prerequisites we’re ready to go. If you do a minor release update (1.28.5 -> 1.28.6 e.g.) or a major release update (1.28 -> 1.29) the steps are basically the same. First update the Kubernetes controller nodes node by node and afterwards the Kubernetes worker nodes.

One additional hint: Upgrading a major release while skipping one major release is really a bad idea and calls for trouble 😉 So if you want upgrade from 1.27 to 1.29 your upgrade path should be 1.27 -> 1.28 -> 1.29.

From time to time the recommended and therefore tested/supported version of etcd changes. This is the case if you upgrade from K8s 1.28 to 1.29 e.g. The recommended etcd version for K8s 1.29 is 3.5.10. As time of writting this article etcd release 3.5.13 was already available so I used that one right away. In general it’s normally not a problem to use the latest etcd patch release. (search for etcd and kops or kubeadm in the CHANGELOG-1.29.md). So before upgrading Kubernetes I upgraded etcd first. The last time a major etcd upgrade happened was from Kubernetes 1.16 -> 1.17. In case of a major etcd version change upgrading might need more effort.

Have a look at the etcd upgrade guides. In this example it’s Upgrade etcd from 3.4 to 3.5. One of the first lines in the upgrade guide is: In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade. That’s cool because in that case etcd can be upgraded node by node (or service by service). But before moving on make sure to read the upgrade guide as a whole to catch changes regarding flags e.g. Also the CHANGELOG-3.5 might contain important information.

First check the etcd cluster state. Before you continue make sure that the cluster is a healthy state! Since I’m using Ansible to manage my Kubernetes cluster I can do like so:

bash

ansible -m shell -e "etcd_conf_dir=/etc/etcd" -a 'ETCDCTL_API=3 etcdctl endpoint health \
--endpoints=https://{{ ansible_wg0.ipv4.address }}:2379 \
--cacert={{ etcd_conf_dir }}/ca-etcd.pem \
--cert={{ etcd_conf_dir }}/cert-etcd-server.pem \
--key={{ etcd_conf_dir }}/cert-etcd-server-key.pem' \
k8s_etcd

I use Ansible’s shell module here. I also set a variable etcd_conf_dir which points to the directory where the etcd certificate files are located. That should be the same value as the value of etcd_conf_dir variable of the etcd role. Since my etcd processes listen on the WireGuard interface (in my case) I use ansible_wg0.ipv4.address here as wg0 is the name of my WireGuard interface (yours might be eth0 or ens0 e.g.). If you use a different port than the default etcd port 2379 then of course you need to change that one too. An output similar to this shows a healthy etcd cluster:

bash

etcd-node1 | CHANGED | rc=0 >>
https://10.8.0.101:2379 is healthy: successfully committed proposal: took = 2.807665ms
etcd-node2 | CHANGED | rc=0 >>
https://10.8.0.103:2379 is healthy: successfully committed proposal: took = 2.682864ms
etcd-node3 | CHANGED | rc=0 >>
https://10.8.0.102:2379 is healthy: successfully committed proposal: took = 10.169332ms

You can also check the current etcd API version (this will change if ALL etcd members are upgraded):

bash

ansible -m shell -e "etcd_conf_dir=/etc/etcd" -a 'ETCDCTL_API=3 etcdctl version \
--endpoints=https://{{ ansible_wg0.ipv4.address }}:2379 \
--cacert={{ etcd_conf_dir }}/ca-etcd.pem \
--cert=/etc/etcd/cert-etcd-server.pem \
--key=/etc/etcd/cert-etcd-server-key.pem' \
k8s_etcd

which produces an output like this:

bash

etcd-node1 | CHANGED | rc=0 >>
etcdctl version: 3.5.13
API version: 3.5
etcd-node2 | CHANGED | rc=0 >>
etcdctl version: 3.5.13
API version: 3.5
etcd-node3 | CHANGED | rc=0 >>
etcdctl version: 3.5.13
API version: 3.5

If the cluster is healthy the upgrade process can be started. More information about expected errors in the etcd logs e.g. can be found in Upgrade procedure which also contains information about upgrading etcd manually.

In my case I’ve my Ansible etcd role which will do the upgrade for me. So in that case I set the variable etcd_version from 3.4.14 to 3.5.8 (which is the lastest etcd version at this time) in group_vars/all.yml (or where ever it makes sense for you).

Now I upgrade the first etcd node e.g.:

bash

ansible-playbook --tags=role-etcd --limit=controller01.i.domain.tld k8s.yml

If this was successful restart the etcd daemon on that node:

bash

ansible -m systemd -a "name=etcd state=restarted" controller01.i.domain.tld

Also keep an eye on the etcd logs e.g.:

bash

ansible -m command -a 'journalctl --since=-15m -t etcd' controller01.i.domain.tld

If the logs are ok do the same for the remaining etcd nodes. If all etcd daemons are finally upgraded one should see something like this in the logs:

plain

Jan 09 22:17:52 controller03 etcd[5134]: {"level":"info","ts":"2022-01-09T22:17:52.065Z","caller":"etcdserver/server.go:2481","msg":"updating cluster version using v2 API","from":"3.4","to":"3.5"}
Jan 09 22:17:52 controller03 etcd[5134]: {"level":"info","ts":"2022-01-09T22:17:52.072Z","caller":"membership/cluster.go:576","msg":"updated cluster version","cluster-id":"b187005e7cc46340","local-member-id":"9f6c1751878e9f7c","from":"3.4","to":"3.5"}
Jan 09 22:17:52 controller03 etcd[5134]: {"level":"info","ts":"2022-01-09T22:17:52.072Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
Jan 09 22:17:52 controller03 etcd[5134]: {"level":"info","ts":"2022-01-09T22:17:52.072Z","caller":"etcdserver/server.go:2500","msg":"cluster version is updated","cluster-version":"3.5"}

Again we can check the API version:

bash

ansible -m shell -e "etcd_conf_dir=/etc/etcd" -a 'ETCDCTL_API=3 etcdctl version \
--endpoints=https://{{ ansible_wg0.ipv4.address }}:2379 \
--cacert={{ etcd_conf_dir }}/ca-etcd.pem \
--cert=/etc/etcd/cert-etcd.pem \
--key=/etc/etcd/cert-etcd-key.pem' \
k8s_etcd

bash

etcd-node1 | CHANGED | rc=0 >>
etcdctl version: 3.5.8
API version: 3.5
etcd-node2 | CHANGED | rc=0 >>
etcdctl version: 3.5.8
API version: 3.5
etcd-node3 | CHANGED | rc=0 >>
etcdctl version: 3.5.8
API version: 3.5

Now you have a shiny new etcd cluster running version 3.5! 😄 Afterwards consider restarting all kube-apiserver and make one or two test deployments. I have had the case in the past that everything looked fine after the upgrade e.g. kubectl get pods -o wide -A worked fine but later I figured out that it was not possible to create or change deployments…

Keep an eye in the changelog external dependencies. That’s mainly CNI plugins, containerd (or cri-o if you use that), runc and CoreDNS (Docker/dockershim is deprecated in K8s 1.20 and was removed with K8s 1.24). You may need to upgrade them too. If you need to upgrade CNI plugins, containerd and/or runc I would recommend to drain the node to do this kind of upgrade (see further down the text). This way you can easily upgrade node by node.

You can use my Ansible roles if you need one or all of the components mentioned above:

Updating kubectl utility before you upgrade the controller and worker nodes makes sense. Normally a newer client version can talk to older server versions. The other way round isn’t always true. When I update my K8s controller and worker roles I also update my kubectl role. In general kubectl and Kubernetes server version should only differ by one major release. So have a look if you can find the version you look for and upgrade kubectl locally first. E.g. if you followed my blog series to update kubectl the command would be:

bash

ansible-playbook --tags=role-kubectl k8s.yml

Update your inventory cache with ansible -m setup all.

The next thing to do is to set k8s_ctl_release. Let’s assume we currently have set k8s_ctl_release: "1.26.7" and want to upgrade to 1.27.4 so we set k8s_ctl_release: "1.27.4" in group_vars/all.yml (or whatever place you defined this variable).

Normally my Kubernetes controller role has also tagged various releases e.g. 21.0.0+1.27.4. In that case you can just update the role. Have a look at the CHANGELOG what changed and adjust your variables and maybe other things accordingly. If you don’t find the tag with the K8s release you need you have to adjust the settings by yourself according to the K8s changelog.

Next we deploy the controller role one by one to every controller node e.g.:

bash

ansible-playbook --tags=role-kubernetes-controller --limit=controller01.i.domain.tld k8s.yml

Of course replace controller01.i.domain.tld with the hostname of your first controller node. This will download the Kubernetes binaries, updates the old one and finally restarts kube-apiserver, kube-controller-manager and kube-scheduler.

After the role is deployed you should have a look at the logfiles (with journalctl e.g.) on controller01 to verify everything worked well. E.g. to get the logs of the three components from the last 15 minutes:

bash

ansible -m command -a 'journalctl --since=-15m -t kube-apiserver' controller01.i.domain.tld
ansible -m command -a 'journalctl --since=-15m -t kube-controller-manager' controller01.i.domain.tld
ansible -m command -a 'journalctl --since=-15m -t kube-scheduler' controller01.i.domain.tld

Also check if the services are still listen in the ports they usually do (netstat -tlpn or ss -tlpn e.g.). You could also do a small Kubernetes test deployment via kubectl to see if this still works.

If you see errors like this one

bash

v1beta1.apiextensions.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.apiextensions.k8s.io": the object has been modified; please apply your changes to the latest version and try again

that should be ok at the moment. At this point you’ve one controller node with a newer version of K8s and two other nodes with a older K8s version. This message should disappear if you’ve updated all controller nodes.

If everything is ok go ahead and update controller02 and controller03 e.g.:

bash

ansible-playbook --tags=role-kubernetes-controller --limit=controller02.i.domain.tld k8s.yml

# Wait until controller role is deployed on controller02...

ansible-playbook --tags=role-kubernetes-controller --limit=controller03.i.domain.tld k8s.yml

Now your controller nodes should be up2date!

As with the Kubernetes controller role mentioned above I also tag the Kubernetes worker role accordingly.

First set k8s_worker_release accordingly (e.g. k8s_worker_release: "1.26.7")

For the worker nodes it’s basically the same as with the controller nodes. We start with worker01

bash

ansible-playbook --tags=role-kubernetes-worker --limit=worker01.i.domain.tld k8s.yml

Of course replace worker01.i.domain.tld with the hostname of your first worker node. This will download the Kubernetes binaries, updates the old ones and finally restarts kube-proxy and kubelet. While the two services are updated they won’t be able to start new pods or change network settings. But that’s only true while the services are restarted which takes only a few seconds and they will catch up the changes afterwards. Shouldn’t be a big deal as long as you don’t have a few thousand pods running ;-)

You can also drain a node before you start upgrading that node to avoid possible problems (see Safely Drain a Node while Respecting Application SLOs). You can use kubectl drain to safely evict all of your pods from a node before you perform maintenance on the node (e.g. kernel upgrade, hardware maintenance, etc.). Safe evictions allow the pod’s containers to gracefully terminate and will respect the PodDisruptionBudgets you might have specified.

Again check the logs and if everything is ok continue with the other nodes:

bash

ansible-playbook --tags=role-kubernetes-worker --limit=worker02.i.domain.tld k8s.yml

# Wait until worker role is deployed on worker02...

ansible-playbook --tags=role-kubernetes-worker --limit=worker03.i.domain.tld k8s.yml

If the worker role is deployed to all worker nodes we’re basically done with the Kubernetes upgrade!

Updated on 2024-04-10
 Kubernetes, Update, Upgrade, Etcd, Kube-Apiserver, Kube-Controller-Manager, Kube-Scheduler, Kube-Proxy, Kubelet, Cni, Plugins, Containerd, Runc
 | Home