Kubernetes the not so hard way with Ansible - WireGuard - (K8s v1.28)

This post is based on Kelsey Hightower’s Kubernetes The Hard Way - Provisioning Compute Resources. Since I don’t use AWS or Google Cloud I don’t have the feature set of this two platforms at hand. But one can work around the shortcoming’s with some tools and one of this tool is WireGuard.

In harden the instances I secured the VM instances. Since we don’t have networking features like AWS VPC or Google Cloud Engine VPC I create a secure network with WireGuard. All Kubernetes services will only listen on the WireGuard interface (wg0 by default) for incoming requests and all communication between the Kubernetes hosts will be secured by WireGuard VPN. I’ll configure the Ansible WireGuard role to create a fully meshed VPN which includes at least all Kubernetes hosts. But you can also include additional hosts and even your laptop on which you execute Ansible commands or later even more important the Kubernetes control utility kubectl.

Setup WireGuard

I’ve prepared a Ansible role for installing WireGuard. You can again use

ansible-galaxy install githubixx.ansible_role_wireguard

to install the role or just clone it via git if you like.

By default port 51820 (protocol UDP) should be accessible from the outside. But you can adjust the port by changing the variable wireguard_port in group_vars, host_vars or whereever you specify your variables for a specific host or a hosts group. Also IP forwarding needs to be enabled e.g. via echo 1 > /proc/sys/net/ipv4/ip_forward. I decided not to implement this tasks in this Ansible role. IMHO that should be handled elsewhere. But you can use the wireguard_(preup|predown|postup|postdown) hooks for such kind of tasks. E.g.:

wireguard_preup:
  - echo 1 > /proc/sys/net/ipv4/ip_forward
  - ufw allow 51820/udp

This would enable IP forwarding and open port 51820 for UDP protocol before the WireGuard interfaces are created (in this case I assumed that UFW - Uncomplicated Firewall is used as an iptables fronted which is the case if you use harden_linux).

If you followed my blog series so far then you already know that my Ansible role harden_linux is also able to do this for you. See my previous blog post Kubernetes the not so hard way with Ansible - Harden the instances how to implement it. Besides changing sysctl entries (which you need to enable IP forwarding) it also manages firewall settings among other things. As a reminder the following settings should be set and deployed in case harden_linux role is used (if you use the default SSH port change 22222 to 22):

harden_linux_ufw_rules:
  - rule: "allow"
    to_port: "22222"
    protocol: "tcp"
  - rule: "allow"
    to_port: "51820"
    protocol: "udp"

harden_linux_sysctl_settings_user:
  "net.ipv4.ip_forward": 1
  "net.ipv6.conf.default.forwarding": 1
  "net.ipv6.conf.all.forwarding": 1

harden_linux_ufw_defaults_user:
  "^DEFAULT_FORWARD_POLICY": 'DEFAULT_FORWARD_POLICY="ACCEPT"'

For more information about other possible settings of the Ansible WireGuard role have a look at the README of that role.

You should already have a Ansible host_vars file for every host that is a member of your Kubernetes cluster and maybe also for your workstation you execute ansible-playbook and kubectl later e.g.:

k8s-010101.i.example.com # etcd #1
k8s-010102.i.example.com # K8s controller #1
k8s-010103.i.example.com # K8s worker #1
k8s-010201.i.example.com # etcd # 2
k8s-010202.i.example.com # K8s controller #2 
k8s-010203.i.example.com # K8s worker #2
k8s-010301.i.example.com # etcd #3
k8s-010302.i.example.com # K8s controller #3
k8s-010303.i.example.com # K8s worker #3

In Kubernetes the not so hard way with Ansible - Harden the instances I already showed you examples what settings you should set for every host so I won’t repeat everything here again. Just the example for the first controller node (which is k8s-010102 according to the list above and the file is host_vars/k8s-010102.i.example.com):

---
# Ansible
ansible_host: "k8s-010102.p.example.com"

# WireGuard
wireguard_address: "10.0.11.3/24"
wireguard_port: "51820"
wireguard_persistent_keepalive: "30"
wireguard_endpoint: "{{ ansible_host }}"

You need to set wireguard_address for every node. Of course every node needs a different IP but make sure to use the same network like /24 in the example above. You also need to set wireguard_endpoint for every node that will be part of your Kubernetes cluster. Only the workstation will have a wireguard_endpoint set to "" (empty string). The reason for this is that the workstation wont have a WireGuard VPN endpoint where the other hosts would connect to. You just want to access the kube-apiserver from your workstation or your CD host (like Jenkins) and not the other way round. Note: You don’t need to include your workstation/laptop in the WireGuard VPN mesh. By default kube-apiserver (the service you need to “talk” to with kubectl e.g. if you want to create Kubernetes resources) only listens on the WireGuard interface. But you can adjust that later so that it also listens on your public interface and make the service available to your network e.g.

But you want to enable all Kubernetes hosts to communicate with each other. That’s what wireguard_endpoint is needed for. Besides other things the Ansible role will create a WireGuard configuration for every host. That configuration will include the information what WireGuard IP can be reached via what endpoint. You can find a complete example in the README of the WireGuard role but here is an example of the WireGuard configuration of the workstation:

[Interface]
Address = 10.0.11.254/24
PrivateKey = ....
ListenPort = 51820

[Peer]
PrivateKey = ....
AllowedIPs = 10.0.11.3/32
Endpoint = k8s-010201.p.example.com:51820 # K8s controller #1

[Peer]
PrivateKey = ....
AllowedIPs = 10.0.11.6/32
Endpoint = k8s-010202.p.example.com:51820 # K8s controller #2
...

The AllowedIPs parameter is also some kind of routing information and WireGuard will create routes accordingly. So on my workstation with the IP 10.0.11.254 there will be a route that tells WireGuard that if I want to connect to 10.0.11.3 (which is the first Kubernetes controller node) send the traffic to k8s-010201.p.example.com:51820.

Include the role into your playbook (k8s.yml) like in this example:

-
  hosts: vpn
  roles:
    -
      role: githubixx.wireguard
      tags: role-wireguard

vpn is a host group which contains all hosts that WireGuard should be installed on. So lets add the following to Ansible’s hosts file. E.g.:

vpn:
  hosts:
    k8s-01[01:03][01:03].i.example.com:
    k8s-01-ansible-ctrl.i.example.com:

Now you can deploy the role:

ansible-playbook --tags=role-wireguard k8s.yml

That limits the tasks which gets executed and in this case that’s just the ones from the WireGuard role.

If you use Ansible’s host facts caching as mentioned in part 2 make sure that you refresh the cache afterwards as there is now a new WireGuard interface that Ansible isn’t aware of yet. Use the following command to gather the new host facts (replace all with k8s_all if you haven’t installed WireGuard role on your workstation e.g.):

ansible -m setup all

To check if all WireGuard interfaces are up run this command:

ansible -m command -a "ip addr show dev wg0" k8s_all

This should produce a output like this:

k8s-010301.i.example.com | CHANGED | rc=0 >>
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.0.11.8/24 scope global wg0
       valid_lft forever preferred_lft forever
k8s-010101.i.example.com | CHANGED | rc=0 >>
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.0.11.2/24 scope global wg0
       valid_lft forever preferred_lft forever
...

This should also work:

ansible -m command -a "ping -c 1 k8s-010201.i.example.com" k8s-010101.i.example.com

k8s-010101.i.example.com | CHANGED | rc=0 >>
PING k8s-010201.i.example.com (10.0.11.5) 56(84) bytes of data.
64 bytes from 10.0.11.5 (10.0.11.5): icmp_seq=1 ttl=64 time=2.33 ms

--- k8s-010201.i.example.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.330/2.330/2.330/0.000 ms

And to check the state of all WireGuard interfaces (this produces a lot of output):

ansible -m command -a "wg show" k8s_all

or

ansible -m command -a "systemctl status wg-quick@wg0.service" k8s-010101.i.example.com

You should now be able to ping all WireGuard IP’s from your workstation (if you configured that). The same should be true for all hosts of course. You should be able to ping every Kubernetes host from every other Kubernetes node. You just can’t ping your workstation from the Kubernetes nodes as the workstation has no WireGuard endpoint (which was intended).

That’s it for part 3! In part 4 we’ll install a certificate authority which is needed for Kubernetes.