Virtualization with Arch Linux and Qemu/KVM - part 1
Introduction
Today I want to setup three Mini PCs with Arch Linux OS. These Mini PCs are really great for a small Kubernetes homelab, home automation, virtualization in general or stuff like that. I personally not so much into ARM processors and Raspberry Pi or similar architectures and like to stick to the amd64/x86_64 architecture.
There are actually hundreds of different Mini PC (or NUC) available. Just search on Amazon e.g. I personally have GEEKOM MiniAir 11 and GEEKOM Mini IT12. These Mini PC have the advantage to not use that much power as they have either an older processor like the Intel Celeron N5095 (as for the Mini Air 11) or an Intel Core i5-1240P/i7-1260P (as for the Mini IT12). The later one you normally only find in laptops e.g. They have an idle power consumption of around 6-8 Watts. Even with a running Ubuntu VM it’s not going much higher. Additionally the Mini IT12 has 2.5 GB Ethernet (Intel I225-V). That makes it interesting if you want to run something like Longhorn which provides cloud native distributed block storage for Kubernetes. In this case is “faster is better” 😉 For a homelab it’s also just fine to have one host and run six VMs on it. What’s mainly important is memory. Try to get as much as you can.
What I want to achieve
- Arch Linux OS on every host
- QEMU/KVM installed on every host for virtualization
- Three virtual machines on every host running Ubuntu 22.04 (cloud image)
- Prepare one VM on every host for a etcd cluster (where Kubernetes stores it’s state)
- Prepare one VM on every host for Kubernetes control plane (
kube-apiserver,kube-schedulerandkube-controller-manager) - Prepare one VM on every host for Kubernetes worker nodes (
kube-proxyandkubelet)
Arch Linux ISO image
To install Arch Linux we need to download the ISO image and write in on a USB flash drive. It can be downloaded here. How to get the image with the Arch Linux Installer on a USB stick please read USB flash installation medium from the Arch Linux Wiki. I’m normally using dd e.g.:
dd bs=4M if=path/to/archlinux-version-x86_64.iso of=/dev/disk/by-id/usb-My_flash_drive conv=fsync oflag=direct status=progress
In my case the path of the USB stick looks like this /dev/disk/by-id/usb-_USB_DISK_3.0_90008A00FB5C8916-0:0 (lsblk might also help to identify your USB stick). After dd is done execute sudo sync to make sure that buffers are fully written before you remove the USB stick.
Booting Arch Linux from USB stick/drive
So put the stick into the host where you want to install Arch Linux. Make sure that the BIOS is able to boot from USB (see device order in the Boot section for you BIOS). Sooner or later you’ll end up on the command prompt. If you have a non-english keyboard you can load a different keyboard translation table with loadkeys de for a German keyboard e.g. You can now continue on the command line. At that point I normally set a password for root user with passwd command. This is only temporary used to be able to login via SSH and is not the one for the installation that happens next. Having a password set allows me to login via SSH from my laptop and continue working from that one. For this sshd needs to be running: systemctl start sshd.
With lsblk you can display how your HDD, SSD or NVMe is setup e.g.:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 673M 1 loop /run/archiso/airootfs
sda 8:0 1 29.3G 0 disk
├─sda1 8:1 1 778M 0 part
└─sda2 8:2 1 15M 0 part
nvme0n1 259:0 0 476.9G 0 disk
├─nvme0n1p1 259:1 0 100M 0 part
├─nvme0n1p2 259:2 0 16M 0 part
├─nvme0n1p3 259:3 0 476G 0 part
└─nvme0n1p4 259:4 0 800M 0 part
We’ve two drives here in my case. nvme0n1 is a NVMe SSD disk where I’ll put Arch Linux on. As you can see it already contains four partions. That’s because my Mini PC was delivered with Windows 11 installed. We’ll get rid of it in just a second 😉. Later it will also contain partitions for the Virtual Machines. sda is the USB stick I booted Arch Linux from.
While installing Arch Linux in the past was a little bit complicated you can just use the Arch Linux installer nowadays. It’s already included on the USB stick.
But before starting archinstall I’ll already prepare the disk and its partitions. While you can do that with archinstall too it’s pretty cumberstone IMO as you’ve to work with sectors (I still don’t get why an installer works that way…). So I’ll use cgdisk. In my case I get this partition list when running cgdisk /dev/nvme0n1:
Part. # Size Partition Type Partition Name
----------------------------------------------------------------
1007.0 KiB free space
1 100.0 MiB EFI system partition EFI system partition
2 16.0 MiB Microsoft reserved Microsoft reserved partition
3 476.0 GiB Microsoft basic data Basic data partition
4 800.0 MiB Windows RE Basic data partition
327.5 KiB free space
I’ll delete all partions. Make sure that you really want to delete that partition you selected and that you using the correct disk drive! Just use the cursor keys to move up and down between the partions. Then use the left and right cursor keys to move to [Delete]. After deleting all existing partions the list looks like this e.g.:
Part. # Size Partition Type Partition Name
----------------------------------------------------------------
476.9 GiB free space
Now I’ll create two new partions. One for /boot and one for /. So I’ll select [New]. First sector starts at 2048 (yeah, I know… sectors again 😉). I’ll keep the default. Next the size for /boot will be 512M (no sectors here…). Partition type for /boot is ef00 (EFI system partition). As partition name I’ll enter /boot.
Next select the free space below /boot partition and select [New] again. Again I’ll keep the default which is 1050624 in my case. cgdisk will just contine with the next sector after the last partition. For / I’ll choose a size of 76G. That’s just because I’ve 476.9GiB and to make the remaining free space more or less even 😉 In general somewhere between 30-50GB for / should be fine (depends of course what you want to install there). For the filesystem type I’ll keep the default 8300 (Linux filesystem) and partition name will be /. The partition schema looks now like this:
Part. # Size Partition Type Partition Name
----------------------------------------------------------------
1007.0 KiB free space
1 512.0 MiB EFI system partition /boot
2 76.0 GiB Linux filesystem /
400.4 GiB free space
The remaining space will be stay left untouched for now. I’ll use Linux Logical Volume Manager (LVM) later to setup that space. The partitions created with LVM will be used for my virtual machines (more information on that will follow below).
If you’re fine with the partitions move the cursor to [Write] and (again think about if you change the correct disk drive and partions!) hit enter. And finally we can [Quit].
So just run archinstall now. If you do so you’ll get a menu like this (this is true for archinstall version 2.6.0 - other versions might look different!):
Set/Modify the below options
> Archinstall language English (100%)
Mirrors
Locales Defined
Disk configuration
Bootloader Systemd-boot
Swap True
Hostname archlinux
Root password
User account
Profile
Audio No audio server
Kernels linux
Additional packages
Network configuration Not configured, unavailable unless setup manually
Timezone UTC
Automatic time sync (NTP) True
Optional repositories
Save configuration
Install
Abort
(Press "/" to search)
So select your Archinstall language accordingly. I’ll stay with English. For Mirrors select your country. In Locales I’ll keep Locale language and Locale encoding with en_US and utf-8 as this makes most sense for servers. You can also change this now or later. Keyboard layout will be de in my case.
Next Disk configuration: As mentioned I’ll install Arch Linux on the NVMe SSD drive /dev/nvme0n1. So if I enter Manual configuration the drive list looks like this:
Model | Path | Type | Size | Free space | Sector size | Read only
------------------------------------------------------------------------------------
> [ ] KINGSTON OM8SEP4512N-A0 | /dev/nvme0n1 | nvme | 488386 MiB | 410050 | 512 | False
[ ] USB DISK 3.0 | /dev/sda | scsi | 30000 MiB | 29984 | 512 | False
Existing Partitions
Name | Type | Filesystem | Path | Start | Length | Flags -------------------------------------------------------------------------------- /boot | primary | fat32 | /dev/nvme0n1p1 | 1 MiB | 512 MiB | Boot, ESP / | primary | Unknown | /dev/nvme0n1p2 | 513 MiB | 77824 MiB |
I select the /dev/nvme0n1, press space key to mark the disk and hit enter. You see the partions we created before with cgdisk. Select the first one (/dev/nvme0n1p1), hit enter and select Mark/Unmark to be formatted (wipes data). Select the first one again, hit enter and select Assign mountpoint and enter /boot. Hit enter. archinstall will automatically set the Boot and ESP flag if a partition is called /boot. So that’s fine and we’ll stay with it. Also FS type (file system type) is fat32 which is also needed for /boot.
Next select the second partition /dev/nvme0n1p2, hit enter and select Mark/Unmark to be formatted (wipes data). Select the second partition again, hit enter and select Assign mountpoint which will be just /. And one more time select the second partition, hit enter and select Change filesystem. I’ll choose ext4 for / but xfs or btrfs are also valid options (from the other options I’d stay away).
So we now have something like this:
Status | Device | Type | Start | Length | FS type | Mountpoint | Mount options | Flags
--------------------------------------------------------------------------------------
> modify | /dev/nvme0n1p1 | primary | 1 MiB | 512 MiB | fat32 | /boot | | Boot, ESP
modify | /dev/nvme0n1p2 | primary | 513 MiB | 77824 MiB | ext4 | / | |
The rest of the disk will stay unallocated for now. Later I’ll put logical volumes (LVM) on that remaining space. The idea is to allocate a certain amount of disk space for every virtual machine later. LVM allows to easily extend disk space e.g. Also raw disks are way faster then the qcow2 images. Both formats have their pros and cons. For me performance is most important and I don’t intend to move VMs between hosts. By using LVM I can get rid of some disadvantages of raw. E.g. you can create snapshots of raw disks with LVM. So you’ve to decide which makes most sense for you. qcow2 images are just normal files and are easier to handle of course.
Hit Confirm and exit now. If you want you can also encrypt disks by selecting Disk encryption but I’ll skip that. For Bootloader I’ll keep systemd-bootctl as I’m a systemd fan boy 😉 For Swap I’ll keep True. This will create a zram compressed block device during host startup (so no swap partition or a swap file on disk as one might expect). Set your Hostname accordingly. Mine will be k8s-010100 (for Kubernetes Cluster 01 and host 01 of that cluster. So there will also be host k8s-010200 and k8s-010300). I’ll set Root password to None because I’ll create a User account next that has sudo permissions. Select Add a user and Enter username and Password for that user. Answer Should "..." be a superuser (sudo)? with yes (that’s important as root user has no password!). Hit Confirm and exit.
For Profile and Audio I’ll keep None (normally you don’t need audio on a server). Kernels will also stay default linux. If you want to be a little bit more conservative linux-lts might be an option.
In Additional packages I’ll add openssh bridge-utils vim python (just copy&paste). openssh is definitely needed for remote access later. bridge-utils is used to work with Bridges. A network bridge is a virtual network device that forwards packets between two or more network segments. So instead of a “normal” network interface like eth0 I’ll configure a bridge called br0. This makes it pretty easy later to add network connectivity to a virtual machine. python is needed later in order to manage the hosts with Ansible.
Next is Network configuration. I’ll keep Not configured, unavailable unless setup manually for now. After the installation of the base system I’ll configure the bridge mentioned above manually.
For Timezone I’ll keep UTC as this makes most sense for a server. For Automatic time sync (NTP) I’ll keep True as keeping time in sync is pretty critical for some some cryptographic functions e.g.
Finally hit Install. This will format your partitions and install the latest Arch Linux packages for the base system and the optional packages you requested. Depending on your Internet connection this might take a while to download the OS packages.
After this is done you’ll be asked Would you like to chroot into the newly created installation and perform post-installation configuration?. Choose yes. You’ve now entered your shiny new OS 😉 Now it’s time to finish the network configuration as mentioned above. For this we need to know the interface name of your network card. Execute ip link. In my case the output looks like this:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 38:f7:cd:c5:a7:47 brd ff:ff:ff:ff:ff:ff
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
link/ether 14:f5:f9:83:e5:e3 brd ff:ff:ff:ff:ff:ff
I don’t care about lo (the loopback interface) and also not about the WiFi interface wlan0. So enp3s0 is my Ethernet interface. That one we need for the bridge. How to setup a bridge is described in Arch Linux wiki but here is a basic setup that should be already good enough for the purpose:
Switch to directory /etc/systemd/network. First, create a virtual bridge interface with a netdev unit file. We tell systemd to create a device named br0 that functions as an ethernet bridge. I’ll create a file called 99-br0.netdev with the following content:
[NetDev]
Name=br0
Kind=bridge
Next we the bridge needs a physical interface assigned. That’s enp3s0 which we figured out with ip link above. This file needs to be loaded before 99-br0.netdev. So the first number fo the file name needs to be lower. I’ll call the file 98-enp3s0.network with this content:
[Match]
Name=enp3s0
[Network]
Bridge=br0
If all your interfaces should be part of the bridge you can also specify Name=en* e.g.
And finally the bridge needs a network configuration. So I’ll create a file called 99-br0.network with this content:
[Match]
Name=br0
[Network]
Address=192.168.10.2/23
Gateway=192.168.10.1
DNS=1.1.1.1
DNS=9.9.9.9
Search=example.com
If your host should get its IP configuration via DHCP only specify DHCP=ipv4 in the [Network] section. In my case it’s a static IP with 192.168.10.2/23 and the default gateway 192.168.10.1. So the network my physical hosts and the VMs are located in is 192.168.10.0/23. DNS server is Cloudflare’s 1.1.1.1. Additionally it has Quad9 DNS server configured as fallback. You can also add Domains= here as search domains.
Next we should have a few entries in /etc/hosts e.g.:
127.0.0.1 localhost
127.0.1.1 localhost
::1 localhost
192.168.10.2 k8s-010100
You want to change the hostname k8s-010100 of course to your hostname and adjust the IP address accordingly 😉
Next we need to make sure that we have the DNS resolver running once the host is up and running. The same is true for networking and SSH of course:
systemctl enable systemd-resolved.service
systemctl enable systemd-networkd.service
systemctl enable sshd.service
The next is optional and you might skip that if you think it hurts security. Normally if you type sudo ... you’ve to enter the password of that user (at least from time to time). If you want to avoid this we need to adjust sudoers. So lets create a drop-in file /etc/sudoers.d/wheel (it’s important to use visudo to make sure the syntax is correct!):
visudo /etc/sudoers.d/wheel
The file will have this content:
%wheel ALL=(ALL) NOPASSWD: ALL
During installation we created a user and that user is part of the wheel group (see /etc/group). So here we state that every user that is part of the wheel group can run sudo without entering a password. Now leave the chroot with exit and reboot with reboot and remove the USB stick.
So I’m done now with basic preperation for all three hosts. The next blog post will be about setting up Ansible to automate a few tasks e.g. harden the hosts a bit and setting up Linux Logical Volume Manager (LVM).