Virtualization with Arch Linux and Qemu/KVM - part 3
Introduction
For the Virtual Machines I need a Operating System. I’ll use Ubuntu Cloud Images. In my case Ubuntu 22.04. These kind of images are already well prepared to run as Virtual Machines. That means they only have important packages installed e.g. They also have cloud-init installed and enabled. This tool helps to setup a VM during startup e.g. by configuring the hostname, networking, and other stuff.
Download cloud image
I’ll use Ubuntu 22.04 LTS as operation system (OS) for my VMs. There is a Ubuntu 22.04 server cloud image available which I’ll use. You need around 650 MByte disk space for the image. So lets download it to a directory on the laptop:
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
Convert cloud image from qcow2 to raw format
The downloaded image is of format qcow2. This one can’t be used to transfer it to the Logical Volumes. E.g.:
qemu-img info jammy-server-cloudimg-amd64.img
image: jammy-server-cloudimg-amd64.img
file format: qcow2
virtual size: 2.2 GiB (2361393152 bytes)
disk size: 643 MiB
cluster_size: 65536
Format specific information:
compat: 0.10
compression type: zlib
refcount bits: 16
Child node '/file':
filename: jammy-server-cloudimg-amd64.img
protocol type: file
file length: 643 MiB (673972224 bytes)
disk size: 643 MiB
So lets convert it to raw format. This needs around 2.3 GByte additional disk space temporary:
qemu-img convert -f qcow2 -O raw jammy-server-cloudimg-amd64.img jammy-server-cloudimg-amd64.raw
This file needs to be copied to all hosts in k8s_01 group (so all physical hosts). Ansible’s copy module can be used for this task. If the file is located at /tmp directory e.g. and should be stored in /tmp directory on the hosts the command looks like this:
ansible -m copy -a "src=/tmp/jammy-server-cloudimg-amd64.raw dest=/tmp/jammy-server-cloudimg-amd64.raw" k8s_01
Transfer Ubuntu 22.04 LTS image to LVM
The raw image now needs to be transferred to every Logical Volume (used for the Virtual Machines) on every Physical Host. This can be done with a little Bash script and Ansible (be VERY careful as this will override ANY data in the specified devices! But it also allows you to quickly “reset” a VM in case you want to start from the very beginning 😉):
for host in 01 02 03; do
for vm in 01 02 03; do
ansible -m command -a "dd if=/tmp/jammy-server-cloudimg-amd64.raw of=/dev/sdd01/k8s-01${host}${vm} bs=4M status=progress" k8s_01
done
done
Preparing cloud-init
While it now would be already possible to start the Virtual Machines they wouldn’t be of much use as a few details are missing like a user to login, the hostname, a network configuration, and so on. This is where cloud-init becomes handy. It’s basically the standard for customizing cloud instances during startup. When a VM with a Ubuntu cloud image starts, cloud-init is enabled by default. So lets prepare a cloud-init.cfg file for all VMs.
First I need a little directory structure in my playbooks directory which will look like this when very thing is done (don’t care about the files yet it’s explained below):
playbooks/
├── bootstrap_python.yml
├── files
│ └── cloud-init
│ ├── k8s-010101
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010102
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010103
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010201
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010202
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010203
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010301
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ ├── k8s-010302
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
│ └── k8s-010303
│ │ ├── cloud-init.iso
│ │ ├── network-config.yml
│ │ └── user-data.yml
└── libvirt.yml
The files bootstrap_python.yml and libvirt.yml are already in place and were used earlier already. With in playbooks directory I created two new directories files/cloud-init. And within that directory there is now a directory for every Virtual Machine host. The directory structure can be easily created with this command:
mkdir -p files/cloud-init/k8s-010{1,2,3}0{1,2,3}
So lets create a user-data.yml file for the the first Virtual Machine k8s-010101.example.com. I’ll put that file into files/cloud-init/k8s-010101 directory. Here is an example user-data file:
#cloud-config
users:
- name: my_user
groups: adm, cdrom, sudo, dip, plugdev, lxd
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_authorized_keys:
- ssh-rsa AAAAB...
hostname: k8s-010101
manage_etc_hosts: true
locale: en_US.UTF-8
timezone: Europe/Berlin
For a full example see Cloud config examples. But lets get quickly through the example above. user-data always has to start with #cloud-config. Then I specified that cloud-init should create a user my_user and add it to the groups specified in groups. We don’t need to specify a group or home directory. By default cloud-init will create a primary group that has the same name as the user and it will create a directory in /home that is also called like the user. Login shell will be /bin/bash. Insudo I specified that this user can run sudo command without password without any restrictions. This will create a file called /etc/sudoers.d/90-cloud-init-users later during VM startup. To be able to login via SSH I also added my public SSH key in ssh_authorized_keys. I haven’t specified a password for security reasons. So I won’t be able to login via QEMU console e.g. This is of course a tradeoff. You might also want to specify a passwd key here. The password needs to be hashed. Use mkpasswd --method=SHA-512 --rounds=4096 to create such a password. Then there is also the hostname. manage_etc_hosts: true will basically add a line to /etc/hosts like 127.0.1.1 k8s-010101 k8s-010101. For further information see Update Etc Hosts module. And finally locale generates the locale specified and makes it default and timezone sets the timezone for this host.
That would be already good enough. But I want to have a static IPv4 network configuration (this could also be achieved by using DHCP and the MAC address of the VM). By default cloud-init will configure networking to fetch an IP, Gateway and DNS servers from DHCP. So I also need a file called network-config.yml (same directory as user-data.yml) for the network configuration. E.g.:
network:
version: 2
ethernets:
enp1s0:
addresses: [192.168.11.2/23]
gateway4: 192.168.10.1
dhcp4: false
nameservers:
search: [example.com]
addresses: [9.9.9.9, 1.1.1.1]
For all possible options see Networking config version 2. I guess the file is not that hard to understand. Ubuntu Virtual Machines (at least for Ubuntu 20.04 and 22.04) normally have a network interface called enp1s0 if a virtio “network card” is used (more on this later). That’s why enp1s0 is used as network interface here. For addresses I only specified an IPv4 address but you can add more addresses and also add IPv6 addresses. gateway4 is the IPv4 gateway IP e.g. of the Internet router. DHCP is disabled via dhcp4: true. I used example.com as the DNS zone and therefore the search list for domains is example.com but you can add more of course. And finally I specified the Quad9 and Cloudflare DNS services for DNS resolution.
With that two files in place I’ll create an ISO file that will be later mounted as CDROM by the VMs. cloud-init can fetch the files I created above from various datasources. One of these data sources is NoCloud. cloud-init will search for a disk (or CDROM in my case) with the volume label cidata and retrieve the files from it.
To create such a disk I installed a litte tool called cloud-localds (for further possibilities like retrieving files form a HTTP server see NoCloud). So while still in files/cloud-init/k8s-010101 directory I execute the following command:
cloud-localds cloud-init.iso user-data.yml -N network-config.yml
Creating user-data.yml, network-config.yml and running cloud-localds command has to be done for every Virtual Machine in their directories. Esp. make sure to adjust hostname and the network address(es) accordingly. Of course that can also be automated with Ansible 😉
The playbook (which comes next) which transfers these .iso files to the hosts, needs to know which files it needs to transfer for every host. So I extend the host_vars accordingly. E.g. for host_vars/k8s-010100.example.com I add:
vms:
- k8s-010101
- k8s-010102
- k8s-010103
Do the same for host_vars/k8s-010200.example.com and host_vars/k8s-010200.example.com and adjust the list accordingly.
When done then these .iso files have to be transferred to the Physical Hosts. So I create an Ansible playbook file called playbooks/cloud-init.yml and it looks like this:
---
- name: Setup cloud-init
hosts: k8s_01
gather_facts: true
vars:
cloud_init_directory: "/var/lib/libvirt/cloud-init"
tasks:
- name: Create directory for cloud-init ISO files
ansible.builtin.file:
state: directory
path: "{{ cloud_init_directory }}"
mode: "0750"
owner: "libvirt-qemu"
group: "libvirt-qemu"
- name: Copy cloud-init ISO files
ansible.builtin.copy:
src: "cloud-init/{{ vm_name }}/cloud-init.iso"
dest: "{{ cloud_init_directory }}/{{ vm_name }}.iso"
owner: "libvirt-qemu"
group: "libvirt-qemu"
mode: "0640"
loop: "{{ vms }}"
loop_control:
loop_var: "vm_name"
On Physical Host it will create a directory /var/lib/libvirt/cloud-init and copies the .iso files relevant for only this host to this directory. Later I’ll create a libvirt Directory Storage Pool which uses this directory with all the .iso files in it.
If everything is in place execute the playbook:
ansible-playbook playbooks/cloud-init.yml
Before finishing this part one final note: As you may remember I’ve created a Logical Volume for every Virtual Machine. The sizes are 25G, 25G and 100G for the three VMs on every host. At the beginning of this part of the blog series I transferred the Ubuntu cloud image to these Logical Volumes with dd utility. This raw image was only 2.2 GByte. So one would expect that after launching the VMs the root / disk is only about that size. But actually they still will have 25G, 25G and 100G. Why is this? That’s because of a cloud-init module called resizefs that’s enabled by default. It will automatically resize the root / partition to the maximum available space of the underlying disk which is 25G, 25G and 100G.
In the next part I’ll finally create the Virtual Machines with virt-manager.