Testing Ansible roles with Molecule, libvirt (vagrant-libvirt) and QEMU/KVM

I maintain a small Ansible role for WireGuard. It was mainly made for Ubuntu. After a while people started contributing code esp. for other OSes. As I only use Ubuntu and Archlinux personally I needed to trust contributors that their code works as I couldn’t test the parts that were special for a specific OS. That wasn’t great. I needed something to test the role for all OSes. That’s were Molecule comes in.

To quote the description of Molecule: “Molecule project is designed to aid in the development and testing of Ansible roles. Molecule provides support for testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks and testing scenarios.”

Most people will use Docker as a driver which is also Molecule’s default. That means if you want to test your role with Ubuntu 20.04 e.g. Molecule will instruct the Docker driver to launch a Docker container with Ubuntu 20.04. But for my WireGuard role Docker isn’t good enough. I needed a “real” OS as some OSes still don’t have a kernel that supports WireGuard out of the box like Ubuntu 18.04 e.g. In such a case we need a DKMS module package and for this you need the correct kernel header package installed. If I would do this with Docker Ansible would report the kernel version of the host where the Docker container runs on. But that could (and probably will) be different from the Docker container. Also I needed systemd which also requires some changes to make it work with Docker.

The next natural fit would be to use the Vagrant driver with the VirtualBox provider to create virtual machines (VM) with a fully installed Linux operating system. But why a second virtualization solution if I already have a few virtual machines running locally with libvirt and KVM (Kernel-based Virtual Machine)? That two components provide fast virtualization out of the box that basically comes with every Linux OS.

This instructions are mainly for Archlinux here but that only matters for installing the required packages. Everything else is the same no matter which Linux distribution you use. I’ll mention the required packages for Ubuntu 20.04 too but that’s mostly an educated guess (but I tried to install the packages in a virtual machine at least) ;-)

Before you start make sure that you add yourself to the libvirt group (for Ubuntu 20.04 you may also need to be part of group libvirt-qemu).

Next we need a few packages:

For Archlinux that’s

sudo pacman -S \
  ansible \
  qemu \
  vagrant \
  ebtables \
  dnsmasq \
  libvirt \
  python-pip

and for Ubuntu 20.04 it’s most probably:

sudo apt-get install \
  ansible \
  qemu \
  vagrant \
  ebtables \
  dnsmasq \
  libvirt-daemon-system \
  libvirt-clients \
  libvirt-dev \
  pkg-config \
  python3-pip

There are also a few Python packages needed. While they are partly also part of the OS package repository I’d recommend to install them via Python’s package manger PIP to get the latest versions. Archlinux packages are normally quite up2date but in case for molecule-vagrant that’s currently (20200930) not the case but we definitely need the latest version here. Using the --user flag makes sure that the Python packages are installed to the Python user install directory for your platform (typically ~/.local/). You can also create your own virtual env. with python3 -m venv ... (at least since version 3.3 and up) but that’s out of scope of this blog post. Either way it helps to don’t mess around with the system Python packages. So:

pip3 install --user python-vagrant
pip3 install --user testinfra
pip3 install --user libvirt-python
pip3 install --user molecule
pip3 install --user molecule-vagrant
pip3 install --user rich

molecule-vagrant needs to be at least version 0.3 or higher. With a lower version you’ll be out of luck!

Now I installed the libvirt plugin for Vagrant:

vagrant plugin install vagrant-libvirt

Also make sure that $HOME/.local/bin is part of your $PATH e.g. export PATH="$HOME/.local/bin:$PATH".

To demonstrate how I’ve implemented Molecule in my WireGuard role lets have a look how the role directory looked before I started:

CHANGELOG.md
defaults
.gitignore
handlers
meta
README.md
tasks
templates

Since the role implementation was already there the initialization of Molecule looks like this:

molecule init scenario kvm --role-name githubixx.ansible_role_wireguard --driver-name vagrant

This command adds a few files and directories:

├── molecule
│   └── kvm
│       ├── converge.yml
│       ├── INSTALL.rst
│       ├── molecule.yml
│       └── verify.yml
└── .yamllint

As you can see above inside the molecule directory there is currently a folder called kvm. That’s the name of the scenario I defined above with molecule init scenario kvm .... If you don’t specify a scenario name here the directory would have been called default.

Scenarios are the starting point for a lot of powerful functionality that Molecule offers. For now, we can think of a scenario as a test suite for the role. You can create as many scenarios as you like and Molecule will run one after the other. One example could be that maybe later I’ll decide that I don’t only want to test with KVM virtual machines but also within a Google Cloud project. So I’ll just create an additional scenario google-cloud e.g. When you create more scenarios it might make sense to share some files between them by creating a shared folder and point from the scenarios to the files in that folder. But that’s a different story… ;-)

The content of the files in molecule/kvm/ directory is pretty rudimentary. So I replaced the content. Lets first have a look at molecule.yml. That’s basically the configuration file for the resources that are needed to run the test suite.

The first thing that is defined in this file is the dependency manager:

dependency:
  name: galaxy

Since we’re using Ansible it of course makes sense that ansible-galaxy is used as dependency manager. As the WireGuard role don’t need any other Ansible roles nothing more needs to be defined here.

Next I defined the driver. Molecule uses the driver to delegate the task of creating instances:

driver:
  name: vagrant
  provider:
    name: libvirt
    type: libvirt
    options:
      memory: 192
      cpus: 2

By default Molecule would use Docker but as already stated above I used the vagrant driver with the libvirt provider. So instead of Docker container this configuration will start virtual machines via libvirt.

platforms:
  - name: test-wg-ubuntu2004
    box: generic/ubuntu2004
    interfaces:
      - auto_config: true
        network_name: private_network
        type: static
        ip: 192.168.10.10
    groups:
      - vpn
  - name: test-wg-ubuntu1804
    box: generic/ubuntu1804
    interfaces:
      - auto_config: true
        network_name: private_network
        type: static
        ip: 192.168.10.20
    groups:
      - vpn

...

  - name: test-wg-arch
    box: archlinux/archlinux
    interfaces:
      - auto_config: true
        network_name: private_network
        type: static
        ip: 192.168.10.80
    groups:
      - vpn

In platforms the virtual machines are defined that should be started. In the example above you only see three VMs defined but the WireGuard role defines a few more but that doesn’t matter. After the name box is defined. That’s basically preconfigured virtual machine images. You can find quite a few of them at https://app.vagrantup.com/boxes/search . Since in my case I’m only interested in VM images that can be used with the libvirt provider you can use this URL: https://app.vagrantup.com/boxes/search?provider=libvirt

I mostly use the generic/* boxes. Besides the boxes for Ubuntu 18 and 20 and Archlinux I also defined VMs for CentOS, Fedora and Debian in different versions. I also tried to use generic/arch but this box has a problem that was not that easy to solve. It doesn’t contain Python by default and that’s rather bad if you want to configure a VM with Ansible ;-) Luckily there is also an official Archlinux box archlinux/archlinux which contained Python out of the box. But that one had another problem. But I’ll come back to that later.

For most roles you normally don’t need to define static IPs for the interfaces. But in case of WireGuard an endpoint needs to be defined so that WireGuard can contact its peers. That’s why I configured an interface with static IPs for every VM.

And finally every VM is part of the vpn group. That’s basically the host groups which you normally define in Ansible’s hosts file.

While not strongly needed it may make sense to already download all the Vagrant VM images/boxes. This speeds up the process later. The VM images are quite large so be prepared to wait a little bit in case your internet connection isn’t that fast.

In case of the WireGuard role the following VM images/boxes are needed:

vagrant box add generic/ubuntu2004 --provider libvirt
vagrant box add generic/ubuntu1804 --provider libvirt
vagrant box add generic/debian10 --provider libvirt
vagrant box add generic/fedora31 --provider libvirt
vagrant box add generic/fedora32 --provider libvirt
vagrant box add generic/centos8 --provider libvirt
vagrant box add generic/centos7 --provider libvirt
vagrant box add archlinux/archlinux --provider libvirt

But lets continue with the next part of molecule.yml:

provisioner:
  name: ansible
  connection_options:
    ansible_ssh_user: vagrant
    ansible_become: true
  log: true
  lint:
    name: ansible-lint
  inventory:
    host_vars:
      test-wg-ubuntu2004:
        wireguard_address: "10.10.10.10/24"
        wireguard_port: 51820
        wireguard_persistent_keepalive: "30"
        wireguard_endpoint: "192.168.10.10"
      test-wg-ubuntu1804:
        wireguard_address: "10.10.10.20/24"
        wireguard_port: 51820
        wireguard_persistent_keepalive: "30"
        wireguard_endpoint: "192.168.10.20"

...

The sample above don’t shows all host_vars. As already mentioned above basically the same host variables were also defined for CentOS, Fedora, Debian and Archlinux.

All the Vagrant images contain a user vagrant that has sudo permissions. That’s why I defined ansible_ssh_user and ansible_become to tell Ansible to use that user to login and to use sudo (which is the default method for become.)

For all hosts defined in platforms above I defined a few host variables. As you can see wireguard_endpoint host variable matches interfaces.ip defined in platforms. And every host becomes a private WireGuard interface IP defined in wireguard_address. So if everything works well later we should be able to run ping 10.10.10.20 from test-wg-ubuntu2004 and should get some response if the VPN works as intended.

scenario:
  name: kvm
  test_sequence:
    - prepare
    - converge

As mentioned above I created a scenario kvm. Molecule treats scenarios as a first-class citizens, with a top-level configuration syntax. A scenario allows Molecule test a role in a particular way. A scenario is a self-contained directory containing everything necessary for testing the role in a particular way.

If you’ve a look at the documentation various sequences can be defined like create, check, converge, destroy and test sequence. For now the test_sequence defined above is good enough to prepare/start the VMs and install the role on all virtual machines by using the host variables defined in the inventory.

That’s basically what we need for molecule.yml for now. As already mentioned in platforms above I had one problem with the Archlinux box. During my first run I figured out that Ansible was not able to install the required packages as Archlinux package manager pacman was not initialized. But I don’t wanted to add this task to the WireGuard role because this is not the responsibility of that role to init a package manager.

So I finally came up with the idea to modify converge.yml. So the result looked like this:

---
- hosts: all
  remote_user: vagrant
  become: true
  gather_facts: true
  tasks:
    - name: Init pacman
      raw: |
        pacman-key --init
        pacman-key --populate archlinux        
      changed_when: false
      ignore_errors: true
      when: ansible_distribution|lower == 'archlinux'

    - name: Include WireGuard role
      include_role:
        name: githubixx.ansible_role_wireguard

As you can see there is task that gets only executed if Ansible sets the internal ansible_distribution variable to archlinux. And it does exactly what I needed. So afterwards only the WireGuard role needs to be included.

Now that everything is setup

molecule converge -s kvm

should now start the test VMs and install the role on all hosts accordingly. If you don’t already downloaded the VM images/boxes as mentioned above it could take quite a while the first time to execute this command as all VM images needs to be downloaded first.

Now currently Molecule only creates the virtual machines and installs the role with the provided variables. This only makes sure that the role can be installed but that doesn’t mean that everything also works as intended. This is where Testinfra comes in. With Testinfra you can write unit tests in Python to test actual state of your servers configured by management tools like Ansible. But since I only wanted to demonstrate how to use Molecule with libvirt this blog post ends here ;-)