QEMU/KVM + virtio-fs - Sharing a host directory with a virtual machine

2020-11-12 2020-11-12 1101 words 5 minutes

Contents

Before virtio-fs if someone wanted to share files between a virtual machine (VM) and the host the VM runs on there weren’t too much options that worked very well performance-wise. Among other options that were mostly SMB-Share (Samba), NFS or virto-9p. As already mentioned they all have in common that they are not that fast. One reason is that they are all depended on some network related stuff which of course adds overhead that costs performance.

virtio-fs on the other side is designed to offer local file system semantics and performance. virtio-fs takes advantage of the virtual machine’s co-location with the hypervisor to avoid overheads associated with network file systems. virtio-fs uses FUSE as the foundation. Unlike traditional FUSE where the file system daemon runs in userspace, the virtio-fs daemon runs on the host. A VIRTIO device carries FUSE messages and provides extensions for advanced features not available in traditional FUSE.

The prerequisites to use virtio-fs are:

The guest VM must run a Linux kernel >= 5.4 (which is true for Ubuntu 20.04 e.g. - also Ubuntu 20.04 supports kernel 5.8 meanwhile)
On the host QEMU 5.0 and libvirt 6.2 must be installed. At least Archlinux supports this requirements. For other distributions please check with your package manager which versions are available.

So lets start with the directory that should be shared on the host. The presentation A Shared File System for Virtual Machines] (PDF / slides) mentions a few security best practices that should be taken into consideration:

Guests have full uid/gid access to shared directory
Guests have no access outside shared directory
Use dedicated file system for shared directory to prevent inode exhaustion or other Denial-of-Service attacks
Parent directory of shared directory should have rwx------ permissions to prevent non-owners from accessing untrusted files
Mount shared directory nosuid,nodev on host

It’s perfectly possible to share e.g. /srv/http which should be available on most Linux installations. This directory is normally located at the root (/) filesystem. So if you share this directory with the virtual machine it would be possible to create lots of files which could result in running out of inodes on the host root partition. So it depends on your needs if that is an issue you want to avoid because of production requirements or if you just run a VM on your private laptop e.g.

I want to share three host directories with three VMs. To adhere the best practices I decided to create three partitions on a disk that had free space left e.g.:

bash

cfdisk /dev/sda

sda           8:0    0 931.5G  0 disk
├─sda1        8:1    0   100G  0 part
├─sda2        8:2    0   100G  0 part
└─sda3        8:3    0   100G  0 part

Now sda(1-3) can be formatted e.g. with a ext4 filesystem (mkfs.ext4). -m 1 specifies the percentage of the filesystem blocks reserved for the super-user. In my case one percent is good enough. Additionally I also put a label (-L) on it which can be used later to mount the disk accordingly:

bash

sudo mkfs.ext4 -m 1 -L shared_vm1 /dev/sda1
sudo mkfs.ext4 -m 1 -L shared_vm2 /dev/sda2
sudo mkfs.ext4 -m 1 -L shared_vm3 /dev/sda3

Now I create a parent directory with rwx------ permissions for all mountpoints as mentioned above in best practices. E.g.:

bash

sudo mkdir /shared
sudo chmod 700 /shared

sudo mkdir /shared/vm1
sudo mkdir /shared/vm2
sudo mkdir /shared/vm3

sudo chown 700 /shared/vm1
sudo chown 700 /shared/vm2
sudo chown 700 /shared/vm3

To mount the new partitions I added the following entries in /etc/fstab:

bash

LABEL=shared_vm1 /shared/vm1 ext4     rw,relatime,data=ordered,noatime,nosuid,nodev 0 2
LABEL=shared_vm2 /shared/vm2 ext4     rw,relatime,data=ordered,noatime,nosuid,nodev 0 2
LABEL=shared_vm3 /shared/vm3 ext4     rw,relatime,data=ordered,noatime,nosuid,nodev 0 2

Now the more interesting part on how to configure QEMU and finally share a host directory with a VM. The document libvirt: Sharing files with virtio-fs describes the process in more detail. Here is the path that I implemented.

QEMU needs to allocate the backing memory for all the guest RAM as shared memory. I have chosen to use file-backed memory (the other option would be hugepage-backed memory). This needs to be configured in /etc/libvirt/qemu.conf e.g.:

bash

# NOTE: big files will be stored here
memory_backing_dir = "/var/lib/libvirt/qemu/ram"

Make sure that the directory specified here have enough space left. For a VM that is configured to have 8 GB RAM you’ll see a file libvirt/qemu/vm1/ram-node0 later e.g. that also has 8 GB file size (in this case for a VM called vm1). This doesn’t mean that the whole 8 GB of that file will be allocated right from the start. So it’s perfectly possible that du -sh /var/lib/libvirt/qemu/ram/ might report a way lower value of space used.

Now restart libvirtd e.g.:

bash

sudo systemctl restart libvirtd

Next I changed the configuration of the VM accordingly e.g. sudo virsh edit vm1:

xml

<domain>
  ...
  <cpu ...>
    <numa>
      <cell id='0' cpus='0-1' memory='8' unit='GiB' memAccess='shared'/>
    </numa>
  </cpu>
 ...
</domain>

The <cpu> element probably already exists. So just the <numa> element is new in my case. The VM in question has 2 CPUs and 8 GB RAM. This numbers are also reflected for the <cell> properties accordingly. In case of a 4 CPU VM the property value of cpus would be 0-3 e.g.

For file-backed memory also a new element needs to be inserted at the same intention as the <cpu> element:

xml

<domain>
  ...
  <memoryBacking>
    <access mode='shared'/>
  </memoryBacking>
  ...
</domain>

Finally within the <devices> XML node a new <filesystem> element needs to be inserted e.g.:

xml

<domain>
  ...
  <devices>
    ...
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs'/>
      <source dir='/shared/vm1'/>
      <target dir='vm1'/>
    </filesystem>
    ...
  </devices>
</domain>

passthrough is the only supported access mode at the moment (which requires to run virtiofsd daemon on the host as root for now - this daemon will be started automatically). <source dir='/shared/vm1'/> specifies the host directory that should be shared with the VM. <target dir='vm1'/> is a bit misleading. The value isn’t a directory but just a tag which will be specified when the filesystem is mounted later within the VM (see below).

Now the VM needs to be restarted e.g.:

bash

sudo virsh shutdown vm1
sudo virsh start vm1

After the VM is up again the shared directory can be mounted:

bash

sudo mount -t virtiofs vm1 /mnt

Or put the mount into /etc/fstab to make it permanent:

bash

vm1 /mnt virtiofs rw,_netdev 0 0

To mount immediately run sudo mount -a.

Happy file sharing ;-)

Useful links: