Linux, AMD Ryzen 3900X, X570, NVIDIA GTX 1060, AMD 5700XT, Looking Glass, PCI Passthrough and Windows 10

2020-08-03

• added comment about using <feature policy='disable' name='amd-stibp'/> in case of KERNEL_SECURITY_CHECK_FAILED error during Windows boot with Qemu 5.0 and Linux kernel >= 5.6

2020-05-08

• added comment about KERNEL_SECURITY_CHECK_FAILED error during Windows boot with Qemu 5.0 and Linux kernel >= 5.6

I didn’t expected to write this post at all but it turned out it’s quite important esp. if playing games ;-) While not really a Fortnite fan it’s quite good to test graphics performance as it’s free to download. Without modifications to the Windows 10 KVM configuration it was working but sometimes the whole Windows VM paused for a few seconds which isn’t quite nice while playing.

So I played around a little bit. Turns out that pinning vCPUs to real CPU cores that share the same L3 cache is one thing that improves performance. So let’s have a look at how the CPU is organized. This can be done with lstopo -l command. This generates the following diagram for the Ryzen 3900X CPU:

Now the idea here is to pin CPU cores that share the same L3 cache. In my case I used the physical Core L#3 - L#5 (with the logical “cores” PU L#6 - L#11) and Core L#9 - L#11 (with the logical “cores” PU L#18 - L#23). So this are my current settings (my VM is called Windows10 so in my case editing the VM configuration is done via virsh edit Windows10 as already mentioned in the previous chapters):

<vcpu placement='static'>12</vcpu>
<iothreads>6</iothreads>
<cputune>
  <vcpupin vcpu='0' cpuset='6'/>
  <vcpupin vcpu='1' cpuset='18'/>
  <vcpupin vcpu='2' cpuset='7'/>
  <vcpupin vcpu='3' cpuset='19'/>
  <vcpupin vcpu='4' cpuset='8'/>
  <vcpupin vcpu='5' cpuset='20'/>
  <vcpupin vcpu='6' cpuset='9'/>
  <vcpupin vcpu='7' cpuset='21'/>
  <vcpupin vcpu='8' cpuset='10'/>
  <vcpupin vcpu='9' cpuset='22'/>
  <vcpupin vcpu='10' cpuset='11'/>
  <vcpupin vcpu='11' cpuset='23'/>
  <emulatorpin cpuset='0,3'/>
</cputune>

The AMD Ryzen 3900X has 12 “real” cores + 12 additional “cores” you get when enabling hyperthreading. So my idea here was to pass 6 cores and 6 of the hyperthreading “cores” to the Windows VM. In <vcpu placement='static'> I therefore specified 12 virtual CPUs (VCPUs). With <emulatorpin cpuset='0,3'/> I specified that CPU core 0 and 3 should be used for I/O. This delegates I/O activity like hard disk read/write activity to these two CPU cores so that the others can do other CPU related computation.

<cpu mode='host-passthrough' check='none'>
  <topology sockets='1' cores='6' threads='2'/>
  <feature policy='require' name='topoext'/>
</cpu>

<cpu mode='host-passthrough' ...> specifies that all the CPU flags should be passed to the Windows VM (well, most of them ;-) ). That gives us max. performance

UPDATE 20200508: While host-passthrough worked fine with Qemu 4.2 it causes problems with newly released Qemu 5.0. In case you get on KERNEL_SECURITY_CHECK_FAILED on Windows boot try host-model. For more information see Upgrading to QEMU 5 broke my setup. Windows BSODs to KERNEL_SECURITY_CHECK_FAILED on boot.

UPDATE 20200803: Turns out that meanwhile even the fix mentioned above didn’t work anymore. The KERNEL_SECURITY_CHECK_FAILED is back again. That means using host-model didn’t work anymore for whatever reasons. With libvirt 6.5 or higher installed adding <feature policy='disable' name='amd-stibp'/> fixed it for me. So cpu now looks like this:

<cpu mode='host-passthrough' check='none'>
  <topology sockets='1' dies='1' cores='6' threads='2'/>
  <feature policy='require' name='topoext'/>
  <feature policy='disable' name='amd-stibp'/>
</cpu>

Other possible workarounds:

• Use linux-lts instead of linux package.
• Manually patch qemu in order to revert this commit.
• On qemu commandline, add amd-stibp=off to the cpu flags string. This can also be invoked through libvirt via a <qemu:commandline> entry (do not forget that in this case you need to replace <domain type='kvm'> with <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> at the very beginning of the XML file).

If you want to migrate a VM from one host to another like Cloud providers do then this is definitely something you shouldn’t do because that means that you only can migrate VMs to other hosts that basically have to have the same CPU. But since I don’t do any VM migrations performance is way more important. So this setting is save to use.

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native' queues='4'/>

Here I added queues='4' parameter to the disk driver.

With this additional settings Fortnite now works without performance issues. I guess disk I/O can be further improved but if I can trust the numbers that the Epic Game Launcher gives me during updating Fortnite I’m already quite near the max. performance of my SSD NVMe disk.

One word about passing through mouse and keyboard to your Windows VM: During playing Fortnite I always had the problem that suddenly “something” switched between weapon and building mode. As you can imaging that’s quite annoying if you’re in the middle of a fight and suddenly you don’t have a weapon anymore ;-) I passed through my mouse and keyboard that I normally use with Linux to the Windows VM. By pressing both CTRL keys I was able to hand over mouse and keyboard control between Linux and Windows. That works quite well in general but with the annoying effect described above. Since I didn’t wanted to debug that much I finally ended up in buying a separate mouse and keyboard and passed the USB devices to the Windows VM. That “solved” the problem.

I tried everything to pass through my Xbox controller connected via USB as normal USB device to the Windows VM. But no luck. Windows didn’t even found the device. I even managed to pass a whole USB hub with the Xbox controller attached as PCI device to the Windows VM. But even that didn’t worked. So I ended up in buying a Xbox Wireless Adapter. That’s also just a small USB stick. Just pass it to the Windows VM as normal USB device. After starting Windows you can connect the Xbox Controller and the Xbox Wireless Adapter. That’s normally enough for Windows to recognize the controller. But be aware that the connect doesn’t always work right away (and this is even true if you have Windows only computer). Try restarting the Windows VM and try again. Also you can try removing the device in Windows USB manager and adding it again. Maybe installing the latest drivers manually might help. Seems that people have some problems getting it working if you search the web. But once it’s working it works well IF(!) you disable power savings for the Xbox Wireless USB Adapter! Because otherwise the stick will just go straight into power saving mode while you’re playing ;-) To do so right click on Windows Start Button. Then click on Device Manager. Go to Network adapters. Then right click on Xbox Wireless Adapter for Windows. Then click on Properties. Click on Power Management tab. Then uncheck Allow the computer to turn off this device to save power and click OK.

And one final word regarding sound: As already mentioned in a previous blog post I bought a Asus Xonar U3 USB sound stick and passed it through to the Windows VM. Passing VM audio to host via PulseAudio basically works but sometimes during gaming the sound completely disappeared. With the Asus Xonar U3 I don’t have this problem. Only now and then during playing Fortnite there is silence for 2-3 seconds but afterwards sound finally works again. Also recording works without issues so you can communicate easily with other gamers.

With the tuning options implemented mentioned in this blog post everything works perfectly now (besides that very minor sound issue). Happy gaming! ;-)