Part 2 -Preparation for QEMU/KVM on Arch Linux

All things related to software and development.
Windows, Linux and Mac.
User avatar
UraharaUlquiorra
Game Development Club Leader
Posts: 10
Joined: Wed Feb 24, 2016 7:43 pm
Country: South Africa
Firstname: Alano

Part 2 -Preparation for QEMU/KVM on Arch Linux

Postby UraharaUlquiorra » Sat Nov 19, 2016 1:02 am

Hi all,

This is a super long post. Hope it helps someone. :D

Synopsis
Preparing Arch for QEMU and passthrough of GPU.
Installing Qemu and setting up a Windows 10 virtual machine.

Disclaimer
Please research all Linux commands before using them, also (if done incorrectly) this guide can destroy your Linux build. I suggest you either use this guide on a new install of Arch or backup all data from existing install.
Common result can be a black screen at boot or a crashing GUI.

*Also note I have copy pasta'd a lot of this content and do not take any credit for it. I am only compiling sources and in the order of proceedings that would help people who are trying this for the first time.

Requirements
  • AMD-VI/VT-D enabled and working (On intel systems both your MB and you CPU must support it, to find out if you CPU has VT-D support go here: http://ark.intel.com/, you also need to boot with intel_iommu=on to enable it)
  • At least 2 GPU's, one primary boot device and the card you wish to pass-through (can use onboard as primary)
  • Qemu=>2.0
  • Aditional kernel patches might be required if you're using an Intel CPU: ACS override patch and i915 VGA arbiter patch, you can find a kernel package with these patches included on AUR: linux-vfio
  • Aditional requirements for OVMF:
    An UEFI compatible GPU (Most modern gpus support this, SEE: Does my graphics card ROM support EFI?
    An UEFI compatible GUEST (ex: Windows 7 and up)
-Check if my graphics card is supported by QEMU: here
-Source

Preparation
  1. Turn AMD-VI/VT-D on in the BIOS (should be under advanced features under the name "virtualisation technology" or Overclocking menu with the name "Vt-d" or "AMD-VI". Legacy names could also appear: "Vanderpool" for Vt-x, "Pacifica" for AMD-V)
  2. Turn on VT-d in Linux. [Scroll down to the VT-d Bootloader section]
  3. Get the pci-e bus id for the gpu you want to passthrough. [Passthrough section]
  4. If you are using official nvidia drivers, it might break in the new kernal. [Nvidia Drivers section]

VT-d Bootloader
If you followed part 1 and are using Syslinux bootloader then try this to edit the bootloader:
*First let us run Arch in a test mode with this command attached "intel_iommu=on" at the end of the string.
Press e when the bootmenu appears (if you have not set a timeout delay hold space then press e)
You can now add a command to the bootloader. Press enter to continue the boot.

Code: Select all

initrd=\initramfs-linux.img root=/dev/sda2 intel_iommu=on"


*To make things permanent we can edit the bootloader inside Arch
I like to use the nano editor, but you can use whatever. (nano commands: ctrl+o to save. ctrl+x to exit)

Code: Select all

sudo nano /boot/loader/entries/arch.conf

Add

Code: Select all

intel_iommu=on
for intel CPU's and

Code: Select all

amd_iommu=on
for AMD CPU's to the end of UUID as shown in this picture.
Don't worry about

Code: Select all

pcie_acs_override=downstream pci-stub.ids=10de
for the moment.
Reboot after you have added it.
Image

Passthrough section
We need to get the pci-e id of the GPU.
Run

Code: Select all

lspci -nnk

This is the result on my machine:

Code: Select all

00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)
   Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7976]
   Kernel driver in use: skl_uncore
libkmod: kmod_config_parse: /etc/modprobe.d/kvm.conf line 1: ignoring bad line starting with 'kvm_intel'
00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)
   Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7976]
   Kernel driver in use: i915
   Kernel modules: i915
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 950] [10de:1402] (rev a1)
   Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3208]
   Kernel driver in use: vfio-pci
   Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
   Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3208]
   Kernel driver in use: vfio-pci
   Kernel modules: snd_hda_intel
   


Take note of the id of the VGA controller.
Run

Code: Select all

ls -lha /sys/bus/pci/devices/YOUR_BUS/iommu_group/devices/

In my case it would be:

Code: Select all

ls -lha/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/

Image

Different Approach to checking IOMMU Groups:
Run:

Code: Select all

dmesg|grep -e DMAR -e IOMMU

Check for this in output:

Code: Select all

0.000000] Intel-IOMMU: enabled


Run this script for checking IOMMU Group mapping:

Code: Select all

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;


Note if your output contains the PCI root as shown here:

Code: Select all

IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)


You will need to pass on ALL of the devices listed here.
Note: If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.

  1. If the terminal complains about iommu not existing, then VT-d failed. Recheck your bootloader for the added command..
  2. If you have 2 GPU's and they show up as grouped, go to [Fixing grouped GPU]

Continue passthrough here
For the Windows VM to utilise the GPU we need to blacklist it in Linux.
Why, you ask? Well, Nvidia binds to the GPU at startup making it impossible to use in the VM as it is technically already in use. Vfio-pci is what we will use to take control of the card before Nvidia does, making it easier for us to pass the card on to the VM.

"Since Linux 4.1 the kernal includes vfio-pci" -Arch Wiki
Run:

Code: Select all

modinfo vfio-pci

If it returns an error you will need to use pci-stub instead. [PCI stub section]

Vfio-pci uses the GPU ID to grab hold of it at boot time.
Run this to find your gpu id:

Code: Select all

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;


Check for your VGA controller group. Output should look something like this:

Code: Select all

IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}


The ID's part is what we are interested in:

Code: Select all

[10de:13c2], [10de:0fbb]


Warning: Please make sure you are isolating the correct GPU. The isolated GPU will not be usable inside Linux after the next reboot.

Add the id's of the GPU you want to passthrough to the bootloader options. (we did this a few times already)

Code: Select all

options vfio-pci ids=10de:13c2,10de:0fbb


Edit mkinitcpio.conf :

Code: Select all

sudo nano /etc/mkinitcpio.conf


Add

Code: Select all

vfio vfio_iommu_type1 vfio_pci vfio_virqfd
before the rest of the modules.
Output:

Code: Select all

 MODULES="... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ..."

Also ensure

Code: Select all

modconf
is added to the HOOKS.
Output:

Code: Select all

 HOOKS="...modconf..."


Since new modules have been added to the initramfs configuration, it must be regenerated. Should you change the IDs of the devices in /etc/modprobe.d/vfio.conf, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.

Run:

Code: Select all

mkinitcpio -p linux


Note: If you are using a non-standard kernel, such as linux-vfio, replace linux with whichever kernel you intend to use.

Reboot Linux. Now check if vfio-pci has been properly loaded:
Run:

Code: Select all

dmesg | grep -i vfio

Output should look something like this:

Code: Select all

[    0.329224] VFIO - User Level meta-driver version: 0.3
[    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
[    2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)


Making sure as vfio.conf doesnt always show all devices:

*just replace the id in the code with your own gpu id
Run:

Code: Select all

lspci -nnk -d 10de:13c2


Output:

Code: Select all

06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
   Kernel driver in use: vfio-pci
   Kernel modules: nouveau nvidia
   


Run:

Code: Select all

lspci -nnk -d 10de:0fbb

Output:

Code: Select all

   06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
   Kernel driver in use: vfio-pci
   Kernel modules: snd_hda_intel
   


PCI-stub

Most linux distros (including Arch Linux) have pci-stub built statically within the kernel image. If for any reason it needs to be loaded as a module in your case, you will need to bind it yourself using whatever tool your distro provides for this, such as mkinitpcio for Arch.

Run:

Code: Select all

sudo nano /etc/mkinitcpio.conf

Add

Code: Select all

pci-stub
to MODULES
Output:

Code: Select all

MODULES="... pci-stub ..."


Regenerate kernal image if you had to add pci-stub
Run:

Code: Select all

mkinitcpio -p linux


Note: If you are using a non-standard kernel, such as linux-vfio, replace linux with whichever kernel you intend to use.


Run this to find your gpu id:

Code: Select all

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;


Add the device Id's to the kernal bootloader. (we have done this before)
Output:

Code: Select all

options "...pci-stub.ids=10de:13c2,10de:0fbb..."


Reboot Arch.

Check to see if pci-stub was successfully applied:
Run:

Code: Select all

dmesg | grep pci-stub

Output:

Code: Select all

[    2.390128] pci-stub: add 10DE:13C2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.390143] pci-stub 0000:06:00.0: claimed by stub
[    2.390150] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.390159] pci-stub 0000:06:00.1: claimed by stub


Fixing grouped GPU
If you followed the video in part 1 you should have yaourt installed.
If not install yaourt:

Code: Select all

sudo pacman -S yaourt

Then run yaourt:

Code: Select all

yaourt -S linux-vfio

Just say no if it asks you to edit things.
Reboot Arch.

We now need to add

Code: Select all

pcie_acs_override=downstream
to the bootloader.
Run:

Code: Select all

sudo nano /boot/loader/entries/arch.conf

Add

Code: Select all

pcie_acs_override=downstream
at the end of options.
Reboot again.
Recheck iommu groups

Code: Select all

ls -lha /sys/bus/pci/devices/YOUR_BUS/iommu_group/devices/


Nvidia Drivers
Just copy pasta'd this...cause I am using Nouveau drivers.
If you installed the “nvidia” package from the arch repo your driver will probably break in the new kernel. I simply installed the binary from nvidia’s website. A simple way to do that against the new kernel is to download the binary, reboot, edit grub by pressing “e” with linux-vfio selected, and then append “nomodeset systemd.unit=multi-user.target” to the end of the long line that begins with “linux …” so you do a one time edit of the boot parameters. Then navigate to the binary and “sh NVIDIA-###…sh” It should disable nouveau if needed and install. Reboot and continue. You may also want to look into nvidia-dkms if you plan on updating your linux-vfio kernel regularly.

Sources
  • Useful read: here
  • Main source: here
  • Arch Kernal Parameters: here
  • PCI Passthrough via OVMF: here

A Final Word
I will add Part 3 soon (installing QEMU and setting it up for Windows 10)
Please comment in the section below if you have tried this and if it worked or not. Also if you have any questions feel free to ask. One last thing, if I made mistakes could you just point them out so I can correct them.


Kind Regards,
Alano (benehiko)
Summoner Name: Arki Benehiko
:D
Image

MarukoViBrittania
Posts: 18
Joined: Mon Feb 22, 2016 10:04 pm
Country: South Africa
Firstname: Marco

Re: Part 2 -Preparation for QEMU/KVM on Arch Linux

Postby MarukoViBrittania » Sat Nov 19, 2016 12:18 pm

Thanks. Busy working on something else at the moment but will probably eventually give this a try. :D :P
=^.^=


Return to “Software”

Who is online

Users browsing this forum: No registered users and 14 guests