This is a super long post. Hope it helps someone.
Synopsis
Preparing Arch for QEMU and passthrough of GPU.
Installing Qemu and setting up a Windows 10 virtual machine.
Disclaimer
Please research all Linux commands before using them, also (if done incorrectly) this guide can destroy your Linux build. I suggest you either use this guide on a new install of Arch or backup all data from existing install.
Common result can be a black screen at boot or a crashing GUI.
*Also note I have copy pasta'd a lot of this content and do not take any credit for it. I am only compiling sources and in the order of proceedings that would help people who are trying this for the first time.
Requirements
- AMD-VI/VT-D enabled and working (On intel systems both your MB and you CPU must support it, to find out if you CPU has VT-D support go here: http://ark.intel.com/, you also need to boot with intel_iommu=on to enable it)
- At least 2 GPU's, one primary boot device and the card you wish to pass-through (can use onboard as primary)
- Qemu=>2.0
- Aditional kernel patches might be required if you're using an Intel CPU: ACS override patch and i915 VGA arbiter patch, you can find a kernel package with these patches included on AUR: linux-vfio
- Aditional requirements for OVMF:
An UEFI compatible GPU (Most modern gpus support this, SEE: Does my graphics card ROM support EFI?
An UEFI compatible GUEST (ex: Windows 7 and up)
-Source
Preparation
- Turn AMD-VI/VT-D on in the BIOS (should be under advanced features under the name "virtualisation technology" or Overclocking menu with the name "Vt-d" or "AMD-VI". Legacy names could also appear: "Vanderpool" for Vt-x, "Pacifica" for AMD-V)
- Turn on VT-d in Linux. [Scroll down to the VT-d Bootloader section]
- Get the pci-e bus id for the gpu you want to passthrough. [Passthrough section]
- If you are using official nvidia drivers, it might break in the new kernal. [Nvidia Drivers section]
VT-d Bootloader
If you followed part 1 and are using Syslinux bootloader then try this to edit the bootloader:
*First let us run Arch in a test mode with this command attached "intel_iommu=on" at the end of the string.
Press e when the bootmenu appears (if you have not set a timeout delay hold space then press e)
You can now add a command to the bootloader. Press enter to continue the boot.
Code: Select all
initrd=\initramfs-linux.img root=/dev/sda2 intel_iommu=on"
*To make things permanent we can edit the bootloader inside Arch
I like to use the nano editor, but you can use whatever. (nano commands: ctrl+o to save. ctrl+x to exit)
Code: Select all
sudo nano /boot/loader/entries/arch.conf
Add
Code: Select all
intel_iommu=on
Code: Select all
amd_iommu=on
Don't worry about
Code: Select all
pcie_acs_override=downstream pci-stub.ids=10de
Reboot after you have added it.
Passthrough section
We need to get the pci-e id of the GPU.
Run
Code: Select all
lspci -nnk
This is the result on my machine:
Code: Select all
00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7976]
Kernel driver in use: skl_uncore
libkmod: kmod_config_parse: /etc/modprobe.d/kvm.conf line 1: ignoring bad line starting with 'kvm_intel'
00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7976]
Kernel driver in use: i915
Kernel modules: i915
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 950] [10de:1402] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3208]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3208]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
Take note of the id of the VGA controller.
Run
Code: Select all
ls -lha /sys/bus/pci/devices/YOUR_BUS/iommu_group/devices/
In my case it would be:
Code: Select all
ls -lha/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/
Different Approach to checking IOMMU Groups:
Run:
Code: Select all
dmesg|grep -e DMAR -e IOMMU
Check for this in output:
Code: Select all
0.000000] Intel-IOMMU: enabled
Run this script for checking IOMMU Group mapping:
Code: Select all
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;
Note if your output contains the PCI root as shown here:
Code: Select all
IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
You will need to pass on ALL of the devices listed here.
Note: If they are grouped with other devices in this manner, pci root ports and bridges should neither be bound to vfio at boot, nor be added to the VM.
- If the terminal complains about iommu not existing, then VT-d failed. Recheck your bootloader for the added command..
- If you have 2 GPU's and they show up as grouped, go to [Fixing grouped GPU]
Continue passthrough here
For the Windows VM to utilise the GPU we need to blacklist it in Linux.
Why, you ask? Well, Nvidia binds to the GPU at startup making it impossible to use in the VM as it is technically already in use. Vfio-pci is what we will use to take control of the card before Nvidia does, making it easier for us to pass the card on to the VM.
"Since Linux 4.1 the kernal includes vfio-pci" -Arch Wiki
Run:
Code: Select all
modinfo vfio-pci
If it returns an error you will need to use pci-stub instead. [PCI stub section]
Vfio-pci uses the GPU ID to grab hold of it at boot time.
Run this to find your gpu id:
Code: Select all
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;
Check for your VGA controller group. Output should look something like this:
Code: Select all
IOMMU Group 13 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU Group 13 06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)}}
The ID's part is what we are interested in:
Code: Select all
[10de:13c2], [10de:0fbb]
Warning: Please make sure you are isolating the correct GPU. The isolated GPU will not be usable inside Linux after the next reboot.
Add the id's of the GPU you want to passthrough to the bootloader options. (we did this a few times already)
Code: Select all
options vfio-pci ids=10de:13c2,10de:0fbb
Edit mkinitcpio.conf :
Code: Select all
sudo nano /etc/mkinitcpio.conf
Add
Code: Select all
vfio vfio_iommu_type1 vfio_pci vfio_virqfd
Output:
Code: Select all
MODULES="... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ..."
Also ensure
Code: Select all
modconf
Output:
Code: Select all
HOOKS="...modconf..."
Since new modules have been added to the initramfs configuration, it must be regenerated. Should you change the IDs of the devices in /etc/modprobe.d/vfio.conf, you will also have to regenerate it, as those parameters must be specified in the initramfs to be known during the early boot stages.
Run:
Code: Select all
mkinitcpio -p linux
Note: If you are using a non-standard kernel, such as linux-vfio, replace linux with whichever kernel you intend to use.
Reboot Linux. Now check if vfio-pci has been properly loaded:
Run:
Code: Select all
dmesg | grep -i vfio
Output should look something like this:
Code: Select all
[ 0.329224] VFIO - User Level meta-driver version: 0.3
[ 0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[ 0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
[ 2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)
Making sure as vfio.conf doesnt always show all devices:
*just replace the id in the code with your own gpu id
Run:
Code: Select all
lspci -nnk -d 10de:13c2
Output:
Code: Select all
06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
Kernel driver in use: vfio-pci
Kernel modules: nouveau nvidia
Run:
Code: Select all
lspci -nnk -d 10de:0fbb
Output:
Code: Select all
06:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
PCI-stub
Most linux distros (including Arch Linux) have pci-stub built statically within the kernel image. If for any reason it needs to be loaded as a module in your case, you will need to bind it yourself using whatever tool your distro provides for this, such as mkinitpcio for Arch.
Run:
Code: Select all
sudo nano /etc/mkinitcpio.conf
Add
Code: Select all
pci-stub
Output:
Code: Select all
MODULES="... pci-stub ..."
Regenerate kernal image if you had to add pci-stub
Run:
Code: Select all
mkinitcpio -p linux
Note: If you are using a non-standard kernel, such as linux-vfio, replace linux with whichever kernel you intend to use.
Run this to find your gpu id:
Code: Select all
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;
Add the device Id's to the kernal bootloader. (we have done this before)
Output:
Code: Select all
options "...pci-stub.ids=10de:13c2,10de:0fbb..."
Reboot Arch.
Check to see if pci-stub was successfully applied:
Run:
Code: Select all
dmesg | grep pci-stub
Output:
Code: Select all
[ 2.390128] pci-stub: add 10DE:13C2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.390143] pci-stub 0000:06:00.0: claimed by stub
[ 2.390150] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.390159] pci-stub 0000:06:00.1: claimed by stub
Fixing grouped GPU
If you followed the video in part 1 you should have yaourt installed.
If not install yaourt:
Code: Select all
sudo pacman -S yaourt
Then run yaourt:
Code: Select all
yaourt -S linux-vfio
Just say no if it asks you to edit things.
Reboot Arch.
We now need to add
Code: Select all
pcie_acs_override=downstream
Run:
Code: Select all
sudo nano /boot/loader/entries/arch.conf
Add
Code: Select all
pcie_acs_override=downstream
Reboot again.
Recheck iommu groups
Code: Select all
ls -lha /sys/bus/pci/devices/YOUR_BUS/iommu_group/devices/
Nvidia Drivers
Just copy pasta'd this...cause I am using Nouveau drivers.
If you installed the “nvidia” package from the arch repo your driver will probably break in the new kernel. I simply installed the binary from nvidia’s website. A simple way to do that against the new kernel is to download the binary, reboot, edit grub by pressing “e” with linux-vfio selected, and then append “nomodeset systemd.unit=multi-user.target” to the end of the long line that begins with “linux …” so you do a one time edit of the boot parameters. Then navigate to the binary and “sh NVIDIA-###…sh” It should disable nouveau if needed and install. Reboot and continue. You may also want to look into nvidia-dkms if you plan on updating your linux-vfio kernel regularly.
Sources
A Final Word
I will add Part 3 soon (installing QEMU and setting it up for Windows 10)
Please comment in the section below if you have tried this and if it worked or not. Also if you have any questions feel free to ask. One last thing, if I made mistakes could you just point them out so I can correct them.
Kind Regards,
Alano (benehiko)