Fedora 28 QEMU-KVM OVMF GPT Passthrough
I . Packages to install :
pacmansudo -Sdnf qemuinstall virt-manager virtviewerqemu-kvm dnsmasqqemu-img iptableslibvirt ebtablesvirt-install
sudo usermod -a -G libvirt username
sudo systemctl enable libvirtdII. Configuring Host before passing through :
modprobe -r kvm_intel
modprobe kvm_intel nested=1Make sure you do not have the GPU you want to passthrough in your slot #0 of your PCI lanes. This will alter the ROM as soon as the host is booted, and you will be unable to use your GPU properly on your guest.
III. The script below will show you all PCI devices and their mapping to their respective IOMMU groups. If the output is blank, you do not have IOMMU enabled.
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;Enabling IOMMU :
You will need to add a boot load kernel option :
Forvim Intel/etc/sysconfig/grubAdd: rd.driver.pre=vfio-pci : i915.alpha_support=1 intel_iommu=on iommu=pt at the end of your GRUB_CMDLINE_LINUX=
The grub config will look like this:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=UUID=90cb68a7-0260-4e60-ad10-d2468f4f6464 rhgb quiet rd.driver.pre=vfio-pci i915.alpha_support=1 intel_iommu=on iommu=pt"
GRUB_DISABLE_RECOVERY="true"
Then re-gen your grub2
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfgFor AMD CPUs: amd_iommu=onrebootUse script above // Example output :
IOMMU Group 30Group 0 00:1d.00.0 USBHost controllerbridge [0c03]0600]: Intel Corporation C610/X998th seriesGen chipsetCore USB EnhancedProcessor Host ControllerBridge/DRAM #1Registers [8086:8d26]3ec2] (rev 05)07)
IOMMU GroupGroup 10 3100:1c.2 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #3 [8086:a292] (rev f0)
IOMMU Group 11 00:1c.3 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #4 [8086:a293] (rev f0)
IOMMU Group 12 00:1c.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #5 [8086:a294] (rev f0)
IOMMU Group 13 00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
IOMMU Group 14 00:1d.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #9 [8086:a298] (rev f0)
IOMMU Group 15 00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99Z370 seriesChipset chipset LPCLPC/eSPI Controller [8086:8d47] (rev 05)a2c9]
IOMMU Group 31Group 15 00:1f.2 SATAMemory controller [0106]0580]: Intel Corporation C610/X99200 seriesSeries/Z370 chipsetChipset 6-PortFamily SATAPower Management Controller [AHCI mode] [8086:8d02] (rev 05)a2a1]
IOMMU Group 31Group 15 00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU Group 15 00:1f.4 SMBus [0c05]: Intel Corporation C610/X99200 seriesSeries/Z370 chipsetChipset Family SMBus Controller [8086:8d22]a2a3]
IOMMU Group 16 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
IOMMU Group 17 02:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black NVMe SSD [15b7:5001]
IOMMU Group 18 07:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:2142]
IOMMU Group 19 08:00.0 Network controller [0280]: Intel Corporation Wireless 3165 [8086:3165] (rev 05)81)
IOMMU GroupGroup 1 3200:01.0 06:PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104TU104 [GeForce GTXRTX 1080]2080] [10de:1b80]1e87] (rev a1)
IOMMU GroupGroup 1 32 06:01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio ControllerDevice [10de:10f0]10f8] (rev a1)
IOMMU GroupGroup 1 3301:00.2 07:USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
IOMMU Group 1 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: NVIDIAIntel Corporation GP107UHD Graphics 630 (Desktop) [GeForce GTX 1050] [10de:1c81] (rev a1)8086:3e92]
IOMMU GroupGroup 3 3300:08.0 07:00.1System Audio deviceperipheral [0403]0880]: NVIDIAIntel Corporation GP107GLXeon HighE3-1200 Definitionv5/v6 Audio/ ControllerE3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model [10de:0fb9] (rev a1)8086:1911]
IOMMU GroupGroup 4 34 09:00.00:14.0 USB controller [0c03]: VIAIntel Technologies,Corporation Inc.200 VL805Series/Z370 Chipset Family USB 3.0 HostxHCI Controller [1106:3483]8086:a2af]
IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
IOMMU Group 6 00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
IOMMU Group 7 00:1b.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #17 [8086:a2e7] (rev 01)f0)
IOMMU GroupGroup 8 3500:1b.4 0a:00.PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #21 [8086:a2eb] (rev f0)
IOMMU Group 9 00:1c.0 USBPCI controllerbridge [0c03]0604]: ASMediaIntel TechnologyCorporation Inc.200 ASM1042ASeries USBPCH 3.0PCI HostExpress ControllerRoot Port #1 [1b21:1142]8086:a290] (rev f0)
IV. Isolating your GPU
To assign a GPU device to a Virtual machine, you will need to use a place holder driver to prevent the host from interacting with it on boot. You cannot dynamically re-assign a GPU device on a VM after you booted due to its complexity. You can use either VFIO or pci-stub.
Most newer machines will have VFIO by default, which we will be using here.
If your system supports it, which you can try by running the following command, you should use it. If it returns an error, use pci-stub instead.
[]# modinfo vfio-pci
-----
filename: /lib/modules/4.9.53-1-lts/kernel/drivers/vfio/pci/vfio-pci.ko.gz
description: VFIO PCI - User Level meta-driver
author: Alex Williamson <alex.williamson@redhat.com>
license: GPL v2
In this case here I'm interested in the following groups to passthrough
IOMMU GroupGroup 1 32 06:01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104TU104 [GeForce GTXRTX 1080]2080] [10de:1b80]1e87] (rev a1)
IOMMU GroupGroup 1 32 06:01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio ControllerDevice [10de:10f0]10f8] (rev a1)
IOMMU Group 1 01:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
IOMMU Group 1 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)Adding their relevant IDs to the VFIO driver
After you completed the below steps, your GPU will no longer be detected by your host, make sure you have a secondary GPU available.
vim /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1b80,10de:10f01e87,0de:10f8,0de:1ad8,0de:1ad9 MakeRegenerate sure that your vfio-pci driver will be loaded before other graphics driver. Their modules need to be added before any other graphical driversinitramfs
vim /etc/mkinitcpio.conf
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
If you added modules, make sure to regenerate your kernel image configuration
mkinitcpiodracut -pf linux-lts--kver `uname -r`V. Passing the GPU to your VM
Qemu-KVMExample optimizationsof toXML followfile from my VM
(NOTESCPU TO ADD)Pinning
grep -e "processor" -e "core id" -e "^$" /proc/cpuinfoprocessor : 0
core id : 0
processor : 1
core id : 1
processor : 2
core id : 2
processor : 3
core id : 3
processor : 4
core id : 4
processor : 5
core id : 5
processor : 6
core id : 0
processor : 7
core id : 1
processor : 8
core id : 2
processor : 9
core id : 3
processor : 10
core id : 4
processor : 11
core id : 5sudo EDITOR=gedit virsh edit <vm-name> <cputune>
<vcpupin vcpu='0' cpuset='6'/>
<vcpupin vcpu='1' cpuset='7'/>
<vcpupin vcpu='2' cpuset='8'/>
<vcpupin vcpu='3' cpuset='9'/>
<vcpupin vcpu='4' cpuset='10'/>
<vcpupin vcpu='5' cpuset='11'/>
</cputune> <hyperv>
<vendor_id state='on' value='3645293847ab'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>