Single Gpu Passthrough

Single GPU Passthrough — the author's Setup

System Overview

Hardware Topology

IOMMU Groups

The GPU and its HDMI audio controller share a clean IOMMU group (Group 12):

• pci_0000_01_00_0 — GPU (vendor 0x10de, product 0x1f03, class 0x030000)
• pci_0000_01_00_1 — HDMI Audio (vendor 0x10de, product 0x10f9, class 0x040300)

Both devices must be passed together as they share IOMMU Group 12. They connect via PCI bridge at 0000:00:01.0.

Other Devices

• 03:00.0 — Realtek RTL8111H Gigabit Ethernet
• 04:00.0 — MAXIO NVMe SSD Controller

Kernel Configuration

Kernel Cmdline (bootc/ostree)

Kernel parameters are set via rpm-ostree kargs (NOT /etc/default/grub — bootc is immutable):

sudo rpm-ostree kargs \
  --append='intel_iommu=on' \
  --append='iommu=pt' \
  --append='rd.driver.blacklist=nouveau' \
  --append='modprobe.blacklist=nouveau' \
  --append='nvidia-drm.modeset=1'

Current cmdline (from /proc/cmdline):

intel_iommu=on iommu=pt rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1

IMPORTANT — bootc atomic context: On a bootc/ostree system:

• /etc is writable (persists across updates)
• /usr is immutable (resets on each deployment update)
• Kernel params are applied via rpm-ostree kargs — add/remove requires a reboot
• To add: sudo rpm-ostree kargs --append='new_param=value'
• To remove: sudo rpm-ostree kargs --delete='old_param=value'
• To replace: sudo rpm-ostree kargs --replace='old=value' --replace-to='new=value'
• To view: rpm-ostree kargs

modprobe.d /etc Configs

/etc/modprobe.d/ is writable on bootc. The following reside there:

• nvidia.conf (performance profile): NVreg_RegistryDwords, NVreg_EnableGpuFirmware, NVreg_DynamicPowerManagement, nvidia-drm modeset=1
• vhost.conf: max_mem_regions=509

VFIO modules are not loaded at boot — they are loaded dynamically by the hook scripts on VM prepare.

IOMMU Verification Commands

# Check IOMMU is enabled
dmesg | grep -i iommu
cat /proc/cmdline | grep -o 'intel_iommu\|iommu'

# Check IOMMU groups
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
  echo "IOMMU Group $(basename $g):"
  for d in $g/devices/*; do
    echo -n "  $(basename $d): "
    lspci -nns "$(basename $d)" 2>/dev/null || echo "unknown"
  done
done

# Via virsh
virsh nodedev-list --cap pci
virsh nodedev-dumpxml pci_0000_01_00_0  # GPU
virsh nodedev-dumpxml pci_0000_01_00_1  # HDMI Audio

Hook Architecture

Overview

The passthrough uses libvirt qemu hooks — scripts that fire on VM lifecycle events. Three scripts form the system:

1. /etc/libvirt/hooks/qemu — dispatcher: called by libvirt on VM prepare/release
2. ~/.local/sbin/vfio-startup — detaches GPU from host, called on VM prepare
3. ~/.local/sbin/vfio-teardown — reattaches GPU to host, called on VM release
4. libvirt-nosleep@.service — systemd inhibit sleep while VM runs

File Locations (Deployed)

NOTE: The repo's install_hooks.sh installs to /usr/local/bin/ but The author modified the qemu hook to call ~/.local/sbin/ instead. Always check which paths are active before modifying.

Script Sources

The source repo is at ~/projects/single-gpu-passthrough/ but the deployed scripts have been customized — they differ from the repo versions significantly.

Libvirt Hook Lifecycle

The qemu hook receives 4 positional arguments:

/etc/libvirt/hooks/qemu <vm_name> <operation> <sub-operation> <extra>

The domain XML is passed on stdin during all guest hook calls.

the author's hook uses prepare and release — the outermost lifecycle phases.

qemu Hook (Dispatcher)

Path: /etc/libvirt/hooks/qemu

#!/bin/bash
OBJECT="$1"
OPERATION="$2"

if [[ $OBJECT == "win10" ]]; then
    case "$OPERATION" in
        "prepare")
            systemctl start libvirt-nosleep@"$OBJECT" 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
            ~/.local/sbin/vfio-startup 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
            ;;
        "release")
            systemctl stop libvirt-nosleep@"$OBJECT" 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
            ~/.local/sbin/vfio-teardown 2>&1 | tee -a /var/log/libvirt/custom_hooks.log
            ;;
    esac
fi

Key details:

• Only activates for VM named win10 (change this for different VM names)
• Calls the startup/teardown scripts
• Starts/stops the no-sleep inhibitor service
• All output is tee'd to /var/log/libvirt/custom_hooks.log

vfio-startup (GPU Detach)

Path: ~/.local/sbin/vfio-startup

Execution flow:

1. Stop display server — systemctl stop greetd, then killall -9 niri qs zen kitty Xwayland pipewire wireplumber
2. Isolate to multi-user.target — prevents systemd from restarting GUI services
3. Unbind VT consoles — echo 0 > /sys/class/vtconsole/vtcon0/bind and vtcon1/bind
4. Unbind EFI framebuffer — echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
5. Unload NVIDIA kernel modules — in order: nvidia_uvm, nvidia_drm, nvidia_modeset, nvidia, i2c_nvidia_gpu, drm_kms_helper
6. Detach PCI devices via virsh:
   • virsh nodedev-detach pci_0000_01_00_0 (GPU)
   • virsh nodedev-detach pci_0000_01_00_1 (HDMI Audio)
7. Logs to /var/log/libvirt/vfio-startup.log with set -x trace

IMPORTANT: The NVIDIA module unload is a single modprobe -r call with all modules listed. If any module is busy (e.g., nvidia_uvm from a container), the entire unload fails. In that case, try unloading individually or find what's holding references:

lsmod | grep nvidia
lsof | grep nvidia  # what processes hold nvidia files

vfio-teardown (GPU Reattach)

Path: ~/.local/sbin/vfio-teardown

Execution flow:

1. Reattach PCI devices via virsh:
   • virsh nodedev-reattach pci_0000_01_00_0 (GPU)
   • virsh nodedev-reattach pci_0000_01_00_1 (HDMI Audio)
2. Reload NVIDIA kernel modules: nvidia_drm, nvidia_modeset, nvidia_uvm, nvidia
3. Rebind VT consoles — echo 1 > vtcon0/bind and vtcon1/bind
4. Start greetd — systemctl start greetd
5. Logs to /var/log/libvirt/vfio-teardown.log with set -x trace

libvirt-nosleep@.service

Path: /etc/systemd/system/libvirt-nosleep@.service

[Unit]
Description=Preventing sleep while libvirt domain "%i" is running

[Service]
Type=simple
ExecStart=/usr/bin/systemd-inhibit --what=sleep --why="Libvirt domain \"%i\" is running" --who=%U --mode=block sleep infinity

This uses systemd-inhibit to block system sleep (suspend/hibernate) while the VM runs.

Installation Steps

To deploy hooks from scratch:

# Become root
sudo -i

# Copy scripts
cp ~/projects/single-gpu-passthrough/hooks/vfio-startup ~/.local/sbin/vfio-startup
cp ~/projects/single-gpu-passthrough/hooks/vfio-teardown ~/.local/sbin/vfio-teardown
cp ~/projects/single-gpu-passthrough/hooks/qemu /etc/libvirt/hooks/qemu
cp ~/projects/single-gpu-passthrough/systemd-no-sleep/libvirt-nosleep@.service /etc/systemd/system/libvirt-nosleep@.service

# Make executable
chmod +x ~/.local/sbin/vfio-startup ~/.local/sbin/vfio-teardown /etc/libvirt/hooks/qemu

# Create log file
mkdir -p /var/log/libvirt
touch /var/log/libvirt/custom_hooks.log /var/log/libvirt/vfio-startup.log /var/log/libvirt/vfio-teardown.log

# Reload systemd
systemctl daemon-reload

# Verify hooks directory exists
ls -la /etc/libvirt/hooks/qemu

NOTE: The repo scripts are the base versions. The deployed scripts at ~/.local/sbin/ have been significantly customized for the author's Wayland+niri setup. Always check the deployed versions before modifying.

Libvirt / QEMU Configuration

Modular Daemons

Fedora 44 libvirt uses modular daemons (not the monolithic libvirtd):

systemctl status virtqemud      # QEMU-specific daemon
systemctl status virtnodedevd    # Node device management (needed for nodedev-detach)
systemctl status virtnetworkd    # Network management
systemctl status virtstoraged    # Storage management

Ensure virtnodedevd is running — the hooks depend on it for virsh nodedev-detach and nodedev-reattach.

VM Domain XML Skeleton

The VM should be defined in virsh. Key passthrough-specific XML elements:

<domain type="kvm">
  <name>win10</name>
  <memory unit="GiB">16</memory>
  <vcpu>12</vcpu>
  <os>
    <type>hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE_4M.qcow2</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" dies="1" cores="6" threads="2"/>
  </cpu>
  <devices>
    <!-- VFIO GPU Passthrough -->
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
      </source>
      <rom bar="off"/>
    </hostdev>
    <!-- VFIO HDMI Audio -->
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
      </source>
    </hostdev>
  </devices>
</domain>

Key points:

• managed="yes" — libvirt manages driver binding (detach/attach), which is what the hooks use via nodedev-detach
• rom bar="off" on GPU — prevents QEMU from trying to map the GPU ROM BAR (often causes issues with NVIDIA cards)
• kvm hidden state=on — hides KVM hypervisor signature from NVIDIA drivers (avoids Code 43)
• OVMF (UEFI) is required for GPU passthrough
• cpu mode="host-passthrough" — pass all host CPU features to guest (gaming performance)

Alternatively, for finer control, the QEMU command-line equivalent (used when not via libvirt):

-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,rombar=0
-device vfio-pci,host=01:00.1

Where:

• multifunction=on — tells QEMU the GPU is a multi-function device (required when passing both functions 0 and 1)
• x-vga=on — marks this device as the primary VGA (needed for boot-time video)
• rombar=0 — don't expose the option ROM BAR (avoids conflicts)

Storage Configuration

The VM disk should be on the ext4 partition for better performance:

• Path candidate: /mnt/media/win10.qcow2
• HDD storage: 1.4TB available on niaext (ext4)
• NTFS partition (niabc, 452GB) for Windows data sharing

# Create qcow2 disk (example: 120GB sparse)
qemu-img create -f qcow2 /mnt/media/win10.qcow2 120G

Network Configuration

Default libvirt NAT network is sufficient:

virsh net-start default
virsh net-autostart default

Use virtio NIC for best performance:

<interface type="network">
  <mac address="52:54:00:xx:xx:xx"/>
  <source network="default"/>
  <model type="virtio"/>
</interface>

Workflow

Starting the VM

virsh start win10

Libvirt will:

1. Call /etc/libvirt/hooks/qemu win10 prepare begin -
2. Hook runs vfio-startup (stops GUI, unloads NVIDIA, detaches GPU via virsh)
3. libvirt starts QEMU with VFIO devices
4. GPU output switches to VM

Shutting Down and Returning to Host

virsh shutdown win10     # Graceful shutdown
# or
virsh destroy win10      # Force power off

Libvirt will:

1. QEMU exits
2. Call /etc/libvirt/hooks/qemu win10 release end <reason>
3. Hook runs vfio-teardown (reattaches GPU, reloads NVIDIA, restarts greetd)
4. Display output returns to host (niri on tty1)

Monitoring

# Hook logs
tail -f /var/log/libvirt/custom_hooks.log
tail -f /var/log/libvirt/vfio-startup.log
tail -f /var/log/libvirt/vfio-teardown.log

# virsh status
virsh list
watch -n 1 virsh list

# GPU status
lspci -nnk -s 01:00
nvidia-smi      # shows when host driver is bound

Troubleshooting

GPU Not Detaching (VFIO busy)

# Check current driver
lspci -nnk -s 01:00

# Check if NVIDIA modules still loaded
lsmod | grep nvidia

# Check if display server still running
systemctl is-active greetd

# Check what holds the GPU
fuser -v /dev/dri/* 2>/dev/null
lsof | grep -i nvidia

# Force detach (if not managed)
virsh nodedev-detach pci_0000_01_00_0
virsh nodedev-detach pci_0000_01_00_1

vm starts but no display output (black screen)

• Ensure monitor is connected to the same GPU port used by host
• Check rom bar="off" is set in VM XML
• Try adding a GPU VBIOS ROM file (dump from Windows with GPU-Z, or from motherboard)
• Try x-vga=on in the QEMU args
• Ensure NVIDIA driver is installed in Windows VM

Error Code 43 in Windows VM

NVIDIA drivers detect the KVM hypervisor and disable themselves. Fixes:

• kvm hidden state=on in domain XML
• Set hyperv features: relaxed, vapic, spinlocks
• CPU mode host-passthrough
• Some newer NVIDIA drivers also check for: <vendor_id state="on" value="1234567890ab"/>

Module Unload Fails During Startup

# Check usage
lsmod | grep nvidia

# Find what's using nvidia_uvm
cat /sys/devices/virtual/misc/uvm/uevent

# Kill remaining processes
ps aux | grep -i nvidia
killall nvidia-persistenced 2>/dev/null

Greetd / Niri Won't Restart After Teardown

# Check greetd status
systemctl status greetd

# Manually start
sudo systemctl start greetd

# Check niri log
journalctl -u greetd -n 50 --no-pager

Bootc-specific Issues

On Fedora bootc (ostree atomic host):

• If you layer packages with rpm-ostree install, they persist through updates
• If you use rpm-ostree usroverlay for /usr modifications, those are temporary (lost on reboot)
• libvirt is part of the base image (container layer), not layered
• Hook scripts persist in /etc (writable state directory)
• For kernel args changes: rpm-ostree kargs --append/--delete/--replace requires a reboot

Advanced: Manual VFIO Binding

If libvirt's managed mode isn't working, bind VFIO-PCI manually at startup:

# Bind GPU to vfio-pci (instead of virsh nodedev-detach)
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo "10de 1f03" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "10de 10f9" > /sys/bus/pci/drivers/vfio-pci/new_id

But this is redundant with virsh nodedev-detach — prefer the virsh method which handles the bind/unbind lifecycle correctly.

Key Paths Reference

References

• Risingprism Single GPU Passthrough guide (repo): https://gitlab.com/risingprismtv/single-gpu-passthrough
• QEMU system emulation docs: https://www.qemu.org/docs/master/system/
• libvirt hooks docs: https://libvirt.org/hooks.html
• libvirt domain XML format: https://libvirt.org/formatdomain.html
• OVMF firmware location: /usr/share/edk2/ovmf/