How to Run an NVIDIA Tesla Architecture GPU in a QEMU Windows VM

For those who don't know, there is something called the infamous code 43 error for NVIDIA GPUs when a user tries to run them in a virtualized environment. For those already familiar, you've probably been frustrated and tried to search for what's going on.

I was able to fix Code 43 on my Proxmox VE 7.3 environment with a GT210 (Tesla architecture) GPU. I'll probably update this post when I try this again with my GTX 1060 3GB next week, but no ETA on the update yet. Honestly, I have no idea which patches actually did something and which are just useless. But it's been stable for about 5 days now.

My System Configuration 

Patch 1: Proxmox Host 

Update /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=vesafb:off video=efifb:off video=simplefb:off initcall_blacklist=sysfb_init nofb nomodeset"

Modify (or Create) /etc/modprobe.d/vfio.conf: 

options vfio-pci ids=[yourid],[yourid] disable_vga=1

Modify (or Create) /etc/modprobe.d/blacklist.conf:

blacklist nvidiafb
blacklist nouveau
options nouveau modeset=0
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm ignore_msrs=1

Create GPU Reset Script on /root/fix_gpu_pass.sh: 

#!/bin/bash
echo 1 > /sys/bus/pci/devices/[your gpu pci id]/remove
sleep 1
echo 1 > /sys/bus/pci/rescan

and add @reboot /root/fix_gpu_pass.sh to your crontab. 

Now, you can do update-initramfs -u -k all && update-grub && reboot to deploy all these patches. 

Patch 2: QEMU Configuration

I CANNOT emphasize this enough: you must set your VM BIOS setting to OVMF (UEFI) if you are trying to use your NVIDIA GPU as the primary GPU (or x-vga=1)!

Dump your GPU's ROM to /usr/share/kvm/gt210.rom: 

cd /usr/share/kvm
echo 1 > /sys/bus/pci/devices/[gpu pcie id]/rom
cat /sys/bus/pci/devices/[gpu pcie id]/rom > gt210.rom
echo 0 > /sys/bus/pci/devices/[gpu pcie id]/rom

Now, update your /etc/pve/qemu-server/[vmid].conf:

args: -cpu host,kvm=off,-hypervisor,+invtsc -smbios type=0,vendor=American\ Megatrends,version=1.0 -smbios type=1,manufacturer=ASUS,product=PRIME\ X570-P -machine q35,kernel_irqchip=on -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet
hostpci0: 0000:03:00,pcie=1,romfile=gt210.rom,x-vga=1

And... that's about it for QEMU Configuration Patches. (However, you'll need to add vga: std at this stage, as this configuration disables noVNC.)

Patch 3: Windows Virtual Machine 

I completely disabled Windows Update for this driver installation, as Windows kept trying to install the NVIDIA driver by itself.

In Command Prompt (Administrator): 

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate" /v ExcludeWUDriversInQualityUpdate /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\DriverSearching" /v SearchOrderConfig /t REG_DWORD /d 0 /f
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Device Metadata" /v PreventDeviceMetadataFromNetwork /t REG_DWORD /d 1 /f

sc stop wuauserv
sc config wuauserv start= disabled

sc stop bits
sc config bits start= disabled

sc stop TrustedInstaller
sc config TrustedInstaller start= disabled

In Command Prompt (Administrator): 

gpupdate /force

Then reboot your Windows machine. If a graphics driver is already installed, uninstall it using DDU.

Install NVIDIA Graphics Driver version 341.92 from their official website. When rebooting the Windows VM after driver installation, make sure to completely reboot the Proxmox host as well.

Now, you should see the NVIDIA graphics card in Device Manager without any errors, with Windows detecting it as the primary GPU.

Uncertain Patches: Windows Virtual Machine

While decompiling nvlddmkm.sys using Ghidra, I found that NVIDIA checks for the following registry keys:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e968-e325-11ce-bfc1-08002be10318}\0000" /v ForceEnableHypervisor /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\NVIDIA Corporation\Global\NVTweak" /v ForceEnableHypervisor /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\nvlddmkm" /v ForceEnableHypervisor /t REG_DWORD /d 1 /f

However, I am not sure if these registry entries helped at all.

Very Big Caveat: You should not reboot your Windows VM alone

If you reboot your Windows VM on Proxmox without rebooting the Proxmox host, you will see the Code 43 error again. Executing fix_gpu_pass.sh would not help solve this, though I'm not sure why. However, Windows seems to be using the GT210 graphics card, as GPU load goes up to 100% when I play a 1080p YouTube video over RDP.

That's it! I hope this helped.