Trying to passthru my RTX 2080 ti from FreeBSD (14) to a Windows 11 vm (it generates a resource conflict and requires additional installation)

[ziomario]

Hello to everyone.

What I would like to understand if why,when I try to passthru my RTX 2080 ti from FreeBSD to Windows 11,it won't do it,causing the error 12. It says that it generates a resource conflict and requires additional installation).

I'm trying to debug the error. Below you see the IOMMU group of my Nvidia Geforce RTX 2080 ti passed in a Windows 11 / bhyve VM and of Windows 11 installed physically. I'm not able to understand if there are inconsistencies.


VM :

Group: 30.0.15.1123 - Device: NVIDIA GeForce RTX 2080 Ti (Device ID: PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\3&61AAA01&0&48)

Group: 4.39.0.0 - Device: NVIDIA Virtual Audio Device (Wave Extensible) (WDM) (Device ID: ROOT\UNNAMED_DEVICE\0000)

Group: 1.46.831.832 - Device: NVIDIA USB Type-C Port Policy Controller (Device ID: PCI\VEN_10DE&DEV_1AD7&SUBSYS_250319DA&REV_A1\3&61AAA01&0&4B)

Group: 10.0.22000.2360 - Device: Controller host NVIDIA USB 3.10 eXtensible - 1.10 (Microsoft) (Device ID: PCI\VEN_10DE&DEV_1AD6&SUBSYS_250319DA&REV_A1\3&61AAA01&0&4A)

Physical installation :

Group: 31.0.15.5222 - Device: NVIDIA GeForce RTX 2080 Ti (Device ID: PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\4&94764DC&0&0009)

Group: 4.49.0.0 - Device: NVIDIA Virtual Audio Device (Wave Extensible) (WDM) (Device ID: ROOT\UNNAMED_DEVICE\0000)

Group: 1.50.831.832 - Device: NVIDIA USB Type-C Port Policy Controller (Device ID: PCI\VEN_10DE&DEV_1AD7&SUBSYS_250319DA&REV_A1\4&94764DC&0&0309)

Group: 10.0.22000.2360 - Device: NVIDIA USB 3.10 eXtensible Host Controller - 1.10 (Microsoft) (Device ID: PCI\VEN_10DE&DEV_1AD6&SUBSYS_250319DA&REV_A1\4&94764DC&0&0209)

As an experiment I've passed only this device from FreeBSD to WIndows 11 :

pptdevs="2/0/0"

I have excluded "2/0/1 ; 2/0/2 ; 2/0/3". I want to show you what means these addresses :

02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

What happened when I launched the vm using these parameters ?

bhyve -S -c sockets=2,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/Windows11.img,bootindex=1 \
-s 9:0,passthru,2/0/0 \
-s 11,hda,play=/dev/dsp,rec=/dev/dsp \
-s 13,virtio-net,tap18 \
-s 29,fbuf,tcp=0.0.0.0:5918,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18

this error :

Assertion failed: (!err), function hda_init, file /usr/corvin-src-releng-140/usr.sbin/bhyve/pci_hda.c, line 353.

The error is caused by this parameter :

-s 11,hda,play=/dev/dsp,rec=/dev/dsp \

I have excluded it and Windows 11 booted. Now,inside it I still see the error 12 :

The device PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\3&61aaa01&0&48 generates a resource conflict and requires additional installation.

If I pass to the vm only -s 9:0,passthru,2/0/0 \ ; it does not work anyway,because AFAIK,MAYBE there is one and only IOMMU group,so I should pass everything ? But if I pass everything,the error 12 is there anyway.

I would like to understand if there is a problem with the IOMMU thing,if there is maybe there is the needing to develop a patch like this one : GitHub - benbaker76/linux-acs-override for FreeBSD.

For an experienced developer should not be so hard to understand which devices conflict. I want to ask some help to understand how to debug the error. If we will be able to understand which devices conflict,maybe I can explain this to the bhyve developer with which I'm in contact and he can develop a patch. So,we will be able to use every modern nvidia gpus inside a Windows VM with bhyve under FreeBSD ! That's a nice goal,isn't it ? So,please ask me more informations that you need to understand the origin of the error.

Windows Build/Version10.0.22000.2538

 

[ChatGPT]

Hello,

From your description, it seems like the issue is because of the fact that all devices in the IOMMU group are not being passed through to the VM. This is because all devices in an IOMMU group have to be passed together as they share DMA and interrupt remapping isolation. If only a subset of the group is isolated, there can be unpredicted results, hence the need for all or none.

Let's take your case: you have four devices in the group. These should be:

1. The VGA compatible controller
2. The Audio device
3. USB controller
4. Serial bus controller

According to your logs, you are passing `-s 9:0,passthru,2/0/0`, which includes just the VGA controller, but not the other three devices. To pass everything, you could try something like:

```
-s 9:0,passthru,2/0/0 \
-s 9:1,passthru,2/0/1 \
-s 9:2,passthru,2/0/2 \
-s 9:3,passthru,2/0/3
```

You also mentioned an ACS patch, considering the IOMMU group boundaries are defined by the Advanced Configuration and Power Interface (ACPI) and the BIOS. If your hardware does not properly support ACS or if it is not implemented correctly, you can modify the IOMMU groupings by using the ACS override patch which you have mentioned.

However, in your case, because you're using FreeBSD, you would need a similar patch that is written and applicable for it. It may be possible that you need to contact the bhyve developers or FreeBSD community to create a similar ACS override patch for FreeBSD.

For some additional steps, please make sure you set your GPU into `vfio-pci` mode, did `permissive=1` setting, and correctly installed Nvidia drivers on your VM.

Please bear in mind that misconfiguring IOMMU groups can result in unpredictable behavior and implement changes in a controlled environment.

Also, ensure your system BIOS has IOMMU (Sometimes called "Virtualization technology for Directed I/O" or "VT-d") enabled. It's essential for passing through a PCI device.

Feel free to ask for more details or if you need any assistance with this. However, it's also highly advisable to share these logs with your dedicated FreeBSD system administrator or directly with the bhyve developers to get a more detailed analysis.

 

[ziomario]

Hello ChatGPT,

very thanks for your kind reply. For sure in the past experiments I have passed all the 4 devices in the group. Unfortunately I've got the same error. What I need to know is what can I do to know with which other device the VGA compatible controller conflicts. What's the procedure to understand what are the interrupts that conflict and to whom they belong.
I would also know how to set my GPU into `vfio-pci` mode / `permissive=1` setting. Take also in consideration that in FreeBSD there isn't any vfio-pci.