Announcement

Collapse
No announcement yet.

Nvidia - PCIe passthrough Blackscreen Windows and Linux VMs

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    [SOLVED] Nvidia - PCIe passthrough Blackscreen Windows and Linux VMs

    I just noticed that all my VMs, Linux and Windows, just show a blackscreen, HDMI and DisplayPort.
    The monitor shows he receives (VRR info) 3840x2160@120Hz, RGB 10b 4L10, GSYNC.​

    It looks to me as if a Kubuntu update has triggered a function in the Nvidia driver, which then only outputs a black image
    The last time I think it worked was 4 days ago, since then I only used the Ubuntu Server VM, for which I don't use a monitor​

    I haven't made any changes to the Libvirt configuration or the VMs, no driver updates, since installing the 5090 and the GPU has been working without problems for weeks.
    The only change has been some updates to Kubuntu 25.04 in the last few days.
    The host sytem also doesn't show any errors when starting a VM.​​


    System:
    Kubuntu 25.04 - 6.14.0-24-generic
    QEMU emulator version 9.2.1 (Debian 1:9.2.1+ds-1ubuntu5)
    AMD 9950x iGPU for host. RTX 5090 for guests.


    I checked some Windows VMs via RDP, but I don't see any problems, the GPU works apart from the black screen normal.
    My Ubuntu Server VM with Ollama also works normally, apart from the fact that the monitor does not display any image.
    Display output works with a normal Windows installation, so it seems to be something with the virtualization in Kubuntu 25.04

    I tried the latest Nvidia driver, but it doesn't help, the monitors remain black.
    I deleted "<hidden state="on"/>" from the Libvirt configuration, because this is usually needed to get rid of the black screen error (code43), but it has no effect, the Nvidia driver still works.


    Code:
    root@kubu01:~# virt-host-validate
      QEMU: Checking for hardware virtualization                                 : PASS
      QEMU: Checking if device '/dev/kvm' exists                                 : PASS
      QEMU: Checking if device '/dev/kvm' is accessible                          : PASS
      QEMU: Checking if device '/dev/vhost-net' exists                           : PASS
      QEMU: Checking if device '/dev/net/tun' exists                             : PASS
      QEMU: Checking for cgroup 'cpu' controller support                         : PASS
      QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
      QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
      QEMU: Checking for cgroup 'memory' controller support                      : PASS
      QEMU: Checking for cgroup 'devices' controller support                     : PASS
      QEMU: Checking for cgroup 'blkio' controller support                       : PASS
      QEMU: Checking for device assignment IOMMU support                         : PASS
      QEMU: Checking if IOMMU is enabled by kernel                               : PASS
      QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
       LXC: Checking for Linux >= 2.6.26                                         : PASS
       LXC: Checking for namespace 'ipc'                                         : PASS
       LXC: Checking for namespace 'mnt'                                         : PASS
       LXC: Checking for namespace 'pid'                                         : PASS
       LXC: Checking for namespace 'uts'                                         : PASS
       LXC: Checking for namespace 'net'                                         : PASS
       LXC: Checking for namespace 'user'                                        : PASS
       LXC: Checking for cgroup 'cpu' controller support                         : PASS
       LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
       LXC: Checking for cgroup 'cpuset' controller support                      : PASS
       LXC: Checking for cgroup 'memory' controller support                      : PASS
       LXC: Checking for cgroup 'devices' controller support                     : PASS
       LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
       LXC: Checking for cgroup 'blkio' controller support                       : PASS
       LXC: Checking if device '/sys/fs/fuse/connections' exists                 : PASS​


    Libvirt Configuration, the configuration is essentially the same for all VMs and has probably been in use for two years

    Code:
     <features>
        <acpi/>
        <apic/>
        <hyperv mode="custom">
          <relaxed state="on"/>
          <vapic state="on"/>
          <spinlocks state="on" retries="8191"/>
          <vpindex state="on"/>
          <runtime state="on"/>
          <synic state="on"/>
          <stimer state="on">
            <direct state="on"/>
          </stimer>
          <reset state="on"/>
          <vendor_id state="on" value="AuthenticAMD"/>
          <frequencies state="on"/>
          <reenlightenment state="on"/>
          <tlbflush state="on"/>
          <ipi state="on"/>
          <evmcs state="off"/>
        </hyperv>
        <kvm>
          <hidden state="on"/>
        </kvm>
        <vmport state="off"/>
        <smm state="on"/>
        <ioapic driver="kvm"/>
      </features>
    
    
    
       <cpu mode="host-passthrough" check="none" migratable="off">
        <topology sockets="1" dies="2" clusters="1" cores="8" threads="1"/>
        <feature policy="require" name="topoext"/>
        <feature policy="require" name="invtsc"/>
        <feature policy="disable" name="monitor"/>
      </cpu>
      <clock offset="localtime">
        <timer name="rtc" tickpolicy="catchup"/>
        <timer name="pit" tickpolicy="delay"/>
        <timer name="hpet" present="no"/>
        <timer name="kvmclock" present="no"/>
        <timer name="hypervclock" present="yes"/>
        <timer name="tsc" present="yes" mode="native"/>
      </clock>
    
    
          <hostdev mode="subsystem" type="pci" managed="yes">
          <source>
            <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
          </source>
          <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0" multifunction="on"/>
        </hostdev>
        <hostdev mode="subsystem" type="pci" managed="yes">
          <source>
            <address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
          </source>
          <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x1"/>
        </hostdev>
    
    ​
    This is the log when a VM starts

    Code:
    journalctl --since "now" -f
    Jul 22 01:12:43 kubu01 kernel: xhci_hcd 0000:69:00.0: remove, state 4
    Jul 22 01:12:43 kubu01 kernel: usb usb6: USB disconnect, device number 1
    Jul 22 01:12:43 kubu01 kernel: xhci_hcd 0000:69:00.0: USB bus 6 deregistered
    Jul 22 01:12:43 kubu01 kernel: xhci_hcd 0000:69:00.0: remove, state 4
    Jul 22 01:12:43 kubu01 kernel: usb usb5: USB disconnect, device number 1
    Jul 22 01:12:43 kubu01 kernel: xhci_hcd 0000:69:00.0: USB bus 5 deregistered
    Jul 22 01:12:43 kubu01 kernel: br0: port 2(vnet3) entered blocking state
    Jul 22 01:12:43 kubu01 kernel: br0: port 2(vnet3) entered disabled state
    Jul 22 01:12:43 kubu01 kernel: vnet3: entered allmulticast mode
    Jul 22 01:12:43 kubu01 kernel: vnet3: entered promiscuous mode
    Jul 22 01:12:43 kubu01 systemd-networkd[1793]: vnet3: Link UP
    Jul 22 01:12:43 kubu01 systemd-networkd[1793]: vnet3: Gained carrier
    Jul 22 01:12:43 kubu01 kernel: br0: port 2(vnet3) entered blocking state
    Jul 22 01:12:43 kubu01 kernel: br0: port 2(vnet3) entered listening state
    Jul 22 01:12:43 kubu01 systemd-machined[1857]: New machine qemu-4-win11-24h2-off.
    Jul 22 01:12:43 kubu01 systemd[1]: Started machine-qemu\x2d4\x2dwin11\x2d24h2\x2doff.scope - Virtual Machine qemu-4-win11-24h2-off.
    Jul 22 01:12:45 kubu01 avahi-daemon[1821]: Joining mDNS multicast group on interface vnet3.IPv6 with address fe80::fc54:ff:fe68:3350.
    Jul 22 01:12:45 kubu01 avahi-daemon[1821]: New relevant interface vnet3.IPv6 for mDNS.
    Jul 22 01:12:45 kubu01 systemd-networkd[1793]: vnet3: Gained IPv6LL
    Jul 22 01:12:45 kubu01 avahi-daemon[1821]: Registering new address record for fe80::fc54:ff:fe68:3350 on vnet3.*.
    Jul 22 01:12:45 kubu01 kernel: vfio-pci 0000:03:00.0: Enabling HDA controller
    Jul 22 01:12:45 kubu01 kernel: vfio-pci 0000:03:00.0: resetting
    Jul 22 01:12:45 kubu01 kernel: br0: port 2(vnet3) entered learning state
    Jul 22 01:12:45 kubu01 kernel: vfio-pci 0000:03:00.0: reset done
    Jul 22 01:12:45 kubu01 kernel: vfio-pci 0000:02:00.0: resetting
    Jul 22 01:12:45 kubu01 kernel: vfio-pci 0000:02:00.0: reset done
    Jul 22 01:12:45 kubu01 kernel: vfio-pci 0000:69:00.0: resetting
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:69:00.0: reset done
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:03:00.0: resetting
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:03:00.1: resetting
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:03:00.0: reset done
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:03:00.1: reset done
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:69:00.0: resetting
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:69:00.0: reset done
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:03:00.0: resetting
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:03:00.0: reset done
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:02:00.0: resetting
    Jul 22 01:12:46 kubu01 kernel: vfio-pci 0000:02:00.0: reset done
    Jul 22 01:12:47 kubu01 kernel: br0: port 2(vnet3) entered forwarding state
    Jul 22 01:12:47 kubu01 kernel: br0: topology change detected, propagating​

    #2
    Problem solved, but somehow strange...
    The GPU suddenly needs a copy of the VBios via Libvirt, but previously worked without it for four weeks.​

    Comment

    Users Viewing This Topic

    Collapse

    There are 0 users viewing this topic.

    Working...
    X