Add fabric manager configuration support #2045

mresvanis · 2026-01-15T15:01:06Z

Description

This PR enables Fabric Manager (FM) configuration for vm-passthrough workloads using the Shared NVSwitch virtualization model.

It enables users to configure the Fabric Manager mode (i.e. FABRIC_MODE=[0,1,2], 0 - full-passthrough, 1 - shared NVSwitch, 2 - vGPU) through the ClusterPolicy CRD, providing better support for NVIDIA multi-GPU systems in virtualized environments.

In the FM shared NVSwitch virtualization model the NVIDIA driver on the host is used for the NVSwitch devices, while the GPU devices are bound to the vfio-pci driver. The goal is for the GPU devices to be passed-through to kubevirt VMs, while the respective fabric is managed on the host.

Depends on / relates to: NVIDIA/gpu-driver-container#538

Changes

ClusetrPolicy API

add FabricManagerSpec to the ClusterPolicy CRD with support for two modes:
- full-passthrough (FABRIC_MODE=0) - default mode.
- shared-nvswitch (FABRIC_MODE=1) - shared NVSwitch virtualization mode.
update all CRD manifests across bundle, config, and deployment directories to include the new Fabric Manager configuration fields.

Controller logic

enable driver installation when using vm-passthrough with FM shared NVSwitch mode and pass an env var to the driver container to indicate the selected fabric mode (the driver container is the one configuring and starting the FM).
integrate FM configuration checks into the state manager workflow.

Driver state management

add logic to detect and handle Fabric Manager shared NVSwitch mode.
update driver startup probe behavior for vm-passthrough and FM shared NVSwitch mode case.
adjust the driver startup probe to accommodate Fabric Manager requirements in vm-passthrough with shared NVSwitch mode.

Sandbox validator

add driver validation as the first init container when FM shared NVSwitch mode.
add wait flow to the vfio-pci validation.

VFIO manager

wait for driver to be ready when FM shared NVSwitch mode - this step is required because we need a mapping of GPU physical module ID to the respective PCIe address, as FM identifies GPUs by their physical module ID. The latter can be found via nvidia-smi, which requires the driver to be loaded and bound to the GPU devices. Once that's done we can bind the GPU devices to vfio-pci.
replace init container with vfio-manage unbind --all when FM shared NVSwitch mode.

Checklist

No secrets, sensitive information, or unrelated changes
Lint checks passing (make lint)
Generated assets in-sync (make validate-generated-assets)
Go mod artifacts in-sync (make validate-modules)
Test cases are added for new code paths

Testing

TBD

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

copy-pr-bot · 2026-01-15T15:01:10Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

LandonTClipp · 2026-01-15T17:51:07Z

How coincidental that I resolved to implement something like this and 2 hours ago you submitted this draft!

I want to ask what the plan is for the CDI-side. The ideal scenario is that the fabricmanager can be spawned as a Kata container, which means we need to inject the NVSwitch VFIO cdevs just like how we do for passthrough GPUs. When I tried to use GPU operator a few months ago, this was simply not possible at the time so I used libvirt instead. Does the GPU Operator CDI already expose the NVswitches to k8s now? I apologize if my knowledge is a little out of date.

…-passthrough Signed-off-by: Michail Resvanis <mresvani@redhat.com>

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

When clusterPolicy.fabricManager.mode=shared-nvswitch and workload=vm-passthrough, the vfio-manager now preserves the NVIDIA driver for fabric management while enabling GPU device passthrough to VMs. Changes: - Modify TransformVFIOManager to detect shared-nvswitch mode. - Replace driver uninstall init container with device unbind init container. - Use vfio-manage unbind --all to detach devices from nvidia driver. - Keep nvidia driver loaded for fabric management functionality. - Add comprehensive unit tests for both normal and shared-nvswitch modes. The new flow for shared-nvswitch mode for the vfio-manager: 1. InitContainer: vfio-manage unbind --all (unbind from nvidia driver) 2. Container: vfio-manage bind --all (bind to vfio-pci) This enables simultaneous fabric management and VM passthrough capabilities.

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

cdesiniotis · 2026-01-28T17:43:43Z

assets/state-driver/0400_configmap.yaml

+      # For vm-passthrough with shared-nvswitch mode, nvidia-smi may fail due to unbound devices
+      # Fall back to checking if nvidia module is loaded when FABRIC_MANAGER_FABRIC_MODE=1


Question (for my understanding) -- GPUs may not be bound to the nvidia driver since there is a chance that the vfio-manager ran already and unbound the devices? Am I understanding this correct?

cdesiniotis · 2026-01-28T17:44:54Z

assets/state-driver/0400_configmap.yaml

-      exit 1
+      # For vm-passthrough with shared-nvswitch mode, nvidia-smi may fail due to unbound devices
+      # Fall back to checking if nvidia module is loaded when FABRIC_MANAGER_FABRIC_MODE=1
+      if [ "${FABRIC_MANAGER_FABRIC_MODE:-}" = "1" ] && lsmod | grep -q "^nvidia "; then


Question -- isn't the right-hand-side of this if statement redundant? Don't we already know the nvidia module is loaded prior to this (see L19-22)?

cdesiniotis · 2026-01-28T17:47:25Z

assets/state-driver/0500_daemonset.yaml

This file is hard for me to review since the diff is large and it appears to mostly be a change in indentation. Did anything meaningful change here besides the indentation? (if not, I'd prefer if we reverted this to minimize the diff)

cdesiniotis · 2026-01-28T17:58:26Z

controllers/object_controls.go

+	if config.FabricManager.IsSharedNVSwitchMode() {
+		// In shared-nvswitch mode, replace driver uninstall with device unbind
+		// Find the k8s-driver-manager init container and replace it with vfio-manage unbind
+		for i := range obj.Spec.Template.Spec.InitContainers {


We have a helper findContainerByName() that we should use here:

container := findContainerByName(obj.Spec.Template.Spec.InitContainers, "k8s-driver-manager") container.Name = "vfio-device-unbind" // ... all other transformations ...

cdesiniotis · 2026-01-28T18:21:40Z

controllers/object_controls.go

+				initContainer.Command = []string{"/bin/sh"}
+				initContainer.Args = []string{"-c", `
+# For shared-nvswitch mode, wait for driver to be ready before unbinding
+echo "Shared NVSwitch mode detected, waiting for driver readiness..."
+until [ -f /run/nvidia/validations/driver-ready ]
+do
+  echo "waiting for the driver validations to be ready..."
+  sleep 5
+done
+
+set -o allexport
+cat /run/nvidia/validations/driver-ready
+. /run/nvidia/validations/driver-ready
+
+echo "Driver is ready, proceeding with device unbind"
+exec vfio-manage unbind --all`}


Instead of adding this in code, what about encapsulating this logic in a custom entrypoint script that is stored in a ConfigMap? The fabric manager mode can be indicated via an envvar and the entrypoint script can check the envvar to determine what actions to take (and what command to run -- k8s-driver-manager uninstall_driver vs vfio-manage unbind --all).

cdesiniotis · 2026-01-28T19:14:30Z

Question -- do we need to add an extra host path volume to the sandbox-device-plugin so that it can access the fabric manager UNIX socket?

Add Fabric Manager configuration API types and CRD manifests

c221ff7

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

LandonTClipp mentioned this pull request Jan 15, 2026

Support for NVSwitch in Shared NVSwitch Virtualization Model NVIDIA/sandbox-device-plugin#24

Open

mresvanis force-pushed the fabric-manager-configuration branch 2 times, most recently from 5f8e006 to e27e938 Compare January 21, 2026 15:11

mresvanis added 5 commits January 22, 2026 10:35

Implement controller support for Fabric Manager configuration when vm…

54dd493

…-passthrough Signed-off-by: Michail Resvanis <mresvani@redhat.com>

Adjust driver startup probe for vm-passthrough with shared NVSwitch mode

b994d43

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

Add FM env var to driver container when shared-nvswitch

e974db0

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

Add wait for vfio-pci sandbox validation

c59ba01

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

mresvanis force-pushed the fabric-manager-configuration branch 2 times, most recently from c53ceaa to 70c5d78 Compare January 22, 2026 12:00

Add driver validation in sandbox when FM shared-nvswitch mode

28c95d9

Signed-off-by: Michail Resvanis <mresvani@redhat.com>

mresvanis force-pushed the fabric-manager-configuration branch from 70c5d78 to 28c95d9 Compare January 22, 2026 12:13

mresvanis mentioned this pull request Jan 22, 2026

Add Fabric Manager Shared NVSwitch virtualization model support NVIDIA/gpu-driver-container#538

Draft

cdesiniotis reviewed Jan 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fabric manager configuration support #2045

Add fabric manager configuration support #2045

mresvanis commented Jan 15, 2026 •

edited

Loading

Uh oh!

copy-pr-bot bot commented Jan 15, 2026

Uh oh!

LandonTClipp commented Jan 15, 2026

Uh oh!

cdesiniotis Jan 28, 2026

Uh oh!

cdesiniotis Jan 28, 2026

Uh oh!

cdesiniotis Jan 28, 2026

Uh oh!

cdesiniotis Jan 28, 2026

Uh oh!

cdesiniotis Jan 28, 2026

Uh oh!

cdesiniotis commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# For vm-passthrough with shared-nvswitch mode, nvidia-smi may fail due to unbound devices
		# Fall back to checking if nvidia module is loaded when FABRIC_MANAGER_FABRIC_MODE=1

Add fabric manager configuration support #2045

Are you sure you want to change the base?

Add fabric manager configuration support #2045

Conversation

mresvanis commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

ClusetrPolicy API

Controller logic

Driver state management

Sandbox validator

VFIO manager

Checklist

Testing

Uh oh!

copy-pr-bot bot commented Jan 15, 2026

Uh oh!

LandonTClipp commented Jan 15, 2026

Uh oh!

cdesiniotis Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cdesiniotis Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cdesiniotis Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cdesiniotis Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cdesiniotis Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cdesiniotis commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mresvanis commented Jan 15, 2026 •

edited

Loading