vmm: Introduce option --platform vfio_p2p_dma=on|off

Add a user-configurable option to control whether VFIO device MMIO BAR
regions are DMA-mapped into the host IOMMU address space.

This mapping is required for peer-to-peer DMA between devices (e.g.
NVLink, RDMA NIC accessing GPU VRAM). However, iommufd on upstream
kernels does not support mapping device MMIO pages (VM_PFNMAP), causing
IOMMU_IOAS_MAP to fail with -EFAULT. Kernels with the NVIDIA PFNMAP
workaround or future kernels with DMABUF-based mapping
(IOMMU_IOAS_MAP_FILE) handle this correctly.

The option defaults to `on` to preserve existing behavior. Users on
vanilla kernels using iommufd should set `vfio_p2p_dma=off` to skip
MMIO BAR DMA mapping.

A validation check ensures that `x_nv_gpudirect_clique` (which depends
on P2P DMA) cannot be used when `vfio_p2p_dma=off`.

Signed-off-by: Bo Chen <bchen@crusoe.ai>

vmm/src/device_manager.rs

@@ -3963,6 +3963,14 @@ impl DeviceManager {
 
         let memory_manager = self.memory_manager.clone();
 
+        let vfio_p2p_dma = self
+            .config
+            .lock()
+            .unwrap()
+            .platform
+            .as_ref()
+            .is_none_or(|p| p.vfio_p2p_dma);
+
         let vfio_pci_device = VfioPciDevice::new(
             vfio_name.clone(),
             self.address_manager.vm.clone(),
@@ -3971,6 +3979,7 @@ impl DeviceManager {
             self.msi_interrupt_manager.clone(),
             legacy_interrupt_group,
             device_cfg.iommu,
+            vfio_p2p_dma,
             pci_device_bdf,
             memory_manager.lock().unwrap().memory_slot_allocator(),
             vm_migration::snapshot_from_id(self.snapshot.as_ref(), vfio_name.as_str()),