vmm: Introduce option --platform vfio_p2p_dma=on|off

Add a user-configurable option to control whether VFIO device MMIO BAR
regions are DMA-mapped into the host IOMMU address space.

This mapping is required for peer-to-peer DMA between devices (e.g.
NVLink, RDMA NIC accessing GPU VRAM). However, iommufd on upstream
kernels does not support mapping device MMIO pages (VM_PFNMAP), causing
IOMMU_IOAS_MAP to fail with -EFAULT. Kernels with the NVIDIA PFNMAP
workaround or future kernels with DMABUF-based mapping
(IOMMU_IOAS_MAP_FILE) handle this correctly.

The option defaults to `on` to preserve existing behavior. Users on
vanilla kernels using iommufd should set `vfio_p2p_dma=off` to skip
MMIO BAR DMA mapping.

A validation check ensures that `x_nv_gpudirect_clique` (which depends
on P2P DMA) cannot be used when `vfio_p2p_dma=off`.

Signed-off-by: Bo Chen <bchen@crusoe.ai>

vmm/src/vm_config.rs

@@ -113,6 +113,10 @@ pub fn default_platformconfig_iommu_address_width_bits() -> u8 {
     DEFAULT_IOMMU_ADDRESS_WIDTH_BITS
 }
 
+pub fn default_platformconfig_vfio_p2p_dma() -> bool {
+    true
+}
+
 #[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)]
 pub struct PlatformConfig {
     #[serde(default = "default_platformconfig_num_pci_segments")]
@@ -135,6 +139,8 @@ pub struct PlatformConfig {
     pub sev_snp: bool,
     #[serde(default)]
     pub iommufd: bool,
+    #[serde(default = "default_platformconfig_vfio_p2p_dma")]
+    pub vfio_p2p_dma: bool,
 }
 
 pub const DEFAULT_PCI_SEGMENT_APERTURE_WEIGHT: u32 = 1;