vm-memory 0.18 has renamed GuestMemory to GuestMemoryBackend, and made
GuestMemory refer to something less specific. For simplicity, we keep
using GuestMemoryBackend (formerly GuestMemory) everywhere for now. We
can adjust bounds to be less specific later if we find ourselves needing
the newly enabled flexibility.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Signed-off-by: Rob Bradford <rbradford@meta.com>
A PCI BAR stores its address in the upper bits and type information in
the lower bits. To determine the BAR size, an operating system disables
memory decoding, writes ones to the BAR's address bits, and reads the
value back. Bits that remain zero reveal the required alignment and
therefore the size.
We currently recognize only 0xffffffff as a size probe. OpenBSD instead
writes the memory BAR address mask 0xfffffff0, without setting the lower
four non-address bits [0][1].
We misinterpret that probe as a real BAR relocation. Because OpenBSD has
disabled memory space, the relocation remains pending. We apply it when
OpenBSD enables memory space again, moving the virtio block BAR away
from its mapped address and preventing the guest from booting.
Recognize probes using the address mask for each BAR type and exclude
them from BAR reprogramming. Add a test for the OpenBSD memory BAR
probe.
[0]: 54a2bc1e54/sys/dev/pci/pci_map.c (L154-L180)
[1]: 54a2bc1e54/sys/dev/pci/pcireg.h (L429-L443)
Co-authored-by: Thomas Prescher <thomas.prescher@cyberus-technology.de>
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Wire the previously empty Migratable implementation on VfioPciDevice to
VFIO DMA logging so the pages the device writes to guest memory are
tracked during a live migration. The Vfio trait gains the DMA logging
operations with defaults that report no migration support.
Without a virtual IOMMU the device sees an identity mapping of guest
memory, so the tracked ranges are the guest memory regions with iova
equal to gpa. The requested page size is only a hint, so the device
reports back the granularity it actually applied.
A device that does not support migration, or one behind a virtual
IOMMU, is refused at migration start rather than silently skipping
dirty tracking and producing an incorrect log. Under a virtual IOMMU
the reported IOVAs are not guest addresses and the logged range set
would have to track the mapping changes a guest makes mid migration,
so that configuration is out of scope.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
A failed VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE set can leave the device
in its original state, midway along a combination transition, or in
ERROR. The uAPI leaves recovery to the user, and from ERROR only a
VFIO_DEVICE_RESET brings the device back to RUNNING.
Previously a failed transition left the device wherever it landed, so
one that reached ERROR stayed wedged for the life of the guest even
though a reset would have recovered it.
On a failed transition the caller now names an optional recovery state
to attempt first, falling back to a device reset. Pause, resume, and
the return to STOP after STOP_COPY name no recovery state, so their
failure resets directly. The original transition error is always
propagated and recovery failures are logged.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
```
warning: you seem to be trying to move all elements into a new `Vec`
--> pci/src/configuration.rs:951:24
|
951 | return self.pending_bar_reprogram.drain(..).collect();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use `mem::take` to avoid creating a new allocation: `std::mem::take(&mut self.pending_bar_reprogram)`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#drain_collect
= note: `-D clippy::drain-collect` implied by `-D clippy::all`
```
Signed-off-by: Rob Bradford <rbradford@meta.com>
mmap(NULL, ...) only guarantees page-sized alignment, so the device
MMIO regions mapped for VFIO passthrough were not necessarily aligned
on a huge page boundary and thus could not take fast paths in the
kernel.
Align the address to the largest possible size. This allows VFIO to take
fast paths when walking the page table.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
Assisted-by: Copilot:GPT-5.5
The VFIO save and load paths drive the kernel migration state machine
through a strict transition sequence and must recover correctly when a
transition or the data transfer fails. Validating that on real hardware
needs a migratable device, which CI does not have, so the behavior would
otherwise go unverified until it breaks in the field.
Introduce a mock Vfio wrapper that records state transitions and keeps
the migration blob in memory, then test VfioCommon save and load against
it. This pins the transition ordering, the data round trip, and the
failure recovery in CI on any host, independent of hardware. The trait
defaults and the VfioMigrationState conversions are covered too, so a
non migratable device and an unknown state value stay well defined. A
restore that carries migration state onto a device without migration
support is checked to fail instead of dropping the state.
A shadow sync test guards the related save path change, confirming a non
BAR config write reaches the PciConfiguration shadow so a snapshot
captures the live value instead of the post init zero.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
When a snapshot is loaded, walk the migration v2 state machine
from VfioCommon::set_state() after interrupt state has been
restored. If the device supports migration and a blob is
present, drive RUNNING to RESUMING in a single transition and
write the blob to the data_fd. The kernel handles the
intermediate STOP arc internally. An explicit STOP dwell was
observed to make mlx5_vfio_pci re initialize SQ, CQ, and EQ
indices on top of the just loaded blob, wedging queue state on
resume. The device is left in RESUMING and resume() drives it
to RUNNING during VM resume.
set_state() also pushes PCI_COMMAND to the device via
write_config() after the blob load. Rebuilding the in memory
MSI or MSI-X structs does not touch the kernel's view of
PCI_COMMAND, so without this the VF sits at post reset defaults
with no bus master and mlx5_core ACCESS_REG times out. It
rearms VFIO_DEVICE_SET_IRQS via enable_msi or enable_msix for
the same reason, since replaying the in memory interrupt state
does not reissue the ioctl and the kernel has no eventfds for
this device until it does. Both match QEMU
vfio_pci_load_config().
In allocate_bars, skip add_pci_bar and add_pci_rom_bar on
restore. PciConfiguration::new(Some(state)) already populated
the BAR registers with used=true, so the extra call trips
BarInUse. The bars vec and mmio_regions pushes still need to
happen so the caller can wire bus mappings.
set_state() retrieves the migration blob from the snapshot
unconditionally and rejects a snapshot that carries migration
state when the device does not support migration, rather than
silently dropping the saved state. A device without migration
support and no blob, including vfio-user, still skips the load.
On any transition or write failure during restore, STOP is
attempted as best effort before bubbling the error.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Wire a migratable VFIO device's migration state to the VM lifecycle so
the device's internal state survives snapshot and restore. A device such
as a ConnectX VF bound to mlx5_vfio_pci would otherwise come back blank,
because a plain snapshot saves only the PCI configuration Cloud
Hypervisor owns, not the device's own state.
On save, pause moves the device to STOP and snapshot() drives it through
STOP_COPY to extract the opaque state blob, attached to the device
snapshot as a base64 encoded child. resume() returns it to RUNNING.
All new behavior is gated on migration_flags.is_some(), so devices
without migration support (including vfio-user) retain their previous
snapshot behavior.
If the data read fails after STOP_COPY was entered, the device is
returned to STOP before the error is bubbled, since the STOP_COPY
to STOP arc stays valid. A failed transition into STOP_COPY returns
immediately because a STOP from the resulting ERROR state cannot
help. Full recovery including device reset is deferred.
Since the non BAR write path goes directly to the VFIO device and not
the shadow, the PciConfiguration shadow can get stale. Mirror every
non BAR, non MSI config write into the shadow via write_byte /
write_word / write_reg so snapshot() can capture PCI_COMMAND. Without
this the shadow keeps the values set at device init and snapshot()
encodes PCI_COMMAND as zero.
Use the raw write_byte, write_word, and write_reg helpers rather than
PciConfiguration::write_config_register, which would otherwise drain
pending_bar_reprogram, consumed by the BAR block below, and rerun
MSI-X set_msg_ctl, already done by update_msix_capabilities.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Probe VFIO_DEVICE_FEATURE_MIGRATION during VfioCommon::new() and store
the result in a new migration_flags field so later migration phases can
gate state machine transitions.
The probe runs on every instantiation, including snapshot restore,
because migration capability is a property of the host kernel and its
variant driver rather than of any saved VM state.
migration_flags() is added to the internal Vfio trait with a default
implementation that returns Ok(None), meaning not migratable.
VfioDeviceWrapper overrides it to issue the kernel ioctl, while
vfio-user devices keep the default and are always treated as non
migratable.
Allow the VFIO_DEVICE_FEATURE ioctl in the VMM seccomp filter.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Replace assertions for incorrect access sizes with logged errors. The
write_table() method already handled it like this and this commit
extends the same pattern to read_table() and read_pba().
Signed-off-by: Rob Bradford <rbradford@meta.com>
`PciBus::register_mapping()` operates on `mmio_bus` and `io_bus`
which are passed in as external parameters and have nothing to do
with PciBus internal state. Move this logic into
`DeviceManager::register_bar_mapping()` where it belongs, and move
the `PioInsert`/`MmioInsert` error variants from `PciRootError` to
`DeviceManagerError` accordingly.
Signed-off-by: wuxinyue <wuxinyue.wxy@antgroup.com>
Assisted-by: Claude:Opus-4.6
Drop stale #[allow]s whose lints no longer fire and convert the
rest to #[expect], which warns if they ever stop being needed.
Part of #8326.
Signed-off-by: Tushar Khatri <hello@tusharkhatri.in>
Import the modules used in the crate instead of spelling the
fully-qualified paths at every use site. This covers std paths along
with a few crate-internal and external-crate paths, leaving pci free of
clippy::absolute_paths warnings.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
VfioUserDmaMapping::map panicked when find_region returned an anonymous
mmap region. Change this so a user gets an error instead of a panic.
When the VMM hotplugs a region into a guest that also has a vfio-user
device, all region's handlers are called. With the anonymous memory
backing (no file=, shared=on, or hugepages), region.file_offset()
returns None and the .unwrap() panics the VMM. Replace the unwrap with
an explicit error and use checked_add for the offset combine.
Signed-off-by: Dylan Reid <dgreid@fb.com>
Allow VFIO devices to list BAR indices that should not be
mmapped into the guest. This lets operators skip large BARs that
are known not to be used by their workload.
When a BAR is skipped, the log also calls out that P2P DMA
mapping is skipped because the VFIO DMA map path uses the same
mmap backing.
Signed-off-by: Damian Barabonkov <dbctl@pm.me>
Assisted-by: OpenCode:gpt-5.5
A malicious or buggy guest can issue an MSI-X table write with an
unexpected size (not 4 or 8 bytes), triggering an assert!() that
crashes the VMM process. Replace the assertion with an error log and
early return to maintain VMM stability under adversarial guest
behavior.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
VfioUserDmaMapping::map is reached from the virtio-iommu MAP handler
with (iova, gpa, size) all guest-controlled. Validate the length of the
region fits, not just the start.
Before this change the vfio-user on the other end could get a size that
spans past the end of its backing storage.
Signed-off-by: Dylan Reid <dgreid@fb.com>
Our bus slots are now Reserved/Allocated/Free so change the method to
free it to free_device_id() and update error.
Also update to take u8 to match the other methods.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This can be used in a two pass approach where all configs that can hold
PCI devices are evaluated to reserve any specific PCI device IDs they
may need. Those device IDs will later be allocated when the devices are
added to the bus. The tri-state Free, Reserved, Allocated also catches
the problem of hotplugging a device with a specific, already used,
device ID.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This commit refactors the PCI bus struct. It has two major focuses.
First, we change the type of `device_ids` in `PciBus` to an array. A
fixed-size array better reflects real PCI bus constraints, especially
its limited number of PCI devices. Moreover, it can't be grown
accidentally.
The second focus is changing the type of the key of `devices` in
`PciBus` to `u8`, since device IDs are not allowed to exceed 31. We
furthermore replace magic numbers with constants and make them publicly
available so we can use them in a follow-up change when parsing user
input.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
When BAR reprogramming is detected, detect_bar_reprogramming()
eagerly updates the BAR address in config space before the actual
MMIO remapping occurs. If the subsequent move_bar() fails (e.g.
the new address falls outside the allocator range), the config
register retains the new address while the MMIO bus still uses
the old one, leaving the device broken.
Add restore_bar_addr() to undo the config space update when
move_bar() fails, so the device remains functional at its
original address.
For 64-bit BARs, restore both the low and high BAR slots as well
as the corresponding config registers, mirroring the two-slot
update logic in detect_bar_reprogramming().
Implement restore_bar_addr() for all PciDevice implementations
(VirtioPciDevice, VfioPciDevice, VfioUserPciDevice, IvshmemDevice,
PvPanicDevice, and PvmemcontrolPciDevice) by delegating to their
respective PciConfiguration::restore_bar_addr().
Signed-off-by: CMGS <ilskdw@gmail.com>
On aarch64 with 64K host pages, VFIO passthrough of devices with
sub-page BARs (e.g. 16K NVMe BAR0) crashes with EINVAL from
KVM_SET_USER_MEMORY_REGION, which requires memory_size to be a
multiple of the host page size.
Expand the mmap to page size instead of rejecting it, matching
QEMU's approach. The kernel's vfio_pci_probe_mmaps() already
verifies that sub-page BARs are page-aligned and reserves the
remainder of the page, so expansion is safe at offset 0. Reject
sub-page sparse areas at non-zero offsets where this guarantee
does not apply.
The expanded mmap region will not overlap with the relocated MSI-X
trap region because fixup_msix_region() ensures MSI-X relocation
at >= page_size offset.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Currently, when snapshoting a running VFIO device with MSI-X enabled,
we get a snapshot where `msix_config.state.enabled` is not consistent
with `msix_state.cap.msg_ctl`,
```jsonc
{
"snapshots": {
"vfio_common": {
"snapshots": {
"msix_config": {
"snapshots": {},
"state": {
"enabled": true
// ...
}
},
// ..
},
"state": {
"msix_state": {
"cap": {
"msg_ctl": 3,
"table": 1,
"pba": 2049
},
// ...
}
// ...
}
}
},
// ...
}
```
The root cause is, after a `MsixCap` is parsed from the device PCI
config space and propagated to a corresponding `MsixConfig`,
`MsixCap::msg_ctl` is never get updated at runtime, only
`MsixConfig::msg_ctl` is updated.
This commit makes `VfioMsix::update` update both `VfioMsix::bar` (of
type `MsixConfig`) and `VfioMsix::cap` (of type `MsixCap`).
Signed-off-by: Changyuan Lyu <changyuanl@google.com>
Add a user-configurable option to control whether VFIO device MMIO BAR
regions are DMA-mapped into the host IOMMU address space.
This mapping is required for peer-to-peer DMA between devices (e.g.
NVLink, RDMA NIC accessing GPU VRAM). However, iommufd on upstream
kernels does not support mapping device MMIO pages (VM_PFNMAP), causing
IOMMU_IOAS_MAP to fail with -EFAULT. Kernels with the NVIDIA PFNMAP
workaround or future kernels with DMABUF-based mapping
(IOMMU_IOAS_MAP_FILE) handle this correctly.
The option defaults to `on` to preserve existing behavior. Users on
vanilla kernels using iommufd should set `vfio_p2p_dma=off` to skip
MMIO BAR DMA mapping.
A validation check ensures that `x_nv_gpudirect_clique` (which depends
on P2P DMA) cannot be used when `vfio_p2p_dma=off`.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Some guests update the MSI-X capability through a 32-bit write at
offset 0 instead of a 16-bit write at offset 2. Update the cached
Message Control state for that path as well so MSI-X enablement stays
in sync with the guest configuration.
Add a short comment documenting why the dword write path also updates
the cached MSI-X Message Control state.
This is important for passthrough GPUs, where MSI-X interrupts are used
during NVIDIA Fabric Manager registration. Without updating the cached
state on the dword write path, interrupt delivery can remain stale and
GPU initialization or fabric registration can fail.
Signed-off-by: Damian Barabonkov <dbctl@pm.me>
Refactor PciDevice::allocate_bars trait and all implementations
to take &mut SystemAllocator instead of &Arc<Mutex<SystemAllocator>>,
removing double indirection.
The caller in device_manager.rs now acquires the lock before
calling allocate_bars.
Signed-off-by: Chinmoy <daschinmoyy21@gmail.com>
OVMF can reprogram PCI BARs while memory space decoding is disabled.
Cloud Hypervisor defers the corresponding BAR move in
`pending_bar_reprogram` until the PCI command register enables Memory
Space again.
That deferred state was not part of `PciConfigurationState`. A
snapshot taken in that window restored the new BAR values in PCI
config space, but lost the pending BAR relocation needed to update the
VMM-side BAR mapping.
The restore logs show guest MMIO accesses to the reprogrammed BAR
addresses `0xc0000000`, `0x100000000`, and `0x100080000` hitting
unregistered addresses. The firmware serial output shows OVMF
assigning those same BAR addresses during PCI resource allocation,
then reaching BDS, finding the mass-storage device, and failing to
boot from it.
Serialize and restore `pending_bar_reprogram` so deferred BAR moves
survive snapshot and restore.
Co-authored-by: Thomas Prescher <thomas.prescher@cyberus-technology.de>
Co-authored-by: Julian Schindel <julian.schindel@cyberus-technology.de>
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
For VFIO devices with non page aligned MSI-X offsets,
fixup_msix_region() relocates MSI-X table and PBA offsets
into an enlarged virtual BAR by mutating msix.cap in place.
generate_sparse_areas() later reads those relocated offsets
to carve mmap holes, but receives the physical BAR size as
region_size. The relocated offsets exceed the physical BAR
boundary, and the kernel rejects the mmap with EINVAL.
Guard inter_ranges insertion with an offset < region_size
check so relocated entries are skipped. The full physical
BAR is mmapped as a single region. The relocated MSI-X in
the upper half of the virtual BAR remains trapped because
it has no mmap backing.
Linux kernel commit a32295c612c5 ("vfio-pci: Allow mapping
MSIX BAR") allows mmapping the entire BAR including the
MSI-X region when VFIO_REGION_INFO_CAP_MSIX_MAPPABLE is
advertised. The actual security guarantees come from IOMMU
isolation and interrupt remapping, not from filtering
MSI-X table accesses. QEMU follows the same pattern,
mmapping the entire physical BAR when MsixMappable is
present.
Fixes: #7898
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Following the `VfioContainer` to `VfioOps` trait switch, update the
remaining field names, method names, comments, and log messages to use
`vfio_ops` and "host IOMMU address space" consistently.
No fucntional changes.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Replace the concrete `VfioContainer` type with the `VfioOps` trait
object for device passthrough. This decouples the VFIO DMA mapping
interface from the legacy VFIO container/group implementation, allowing
it to be extended to support VFIO cdev and iommufd in the future.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Bump to the released versions that are compatible wherever possible but
for the vhost and vfio crates they are git hashes as no releases with
compatible versions have yet been made.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The virtio vhost-user device backend prefers to use externally-provided
eventfds as irqfds. This allows the frontend VM to notify the backend
VM directly, without the need for a userspace proxy process. Since the
frontend can provide irqfds at any time, the backend needs to register
and unregister irqfds dynamically.
This is tricky because the functions that access the irqfd table all
take `&self`, not `&mut self`. The obvious solution to this problem is
to wrap the table in a mutex. Most of these functions are not called on
hot paths, but `.notifier()` is called whenever Cloud Hypervisor needs
to inject an interrupt into a guest. Most devices don't need to
register irqfds at runtime, and for them, slowing down interrupt
injection would be wasteful.
Instead, require devices to opt-in to irqfd registration. The irqfd
table now comes in two forms: one that contains a mutex and one that
does not. The one containing a mutex can be mutated freely, while
attempting to mutate the one that does not will panic.
Right now, no code registeres irqfds at runtime, but this will change in
subsequent commits.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
Moves the MSRV requirement to the workspace and expands it to all
cloud-hypervisor dependencies and dev-dependencies.
This improves discoverability for new contributors working on crates
other than the cloud-hypervisor itself and creates consistency regarding
the MSRV of cloud-hypervisor dependencies.
Functionally, this doesn't change anything for dependencies of the
cloud-hypervisor crate as the MSRV requirement is already enforced by CI
when building the cloud-hypervisor with the MSRV versioned compiler.
On-behalf-of: SAP julian.schindel@sap.com
Signed-off-by: Julian Schindel <julian.schindel@cyberus-technology.de>
Change has_matching_slots() to compare two MmioRegion instances
directly rather than requiring callers to construct an intermediate
HashSet of slot numbers. Remove the now-unused
user_memory_region_slots() method and HashSet import.
Signed-off-by: Damian Barabonkov <dbctl@pm.me>
This is a follow-up of [0].
# Advantages
- This saves dozens of unneeded clone()s across the whole code base
- Makes it much easier to reason about how parameters are used
(often we passed owned Arc/Rc versions without actually needing
ownership)
# Exceptions
For certain code paths, the alternatives would require awkward or overly
complex code, and in some cases the functions are the logical owners of
the values they take. In those cases, I've added
#[allow(clippy::needless_pass_by_value)].
This does not mean that one should not improve this in the future.
[0] 6a86c157af
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
This helps to uncover expensive and needless clones in the code base.
For example, I prevented extensive clones in the snapshot path where
(nested) BTreeMap's have been cloned over and over again. Further,
the lint helps devs to much better reason about the ownership of
parameters.
All of these changes have been done manually with the necessary
caution. A few structs that are cheap to clone are now `copy` so that
this lint won't trigger for them.
I didn't enable the lint so far as it is a massive rabbit hole and
needs much more fixes. Nevertheless, it is very useful.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
The kernel will validate that the size is page-aligned. The file offset
is always zero, so the kernel will also validate that the offset is
page-aligned.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
This API passes a u64 to a kernel API that treats the u64 as a userspace
address. Therefore, it should be marked unsafe, but it currently is not
[1]. Wrap the call in an unsafe block to document that invariants must
be upheld to avoid undefined behavior. This causes a compiler warning,
so suppress the warning with #[allow(unused_unsafe)].
[1]: https://github.com/rust-vmm/vfio/issues/100
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
