diff --git a/docs/snapshot_restore.md b/docs/snapshot_restore.md index d23d3f537..4932945c7 100644 --- a/docs/snapshot_restore.md +++ b/docs/snapshot_restore.md @@ -139,9 +139,13 @@ In the example above, the net device with id `net1` will be backed by FDs '23' and '24', and the net device with id `net2` will be backed by FDs '25' and '26' from the restored VM. -## Limitations +## VFIO devices -VFIO devices is out of scope. +Snapshot and restore are supported for VFIO devices that implement the kernel +VFIO migration v2 protocol (e.g. Mellanox NICs bound to the `mlx5_vfio_pci` +driver). + +See [`vfio.md`](vfio.md) for details on requirements and behavior. ## Offload Snapshot and Restore diff --git a/docs/vfio.md b/docs/vfio.md index 3cab21b36..6fb8bdb26 100644 --- a/docs/vfio.md +++ b/docs/vfio.md @@ -194,3 +194,47 @@ mmio address space is available for use by devices on PCI segment 1. --pci-segment pci_segment=0,mmio32_aperture_weight=2 --pci-segment pci_segment=1,mmio32_aperture_weight=1 ``` + +## Snapshot and Restore of VFIO Devices + +Cloud Hypervisor supports snapshotting and restoring VFIO devices that advertise +the kernel VFIO migration v2 protocol. Live migration of VFIO devices, where the +guest keeps running while its state transfers, is a separate effort not covered +by this section. + +### Requirements + +- **Kernel.** Linux 5.18 or newer for the migration v2 ioctls. 6.0 or newer is + recommended for the broadest set of state transitions. +- **Driver.** The device must be bound to a VFIO variant driver that implements + migration v2, for example `mlx5_vfio_pci` on Mellanox NICs. The generic + `vfio_pci` driver does not implement migration and is treated as non + migratable. + +### Behavior + +At device creation time, Cloud Hypervisor probes the VFIO device for migration +support via `VFIO_DEVICE_FEATURE_MIGRATION`. A device that advertises migration +v2 with at least STOP_COPY is driven through the full state machine on snapshot +and restore. + +Pausing the guest transitions the device to STOP. Snapshot then walks STOP_COPY, +captures the opaque device state, and returns the device to STOP. On restore, the +saved device blob is written through RESUMING in a single transition and the +kernel handles the intermediate STOP arc internally. After the blob loads, Cloud +Hypervisor pushes the saved PCI_COMMAND to the device and rearms MSI or MSI-X +eventfds, because the kernel's view of those is not restored by in memory state +replay alone. The device is left in RESUMING and `resume()` drives it to RUNNING +when the guest is resumed. This matches QEMU's `vfio_pci_load_config()` behavior. + +Migration support is logged at device init. + +``` +INFO vfio: VFIO device 0000:01:00.0 supports migration v2 (flags=0x7, ...) +``` + +### Limitations + +The current snapshot format stores the blob as base64 inside the snapshot +JSON. Very large blobs may benefit from a binary transport path in the +future.