docs: Document VFIO snapshot and restore support

Add a Snapshot and Restore section to docs/vfio.md covering the
migration v2 requirements (Linux 5.18 kernel, variant VFIO
driver such as mlx5_vfio_pci) and the restore sequence for
devices that advertise migration v2.

The behavior description covers the full restore sequence. It
documents the RUNNING to RESUMING single transition (the kernel
walks the intermediate STOP arc), the post load PCI_COMMAND push
to the device, and the MSI or MSI-X eventfd rearm that the
kernel state does not carry. Behavior matches QEMU
vfio_pci_load_config().

It notes one limitation, the snapshot format stores the opaque
device blob as base64 inside the snapshot JSON, which may benefit
from a binary transport path for very large state.

docs/snapshot_restore.md replaces its VFIO out of scope
limitation with a short section that points to docs/vfio.md for
the requirements and behavior.

Signed-off-by: Saravanan D <saravanand@crusoe.ai>

docs/snapshot_restore.md

@@ -139,9 +139,13 @@ In the example above, the net device with id `net1` will be backed by FDs '23'
 and '24', and the net device with id `net2` will be backed by FDs '25' and '26'
 from the restored VM.
 
-## Limitations
+## VFIO devices
 
-VFIO devices is out of scope.
+Snapshot and restore are supported for VFIO devices that implement the kernel
+VFIO migration v2 protocol (e.g. Mellanox NICs bound to the `mlx5_vfio_pci`
+driver).
+
+See [`vfio.md`](vfio.md) for details on requirements and behavior.
 
 ## Offload Snapshot and Restore
 

docs/vfio.md

@@ -194,3 +194,47 @@ mmio address space is available for use by devices on PCI segment 1.
 --pci-segment pci_segment=0,mmio32_aperture_weight=2
 --pci-segment pci_segment=1,mmio32_aperture_weight=1
 ```
+
+## Snapshot and Restore of VFIO Devices
+
+Cloud Hypervisor supports snapshotting and restoring VFIO devices that advertise
+the kernel VFIO migration v2 protocol. Live migration of VFIO devices, where the
+guest keeps running while its state transfers, is a separate effort not covered
+by this section.
+
+### Requirements
+
+- **Kernel.** Linux 5.18 or newer for the migration v2 ioctls. 6.0 or newer is
+  recommended for the broadest set of state transitions.
+- **Driver.** The device must be bound to a VFIO variant driver that implements
+  migration v2, for example `mlx5_vfio_pci` on Mellanox NICs. The generic
+  `vfio_pci` driver does not implement migration and is treated as non
+  migratable.
+
+### Behavior
+
+At device creation time, Cloud Hypervisor probes the VFIO device for migration
+support via `VFIO_DEVICE_FEATURE_MIGRATION`. A device that advertises migration
+v2 with at least STOP_COPY is driven through the full state machine on snapshot
+and restore.
+
+Pausing the guest transitions the device to STOP. Snapshot then walks STOP_COPY,
+captures the opaque device state, and returns the device to STOP. On restore, the
+saved device blob is written through RESUMING in a single transition and the
+kernel handles the intermediate STOP arc internally. After the blob loads, Cloud
+Hypervisor pushes the saved PCI_COMMAND to the device and rearms MSI or MSI-X
+eventfds, because the kernel's view of those is not restored by in memory state
+replay alone. The device is left in RESUMING and `resume()` drives it to RUNNING
+when the guest is resumed. This matches QEMU's `vfio_pci_load_config()` behavior.
+
+Migration support is logged at device init.
+
+```
+INFO  vfio: VFIO device 0000:01:00.0 supports migration v2 (flags=0x7, ...)
+```
+
+### Limitations
+
+The current snapshot format stores the blob as base64 inside the snapshot
+JSON. Very large blobs may benefit from a binary transport path in the
+future.