diff --git a/docs/api.md b/docs/api.md index 8f7a10642..cea3f3181 100644 --- a/docs/api.md +++ b/docs/api.md @@ -72,36 +72,37 @@ The Cloud Hypervisor API exposes the following actions through its endpoints: ##### Virtual Machine (VM) Actions | Action | Endpoint | Request Body | Response Body | Prerequisites | -| ---------------------------------- | ----------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------ | -| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | -| Delete the VM | `/vm.delete` | N/A | N/A | N/A | -| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | -| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | -| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | -| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | -| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | -| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | -| Take a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | -| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | -| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | -| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | -| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | -| Resize a disk attached to the VM | `/vm.resize-disk` | `/schemas/VmResizeDisk` | N/A | The VM is created | -| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | -| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | -| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | -| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | -| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | -| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | -| Inject an NMI | `/vm.nmi` | N/A | N/A | The VM is booted | -| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | -| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | +| --------------------------------------- | ---------------------------- | --------------------------------- | ------------------------ | ------------------------------------------------------ | +| Create the VM | `/vm.create` | `/schemas/VmConfig` | N/A | The VM is not created yet | +| Delete the VM | `/vm.delete` | N/A | N/A | N/A | +| Boot the VM | `/vm.boot` | N/A | N/A | The VM is created but not booted | +| Shut the VM down | `/vm.shutdown` | N/A | N/A | The VM is booted | +| Reboot the VM | `/vm.reboot` | N/A | N/A | The VM is booted | +| Trigger power button of the VM | `/vm.power-button` | N/A | N/A | The VM is booted | +| Pause the VM | `/vm.pause` | N/A | N/A | The VM is booted | +| Resume the VM | `/vm.resume` | N/A | N/A | The VM is paused | +| Take a snapshot of the VM | `/vm.snapshot` | `/schemas/VmSnapshotConfig` | N/A | The VM is paused | +| Perform a coredump of the VM* | `/vm.coredump` | `/schemas/VmCoredumpData` | N/A | The VM is paused | +| Restore the VM from a snapshot | `/vm.restore` | `/schemas/RestoreConfig` | N/A | The VM is created but not booted | +| Add/remove CPUs to/from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | +| Add/remove memory from the VM | `/vm.resize` | `/schemas/VmResize` | N/A | The VM is booted | +| Resize a disk attached to the VM | `/vm.resize-disk` | `/schemas/VmResizeDisk` | N/A | The VM is created | +| Add/remove memory from a zone | `/vm.resize-zone` | `/schemas/VmResizeZone` | N/A | The VM is booted | +| Dump the VM information | `/vm.info` | N/A | `/schemas/VmInfo` | The VM is created | +| Add VFIO PCI device to the VM | `/vm.add-device` | `/schemas/VmAddDevice` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add disk device to the VM | `/vm.add-disk` | `/schemas/DiskConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add fs device to the VM | `/vm.add-fs` | `/schemas/FsConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add generic vhost-user device to the VM | `/vm.add-generic-vhost-user` | `/schemas/GenericVhostUserConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add pmem device to the VM | `/vm.add-pmem` | `/schemas/PmemConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add network device to the VM | `/vm.add-net` | `/schemas/NetConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add userspace PCI device to the VM | `/vm.add-user-device` | `/schemas/VmAddUserDevice` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add vdpa device to the VM | `/vm.add-vdpa` | `/schemas/VdpaConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Add vsock device to the VM | `/vm.add-vsock` | `/schemas/VsockConfig` | `/schemas/PciDeviceInfo` | The VM is booted | +| Remove device from the VM | `/vm.remove-device` | `/schemas/VmRemoveDevice` | N/A | The VM is booted | +| Dump the VM counters | `/vm.counters` | N/A | `/schemas/VmCounters` | The VM is booted | +| Inject an NMI | `/vm.nmi` | N/A | N/A | The VM is booted | +| Prepare to receive a migration | `/vm.receive-migration` | `/schemas/ReceiveMigrationData` | N/A | N/A | +| Start to send migration to target | `/vm.send-migration` | `/schemas/SendMigrationData` | N/A | The VM is booted and (shared mem or hugepages enabled) | * The `vmcoredump` action is available exclusively for the `x86_64` architecture and can be executed only when the `guest_debug` feature is diff --git a/docs/device_model.md b/docs/device_model.md index c072dc2eb..ed4577a2c 100644 --- a/docs/device_model.md +++ b/docs/device_model.md @@ -201,6 +201,24 @@ networking device (e.g. DPDK) into the VMM as their virtio network backend. This device is always built-in, and it is enabled when `vhost_user=true` and `socket` are provided to the `--net` parameter. +### vhost-user-generic + +This is a generic vhost-user device. The main use case is to provide a +vhost-user device that Cloud Hypervisor doesn't support natively. However, +there is nothing preventing its use for devices that Cloud Hypervisor does +support. For instance, the tag of a virtio-fs device can be set on the +virtiofsd command line, whereas the built-in virtio-fs support +requires the tag to be set in Cloud Hypervisor's command line. + +If the backend negotiates the `VHOST_USER_PROTOCOL_F_CONFIG` feature, +all configuration space access will be handled by it. Otherwise, +writes will be ignored and reads will return 0xFF. Cloud Hypervisor +warns if this happens. + +This device is always built-in, and it is enabled when the +`--generic-vhost-user` flag is passed. +See [the generic vhost-user documentation](generic-vhost-user.md) for more details. + ## VFIO VFIO (Virtual Function I/O) is a kernel framework that exposes direct device diff --git a/docs/generic-vhost-user.md b/docs/generic-vhost-user.md new file mode 100644 index 000000000..6af813e28 --- /dev/null +++ b/docs/generic-vhost-user.md @@ -0,0 +1,76 @@ +# How to use generic vhost-user devices + +## What is a generic vhost-user device? + +Cloud Hypervisor deliberately does not have support for all types of virtio devices. +For instance, it does not natively support sound or media. + +However, the vhost-user protocol does not require the frontend to have separate +code for each type of vhost-user device. This allows writing a *generic* frontend +that supports almost all of them. + +Any vhost-user device that only uses supported protocol messages is +expected to work. It can (and often will) be of a type that Cloud +Hypervisor does not know about. It can even be of a type that is +not standardized. + +Virtio-GPU is known to *not* work. The version implemented in QEMU +requires `VHOST_USER_GPU_SET_SOCKET`, which is standard but will +never be implemented by Cloud Hypervisor. Other versions require +messages that have not been standardized. In the future, these +versions might be supported. + +## Examples + +virtiofsd meets these requirements if the `--tag` argument is passed. +Therefore, generic vhost-user can be used as an alternative to the built-in +virtio-fs support. See [fs.md](fs.md) for how to build the virtiofs daemon. + +To use generic vhost-user with virtiofsd, use a command line argument +similar to this: + +```bash +/path/to/virtiofsd \ + --tag=myfs \ + --log-level=debug \ + "--socket-path=$path_to_virtiofsd_socket" \ + "--shared-dir=$path_to_shared_directory" \ + "${other_virtiofsd_options[@]}" & + +/path/to/cloud-hypervisor \ + --cpus boot=1 \ + --memory size=1G,shared=on \ + --disk path=your-linux-image.iso \ + --kernel vmlinux \ + --cmdline "console=hvc0 root=/dev/vda1 rw" \ + --generic-vhost-user "socket=\"${path_to_virtiofsd_socket//\"/\"\"}\",virtio_id=26,queue_sizes=[512,512]" \ + "${other_cloud_hypervisor_options[@]}" +``` + +26 is the ID for a virtio-fs device. The IDs for other devices are defined +by the VIRTIO specification. The odd-looking variable expansion escapes +any double quotes in the socket path. It is also possible to provide +the name that is defined by the virtio specification, so `virtio_id=fs` +will also work. + +Inside the guest, you can mount the virtio-fs device with + +```bash +mkdir mount_dir +mount -t virtiofs -- myfs mount_dir/ +``` + +## Limitations + +Cloud Hypervisor does not save, restore, or migrate the PCI configuration +space of a generic vhost-user device. The backend can do it itself, but if +it does not these features will not work. + +Cloud Hypervisor cannot validate the number or size of the queues. Some +guest drivers do not validate these and will crash if they are wrong. +Notably, at least some versions of Linux will crash if one creates a +virtio-fs device (id 26) with only one queue. + +If any access to configuration space fails, Cloud Hypervisor will panic +instead of injecting an exception into the guest. It is unclear what +correct behavior is in this case.