Introducing a migration mode to both sides of the migration (send and
receive), so that a user can desribe which way the memory should be
migrated between the source and destination VMs.
For now, we only introduce `precopy` and `postcopy` as viable options,
but we can expect other modes (more optimized) to be added in the
future.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
`path` is no longer required in the DeviceConfig, since a VFIO
device may also be supplied via a pre-opened cdev FD passed via
SCM_RIGHTS alongside the /vm.add-device request.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
As we now have more than one parameter for the receive migration call,
this commit also adds parsing and validation for those parameters. We
maintain backwards compatibility by also correctly parsing the case
where the caller only provides a URL.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
Cloud Hypervisor maps guest RAM with MAP_NORESERVE, so the kernel never
reserves the backing pages at mmap time. On a host whose hugepage pool
cannot satisfy every guest, a VM is created successfully and then takes
a SIGBUS when the guest faults a page the pool can no longer back. This
is the failure mode reported in #5730 and #7387. As noted on #5730,
checking free pool headroom up front is not a reliable fix: another
process can consume pages between the check and the fault.
Add a reserve=on parameter to --memory and --memory-zone (default off,
preserving the current MAP_NORESERVE behaviour). When set, the region
is mapped without MAP_NORESERVE, so the kernel reserves the backing
pages (swap, or huge pages for hugepage-backed memory) at mmap time,
atomically with the mapping. An over-committed configuration then
fails cleanly at VM creation with an mmap ENOMEM instead of crashing
the guest later. Unlike prefault it does not fault the memory in, so
it does not slow down boot.
This mirrors QEMU's memory-backend reserve property, which has the same
name and meaning (reserve=off maps with MAP_NORESERVE). reserve is
threaded through the same mmap paths as the existing prefault option,
and is exposed in the OpenAPI schema, CLI help and docs. The top-level
--memory reserve=on path is unchanged: the default zone is synthesised
from MemoryConfig and inherits its reserve value.
Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Ian Klemm <hi@ianklemm.de>
Mark serial_number/uuid as deprecated in the OpenAPI schema and emit
warnings when those legacy --platform keys are used, while continuing to
accept them for compatibility.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Extend SMBIOS System Information with manufacturer, product,
version, family, sku, serial, and uuid fields, add a chassis
asset tag, and pass a structured SMBIOS config from --platform
into arch setup. Keep OEM strings and legacy serial_number/uuid
options working for compatibility. The platform option naming
follows `dmidecode -s <field>`.
Fields:
- system_manufacturer
- system_product_name
- system_version
- system_family
- system_serial_number
- system_uuid
- chassis_asset_tag
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
The current OpenAPI schema format is wrong because uint16 and uint32 are
not valid top-level OpenAPI types. Describe queue_size and virtio_id as
integer types with explicit uint formats instead.
Signed-off-by: Roman <roman@vanesyan.com>
This change will allow us to get accurate time over ptp in guests
started from a MSHV-virtualized Linux host. Implementing it as a
virtio device is preferable to using the existing kvm_ptp because:
kvm_ptp relies on hypercalls that only exist on host kernels running
kvm. Virtio-rtc gives us more flexibility in what clock types we want
to provide. We can later extend the device to implement multiple clocks
(smeared UTC, TAI, monotonic, etc.). Virtio-rtc protocol supports
alarms. Alarms may later enable usecases where the guests can do their
own VM lifecycle management without relying on a host-side
orchestrator.
Implement device backend for virtio-rtc. Currently this implementation
encompasses:
1. CONFIG, CAP, READ, CROSSCAP (returns false)
2. One PTP clock is presented of type
VIRTIO_RTC_CLOCK_UTC_MAYBE_SMEARED with leap_second_smearing
VIRTIO_RTC_SMEAR_UNSPECIFIED
The device is disabled by default, requiring --rtc to be passed
Not implemented but theoretically supported by virtio-rtc is:
1. Cross-timestamping support
2. The alarm queue
Fixes#7730
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
The JSON parser expects the field to be named
`iommu_address_width_bits`, but the code declared it as
`iommu_address_width`. This mismatch caused the field to be
unrecognized when deserializing configuration from JSON.
Rename the field to `iommu_address_width_bits` to match the
expected schema.
Signed-off-by: tchaton <thomas.chaton.ai@gmail.com>
Allow VFIO devices to list BAR indices that should not be
mmapped into the guest. This lets operators skip large BARs that
are known not to be used by their workload.
When a BAR is skipped, the log also calls out that P2P DMA
mapping is skipped because the VFIO DMA map path uses the same
mmap backing.
Signed-off-by: Damian Barabonkov <dbctl@pm.me>
Assisted-by: OpenCode:gpt-5.5
Add a user-configurable option to control whether VFIO device MMIO BAR
regions are DMA-mapped into the host IOMMU address space.
This mapping is required for peer-to-peer DMA between devices (e.g.
NVLink, RDMA NIC accessing GPU VRAM). However, iommufd on upstream
kernels does not support mapping device MMIO pages (VM_PFNMAP), causing
IOMMU_IOAS_MAP to fail with -EFAULT. Kernels with the NVIDIA PFNMAP
workaround or future kernels with DMABUF-based mapping
(IOMMU_IOAS_MAP_FILE) handle this correctly.
The option defaults to `on` to preserve existing behavior. Users on
vanilla kernels using iommufd should set `vfio_p2p_dma=off` to skip
MMIO BAR DMA mapping.
A validation check ensures that `x_nv_gpudirect_clique` (which depends
on P2P DMA) cannot be used when `vfio_p2p_dma=off`.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
This option allows user to configure VFIO device pass-through with
iommufd (e.g. vfio cdev mode) or not (e.g. vfio legacy mode).
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Add a MemoryRestoreMode enum (Copy | OnDemand) to RestoreConfig so
the restore path can be selected at restore time. Copy preserves the
existing eager read-copy behavior. OnDemand enables userfaultfd-based
demand paging and fails restore if the kernel does not support it.
Validate that prefault=on is not combined with OnDemand mode.
Update the OpenAPI spec with the new enum field.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
This makes it clearer that these are enums that can be reused and also
helps generation by providing names for those types.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add a per-disk lock_granularity parameter that lets users choose
between byte-range OFD locks and whole-file OFD locks:
--disk path=/foo.img,lock_granularity=byte-range
--disk path=/bar.img,lock_granularity=full
Byte-range is the default and matches QEMU behavior, working
best with storage backends where whole-file OFD locks are treated
as mandatory. The full option restores the original whole-file
locking for environments that depend on it.
The LockGranularityChoice enum and its FromStr impl live in the
block crate alongside the existing LockGranularity type. The
Block device converts the user-facing choice to the internal
LockGranularity at lock time, keeping device_manager.rs simple.
Closes: #7553
Signed-off-by: Victor Vieux <vieux@repl.it>
Add a core_scheduling option to --cpus with three modes of operation.
This feature takes advantage of a kernel feature that restricts
scheduling of processes on the SMT threads on the same core. This is
useful for mitigating certain classes of side-channel attacks and has
better performance that disabling SMT on the CPU.
- vm (default): All vCPU threads share one core scheduling cookie.
They may be co-scheduled on SMT siblings while host threads are
excluded - this has minimal performance impact and can even
potentially improve performance from co-location.
- vcpu: Each vCPU gets a unique cookie preventing any two vCPUs from
sharing SMT siblings. This has the strongest isolation but at some
compromise of performance.
- off: No core scheduling applied (old behaviour).
This isolation is done by the kernel maintaining a "cookie" - threads
with the same cookie can share the same core.
In vCPU mode each vCPU thread the cookie is created when the thread
starts and each gets a unique cookie. For VM mode the first vCPU thread
(the leader) will create the cookie. All other vCPU threads started (via
hotplug or during boot) will have that cookie shared to it.
EINVAL/ENODEV from prctl is silently ignored so this works transparently
on kernels older than 5.14 that lack PR_SCHED_CORE or when SMT disabled.
Full details of this kernel feature can be found at:
https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html
This implementation was inspired by crosvm's implementation - in
particular the enable_core_scheduling() function.
This is challenging to test via integration testing but the logging of
the received cookie shows it working:
VM case:
cloud-hypervisor: 0.243102s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0x33e4c167
cloud-hypervisor: 0.243102s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x33e4c167
vCPU case:
cloud-hypervisor: 0.089356s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x13993ad6
cloud-hypervisor: 0.089380s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0xd48e86e
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add an image_type to DiskConfig to specify the image type. If none is
specified autodetect the image type but disable potentially unsafe
behaviour in the QCOW2 backend by disabling the backing file support.
If the image type is autodetected then fix it in the config so that it
will be persistant across reboots and migrations/snapshot & restores.
This also handles the case where the image type was not specified as
part of the disk configuration.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add an optional device_id string field to NumaConfig for identifying
PCI devices associated with a NUMA node. This is used by the Generic
Initiator support to map devices to their proximity domain.
Update OpenAPI spec (cloud-hypervisor.yaml) to include the
new device_id field in the NumaConfig schema.
The device_id is optional and parsed from the --numa parameter:
--numa "device_id=<device_id>,distances=[...],..."
The optional field is accepted but not used.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
This commit removes the SGX support from cloud hypervisor. SGX support
was deprecated in May as part of #7090.
Signed-off-by: Shubham Chakrawar <schakrawar@crusoe.ai>
Allow tap interfaces to be configured with an IPv6 address. The change
is fairly straightforward: we need to update the API types and CLI
parsing to accept either an IPv6 or IPv4 and then match on the IP
address type when the tap device is configured.
For IPv6 addresses, the netmask (prefix) must be provided at the same
time as the address itself (in the SIOCSIFADDR ioctl). They cannot be
configured separately. So we remove the separate "set_netmask" function
and convert "set_ip_addr" to also accept a netmask. For IPv4 addresses,
the IP address and netmask were already always set together, so this
should have no functional impact for users of IPv4 addresses.
Signed-off-by: Gregory Anders <ganders@cloudflare.com>
Although the CPU manager gives us a quite descriptive error, on the
application side (the part calling Cloud Hypervisor) we have absolutely
no way to distinguish such error from any other error that may happen
when resizing a VM.
With this in mind, let's be more specific and return a TooManyRequests
(429) error, allowing the caller to have a chance to decide whether they
want to retry the operation or not.
https://datatracker.ietf.org/doc/html/rfc6585#section-4
Signed-off-by: Fabiano Fidêncio <fidencio@northflank.com>
This aligns with our CLI syntax. The correctness of `DiskConfig` will be
ensured via `VmConfig::validate()`, e.g. `path` and `socket` are
mutually exclusive.
Fixes: #7016
Signed-off-by: Bo Chen <bchen@crusoe.ai>
This is to resolve the inconsistencies from our openapi specification,
as default values do not make sense for required fields.
Reported-by: James O. D. Hunt <james.o.hunt@intel.com>
Signed-off-by: Bo Chen <chen.bo@intel.com>
When using multiple PCI segments, the 32-bit and 64-bit mmio
aperture is split equally between each segment. Add an option
to configure the 'weight'. For example, a PCI segment with a
`mmio32_aperture_weight` of 2 will be allocated twice as much
32-bit mmio space as a normal PCI segment.
Signed-off-by: Thomas Barrett <tbarrett@crusoeenergy.com>
On platforms where PCIe P2P is supported, inject a PCI capability into
NVIDIA GPU to indicate support.
Signed-off-by: Thomas Barrett <tbarrett@crusoeenergy.com>
Currently the only way to set the affinity for virtio block threads is
to boot the VM, search for the tid of each of the virtio block threads,
then set the affinity manually. This commit adds an option to pin virtio
block queues to specific host cpus (similar to pinning vcpus to host
cpus). A queue_affinity option has been added to the disk flag in
the cli to specify a mapping of queue indices to host cpus.
Signed-off-by: acarp <acarp@crusoeenergy.com>
This commit adds the debug-console (or debugcon) device to CHV. It is a
very simple device on I/O port 0xe9 supported by QEMU and BOCHS. It is
meant for printing information as easy as possible, without any
necessary configuration from the guest at all.
It is primarily interesting to OS/kernel and firmware developers as they
can produce output as soon as the guest starts without any configuration
of a serial device or similar. Furthermore, a kernel hacker might use
this device for information of type B whereas information of type A are
printed to the serial device.
This device is not used by default by Linux, Windows, or any other
"real" OS, but only by toy kernels and during firmware development.
In the CLI, it can be configured similar to --console or --serial with
the --debug-console parameter.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
