GenericVhostUserConfig flattens PciDeviceCommonConfig, so the wire
format accepts an optional id like the other PCI device configs.
FsConfig, which flattens the same struct, already documents id.
Add it here as well so generated clients can name the device and
later target it with /vm.remove-device.
Signed-off-by: leo03164 <leo03164@gmail.com>
The field is declared as generic_vhost_user in vmm/src/vm_config.rs
without a serde rename, so the wire format uses the underscore form.
The OpenAPI document names the property generic-vhost-user, which is
also the only hyphenated key in the VmConfig schema.
VmConfig deserialization does not reject unknown fields, so a client
generated from the document sends a key that serde silently ignores:
the devices are dropped on vm.create without any error.
Signed-off-by: leo03164 <leo03164@gmail.com>
The VmState enum in vmm/src/vm.rs has five variants (Created,
Running, Shutdown, Paused, BreakPoint) but the OpenAPI document
only lists four. BreakPoint is observable through vm.info when a
guest_debug build is paused by the GDB stub (Vm::debug_pause).
Clients generated with strict enum handling (e.g. via
openapi-generator) fail to deserialize the vm.info response in
that state. The document already covers guest_debug-gated surface
such as /vm.coredump, so the enum should be complete as well.
Signed-off-by: leo03164 <leo03164@gmail.com>
Defines and implements the layout of a VMDK extent, the region
of storage that is used by the virtual disk. Each line in the
descriptor file's extent section describes one extent.
Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
Map VmError::ConfigValidation in api_error_status_code() to client-error
codes: 409 Conflict for a duplicate identifier or path, 400 Bad Request
otherwise. Update the OpenAPI responses for the vm.add-* and vm.restore
endpoints, fix the InvalidIdentifier message, and add unit tests.
Signed-off-by: Stepan Rabotkin <epicstyt@gmail.com>
Assisted-by: Claude:Opus-4.8
During a live migration, connections may get interrupted silently,
without being reset or closed. Interrupted idle connections may stay
alive indefinitely, for example if a connection dies during prefaulting
on the receiver side.
To detect dead idle connections, we enable
`SO_KEEPALIVE`. With `SO_KEEPALIVE`, the kernel will send keepalive
probes if a connection is idle and close the connection if the probes
remain unacknowledged.
To detect dead connections when actively sending data in a timely
manner, we enable `TCP_USER_TIMEOUT`, to reduce the timeout for closing
a connection where the peer doesn't acknowledge sent data.
On-behalf-of: SAP julian.schindel@sap.com
Signed-off-by: Julian Schindel <julian.schindel@cyberus-technology.de>
If a device is attached to the iommu individually (not behind an iommu
backed segment) then we shouldn't allow its removal as the static VIOT
ACPI table establishes which devices (or whole segments) are behind the
IOMMU. After a removal that would be incorrect.
An alternative that was considered was to "block" the BDF from being
used for another PCI device that was hotplugged unless it was also IOMMU
attached that however feels unncessary complex when the primary purpose
of the vIOMMU backing is for nested device testing.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Migration threads may be left orphaned and keep the socket bound after
a migration has failed. Prevent this by signaling termination via the
`kill_evt` in `ReceiveAdditionalConnections`'s `Drop` impl.
On-behalf-of: SAP julian.schindel@sap.com
Signed-off-by: Julian Schindel <julian.schindel@cyberus-technology.de>
Replicate the checks that are made against the top level MemoryConfig
into those for the per-zone configuration.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The introduced `TcpStream` accessor allows setting socket options on the
`TcpStream` without going through the file descriptor and unsafe
methods.
On-behalf-of: SAP julian.schindel@sap.com
Signed-off-by: Julian Schindel <julian.schindel@cyberus-technology.de>
The ivshmem device was remapping the host file every time the BAR was
reprogrammed. This occurs on the vCPU thread and so was falling foul of
our improved seccomp filters. Avoid this by maintaining the existing
region when the device BARs are changed and just adjust the guest GPA.
Assisted-by: Codex:GPT-5.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
Unfortunately glibc can read the overcommit sysctl from any thread. This
has lead to us adding a patchwork of openat/read syscalls to our allow
list when those threads don't necessarily need openat for their actual
uses.
Only the VMM and migration worker thread have a strict requirement for
the openat syscall. The syscall was added to the other threads to deal
with this glibc behaviour.
As read() is itself harmless move it to the common syscalls, strip
full openat() from all but the threads that need it and add limited,
read only, openat to all threads.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Vcpus need to be configured according to the selected CPU profile.
We thus store the computed vCPU MSR config update in the CPU manager
and use it whenever a vCPU is created.
Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
When applying a CPU profile we need a way to change the configuration of
each vCPU to respect the requirements of the CPU profile.
This means that we need to set the feature MSRs in accordance with the
CPU profile upon configuring the vCPU and also ensuring that we do not
attempt to restore any MSRs that are not compatible with the profile
upon snapshot/restore.
The first step is to update `Vm::create_vcpu` to take an extra
parameter describing the necessary update.
In the case of KVM we modify the internal MSR state buffer when
constructing the vCPU whenever a VcpuMsrConfigUpdate is present.
The feature MSRs contained in the configuration will be treated in
follow up commits.
The changes to the vmm crate that are part of this commit are just
the minimum necessary to make the crate compile. We will update the
vmm crate to take CPU profiles into account in a follow up commit.
Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
Map NoDeviceToRemove and DeviceManager(UnknownDeviceId) to 404 Not
Found, update the OpenAPI 404 descriptions, and add unit tests.
Signed-off-by: Stepan Rabotkin <epicstyt@gmail.com>
Assisted-by: Claude:Opus-4.8
Extend the migration protocol with a `preserve_source` option so that a
source VM can be preserved. This benefits the snapshot case where the
offload daemon can now snapshot a VM without tearing it down.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
PR #8320 changed several endpoints to return 404 (Not Found) when the VM
has not been created yet, but the OpenAPI specification was not updated
to document these responses. Add the missing 404 entries for vm.info,
resize, resize-zone and the (cold) add-disk/device/pmem/net/vsock/vdpa
endpoints so the spec matches the implemented HTTP status codes.
Signed-off-by: Stepan Rabotkin <epicstyt@gmail.com>
Assisted-by: Claude:Opus-4.8
On `VM::boot` call, lock disks only after the state change has been
checked for validity. Otherwise, on invalid state change, boot is
aborted, but file locks aren't rolled back.
On-behalf-of: SAP julian.schindel@sap.com
Signed-off-by: Julian Schindel <julian.schindel@cyberus-technology.de>
This removes a need to expose a method from the CpuManager back out to
the Vm which is called immediately after creating the CpuManager.
Signed-off-by: Rob Bradford <rbradford@meta.com>
test_create_fdt_with_devices() fails with a divide by zero. The caller
passes Some((0, 0, 0, 0)) for CPU topology as (threads_per_core,
cores_per_die, dies_per_package, packages). The valid default is
(1, 1, 1, 1), not all zero. Pass None because the eventual callee,
create_cpu_nodes(), checks None to get the right default value.
This affects the unit test only. A real VM never hits it because
VmConfig::validate() rejects zero with an error.
Fixes: 7fb980f17 ("arch, vmm: Pass cpu topology configuation to FDT")
Assisted-by: Claude:Opus-4.8 (1M context)
Signed-off-by: Isaku Yamahata <iyamahata@crusoe.ai>
CPU unplug requires co-operation between the guest and the VMM. Ensure
that the CPU the guest marks for eject is the one the VMM expects to be
removed. If the guest were to remove a different vCPU there would be
unexpected behaviour. Further ensure that the boot vCPU is never
unplugged.
Signed-off-by: Rob Bradford <rbradford@meta.com>
vm-memory 0.18 has renamed GuestMemory to GuestMemoryBackend, and made
GuestMemory refer to something less specific. For simplicity, we keep
using GuestMemoryBackend (formerly GuestMemory) everywhere for now. We
can adjust bounds to be less specific later if we find ourselves needing
the newly enabled flexibility.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Signed-off-by: Rob Bradford <rbradford@meta.com>
All error cases used `ValidationError` with a String containing context
about the error. This is suboptimal as the error kind itself does not
contain any context about the actual failure cause. Furthermore, this
does not allow to build error chains as underlaying errors are replaced
with the respective message string.
We improve the situation by introducing specific error variants for
failure case. This also allows us to test the respective code paths
more easily.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
This commit adds support for parsing `zone_updates` from the CLI
for the live migration and restore paths.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.
We consume `zone_updates` from `VmReceiveMigrationData` and
`RestoreConfig` to remap already existing `MemoryZone`s to different
host NUMA nodes. For now, we do not support further changes, such as
altering the size of the respective `MemoryZone`s.
These changes allow to migrate a VM to a host that has the capacity to
host the same `MemoryZone`s on a different NUMA layout.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
We use this function in the commit directly following this one to
update the VM configuration. This commit also introduces a test.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
It is possible to migrate a VM to a host that might have a different
but compatible NUMA configuration. For example, it should be possible
to move a `MemoryZone` from NUMA node 2 and 3 of the sender to nodes
0 and 1 on the receiver, given sender and receiver nodes are compatible
with respect to available memory.
In such a case, we need to adjust the mapping of guest memory to host
nodes. We need this information for live migration as well as for
snapshot (when performing cold migration). We consume this information
in the following commit.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
In case the host MAC address associated with a TAP device wasn't
explicitly provided by the user, Cloud Hypervisor would get the host MAC
associated by default with this TAP device and store it through the
network config. Problem is, in the context of a snapshot/restore, that
meant the network config provided by the user was different on the
destination host compared to the source host. This was causing an issue
when Cloud Hypervisor wasn't started with CAP_NET_ADMIN permissions as
it couldn't set the host MAC address on the destination, while the
source never needed these permissions since the MAC was automatically
allocated by the kernel.
We're fixing this issue by setting the host MAC address when it's
explicitly requested by the user through the network config, and making
the host MAC immutable so that it can't be changed at runtime.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Add sparse region support to remote live migration as it will save some
of the network bandwidth and therefore speed up the migration to the
destination VM.
Instead of sending the entire guest RAM over the network, only send
ranges which have been populated during the first iteration. Pages full
of holes are simply skipped.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
The VmConfig that arrives over the migration stream carries the
source's device paths and stale FDs, none of which are usable on the
destination. A received VFIO device therefore needs fresh descriptors
supplied with the request.
VmReceiveMigrationData gains vfio_fds, pairing each device id with a
cdev FD, and iommufd_fd for the backing iommufd, both arriving over
SCM_RIGHTS. Once the received VmConfig is available, each named
device's path or FD is swapped for the received one and the iommufd is
installed before the VM is built, so the device comes up FD backed.
The request is rejected when a substituted device lacks the iommufd
backend, when an id is unknown or repeated, or when a device names no
replacement in vfio_fds. These checks run against the migrated VmConfig
once it has been received.
ch-remote gains the vfio_fds and iommufd_fd options and forwards the
descriptors over SCM_RIGHTS. The D-Bus transport cannot carry file
descriptors and drops them.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
A VFIO device restored onto a different host has a device path and
iommufd that are meaningless there, and an FD backed device cannot
serialize a live descriptor into the snapshot at all. Restoring one
therefore needs fresh descriptors supplied with the request.
RestoreConfig gains vfio_fds, pairing each device id with a cdev FD,
and iommufd_fd for the backing iommufd. Both arrive over SCM_RIGHTS on
the restore request. vm_restore swaps each named device's stale path
or FD for the received one and installs the iommufd before the VM is
built, so the device comes up FD backed.
The request is rejected when a substituted device lacks the iommufd
backend, when an id is unknown or repeated, or when an FD backed device
names no replacement.
ch-remote gains the vfio_fds and iommufd_fd options and forwards the
descriptors through the SCM_RIGHTS pool.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Wire the previously empty Migratable implementation on VfioPciDevice to
VFIO DMA logging so the pages the device writes to guest memory are
tracked during a live migration. The Vfio trait gains the DMA logging
operations with defaults that report no migration support.
Without a virtual IOMMU the device sees an identity mapping of guest
memory, so the tracked ranges are the guest memory regions with iova
equal to gpa. The requested page size is only a hint, so the device
reports back the granularity it actually applied.
A device that does not support migration, or one behind a virtual
IOMMU, is refused at migration start rather than silently skipping
dirty tracking and producing an incorrect log. Under a virtual IOMMU
the reported IOVAs are not guest addresses and the logged range set
would have to track the mapping changes a guest makes mid migration,
so that configuration is out of scope.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Replace the hard-coded KVM SNP page type constants with the
KVM_SEV_SNP_PAGE_TYPE_* values from kvm-bindings.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
KVM builds the VMSA internally, so the logic in igvm-loader passing a
VMSA page for KVM is misleading. The page is silently dropped later
in import_isolated_pages(). Only MSHV actually imports it.
Make PageTypeConfig::vmsa an Option that is None on KVM and push the
VMSA page only when it is set. As KVM no longer produces a VMSA page,
remove the now-dead skip in import_isolated_pages().
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
We split the existing `Tuple` type into two types, one that represents
a single `Tuple` and one that represents a list of tuples. This allows
us to test tuple parsing and the parsing of tuple lists via distinct
unit tests in follow-up commits.
The renaming also brings consistency by adapting the naming scheme of
the other list types defined in `option_parser`, e.g. `IntegerList`
and `StringList`.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
Use std::thread::available_parallelism() when sizing the prefault
worker threads so the number of threads scales with the host CPU
count instead of relying on unsafe sysconf.
Fixes: #8495
Signed-off-by: Pulak Kanti Bhowmick <pkbhowmick007@gmail.com>
The generic vhost-user device took its virtio device type on the
command line via the `virtio_id` parameter, but the same value is
called `device_type` in the API and the resulting config struct. This
irregularity was due to churn during the review process, `device_type`
was the intended name.
Accept `device_type` on the command line and keep `virtio_id` as a
deprecated alias that logs a warning. The alias will then be removed in
a later release.
Fixes: #8545
Assisted-by: Claude:Opus-4.8
Signed-off-by: Rob Bradford <rbradford@meta.com>
A hot-unplug leaves the PCI node in the device tree until the guest
acknowledges the ejection, but VmConfig drops the device entry
immediately.
Move the config removal into DeviceManager::remove_device() and fail
when it returns false, so a second remove-device request cannot reuse
the stale device-tree node.
Assisted-by: OpenAI:Codex-GPT-5
Signed-off-by: Rob Bradford <rbradford@meta.com>
common_thread_rules() holds the syscalls every thread needs for the
SIGSYS violation handler added in #8449: the handler prints a
diagnostic to stderr (write()) after identifying the offending
thread (gettid()).
#8449 added gettid() to common_thread_rules() but left the per-thread
copies in place (9 rule sets still list it, 2 do not). write() is
likewise listed unconditionally in every thread's rule set.
Add write() to common_thread_rules() and drop the per-thread
duplicates of both gettid() and write(). Every thread already allowed
both, so there is no change in behaviour.
write() suggested by @phip1611 in #8490.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
If there is an active on-demand restoration then reject any requests to
migrate or snapshot this VM as the memory will not be available for the
snapshot.
Signed-off-by: Rob Bradford <rbradford@meta.com>
In order to deny migration or snapshot when currently doing an on-demand
restore it is necessary to track whether the prefaulting is completed.
This is a proxy for on-demand restoring being completed as pages that
have been restored by a userfaultfd request will excluded from the
prefault set.
Fixes: #8525
Signed-off-by: Rob Bradford <rbradford@meta.com>
