This commit adds support for parsing `zone_updates` from the CLI
for the live migration and restore paths.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.
We consume `zone_updates` from `VmReceiveMigrationData` and
`RestoreConfig` to remap already existing `MemoryZone`s to different
host NUMA nodes. For now, we do not support further changes, such as
altering the size of the respective `MemoryZone`s.
These changes allow to migrate a VM to a host that has the capacity to
host the same `MemoryZone`s on a different NUMA layout.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
We use this function in the commit directly following this one to
update the VM configuration. This commit also introduces a test.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
It is possible to migrate a VM to a host that might have a different
but compatible NUMA configuration. For example, it should be possible
to move a `MemoryZone` from NUMA node 2 and 3 of the sender to nodes
0 and 1 on the receiver, given sender and receiver nodes are compatible
with respect to available memory.
In such a case, we need to adjust the mapping of guest memory to host
nodes. We need this information for live migration as well as for
snapshot (when performing cold migration). We consume this information
in the following commit.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
In case the host MAC address associated with a TAP device wasn't
explicitly provided by the user, Cloud Hypervisor would get the host MAC
associated by default with this TAP device and store it through the
network config. Problem is, in the context of a snapshot/restore, that
meant the network config provided by the user was different on the
destination host compared to the source host. This was causing an issue
when Cloud Hypervisor wasn't started with CAP_NET_ADMIN permissions as
it couldn't set the host MAC address on the destination, while the
source never needed these permissions since the MAC was automatically
allocated by the kernel.
We're fixing this issue by setting the host MAC address when it's
explicitly requested by the user through the network config, and making
the host MAC immutable so that it can't be changed at runtime.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Add sparse region support to remote live migration as it will save some
of the network bandwidth and therefore speed up the migration to the
destination VM.
Instead of sending the entire guest RAM over the network, only send
ranges which have been populated during the first iteration. Pages full
of holes are simply skipped.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
The VmConfig that arrives over the migration stream carries the
source's device paths and stale FDs, none of which are usable on the
destination. A received VFIO device therefore needs fresh descriptors
supplied with the request.
VmReceiveMigrationData gains vfio_fds, pairing each device id with a
cdev FD, and iommufd_fd for the backing iommufd, both arriving over
SCM_RIGHTS. Once the received VmConfig is available, each named
device's path or FD is swapped for the received one and the iommufd is
installed before the VM is built, so the device comes up FD backed.
The request is rejected when a substituted device lacks the iommufd
backend, when an id is unknown or repeated, or when a device names no
replacement in vfio_fds. These checks run against the migrated VmConfig
once it has been received.
ch-remote gains the vfio_fds and iommufd_fd options and forwards the
descriptors over SCM_RIGHTS. The D-Bus transport cannot carry file
descriptors and drops them.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
A VFIO device restored onto a different host has a device path and
iommufd that are meaningless there, and an FD backed device cannot
serialize a live descriptor into the snapshot at all. Restoring one
therefore needs fresh descriptors supplied with the request.
RestoreConfig gains vfio_fds, pairing each device id with a cdev FD,
and iommufd_fd for the backing iommufd. Both arrive over SCM_RIGHTS on
the restore request. vm_restore swaps each named device's stale path
or FD for the received one and installs the iommufd before the VM is
built, so the device comes up FD backed.
The request is rejected when a substituted device lacks the iommufd
backend, when an id is unknown or repeated, or when an FD backed device
names no replacement.
ch-remote gains the vfio_fds and iommufd_fd options and forwards the
descriptors through the SCM_RIGHTS pool.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Wire the previously empty Migratable implementation on VfioPciDevice to
VFIO DMA logging so the pages the device writes to guest memory are
tracked during a live migration. The Vfio trait gains the DMA logging
operations with defaults that report no migration support.
Without a virtual IOMMU the device sees an identity mapping of guest
memory, so the tracked ranges are the guest memory regions with iova
equal to gpa. The requested page size is only a hint, so the device
reports back the granularity it actually applied.
A device that does not support migration, or one behind a virtual
IOMMU, is refused at migration start rather than silently skipping
dirty tracking and producing an incorrect log. Under a virtual IOMMU
the reported IOVAs are not guest addresses and the logged range set
would have to track the mapping changes a guest makes mid migration,
so that configuration is out of scope.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Replace the hard-coded KVM SNP page type constants with the
KVM_SEV_SNP_PAGE_TYPE_* values from kvm-bindings.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
KVM builds the VMSA internally, so the logic in igvm-loader passing a
VMSA page for KVM is misleading. The page is silently dropped later
in import_isolated_pages(). Only MSHV actually imports it.
Make PageTypeConfig::vmsa an Option that is None on KVM and push the
VMSA page only when it is set. As KVM no longer produces a VMSA page,
remove the now-dead skip in import_isolated_pages().
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
We split the existing `Tuple` type into two types, one that represents
a single `Tuple` and one that represents a list of tuples. This allows
us to test tuple parsing and the parsing of tuple lists via distinct
unit tests in follow-up commits.
The renaming also brings consistency by adapting the naming scheme of
the other list types defined in `option_parser`, e.g. `IntegerList`
and `StringList`.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
Use std::thread::available_parallelism() when sizing the prefault
worker threads so the number of threads scales with the host CPU
count instead of relying on unsafe sysconf.
Fixes: #8495
Signed-off-by: Pulak Kanti Bhowmick <pkbhowmick007@gmail.com>
The generic vhost-user device took its virtio device type on the
command line via the `virtio_id` parameter, but the same value is
called `device_type` in the API and the resulting config struct. This
irregularity was due to churn during the review process, `device_type`
was the intended name.
Accept `device_type` on the command line and keep `virtio_id` as a
deprecated alias that logs a warning. The alias will then be removed in
a later release.
Fixes: #8545
Assisted-by: Claude:Opus-4.8
Signed-off-by: Rob Bradford <rbradford@meta.com>
A hot-unplug leaves the PCI node in the device tree until the guest
acknowledges the ejection, but VmConfig drops the device entry
immediately.
Move the config removal into DeviceManager::remove_device() and fail
when it returns false, so a second remove-device request cannot reuse
the stale device-tree node.
Assisted-by: OpenAI:Codex-GPT-5
Signed-off-by: Rob Bradford <rbradford@meta.com>
common_thread_rules() holds the syscalls every thread needs for the
SIGSYS violation handler added in #8449: the handler prints a
diagnostic to stderr (write()) after identifying the offending
thread (gettid()).
#8449 added gettid() to common_thread_rules() but left the per-thread
copies in place (9 rule sets still list it, 2 do not). write() is
likewise listed unconditionally in every thread's rule set.
Add write() to common_thread_rules() and drop the per-thread
duplicates of both gettid() and write(). Every thread already allowed
both, so there is no change in behaviour.
write() suggested by @phip1611 in #8490.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
If there is an active on-demand restoration then reject any requests to
migrate or snapshot this VM as the memory will not be available for the
snapshot.
Signed-off-by: Rob Bradford <rbradford@meta.com>
In order to deny migration or snapshot when currently doing an on-demand
restore it is necessary to track whether the prefaulting is completed.
This is a proxy for on-demand restoring being completed as pages that
have been restored by a userfaultfd request will excluded from the
prefault set.
Fixes: #8525
Signed-off-by: Rob Bradford <rbradford@meta.com>
IGVM no longer implies the MSHV backend at the feature layer. Gate the
loader paths that use MSHV page types, MSHV SNP launch defaults, or
MSHV-specific CPUID page rewriting on the MSHV feature.
This preserves existing MSHV behavior while letting KVM SEV-SNP use the
shared IGVM loader without compiling the full MSHV backend.
Assisted-by: Copilot:GPT-5.5
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Read the host cache info from sysfs and pass through the same to the
guest via the PPTT table. This is the same as the approach taken for
FDT. Similar to that, assume that the L3 cache is always shared and the
L2 cache is unique per CPU.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Use the helpers from the acpi_tables crate to construct the PPTT. This
is in preparation for adding cache hierarchy info to the PPTT which is
simpler using the helpers.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
A Windows guest that launches nested Hyper-V (for example to run WSL2)
fails to start its hypervisor on cloud-hypervisor's HW-reduced-ACPI
FADT. hvloader's hypervisor-launch path (0x18000f01c -> 0x180015628 ->
0x180015788) registers every legacy PM register block via 0x1800158dc
and rejects any block whose GAS address is 0 with status 8
(STATUS_INVALID_DEVICE_REQUEST). hvix64 then never launches and
HypervisorPresent stays False. The HW-reduced FADT leaves those blocks
zero.
Emit valid PM1a event/control blocks (I/O ports, lengths and X_GAS) in
the FADT and reserve those ports in the I/O allocator so nothing else
claims them. The HW-reduced guest OS ignores the legacy ports; only
hvloader's ACPI validation reads them.
These blocks are only useful to a guest that itself runs an enlightened
hypervisor, so emit them only when both guest nesting and the Hyper-V
enlightenments are enabled (--cpu nested=on,kvm_hyperv=on).
Signed-off-by: doge <me@crackerben.com>
The `tpm` module in the `devices` crate is disabled on riscv64 (see
commit 0042447fb "devices: Disable tpm module for riscv64"), and every
use of `tpm` in the device manager is already gated behind
`#[cfg(not(target_arch = "riscv64"))]`. However, the import itself was
merged into the unconditional `use devices::{...}` line in commit
025e782e5 "vmm: trim qualified paths", which broke the riscv64 build:
error[E0432]: unresolved import `devices::tpm`
--> vmm/src/device_manager.rs:70:83
Split the `tpm` import out into its own line gated with
`#[cfg(not(target_arch = "riscv64"))]`, matching all of its usages.
Signed-off-by: Gauthier Jolly <contact@gjolly.fr>
Now that IntegerList can handle any width number type use that for the
queue_sizes parameter on `--generic-vhost-user`
Signed-off-by: Rob Bradford <rbradford@meta.com>
This allows its use for all integer types and will generate an error if
the value is too wide for the type.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Rob Bradford <rbradford@meta.com>
Probe VFIO_DEVICE_FEATURE_MIGRATION during VfioCommon::new() and store
the result in a new migration_flags field so later migration phases can
gate state machine transitions.
The probe runs on every instantiation, including snapshot restore,
because migration capability is a property of the host kernel and its
variant driver rather than of any saved VM state.
migration_flags() is added to the internal Vfio trait with a default
implementation that returns Ok(None), meaning not migratable.
VfioDeviceWrapper overrides it to issue the kernel ioctl, while
vfio-user devices keep the default and are always treated as non
migratable.
Allow the VFIO_DEVICE_FEATURE ioctl in the VMM seccomp filter.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Check that the MMIO accesses is 4 bytes long as otherwise it would
be possible for the guest to trigger a panic when the memory ranges base
and length are copied for fulfilling the MMIO read.
This pattern of check matches similar checks in CpuManager and
DeviceManager.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Suggested by phip1611 on #8446.
This adds the repo's first clippy.toml, carving arch out of the
absolute_paths deny from #7670. Glob imports and trait imports that
must be in scope for method-call resolution (e.g. DeviceInfoForFdt for
.irq()) are left as-is.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
We cannot reliably send Request::abandon() on every kind of failure on
the sender side, as we might be in the middle of a memory transmission.
The receiver would not reliably know what to do with that. So instead,
when the receiver cannot read from the socket, we log that the migration
sender failed, which is the only likely cause of that failure.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
When the receiver of a live migration encounters an error, it sends an
error response. The sender of the migration would then send an abandon
request and wait for a response. This abandon request is not necessary,
because the receiver already abandoned the migration due to the error it
encountered.
From now on this function will not send an abandon request to the
receiver anymore, thus it was renamed to "ok_or_error".
Also, this case was always broken, because after sending the error
response, the receiver just exits without waiting for the additional
abandon request.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
Receiving a migration happens inside the VMM thread, which blocks the
API until a migration was received. On the other hand, sending a
migration is actually just a dispatch operation. We adjust the wording
to improve clarity of the error messages.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On the receiver side, a live migration with status "aborted" does not
return an error. Thus, management software will think that the live
migration was successful (from just looking at the API response). This
is not expected behaviour.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
So far, we only have seccomp rules for the postcopy-send thread. This
commit introduces the basic plumbing to add seccomp rules also for the
migration worker (the migration coordinator) as well as the TCP workers
(both, send and receive) in the following.
To streamline code setup, all filters are created at a central place
early in the migration code. Although this means that some filters are
created without the need to do so (e.g., postcopy), this massively
simplifies code setup and error handling. This overhead is negligible.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
ReceiveAdditionalConnections got quite complicated, especially with the
many threads involved for precopy and the special-case of postcopy. We
therefore should add comprehensive documentation.
I tried to keep it short and concise - what remains provides high value
and improves the mental model of the code.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Live migration can deadlock if the guest triggers a virtio device
activation while the migration worker owns the VM.
The failure shows up when starting live migrations during boot and
firmware startup, where the guest can reset and reinitialize virtio
devices while precopy is running. In the failing case, the source log
shows a pending virtio activation that never completes:
8.115833s _virtio-pci-net_0: Needs activation; returning barrier
8.115854s vmm/src/vm.rs:464 -- Waiting for barrier
24.875452s Entering downtime phase
24.875481s stopping vcpu throttling thread
...
vCPU thread did not respond in 10ms to signal - retrying
vCPU thread did not respond in 20ms to signal - retrying
...
thread 'throttle-vcpu' (1029) panicked
...
Pause(Error signalling vCPUs: Timeout when waiting for signal
to be acknowledged)
The vCPU blocks on the activation barrier and never reaches the normal
pause checkpoint. Later, migration enters downtime and stops the vCPU
throttle thread. In the failing case, that thread is still inside a
CpuManager::pause() call, which waits for every vCPU to acknowledge
the signal. The blocked vCPU never does, so the pause times out.
Fix this by storing the DeviceManager inside VmOwnership::Migration.
This keeps just enough state on the VMM thread to drain pending virtio
activations while the migration worker owns the Vm. The barrier logic
stays unchanged. The VMM now releases the same activation barrier during
migration that it already released before migration started.
This keeps the guest from getting stuck in the activation wait and
lets the later pause succeed.
Co-authored-by: Leander Kohler <leander.kohler@cyberus-technology.de>
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
When seccomp traps a SIGSYS, print the syscall number that caused it,
the current thread id and thread name to make violations easier to
debug.
This change requires that all threads are allowed to execute the
`gettid` and the `prctl` syscalls, thus the seccomp filters have also
been adjusted.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
