Wire the new `downtime_ms`, `timeout_s`, and `timeout_strategy` fields
from `VmSendMigrationData` into the precopy loop, replacing the previous
hard-coded 5-iteration cap.
Each iteration now evaluates three convergence criteria in order:
- no dirty pages remain;
- the estimated final-iteration downtime is within the configured budget
- or the overall migration timeout has elapsed.
On timeout, `TimeoutStrategy::Cancel` aborts and keeps the VM live on
the source, while `TimeoutStrategy::Force` proceeds regardless of the
downtime target. The convergence callback is updated to return a Result
to propagate the cancel error cleanly up the call stack.
With the recent changes [0], it is fairly easy to implement the new
checks and operate on actual metrics.
These changes are inspired by [1] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7799
[1] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7033
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Management software needs fine-grained control over live migration to
meet QoS requirements for VM guests. Add `downtime_ms`, `timeout_s`, and
`timeout_strategy` fields to `VmSendMigrationData`, exposed via API.
This commit contains the API changes only; the VMM does not yet act on
these values. This follows in the next commit.
For the JSON API, downtime and timeout are represented as plain integers
(downtime_ms and timeout_s) to make the units explicit. Using Duration
directly would require custom (de)serialization logic, so instead the
internal raw integers are exposed as Duration via getters. This
introduces minor conversion overhead but keeps the Rust API clear and
unambiguous.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
This change prepares upcoming options (following commit) that are added
to VmSendMigrationData.
VmSendMigrationData is a special case as it is currently the only
"rich configuration" type that lives outside `config.rs`, as it is
purely API-facing. Therefore, it isn't integrated into the existing
OptionParser infrastructure. We therefore introduce a `parse()` method
to use that in `ch-remote` in the following.
In `ch-remote`, we remove `--local` for `send-migration` and switch to
the new option string parsing constructor (breaking change!). This
prepares the addition of downtime and timeout options in the following
and streamlines the `ch-remote` command line interface with other
commands, such as `ch-remote add-net`.
Lastly, this commit updates the integration tests.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Bump to the released versions that are compatible wherever possible but
for the vhost and vfio crates they are git hashes as no releases with
compatible versions have yet been made.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This now required after Rust-VMM crate bumps. Also reorder some syscalls
so that they are now in alphabetical order.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Change Block to hold DiskBackend instead of
Box<dyn async_io::DiskFile>. In device_manager, existing formats
(raw, vhd, vhdx) are wrapped in DiskBackend::Legacy while
QcowDiskSync uses DiskBackend::Next. The fuzz target is updated
accordingly.
The Error::DiskResize variant now carries BlockError instead of
DiskFileError, matching the BlockResult return type of
DiskBackend::resize().
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Add an option that can be used when restoring to resume the VM. This is
particularly useful when restoring the VM via the direct VMM command
line, when you might not want/have an API socket configured.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The virtio vhost-user device backend prefers to use externally-provided
eventfds as irqfds. This allows the frontend VM to notify the backend
VM directly, without the need for a userspace proxy process. Since the
frontend can provide irqfds at any time, the backend needs to register
and unregister irqfds dynamically.
This is tricky because the functions that access the irqfd table all
take `&self`, not `&mut self`. The obvious solution to this problem is
to wrap the table in a mutex. Most of these functions are not called on
hot paths, but `.notifier()` is called whenever Cloud Hypervisor needs
to inject an interrupt into a guest. Most devices don't need to
register irqfds at runtime, and for them, slowing down interrupt
injection would be wasteful.
Instead, require devices to opt-in to irqfd registration. The irqfd
table now comes in two forms: one that contains a mutex and one that
does not. The one containing a mutex can be mutated freely, while
attempting to mutate the one that does not will panic.
Right now, no code registeres irqfds at runtime, but this will change in
subsequent commits.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
This allows creating an InterruptSourceGroup with an externally provided
file descriptor. It also allows changing the file descriptor
afterwards.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
The InterruptRoute code tried to be thread-safe, but it wasn't. In
particular, concurrently enabling and disabling an InterruptRoute could
result in the route thinking it was enabled (when it was disabled) or
visa versa.
Wrap all operations in a mutex and drop the attempt at being lock-free.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
When memory_restore_mode=ondemand is specified on the restore command,
the memory manager creates a userfaultfd descriptor, registers each
guest RAM range for missing-page fault interception, and spawns a
handler thread that serves page faults from the snapshot file using
UFFDIO_COPY. This avoids reading the entire memory-ranges file into
guest RAM before restore completes.
The handler uses epoll to multiplex the userfaultfd and a stop eventfd
for clean shutdown. Concurrent faults from multiple vCPUs are handled
by treating EEXIST as a benign race and waking blocked threads with
UFFDIO_WAKE. Once all pages have been served the handler exits
automatically. If the handler thread panics the VMM is signalled to
exit since the VM cannot continue without page fault service.
MemoryZone gains a backing_page_size field so the handler resolves
fault granularity from the zone rather than the top-level config.
Errors from the UFFD setup path use a structured UffdError enum
and a new MigratableError::OnDemandRestore variant, with a From
impl to keep call sites concise.
The seccomp filter is updated to allow the userfaultfd syscall and
the four uffd ioctls (UFFDIO_API, UFFDIO_COPY, UFFDIO_REGISTER,
UFFDIO_WAKE) under the VMM thread profile.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Add a MemoryRestoreMode enum (Copy | OnDemand) to RestoreConfig so
the restore path can be selected at restore time. Copy preserves the
existing eager read-copy behavior. OnDemand enables userfaultfd-based
demand paging and fails restore if the kernel does not support it.
Validate that prefault=on is not combined with OnDemand mode.
Update the OpenAPI spec with the new enum field.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Add safe Rust wrappers around the raw userfaultfd ioctls: create
(syscall + API handshake), register (missing-page mode), copy
(resolve fault), and wake (unblock threads after EEXIST race).
These are used by the demand-paged snapshot restore handler in a
subsequent commit.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Add a small constants module with the ioctl numbers and protocol
constants needed for userfaultfd-based demand-paged snapshot restore.
These are derived from the kernel's include/uapi/linux/userfaultfd.h.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Emit a "vm.migration-memory-iteration" event after every precopy memory
iteration to allow management software to observe forward progress
during migration.
This event is primarily intended for integration with management
software such as libvirt, where it maps to
VIR_DOMAIN_EVENT_ID_MIGRATION_ITERATION.
The event is intentionally independent of any upcoming migration
metrics endpoint. Detailed migration statistics will be exposed via
that endpoint, while this event provides a lightweight progress signal
expected by external management layers.
With this event, management software can detect forward progress during
migration without being blocked on any upcoming migration metrics
endpoint.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Add infrastructure to collect metrics during precopy memory migration
iterations.
For each iteration we now track transferred bytes, duration, bandwidth,
and estimate the expected downtime based on the remaining memory of the
current iteration and measured bandwidth. These metrics are logged and
used to decide when to stop the precopy phase.
This also introduces basic termination conditions such as:
- maximum number of iterations
- reaching a target downtime
- maximum migration duration
This is the fundament for an upcoming API call to publicly export
statistics about an ongoing live migration. The changes are, however,
self-contained and helpful by themselves.
The new log now looks somewhat as in the following, providing lots of
helpful insights (especially the bandwidth and estimated downtime are
helpful). The metrics were measured with CHV build with `--release`, a
VM under heavy load (lots of memory writes), same-host TCP
migration and prefault=on:
```
cloud-hypervisor: 12.702682s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=0 total=6144MiB curr=6144MiB bw=1986.83MiB/s transfer=3.09s overhead=0ms est_downtime=0ms elapsed=3.11s avg_bw=1975.41MiB/s
cloud-hypervisor: 15.728419s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=1 total=11562MiB curr=5418MiB bw=1824.44MiB/s transfer=2.97s overhead=56ms est_downtime=2726ms elapsed=6.14s avg_bw=1884.21MiB/s
cloud-hypervisor: 18.710428s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=2 total=16980MiB curr=5418MiB bw=1854.25MiB/s transfer=2.92s overhead=59ms est_downtime=2969ms elapsed=9.12s avg_bw=1862.17MiB/s
cloud-hypervisor: 21.783699s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=3 total=22407MiB curr=5428MiB bw=1799.43MiB/s transfer=3.02s overhead=56ms est_downtime=2926ms elapsed=12.19s avg_bw=1837.92MiB/s
cloud-hypervisor: 25.785696s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=4 total=27825MiB curr=5418MiB bw=1375.53MiB/s transfer=3.94s overhead=62ms est_downtime=3010ms elapsed=16.19s avg_bw=1718.26MiB/s
cloud-hypervisor: 29.000349s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=5 total=33243MiB curr=5418MiB bw=1727.60MiB/s transfer=3.14s overhead=78ms est_downtime=3938ms elapsed=19.41s avg_bw=1712.82MiB/s
cloud-hypervisor: 32.215805s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=6 total=38671MiB curr=5429MiB bw=1724.03MiB/s transfer=3.15s overhead=66ms est_downtime=3142ms elapsed=22.62s avg_bw=1709.33MiB/s
cloud-hypervisor: 32.275215s: <vmm> DEBUG:vmm/src/lib.rs:1286 -- Precopy converged: iter=7 total=38671MiB curr=5418MiB bw=1720.46MiB/s transfer=3.15s overhead=66ms est_downtime=3142ms elapsed=22.68s avg_bw=1704.85MiB/s
...
cloud-hypervisor: 33.411682s: <vmm> INFO:vmm/src/lib.rs:1365 -- Precopy complete: iter=8 total=44339MiB curr=5668MiB bw=1799.98MiB/s transfer=3.15s overhead=66ms est_downtime=3142ms elapsed=23.82s avg_bw=1861.45MiB/s
```
# Outlook
We can add user-configurable downtimes and migration downtimes next.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7033
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Refactor the precopy memory migration path into dedicated helpers that
handle the different migration phases:
- initial full memory transfer
- repeated dirty-page iterations while the VM is running
- final iteration after the VM is paused
This separates concerns in the migration code and provides the
infrastructure needed for collecting migration metrics in the following
changes.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7033
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Lower several informational messages in the dirty logging path to
debug level.
These messages are noisy in practice and provide little value since
dirty logging is known to work reliably. More useful migration metrics
(e.g., dirty size per iteration) is logged per iteration in subsequent
commits.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Emit migration lifecycle events via the event monitor.
This aligns migration with other VM lifecycle operations such as boot,
pause, and resume, allowing external management software to observe
migration progress consistently.
Events emitted:
src:
vm.migration-started
vm.migration-finished
vm.migration-failed
dst:
vm.migration-receive-started
vm.migration-receive-finished
vm.migration-receive-failed
Please note that these features are independent of an upcoming new
endpoint to fetch migration statistics.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Currently, it is not possible to migrate a paused VM. It fails with
the following error:
```
[2026-03-09T14:43:42Z ERROR cloud_hypervisor] Fatal error: HttpApiClient(ServerResponse(InternalServerError, Some("[\"Error from API\",\"Error starting migration sender\",\"Failed to pause migratable component\",\"Invalid transition: InvalidStateTransition(Paused, Paused)\"]")))
Error: ch-remote exited with the following chain of errors:
0: http client error
1: Server responded with InternalServerError
2: Error from API
3: Error starting migration sender
4: Failed to pause migratable component
5: Invalid transition: InvalidStateTransition(Paused, Paused)
```
and even worse, after that, the VM is resumed on the source!
Make the behavior explicit by only allowing migration of VMs in the
Running state. This avoids unintended state transitions during
migration and clarifies the current expected semantics.
Future work could extend the migration protocol to work with paused VMs
and preserve the VM runtime state, allowing paused VMs to be migrated
without altering their state.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Move the error branch to the top and remove unnecessary nesting in
send_migration().
This change is purely mechanical and introduces no functional changes.
It simplifies the control flow and prepares the code for the following
migration-related improvements in this series.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Moves the MSRV requirement to the workspace and expands it to all
cloud-hypervisor dependencies and dev-dependencies.
This improves discoverability for new contributors working on crates
other than the cloud-hypervisor itself and creates consistency regarding
the MSRV of cloud-hypervisor dependencies.
Functionally, this doesn't change anything for dependencies of the
cloud-hypervisor crate as the MSRV requirement is already enforced by CI
when building the cloud-hypervisor with the MSRV versioned compiler.
On-behalf-of: SAP julian.schindel@sap.com
Signed-off-by: Julian Schindel <julian.schindel@cyberus-technology.de>
Remove the architecture check that prevented nested virtualization
control on arm64 and riscv64. This allows nested virtualization to
be disabled where supported, particularly when using MSHV.
Note that on arm64 disabling nested virtualization may not fully
disable the capability depending on the underlying platform.
Use of this functionality is left to the user's discretion.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add a small helper in the block crate that opens a disk image
file and wraps any failure in a BlockError carrying the file
path and operation context. Use it from the vmm device manager
so that a failed open now reports which path couldn't be opened.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Convert detect_image_type() from io::Result to BlockResult so
that I/O failures carry the operation name in the error context.
Update the corresponding vmm error variant to wrap BlockError.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Change QcowDiskSync::new() to return BlockResult instead of
qcow::Result, mapping format specific errors to the appropriate
BlockErrorKind at the crate boundary. The vmm caller attaches
the disk image path to the error so failures identify which
file was being opened.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
This makes it clearer that these are enums that can be reused and also
helps generation by providing names for those types.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add a per-disk lock_granularity parameter that lets users choose
between byte-range OFD locks and whole-file OFD locks:
--disk path=/foo.img,lock_granularity=byte-range
--disk path=/bar.img,lock_granularity=full
Byte-range is the default and matches QEMU behavior, working
best with storage backends where whole-file OFD locks are treated
as mandatory. The full option restores the original whole-file
locking for environments that depend on it.
The LockGranularityChoice enum and its FromStr impl live in the
block crate alongside the existing LockGranularity type. The
Block device converts the user-facing choice to the internal
LockGranularity at lock time, keeping device_manager.rs simple.
Closes: #7553
Signed-off-by: Victor Vieux <vieux@repl.it>
It makes most sense to create the epoll FD and add the kill event before
identifying the transport specific fd.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This is not just used for determine the output but also the input to the
console where this can be bidirectional.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The variant has been unused since commit 12e20effd which
replaced direct QcowFile creation with QcowDiskSync.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Avoid a potential infinite loop where if the leader fails to create a
cookie due to an unexpected error (not one of the SMT/no kernel support
errors) then the other vcpu threads will continue around their
spinloops.
This change also clarifies the state machine for the leader election
with an explicit enum.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The SYNTAX help string for --net was missing a comma between
pci_segment and offload_tso parameters, making the help output
show them as a single run-on token.
Signed-off-by: Victor Vieux <vieux@repl.it>
Add a core_scheduling option to --cpus with three modes of operation.
This feature takes advantage of a kernel feature that restricts
scheduling of processes on the SMT threads on the same core. This is
useful for mitigating certain classes of side-channel attacks and has
better performance that disabling SMT on the CPU.
- vm (default): All vCPU threads share one core scheduling cookie.
They may be co-scheduled on SMT siblings while host threads are
excluded - this has minimal performance impact and can even
potentially improve performance from co-location.
- vcpu: Each vCPU gets a unique cookie preventing any two vCPUs from
sharing SMT siblings. This has the strongest isolation but at some
compromise of performance.
- off: No core scheduling applied (old behaviour).
This isolation is done by the kernel maintaining a "cookie" - threads
with the same cookie can share the same core.
In vCPU mode each vCPU thread the cookie is created when the thread
starts and each gets a unique cookie. For VM mode the first vCPU thread
(the leader) will create the cookie. All other vCPU threads started (via
hotplug or during boot) will have that cookie shared to it.
EINVAL/ENODEV from prctl is silently ignored so this works transparently
on kernels older than 5.14 that lack PR_SCHED_CORE or when SMT disabled.
Full details of this kernel feature can be found at:
https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html
This implementation was inspired by crosvm's implementation - in
particular the enable_core_scheduling() function.
This is challenging to test via integration testing but the logging of
the received cookie shows it working:
VM case:
cloud-hypervisor: 0.243102s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0x33e4c167
cloud-hypervisor: 0.243102s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x33e4c167
vCPU case:
cloud-hypervisor: 0.089356s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x13993ad6
cloud-hypervisor: 0.089380s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0xd48e86e
Signed-off-by: Rob Bradford <rbradford@meta.com>
