Switch NetConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Instrument the two main downtime-phase operations on the destination
side - receiving state and resuming the VM - so their costs are visible
in logs and can be iterated on.
The new log messages may look like this:
```text
cloud-hypervisor: 7.283424s: <vmm> DEBUG:vmm/src/lib.rs:948 -- Migration (incoming): recv_snapshot:3ms restore:10ms
cloud-hypervisor: 7.284824s: <vmm> DEBUG:vmm/src/lib.rs:967 -- Migration (incoming): resume:1ms
cloud-hypervisor: 7.284842s: <vmm> DEBUG:vmm/src/lib.rs:977 -- Migration (incoming): Receiving final state and resuming the VM took 15ms
```
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Use OngoingMigrationContext to measure and log the effective VM downtime
(pause to remote resume) and the cost of each non-trivial step in the
downtime window: snapshotting, sending the snapshot, and awaiting
completion. This makes it straightforward to identify and reduce
downtime as live migration matures.
Example:
```
cloud-hypervisor: 7.703402s: <vmm> INFO:vmm/src/lib.rs:1494 -- Migration completed after 2.2s with a downtime of 298ms (goal was 300ms)
cloud-hypervisor: 7.703453s: <vmm> DEBUG:vmm/src/lib.rs:1500 -- Downtime breakdown: 298ms (final_iter:269ms state:7ms send_state:19ms complete:1ms)
```
Note: downtime is measured on the source only; cross-host clock skew
may cause unreliable results.
# Terminology
At first glance, the use of "state" and "[VM] snapshot" may seem
confusing. As discussed in [0], we use "state" consistently in the
migration code. On the VM side, "snapshotting" is merely the mechanism
used to obtain the VM state.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7979#discussion_r3061359899
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add a small helper that returns both the successful result of an
operation and the time it took to complete.
Subsequent migration instrumentation uses this to keep timing code
compact and consistent.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
It is desirable to be able to track the progress of memory hotplug.
Update the memory_actual_size field to query the current plugged size
from virtio-mem to enable this.
Signed-off-by: Jared White <git@jaredwhite.dev>
That way we avoid having to grab a lock when receiving a chunk of memory
over the migration socket. This is a necessary prerequisite for having
multiple memory receiving threads.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
The memory manager is guarded by a mutex, thus parallel accesses to it
and its members are not possible. But we have to execute this function
in parallel when we introduce multiple TCP connections. Otherwise, the
workers who receive the data and write it into guest memory will block
on each other, and thus slow down the migration.
Also rename the function to receive_memory_ranges for better naming
consistency.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
The actual size of the balloon is taken directly from the guest. A
misbehaving guest can set it to an arbitrary value and cause underflow
on the next vm.info call. Use a saturation_sub instead to avoid a panic
in a debug build or a crazy number in a release build.
Signed-off-by: Dylan Reid <dgreid@dylanreid.com>
Wire the new `downtime_ms`, `timeout_s`, and `timeout_strategy` fields
from `VmSendMigrationData` into the precopy loop, replacing the previous
hard-coded 5-iteration cap.
Each iteration now evaluates three convergence criteria in order:
- no dirty pages remain;
- the estimated final-iteration downtime is within the configured budget
- or the overall migration timeout has elapsed.
On timeout, `TimeoutStrategy::Cancel` aborts and keeps the VM live on
the source, while `TimeoutStrategy::Force` proceeds regardless of the
downtime target. The convergence callback is updated to return a Result
to propagate the cancel error cleanly up the call stack.
With the recent changes [0], it is fairly easy to implement the new
checks and operate on actual metrics.
These changes are inspired by [1] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7799
[1] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7033
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Management software needs fine-grained control over live migration to
meet QoS requirements for VM guests. Add `downtime_ms`, `timeout_s`, and
`timeout_strategy` fields to `VmSendMigrationData`, exposed via API.
This commit contains the API changes only; the VMM does not yet act on
these values. This follows in the next commit.
For the JSON API, downtime and timeout are represented as plain integers
(downtime_ms and timeout_s) to make the units explicit. Using Duration
directly would require custom (de)serialization logic, so instead the
internal raw integers are exposed as Duration via getters. This
introduces minor conversion overhead but keeps the Rust API clear and
unambiguous.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Add an option that can be used when restoring to resume the VM. This is
particularly useful when restoring the VM via the direct VMM command
line, when you might not want/have an API socket configured.
Signed-off-by: Rob Bradford <rbradford@meta.com>
When memory_restore_mode=ondemand is specified on the restore command,
the memory manager creates a userfaultfd descriptor, registers each
guest RAM range for missing-page fault interception, and spawns a
handler thread that serves page faults from the snapshot file using
UFFDIO_COPY. This avoids reading the entire memory-ranges file into
guest RAM before restore completes.
The handler uses epoll to multiplex the userfaultfd and a stop eventfd
for clean shutdown. Concurrent faults from multiple vCPUs are handled
by treating EEXIST as a benign race and waking blocked threads with
UFFDIO_WAKE. Once all pages have been served the handler exits
automatically. If the handler thread panics the VMM is signalled to
exit since the VM cannot continue without page fault service.
MemoryZone gains a backing_page_size field so the handler resolves
fault granularity from the zone rather than the top-level config.
Errors from the UFFD setup path use a structured UffdError enum
and a new MigratableError::OnDemandRestore variant, with a From
impl to keep call sites concise.
The seccomp filter is updated to allow the userfaultfd syscall and
the four uffd ioctls (UFFDIO_API, UFFDIO_COPY, UFFDIO_REGISTER,
UFFDIO_WAKE) under the VMM thread profile.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Emit a "vm.migration-memory-iteration" event after every precopy memory
iteration to allow management software to observe forward progress
during migration.
This event is primarily intended for integration with management
software such as libvirt, where it maps to
VIR_DOMAIN_EVENT_ID_MIGRATION_ITERATION.
The event is intentionally independent of any upcoming migration
metrics endpoint. Detailed migration statistics will be exposed via
that endpoint, while this event provides a lightweight progress signal
expected by external management layers.
With this event, management software can detect forward progress during
migration without being blocked on any upcoming migration metrics
endpoint.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Add infrastructure to collect metrics during precopy memory migration
iterations.
For each iteration we now track transferred bytes, duration, bandwidth,
and estimate the expected downtime based on the remaining memory of the
current iteration and measured bandwidth. These metrics are logged and
used to decide when to stop the precopy phase.
This also introduces basic termination conditions such as:
- maximum number of iterations
- reaching a target downtime
- maximum migration duration
This is the fundament for an upcoming API call to publicly export
statistics about an ongoing live migration. The changes are, however,
self-contained and helpful by themselves.
The new log now looks somewhat as in the following, providing lots of
helpful insights (especially the bandwidth and estimated downtime are
helpful). The metrics were measured with CHV build with `--release`, a
VM under heavy load (lots of memory writes), same-host TCP
migration and prefault=on:
```
cloud-hypervisor: 12.702682s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=0 total=6144MiB curr=6144MiB bw=1986.83MiB/s transfer=3.09s overhead=0ms est_downtime=0ms elapsed=3.11s avg_bw=1975.41MiB/s
cloud-hypervisor: 15.728419s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=1 total=11562MiB curr=5418MiB bw=1824.44MiB/s transfer=2.97s overhead=56ms est_downtime=2726ms elapsed=6.14s avg_bw=1884.21MiB/s
cloud-hypervisor: 18.710428s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=2 total=16980MiB curr=5418MiB bw=1854.25MiB/s transfer=2.92s overhead=59ms est_downtime=2969ms elapsed=9.12s avg_bw=1862.17MiB/s
cloud-hypervisor: 21.783699s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=3 total=22407MiB curr=5428MiB bw=1799.43MiB/s transfer=3.02s overhead=56ms est_downtime=2926ms elapsed=12.19s avg_bw=1837.92MiB/s
cloud-hypervisor: 25.785696s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=4 total=27825MiB curr=5418MiB bw=1375.53MiB/s transfer=3.94s overhead=62ms est_downtime=3010ms elapsed=16.19s avg_bw=1718.26MiB/s
cloud-hypervisor: 29.000349s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=5 total=33243MiB curr=5418MiB bw=1727.60MiB/s transfer=3.14s overhead=78ms est_downtime=3938ms elapsed=19.41s avg_bw=1712.82MiB/s
cloud-hypervisor: 32.215805s: <vmm> DEBUG:vmm/src/lib.rs:1313 -- Precopy: iter=6 total=38671MiB curr=5429MiB bw=1724.03MiB/s transfer=3.15s overhead=66ms est_downtime=3142ms elapsed=22.62s avg_bw=1709.33MiB/s
cloud-hypervisor: 32.275215s: <vmm> DEBUG:vmm/src/lib.rs:1286 -- Precopy converged: iter=7 total=38671MiB curr=5418MiB bw=1720.46MiB/s transfer=3.15s overhead=66ms est_downtime=3142ms elapsed=22.68s avg_bw=1704.85MiB/s
...
cloud-hypervisor: 33.411682s: <vmm> INFO:vmm/src/lib.rs:1365 -- Precopy complete: iter=8 total=44339MiB curr=5668MiB bw=1799.98MiB/s transfer=3.15s overhead=66ms est_downtime=3142ms elapsed=23.82s avg_bw=1861.45MiB/s
```
# Outlook
We can add user-configurable downtimes and migration downtimes next.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7033
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Refactor the precopy memory migration path into dedicated helpers that
handle the different migration phases:
- initial full memory transfer
- repeated dirty-page iterations while the VM is running
- final iteration after the VM is paused
This separates concerns in the migration code and provides the
infrastructure needed for collecting migration metrics in the following
changes.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7033
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Emit migration lifecycle events via the event monitor.
This aligns migration with other VM lifecycle operations such as boot,
pause, and resume, allowing external management software to observe
migration progress consistently.
Events emitted:
src:
vm.migration-started
vm.migration-finished
vm.migration-failed
dst:
vm.migration-receive-started
vm.migration-receive-finished
vm.migration-receive-failed
Please note that these features are independent of an upcoming new
endpoint to fetch migration statistics.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Currently, it is not possible to migrate a paused VM. It fails with
the following error:
```
[2026-03-09T14:43:42Z ERROR cloud_hypervisor] Fatal error: HttpApiClient(ServerResponse(InternalServerError, Some("[\"Error from API\",\"Error starting migration sender\",\"Failed to pause migratable component\",\"Invalid transition: InvalidStateTransition(Paused, Paused)\"]")))
Error: ch-remote exited with the following chain of errors:
0: http client error
1: Server responded with InternalServerError
2: Error from API
3: Error starting migration sender
4: Failed to pause migratable component
5: Invalid transition: InvalidStateTransition(Paused, Paused)
```
and even worse, after that, the VM is resumed on the source!
Make the behavior explicit by only allowing migration of VMs in the
Running state. This avoids unintended state transitions during
migration and clarifies the current expected semantics.
Future work could extend the migration protocol to work with paused VMs
and preserve the VM runtime state, allowing paused VMs to be migrated
without altering their state.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Move the error branch to the top and remove unnecessary nesting in
send_migration().
This change is purely mechanical and introduces no functional changes.
It simplifies the control flow and prepares the code for the following
migration-related improvements in this series.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Add a core_scheduling option to --cpus with three modes of operation.
This feature takes advantage of a kernel feature that restricts
scheduling of processes on the SMT threads on the same core. This is
useful for mitigating certain classes of side-channel attacks and has
better performance that disabling SMT on the CPU.
- vm (default): All vCPU threads share one core scheduling cookie.
They may be co-scheduled on SMT siblings while host threads are
excluded - this has minimal performance impact and can even
potentially improve performance from co-location.
- vcpu: Each vCPU gets a unique cookie preventing any two vCPUs from
sharing SMT siblings. This has the strongest isolation but at some
compromise of performance.
- off: No core scheduling applied (old behaviour).
This isolation is done by the kernel maintaining a "cookie" - threads
with the same cookie can share the same core.
In vCPU mode each vCPU thread the cookie is created when the thread
starts and each gets a unique cookie. For VM mode the first vCPU thread
(the leader) will create the cookie. All other vCPU threads started (via
hotplug or during boot) will have that cookie shared to it.
EINVAL/ENODEV from prctl is silently ignored so this works transparently
on kernels older than 5.14 that lack PR_SCHED_CORE or when SMT disabled.
Full details of this kernel feature can be found at:
https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html
This implementation was inspired by crosvm's implementation - in
particular the enable_core_scheduling() function.
This is challenging to test via integration testing but the logging of
the received cookie shows it working:
VM case:
cloud-hypervisor: 0.243102s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0x33e4c167
cloud-hypervisor: 0.243102s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x33e4c167
vCPU case:
cloud-hypervisor: 0.089356s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x13993ad6
cloud-hypervisor: 0.089380s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0xd48e86e
Signed-off-by: Rob Bradford <rbradford@meta.com>
Set HV_PARTITION_CREATION_FLAG_SMT_ENABLED_GUEST when the guest
topology has more than one thread per core. This allows the
hypervisor to schedule guest VPs correctly on SMT-enabled hosts.
Without this flag, the hypervisor schedules guest VPs incorrectly,
causing SMT unusable.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
In [0] we refactored some Arc<Mutex<T>> parameters to &Mutex<T>> to
satisfy clippy's needless_pass_by_value lint. Nevertheless, this is also
not so idiomatic, so as a follow-up, we put the responsibility to lock
objects to the caller side (only where this is not strictly needed by
the callee).
While on it, I also tried to pass vm_config directly into
pre_create_console_devices() which would clean up some code, but then
we have interleaving mutable and immutable borrows of the Vmm, which
are denied by the borrow checker.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Add an option in the CLI to enable nested support.
Adding an option --cpu nested=on|off to the CPU
argument to enable nested support for Microsoft
Hypervisor. By default it is enabled for both KVM
and MSHV.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Create HypervisorVmConfig early and pass the
struct to VM creation API in the vmm crate. Getting
rid of multiple conditional parameter.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Previously, state that we accumulated during the migration process in
the receiver was kept in `mut Option` variables or HashMaps. The
problem is that it is unclear in the code when these variables can be
safely used. It's also difficult to add new state, such as the state
we need to handle the upcoming feature for performing the migration
using multiple connections.
To solve this, I've refactored the code to use the state machine
pattern. Each state carries the data it needs to. Any state that
didn't arrive yet (memory_files, memory_manager) cannot be accessed
until we are in the proper state.
Some benefits that fall out of this:
- We now respond to all requests, even invalid ones, at least with an
error message.
- Any error handling a request will result in an error message being
sent.
- There is only a single place where responses are sent and thus it's
very hard to mess this up in the code.
- The main protocol state machine fits on a screen.
I would argue that especially the error cases are now much more
consistent. There is still a lot to be done. There is still state
transfer via self.vm and similar. In an ideal world, this would also
be carried by the state machine. I also want to see better handling of
payloads, which are still handled all over the place, but this change
is already too big. :)
Co-authored-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP julian.stecklina@sap.com
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Julian Stecklina <julian.stecklina@cyberus-technology.de>
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
... and nuke some Option<> while I was there. Given that HashMap has a
usable default and we end up passing an empty HashMap anyway, just get
rid of the Option.
On-behalf-of: SAP julian.stecklina@sap.com
Signed-off-by: Julian Stecklina <julian.stecklina@cyberus-technology.de>
This is a follow-up of [0].
# Advantages
- This saves dozens of unneeded clone()s across the whole code base
- Makes it much easier to reason about how parameters are used
(often we passed owned Arc/Rc versions without actually needing
ownership)
# Exceptions
For certain code paths, the alternatives would require awkward or overly
complex code, and in some cases the functions are the logical owners of
the values they take. In those cases, I've added
#[allow(clippy::needless_pass_by_value)].
This does not mean that one should not improve this in the future.
[0] 6a86c157af
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
This helps to uncover expensive and needless clones in the code base.
For example, I prevented extensive clones in the snapshot path where
(nested) BTreeMap's have been cloned over and over again. Further,
the lint helps devs to much better reason about the ownership of
parameters.
All of these changes have been done manually with the necessary
caution. A few structs that are cheap to clone are now `copy` so that
this lint won't trigger for them.
I didn't enable the lint so far as it is a massive rabbit hole and
needs much more fixes. Nevertheless, it is very useful.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
This is continuing the work from [0]. Some places in CHV
"misuse" Result::map_err for side-effects. inspect_err is
a better suited alternative for exactly that use-case.
Unfortunately, there is no clippy lint for this yet.
[0] f02745a7ed
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
