This puts the send-migration action into a dedicated thread, laying the
groundwork for many follow-ups towards first-class live-migration in
CH.
This means:
1. The send-migration call will exit sooner (just trigger the
migration - dispatch semantics)
2. Other API calls can be triggered while a migration is ongoing but
will not be able to alter the VM as the VM's ownership is transferred
from the VMM to the migration thread. Example: hotplugging won't work
(which is good).
3. This is the basis for migration statistics via a dedicated endpoint
(future work).
The whole change was done with a special focus on graceful recover and
cleanup: even if anything on the migration paths go wrong, the proper
cleanups are already executed and the VMM can take back the ownership
of the VM.
The receive-migration API call remains blocking. To observe any status
changes about the migration on the sender side, one can observe the
event-monitor output and look for `vm.migration-{failed,finished}`.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7038
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
This initializes the module and the thread that will handle (control)
the migration. This introduces the new types without the necessary
wiring.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Currently, Cloud Hypervisor round-trips CNTVCT_EL0 through
KVM_GET_REG_LIST/SET_ONE_REG, which leaves a cold-restored or migrated
guest behind real UTC by the downtime. Same-host pause/resume
self-corrects (the physical counter keeps running across the pause), so
only restore and migration cases required the clock to catch up to wall
clock time.
Since ARM has no kernel helper, compute the difference in wall clock
time and compute the ticks so that it can advance the CNTVCT correctly.
It is set via vcpu0 only as it affects a single VM wide value after
Linux 6.4. For older kernels, it was a truly vcpu value which needs to
be invoked for every vcpu.
Gated on all(target_arch = "aarch64", feature = "kvm"); x86 is
unchanged.
Basic manual test case (aarch64 + KVM) verified both in intra host and
inter host snapshot save/restore:
1. Boot a Linux guest; in the guest, `date -u` tracks the host's UTC.
2. Pause and snapshot the VM (ch-remote pause; ch-remote snapshot
file:///<dir>).
3. Leave it down for several minutes (the off-host interval).
4. Restore and resume into a fresh VMM (ch-remote restore
source_url=file:///<dir>,resume=true).
5. In the guest, run `date -u` again and compare to the host: the guest
now tracks current UTC, having advanced by ~the time it spent down.
Before this change the restored guest reads behind real UTC by the
downtime; after it, the guest clock is back in sync (to within the
snapshot-to-restore sampling slop).
Signed-off-by: Atish Patra <atishp@meta.com>
Move next_data_extent and write_region_sparse out of memory_manager.rs
into a new vmm::sparse module so the snapshot writer, the restore
reader, and the offload daemon can share one implementation.
No functional change intended.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Expose VmMigrationConfig as a public facing structure that can be used
by an offload daemon to act as if it was the VM to migrate to, or the VM
to migrate from.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Removal of absolute paths is currently in progress. To avoid regressing
those changes add a clippy deny at the workspace level and at the crate
level override with #[expect(clippy::absolute_paths)]
See: #7670
Signed-off-by: Rob Bradford <rbradford@meta.com>
Introducing a new enum that models the various states of VM ownership
from the perspective of the VMM.
This is an important prerequisite for the asynchronization of the
migration, where the ownership of the Vm struct is transferred to the
migration thread. Specifically, this allows to introduces a new
"Migration(ThreadHandle)" variant and all existing match statements
can be easily extended to react accordingly.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Since igvm is a required feature of sev_snp and also sev_snp is x86-64
only the cfg attributes at build time can be consolidated & simplified.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This test has a copy and paste error where the PCI segment ID was being
set with no extra segments configured.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Teach the migration transport to handle TLS-backed streams alongside
plain TCP and UNIX sockets.
Introduce a Tls variant in SocketStream and implement the necessary
traits.
Also updates the local-migration error path to reject any non-UNIX
transport, which now includes TLS-wrapped TCP connections.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
Remove stale #[allow]s whose lints no longer fire, convert the
unconditionally-firing ones to #[expect], and keep the
arch/feature-conditional ones as #[allow]. Verified across kvm/mshv,
x86_64/aarch64, and --all-features.
Part of #8326.
Signed-off-by: Tushar Khatri <hello@tusharkhatri.in>
Validate the sender's migration protocol version when
handling the initial Start request.
Read the version from the Start command header, accept only
the supported version window n-1..=n, and reject unsupported
versions with Error. A rejected Start moves the receiver to
the aborted state.
This keeps compatibility one-way, from older protocol
versions to newer ones, and leaves later version-based
branching on the receiver side.
Log the protocol version on both sender and receiver to make
the active migration path visible.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
During migration send one final set of changed memory after capturing
the snapshot/state. This captures any memory changed as a side effect of
capturing that state. In particular with vhost-user capturing the device
state can lead to inflight requests being drained/flushed which could
change memory. As this is related to the snapshot account for this
memory transfer in the snapshot metrics.
No equivalent change is needed for snapshot as the memory is written
after the state is snapshotted.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Cloud Hypervisor maps guest RAM with MAP_NORESERVE, so the kernel never
reserves the backing pages at mmap time. On a host whose hugepage pool
cannot satisfy every guest, a VM is created successfully and then takes
a SIGBUS when the guest faults a page the pool can no longer back. This
is the failure mode reported in #5730 and #7387. As noted on #5730,
checking free pool headroom up front is not a reliable fix: another
process can consume pages between the check and the fault.
Add a reserve=on parameter to --memory and --memory-zone (default off,
preserving the current MAP_NORESERVE behaviour). When set, the region
is mapped without MAP_NORESERVE, so the kernel reserves the backing
pages (swap, or huge pages for hugepage-backed memory) at mmap time,
atomically with the mapping. An over-committed configuration then
fails cleanly at VM creation with an mmap ENOMEM instead of crashing
the guest later. Unlike prefault it does not fault the memory in, so
it does not slow down boot.
This mirrors QEMU's memory-backend reserve property, which has the same
name and meaning (reserve=off maps with MAP_NORESERVE). reserve is
threaded through the same mmap paths as the existing prefault option,
and is exposed in the OpenAPI schema, CLI help and docs. The top-level
--memory reserve=on path is unchanged: the default zone is synthesised
from MemoryConfig and inherits its reserve value.
Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Ian Klemm <hi@ianklemm.de>
When Cloud Hypervisor crashed or was killed, the API socket file was
left on disk, so the next start failed with EADDRINUSE ("Address already
in use") and the VMM could not restart. This affects any environment
where the socket directory survives across restarts (systemd services,
Kubernetes emptyDir volumes, and so on).
Before binding the path-based API socket, take an exclusive lock on a
sidecar "<socket>.lock" file using the block crate's OFD-lock helper.
Holding it proves no other instance is bound to this path, so a stale
socket left by a crashed run can be removed safely and race-free. If
the lock is already held, fail with a clear "API socket is already in
use" error instead of clobbering the live instance. The lock is held
for the process lifetime and released by the kernel on exit or crash.
The fd-based (socket-activation) path is left unchanged.
This implements the lock-file approach suggested by @DemiMarie.
Fixes: #7784
Signed-off-by: Max Makarov <maxpain@linux.com>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
The new event allows management software to handle the migration better
via events. The `vm.migration-receive-ready` event tells that the VMM is
ready to accept connections whereas `vm.migration-receive-started` means
a migration is incoming.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Since enabling AMX tile state components affect the result returned by
`Hypervisor::get_supported_cpuid` we want this enabled prior to checking
CPUID compatibility between the source and destination VMs.
Although this is not required right now, it will be necessary once we
introduce CPU profiles and it will also be necessary if we decide to
make `check_cpuid_compatibility` (arguably) more thorough by also taking
state components into account.
Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We integrate the CPU profile into the various configs that
ultimately get set by the user.
This quickly ends up involving multiple files, luckily Rust
helps us find which ones via compilation errors.
Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
This change will allow us to get accurate time over ptp in guests
started from a MSHV-virtualized Linux host. Implementing it as a
virtio device is preferable to using the existing kvm_ptp because:
kvm_ptp relies on hypercalls that only exist on host kernels running
kvm. Virtio-rtc gives us more flexibility in what clock types we want
to provide. We can later extend the device to implement multiple clocks
(smeared UTC, TAI, monotonic, etc.). Virtio-rtc protocol supports
alarms. Alarms may later enable usecases where the guests can do their
own VM lifecycle management without relying on a host-side
orchestrator.
Implement device backend for virtio-rtc. Currently this implementation
encompasses:
1. CONFIG, CAP, READ, CROSSCAP (returns false)
2. One PTP clock is presented of type
VIRTIO_RTC_CLOCK_UTC_MAYBE_SMEARED with leap_second_smearing
VIRTIO_RTC_SMEAR_UNSPECIFIED
The device is disabled by default, requiring --rtc to be passed
Not implemented but theoretically supported by virtio-rtc is:
1. Cross-timestamping support
2. The alarm queue
Fixes#7730
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Several riscv64 paths are compiled by the KVM build but missed
imports or cfg coverage needed by the current code.
Import the vm-memory Bytes trait for the RISC-V UEFI loader, keep
Instant available for migration timing code, and enable the UEFI flash
error path for riscv64.
Assisted-by: OpenAI-Codex:GPT-5
Signed-off-by: wangyf0611 <wangyufeng@iscas.ac.cn>
Only Thread::Vmm and Thread::Vcpu need to know the hypervisor type.
Make the type optional, and then simplify the users.
Assisted-by: Pi-agent:Claude-Opus-4.7
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add common PCI device configuration to the virtio-console device
configuration. This allows setting the device ID (the name), ID, the
PCI segment, and the PCI device ID (BDF), which were previously not
configurable for the virtio-console device.
This gives management software, such as libvirt, more control over PCI
resource assignment and aligns virtio-console with other devices that
already support this functionality [0].
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/issues/8175
On-behalf-of: Philipp Schuster <philipp.schuster@sap.com>
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
`--serial` still accepted `iommu=on|off` from the shared serial/console
configuration shape, even though only virtio-console has a meaningful
IOMMU connection.
Cloud Hypervisor wires virtio-iommu support through DMA-capable
virtio/VFIO PCI endpoints. The serial devices are legacy UARTs accessed
through PIO or MMIO registers, so exposing an IOMMU option there is
misleading.
On-behalf-of: Philipp Schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
This commit introduces a new struct `CommonConsoleConfig` which is the
base for the split into `ConsoleConfig` and `SerialConfig`. This is a
pre-requisite for allowing more configurable PCI options for the
virtio-console device.
The commit doesn't change or add any functionality.
On-behalf-of: Philipp Schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
This extends migration to also support paused VMs, preserving the
paused state on the destination.
Changes:
- Add CompletePaused protocol command that finalizes migration without
resuming the VM on the destination
- Skip the pause step during migration if the VM is already paused
- On migration failure, only restore the running state if
the VM was originally running (not paused)
Signed-off-by: Nguyen Dinh Phi <phind.uet@gmail.com>
This introduces the kernel hashes measured boot table into
cloud hypervisor if a cmdline and kernel is passed into an
SEV-SNP CVM, incorporating a kernel/cmdline/optional initrd
into a memory page that is measured into the launch digest
of a SEV-SNP CVM. If both --kernel and --cmdline are not
provided, we do not insert this data page
Signed-off-by: Kevin Hui <kevinhui@meta.com>
Introduce the SevFd abstraction that wraps /dev/sev and implements the
KVM_SEV_INIT2 and KVM_SEV_SNP_LAUNCH_START ioctls for SEV-SNP VM
initialization on KVM.
Key changes:
- Add sev.rs with KvmSevInit and KvmSevSnpLaunchStart ioctl structs
matching the kernel layout (linux/arch/x86/include/uapi/asm/kvm.h)
- Implement KVM_SEV_INIT2 and KVM_SEV_SNP_LAUNCH_START ioctls
- Set KVM_MEMORY_ATTRIBUTE_PRIVATE on newly created memory regions
when guest_memfd is supported
- Widen SevSnpPageAccessProxy cfg gates from mshv-only to all
sev_snp-enabled builds
- Make sev_snp_init a required trait method (remove default impl)
- Include KVM_SEV_SNP_LAUNCH_START in the seccomp allowlist
- Parse VMSA SEV features from IGVM and include them in the
KVM_SEV_INIT2 ioctl
Co-authored-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Alex Orozco <aorozco@google.com>
Co-authored-by: Rob Bradford <rbradford@meta.com>
Signed-off-by: Rob Bradford <rbradford@meta.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Move IGVM file parsing from load_igvm() into a dedicated parse_igvm()
helper in igvm/mod.rs, and parse the file upfront in Vm::new() so the
resulting IgvmFile struct is available throughout VM initialization.
This is a prerequisite for extracting VMSA SEV features from the parsed
IGVM before issuing KVM_SEV_INIT2, which needs sev_features.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Add a CLI-only --no-shutdown flag that keeps the VMM process alive
after a guest-triggered shutdown.
Management software may still need the Cloud Hypervisor process
after the guest has powered off. Exposing this separately lets
management software, for example libvirt, keep the VMM around in a
way that is closer to QEMU.
The flag only affects the GuestExit path. Fatal exits and other
existing VMM shutdown paths remain unchanged.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Introduce a dedicated guest_exit_evt and a matching epoll dispatch
path for guest-triggered shutdowns.
This series is needed because managment software such as libvirt may
still need the Cloud Hypervisor process to stay alive after the guest
has shut down.
Today a guest-triggered shutdown can make the VMM disappear immediately,
which means the managment software can lose track of the VM run-state.
This must only apply to guest-triggered shutdowns. Fatal error paths
and other internal exit paths must keep using the existing VMM exit
handling.
For now GuestExit still calls vmm_shutdown(), so this commit only adds
the separate plumbing and keeps the current behavior unchanged.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Switch NetConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Instrument the two main downtime-phase operations on the destination
side - receiving state and resuming the VM - so their costs are visible
in logs and can be iterated on.
The new log messages may look like this:
```text
cloud-hypervisor: 7.283424s: <vmm> DEBUG:vmm/src/lib.rs:948 -- Migration (incoming): recv_snapshot:3ms restore:10ms
cloud-hypervisor: 7.284824s: <vmm> DEBUG:vmm/src/lib.rs:967 -- Migration (incoming): resume:1ms
cloud-hypervisor: 7.284842s: <vmm> DEBUG:vmm/src/lib.rs:977 -- Migration (incoming): Receiving final state and resuming the VM took 15ms
```
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Use OngoingMigrationContext to measure and log the effective VM downtime
(pause to remote resume) and the cost of each non-trivial step in the
downtime window: snapshotting, sending the snapshot, and awaiting
completion. This makes it straightforward to identify and reduce
downtime as live migration matures.
Example:
```
cloud-hypervisor: 7.703402s: <vmm> INFO:vmm/src/lib.rs:1494 -- Migration completed after 2.2s with a downtime of 298ms (goal was 300ms)
cloud-hypervisor: 7.703453s: <vmm> DEBUG:vmm/src/lib.rs:1500 -- Downtime breakdown: 298ms (final_iter:269ms state:7ms send_state:19ms complete:1ms)
```
Note: downtime is measured on the source only; cross-host clock skew
may cause unreliable results.
# Terminology
At first glance, the use of "state" and "[VM] snapshot" may seem
confusing. As discussed in [0], we use "state" consistently in the
migration code. On the VM side, "snapshotting" is merely the mechanism
used to obtain the VM state.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7979#discussion_r3061359899
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add a small helper that returns both the successful result of an
operation and the time it took to complete.
Subsequent migration instrumentation uses this to keep timing code
compact and consistent.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
It is desirable to be able to track the progress of memory hotplug.
Update the memory_actual_size field to query the current plugged size
from virtio-mem to enable this.
Signed-off-by: Jared White <git@jaredwhite.dev>
That way we avoid having to grab a lock when receiving a chunk of memory
over the migration socket. This is a necessary prerequisite for having
multiple memory receiving threads.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
The memory manager is guarded by a mutex, thus parallel accesses to it
and its members are not possible. But we have to execute this function
in parallel when we introduce multiple TCP connections. Otherwise, the
workers who receive the data and write it into guest memory will block
on each other, and thus slow down the migration.
Also rename the function to receive_memory_ranges for better naming
consistency.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
