IGVM no longer implies the MSHV backend at the feature layer. Gate the
loader paths that use MSHV page types, MSHV SNP launch defaults, or
MSHV-specific CPUID page rewriting on the MSHV feature.
This preserves existing MSHV behavior while letting KVM SEV-SNP use the
shared IGVM loader without compiling the full MSHV backend.
Assisted-by: Copilot:GPT-5.5
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Read the host cache info from sysfs and pass through the same to the
guest via the PPTT table. This is the same as the approach taken for
FDT. Similar to that, assume that the L3 cache is always shared and the
L2 cache is unique per CPU.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Use the helpers from the acpi_tables crate to construct the PPTT. This
is in preparation for adding cache hierarchy info to the PPTT which is
simpler using the helpers.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
A Windows guest that launches nested Hyper-V (for example to run WSL2)
fails to start its hypervisor on cloud-hypervisor's HW-reduced-ACPI
FADT. hvloader's hypervisor-launch path (0x18000f01c -> 0x180015628 ->
0x180015788) registers every legacy PM register block via 0x1800158dc
and rejects any block whose GAS address is 0 with status 8
(STATUS_INVALID_DEVICE_REQUEST). hvix64 then never launches and
HypervisorPresent stays False. The HW-reduced FADT leaves those blocks
zero.
Emit valid PM1a event/control blocks (I/O ports, lengths and X_GAS) in
the FADT and reserve those ports in the I/O allocator so nothing else
claims them. The HW-reduced guest OS ignores the legacy ports; only
hvloader's ACPI validation reads them.
These blocks are only useful to a guest that itself runs an enlightened
hypervisor, so emit them only when both guest nesting and the Hyper-V
enlightenments are enabled (--cpu nested=on,kvm_hyperv=on).
Signed-off-by: doge <me@crackerben.com>
The `tpm` module in the `devices` crate is disabled on riscv64 (see
commit 0042447fb "devices: Disable tpm module for riscv64"), and every
use of `tpm` in the device manager is already gated behind
`#[cfg(not(target_arch = "riscv64"))]`. However, the import itself was
merged into the unconditional `use devices::{...}` line in commit
025e782e5 "vmm: trim qualified paths", which broke the riscv64 build:
error[E0432]: unresolved import `devices::tpm`
--> vmm/src/device_manager.rs:70:83
Split the `tpm` import out into its own line gated with
`#[cfg(not(target_arch = "riscv64"))]`, matching all of its usages.
Signed-off-by: Gauthier Jolly <contact@gjolly.fr>
Now that IntegerList can handle any width number type use that for the
queue_sizes parameter on `--generic-vhost-user`
Signed-off-by: Rob Bradford <rbradford@meta.com>
This allows its use for all integer types and will generate an error if
the value is too wide for the type.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Rob Bradford <rbradford@meta.com>
Probe VFIO_DEVICE_FEATURE_MIGRATION during VfioCommon::new() and store
the result in a new migration_flags field so later migration phases can
gate state machine transitions.
The probe runs on every instantiation, including snapshot restore,
because migration capability is a property of the host kernel and its
variant driver rather than of any saved VM state.
migration_flags() is added to the internal Vfio trait with a default
implementation that returns Ok(None), meaning not migratable.
VfioDeviceWrapper overrides it to issue the kernel ioctl, while
vfio-user devices keep the default and are always treated as non
migratable.
Allow the VFIO_DEVICE_FEATURE ioctl in the VMM seccomp filter.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Check that the MMIO accesses is 4 bytes long as otherwise it would
be possible for the guest to trigger a panic when the memory ranges base
and length are copied for fulfilling the MMIO read.
This pattern of check matches similar checks in CpuManager and
DeviceManager.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Suggested by phip1611 on #8446.
This adds the repo's first clippy.toml, carving arch out of the
absolute_paths deny from #7670. Glob imports and trait imports that
must be in scope for method-call resolution (e.g. DeviceInfoForFdt for
.irq()) are left as-is.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
We cannot reliably send Request::abandon() on every kind of failure on
the sender side, as we might be in the middle of a memory transmission.
The receiver would not reliably know what to do with that. So instead,
when the receiver cannot read from the socket, we log that the migration
sender failed, which is the only likely cause of that failure.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
When the receiver of a live migration encounters an error, it sends an
error response. The sender of the migration would then send an abandon
request and wait for a response. This abandon request is not necessary,
because the receiver already abandoned the migration due to the error it
encountered.
From now on this function will not send an abandon request to the
receiver anymore, thus it was renamed to "ok_or_error".
Also, this case was always broken, because after sending the error
response, the receiver just exits without waiting for the additional
abandon request.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
Receiving a migration happens inside the VMM thread, which blocks the
API until a migration was received. On the other hand, sending a
migration is actually just a dispatch operation. We adjust the wording
to improve clarity of the error messages.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On the receiver side, a live migration with status "aborted" does not
return an error. Thus, management software will think that the live
migration was successful (from just looking at the API response). This
is not expected behaviour.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
So far, we only have seccomp rules for the postcopy-send thread. This
commit introduces the basic plumbing to add seccomp rules also for the
migration worker (the migration coordinator) as well as the TCP workers
(both, send and receive) in the following.
To streamline code setup, all filters are created at a central place
early in the migration code. Although this means that some filters are
created without the need to do so (e.g., postcopy), this massively
simplifies code setup and error handling. This overhead is negligible.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
ReceiveAdditionalConnections got quite complicated, especially with the
many threads involved for precopy and the special-case of postcopy. We
therefore should add comprehensive documentation.
I tried to keep it short and concise - what remains provides high value
and improves the mental model of the code.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Live migration can deadlock if the guest triggers a virtio device
activation while the migration worker owns the VM.
The failure shows up when starting live migrations during boot and
firmware startup, where the guest can reset and reinitialize virtio
devices while precopy is running. In the failing case, the source log
shows a pending virtio activation that never completes:
8.115833s _virtio-pci-net_0: Needs activation; returning barrier
8.115854s vmm/src/vm.rs:464 -- Waiting for barrier
24.875452s Entering downtime phase
24.875481s stopping vcpu throttling thread
...
vCPU thread did not respond in 10ms to signal - retrying
vCPU thread did not respond in 20ms to signal - retrying
...
thread 'throttle-vcpu' (1029) panicked
...
Pause(Error signalling vCPUs: Timeout when waiting for signal
to be acknowledged)
The vCPU blocks on the activation barrier and never reaches the normal
pause checkpoint. Later, migration enters downtime and stops the vCPU
throttle thread. In the failing case, that thread is still inside a
CpuManager::pause() call, which waits for every vCPU to acknowledge
the signal. The blocked vCPU never does, so the pause times out.
Fix this by storing the DeviceManager inside VmOwnership::Migration.
This keeps just enough state on the VMM thread to drain pending virtio
activations while the migration worker owns the Vm. The barrier logic
stays unchanged. The VMM now releases the same activation barrier during
migration that it already released before migration started.
This keeps the guest from getting stuck in the activation wait and
lets the later pause succeed.
Co-authored-by: Leander Kohler <leander.kohler@cyberus-technology.de>
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
When seccomp traps a SIGSYS, print the syscall number that caused it,
the current thread id and thread name to make violations easier to
debug.
This change requires that all threads are allowed to execute the
`gettid` and the `prctl` syscalls, thus the seccomp filters have also
been adjusted.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
In [0] we agreed on the current format. When ch-remote or
cloud-hypervisor exit with an error, they nicely print the whole chain.
This, however, doesn't work when simply doing `error!("error: {e}")`
- which is what we currently do for migration-related errors.
This commit walks the chain of errors and prints all components in a
single line. This massively improves the quality of error messages and
helps tracing down where an error is originating from. Using ` => ` as
separator is better than `\n` which doesn't work well in our current
log format.
# Example (Before - Bad)
```
cloud-hypervisor: 2.859287s: <vmm> ERROR:vmm/src/lib.rs:2021 -- Migration failed: Failed to send migratable component snapshot
```
# Example (New - Better)
```
cloud-hypervisor: 2.296160s: <vmm> ERROR:vmm/src/lib.rs:2038 -- Migration failed: Failed to send migratable component snapshot => Error connecting to TCP socket => Connection refused (os error 111)
```
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7066
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Keep the original error sources while adding operation context to the
straightforward migration send and receive paths. This keeps up a
chain of errors that can be printed nicely and in a helpful way.
This commit does that for all MigratableError-occurences where this
change is easily applicable.
Co-authored-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
size_of is part of std::prelude as of Rust 1.80 (with size_of_val,
align_of, align_of_val), and the workspace MSRV is 1.89, so qualifying
it (mem::size_of, std::mem::size_of, core::mem::size_of) is unnecessary.
Convert every qualified size_of call-site to the bare prelude form and
drop the now-redundant `use std::mem::size_of;` imports, keeping
`use std::mem;` where it still serves non-prelude items (transmute,
swap, replace, take, zeroed, MaybeUninit, offset_of). size_of is the
only one of the four currently used in the tree.
Pure refactor, no behavioural change. Follow-up to the
clippy::absolute_paths cleanup (#7670), as discussed in #8444.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import the modules used in the crate instead of spelling the full paths
at every use site, and drop the now-unnecessary crate-level
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
We have larger cloud deployments and analyzed the logs. Let's silence
some messages that generally provide little value on the `info!` level.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Applying seccomp filtering to the migration postcopy thread running on
the source VM during migration.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
Wire up the source side of postcopy migration over TCP. When
`mode=postcopy` is requested on vm.send-migration, the source skips
the pre-copy dirty-tracking loop and lets the destination resume early,
then serves guest pages on demand over a dedicated connection.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Plumb the SocketUffdMemorySource into the receiving side of live
migration. When memory_mode=postcopy is requested, the destination
brings up a dedicated fault connection, registers userfaultfd on the
restored memory regions, and serves guest pages on demand over that
connection while the VM resumes early.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Add the socket-backed UffdMemorySource that resolves each fault by
sending a Command::PageFault request to the peer over a dedicated fault
connection.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Introducing a migration mode to both sides of the migration (send and
receive), so that a user can desribe which way the memory should be
migrated between the source and destination VMs.
For now, we only introduce `precopy` and `postcopy` as viable options,
but we can expect other modes (more optimized) to be added in the
future.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
Extract the page content provider out of the userfaultfd handler so it
can be plugged with different backends in followup commits.
No functional change intended.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Restoring a snapshot (snapshot restore or live-migration receive) sets
each vCPU's TSC by writing MSR_IA32_TSC as the vCPU is created.
However because CpuManager creates and restores vCPUs one at a time, the
host TSC advances between the per-vCPU writes and KVM derives a slightly
different TSC offset for each vCPU.
KVM only engages its masterclock when every offset matches. This has a
side effect of breaking the HyperV TSC reference clock page resulting in
significantly reduced performance on Windows.
After restore synchronise all vCPU's TSC offset to the boot vCPU's via
the KVM_VCPU_TSC_CTRL device attribute group (Linux 5.16+) this allows
the KVM TSC masterclock to engage and mitigates performance issues with
the KVM HyperV emulation.
See: #8383
Signed-off-by: Rob Bradford <rbradford@meta.com>
Assisted-by: Claude <claude-opus-4-8>
This puts the send-migration action into a dedicated thread, laying the
groundwork for many follow-ups towards first-class live-migration in
CH.
This means:
1. The send-migration call will exit sooner (just trigger the
migration - dispatch semantics)
2. Other API calls can be triggered while a migration is ongoing but
will not be able to alter the VM as the VM's ownership is transferred
from the VMM to the migration thread. Example: hotplugging won't work
(which is good).
3. This is the basis for migration statistics via a dedicated endpoint
(future work).
The whole change was done with a special focus on graceful recover and
cleanup: even if anything on the migration paths go wrong, the proper
cleanups are already executed and the VMM can take back the ownership
of the VM.
The receive-migration API call remains blocking. To observe any status
changes about the migration on the sender side, one can observe the
event-monitor output and look for `vm.migration-{failed,finished}`.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7038
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
This initializes the module and the thread that will handle (control)
the migration. This introduces the new types without the necessary
wiring.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
This was already handled with user memory zones but not with the default
memory. Make a small refactoring to move the boot RAM check into
MemoryManager rather than split across Vm and MemoryManager.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Currently, Cloud Hypervisor round-trips CNTVCT_EL0 through
KVM_GET_REG_LIST/SET_ONE_REG, which leaves a cold-restored or migrated
guest behind real UTC by the downtime. Same-host pause/resume
self-corrects (the physical counter keeps running across the pause), so
only restore and migration cases required the clock to catch up to wall
clock time.
Since ARM has no kernel helper, compute the difference in wall clock
time and compute the ticks so that it can advance the CNTVCT correctly.
It is set via vcpu0 only as it affects a single VM wide value after
Linux 6.4. For older kernels, it was a truly vcpu value which needs to
be invoked for every vcpu.
Gated on all(target_arch = "aarch64", feature = "kvm"); x86 is
unchanged.
Basic manual test case (aarch64 + KVM) verified both in intra host and
inter host snapshot save/restore:
1. Boot a Linux guest; in the guest, `date -u` tracks the host's UTC.
2. Pause and snapshot the VM (ch-remote pause; ch-remote snapshot
file:///<dir>).
3. Leave it down for several minutes (the off-host interval).
4. Restore and resume into a fresh VMM (ch-remote restore
source_url=file:///<dir>,resume=true).
5. In the guest, run `date -u` again and compare to the host: the guest
now tracks current UTC, having advanced by ~the time it spent down.
Before this change the restored guest reads behind real UTC by the
downtime; after it, the guest clock is back in sync (to within the
snapshot-to-restore sampling slop).
Signed-off-by: Atish Patra <atishp@meta.com>
