Extract the page content provider out of the userfaultfd handler so it
can be plugged with different backends in followup commits.
No functional change intended.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Introducing PageFault as the new wire command needed by both postcopy
live migration and on demand restore from the offload daemon. This new
command describes the need from the destination to fault the page
content in. This request describes the page through a MemoryRange
structure, and the response can be either 0 or the actual page size.
In case it is 0, that means the source had access to the guest memory
and was able to copy the page content directly. In case the response is
the actual page size, there is a payload associated which contains the
page content.
We can expect local live migration and offload restore to run locally
and therefore have access to the guest memory. The remote live migration
over the network is the case where we would expect the page content to
be sent over the wire.
This command is served through an additional connection happening on the
UNIX or TCP socket. The goal is to keep the same codepath between local
and remote migrations. This additional channel allows PageFault commands
to be issued asynchronously so they can be served without blocking the
main connection.
A connection role is introduced in order to identify an additional
connection related to pre-copy memory versus the newly introduced
channel for serving post-copy requests.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
BlockBackend predated the disk_file trait family and only carried
logical_size and physical_size, which the disk backends expose
through the disk_file traits DiskSize and PhysicalSize.
It added no polymorphism while its Read, Write and Seek supertraits
forced an unused cursor. Dropping the trait removes the dead code
it was keeping alive.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Import the modules used in the shared common test helpers instead of
spelling the full paths at every use site, and drop the now-unnecessary
crate-level #![expect(clippy::absolute_paths)] from integration_cvm.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import the modules used in the crate instead of spelling the full paths
at every use site, and drop the now-unnecessary crate-level
generated msr_index.rs was trimmed separately.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import the modules used in the binary instead of spelling the full
paths at every use site, and drop the now-unnecessary crate-level
#![expect(clippy::absolute_paths)].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Restoring a snapshot (snapshot restore or live-migration receive) sets
each vCPU's TSC by writing MSR_IA32_TSC as the vCPU is created.
However because CpuManager creates and restores vCPUs one at a time, the
host TSC advances between the per-vCPU writes and KVM derives a slightly
different TSC offset for each vCPU.
KVM only engages its masterclock when every offset matches. This has a
side effect of breaking the HyperV TSC reference clock page resulting in
significantly reduced performance on Windows.
After restore synchronise all vCPU's TSC offset to the boot vCPU's via
the KVM_VCPU_TSC_CTRL device attribute group (Linux 5.16+) this allows
the KVM TSC masterclock to engage and mitigates performance issues with
the KVM HyperV emulation.
See: #8383
Signed-off-by: Rob Bradford <rbradford@meta.com>
Assisted-by: Claude <claude-opus-4-8>
Import the modules used in the binary instead of spelling the full
paths at every use site, and drop the now-unnecessary crate-level
#![expect(clippy::absolute_paths)].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
This puts the send-migration action into a dedicated thread, laying the
groundwork for many follow-ups towards first-class live-migration in
CH.
This means:
1. The send-migration call will exit sooner (just trigger the
migration - dispatch semantics)
2. Other API calls can be triggered while a migration is ongoing but
will not be able to alter the VM as the VM's ownership is transferred
from the VMM to the migration thread. Example: hotplugging won't work
(which is good).
3. This is the basis for migration statistics via a dedicated endpoint
(future work).
The whole change was done with a special focus on graceful recover and
cleanup: even if anything on the migration paths go wrong, the proper
cleanups are already executed and the VMM can take back the ownership
of the VM.
The receive-migration API call remains blocking. To observe any status
changes about the migration on the sender side, one can observe the
event-monitor output and look for `vm.migration-{failed,finished}`.
These changes are inspired by [0] but differ significantly in details.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7038
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
This initializes the module and the thread that will handle (control)
the migration. This introduces the new types without the necessary
wiring.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Trim the one fully-qualified std::iter::successors path in the library
crate down to an imported module, and drop the now-unnecessary
crate-level #![expect(clippy::absolute_paths)].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
The two synchronous completion paths add the head to the used ring with
desc_chain.memory() but reload self.mem.memory() to enable
notifications. Keep both on the snapshot the chain was parsed from so
the used ring update and the notification enable always act on one
guest memory view rather than two independent atomic loads.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Add a parameterized helper that creates a 1 GiB ext4 loop filesystem
with 4096 byte sectors, populates it with a small data disk in the
requested format, attaches that disk with direct=on, and runs a 4096
byte aligned dd round trip with oflag=direct and iflag=direct
followed by cmp.
Wrappers exercise raw, qcow2, fixed VHD, and vhdx. The qcow2 and vhdx
wrappers expect the guest to see the on disk LBS of 512. The others
expect the host LBS of 4096.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The qcow workers carried their own O_DIRECT alignment requirement
and bounced unaligned cluster accesses through AlignedBuffer. Now
that the data file is an AlignedFile that handles O_DIRECT
transparently, the qcow layer can read and write through plain
buffers and let AlignedFile perform the aligned bounce.
Remove the alignment field and the per cluster AlignedBuffer RMW
branches from both the sync and async workers. The async io_uring
fast path still needs to avoid submitting unaligned guest iovecs
under O_DIRECT, so gate it on is_direct rather than on a stored
alignment value.
Drop the QcowAsync alignment override so it reports the trait
default sector size, matching QcowSync. qcow never submits guest
iovecs to the kernel under O_DIRECT, so reporting a larger value
only forced the request layer into an extra bounce buffer.
This adds one buffer copy per unaligned O_DIRECT cluster but moves
all alignment handling into a single place. The buffered path is
unchanged.
With qcow no longer the only caller, AlignedBuffer::read_exact_from
becomes dead code, so remove it and switch its tests to read_from.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Thread the direct flag from the disk open options through VhdDisk into
the AlignedFile the workers run on, so a fixed VHD opened with direct=on
issues O_DIRECT I/O instead of buffered I/O. Alignment is probed once on
that AlignedFile and reused by the sync and io_uring workers.
Advertise host topology from VhdDisk::topology by probing the underlying
file. On a 4096 byte sector filesystem opened with O_DIRECT this reports
logical_block_size 4096 to the guest, so the guest never issues 512 byte
I/O that the host kernel would reject as misaligned. Falls back to the
default topology with a warning when the probe fails.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Read the trailing footer sector through an AlignedFile rather than
probing the device topology and reading a full logical block. The
AlignedFile bounce buffer serves the trailing sector of an O_DIRECT
fd whose offset is unaligned against the device block size, so the
read no longer fails with EINVAL on a 4k sector backing store.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The raw sync, io_uring and AIO workers now own an AlignedFile and use
it for the O_DIRECT alignment value and for the unaligned fallback.
Aligned operations keep the fast preadv and pwritev iovec path straight
to the kernel. When the offset or an iovec base or length is not a
multiple of the probed alignment, the worker gathers the iovecs into
one contiguous host buffer and runs a synchronous RMW through
AlignedFile, then scatters the result back into guest memory.
RawDisk constructs the AlignedFile from the disk file and the direct
flag and passes it into each worker, so alignment is probed once at
open time. The fixed VHD workers are threaded through the same
AlignedFile based constructors using a non-direct AlignedFile to
preserve current behavior.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
RawFile wrapped AlignedFile only to add a seek position and the file
trait impls that the qcow and vhost_user_block code expects. Fold that
position and every impl onto AlignedFile so the wrapper layer goes away
and callers work with a single O_DIRECT aware file type.
AlignedFile now tracks a cursor and implements Read, Write, Seek,
WriteZeroesAt, PunchHole, FileSync, SeekHole, BlockBackend, Clone,
AsRawFd and AsFd in addition to the positional FileExt path. The
direct_io flag is dropped because alignment already encodes it, where
a zero alignment means the file was not opened with O_DIRECT.
All RawFile uses in the qcow internals and vhost_user_block move to
AlignedFile, and raw_file.rs is removed.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Move the statx STATX_DIOALIGN probe out of DiskTopology into a free
probe_direct_alignment helper keyed on a raw fd. The helper gates on the
O_DIRECT open flag and returns the kernel reported alignment only when
direct I/O is in effect, and None otherwise. DiskTopology::probe keeps
the same call path and result.
AlignedFile::new now determines its O_DIRECT block alignment from
probe_direct_alignment instead of trial reads at 512 and 4096, falling
back to SECTOR_SIZE when the kernel does not report a value. This
matches how the raw and fixed VHD workers determine alignment, so all
backends agree on one source of truth.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The boot time tests inferred guest readiness from the two debug I/O
port markers on stderr. Heavier boots could miss the fixed sleep
window, capture a single marker, and panic.
Wait on the cloud-init injected notify-booted callback through
guest.wait_vm_boot() instead. Once the callback fires, both markers
are guaranteed present, so they are parsed only for the metric. The
host side keeps an overall timeout, so a guest that never boots is
still reaped.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
This was already handled with user memory zones but not with the default
memory. Make a small refactoring to move the boot RAM check into
MemoryManager rather than split across Vm and MemoryManager.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Import the std modules used in the crate instead of spelling the full
paths at every use site, and drop the now-unnecessary crate-level
#![expect(clippy::absolute_paths)].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Add an aarch64 test that pauses a running VM, waits out an interval, and
resumes it on the same host, then asserts the guest wall clock still
matches the host. On aarch64 the architected counter free-runs across
the pause, so the guest self-corrects.
The downtime and skew tolerance are shared with the snapshot clock test.
x86_64 has its own kvmclock path and is covered by the snapshot clock
test.
Signed-off-by: Atish Patra <atishp@meta.com>
Add a variation of _test_snapshot_restore that, after taking a snapshot,
waits out a simulated off-host interval and then restores and resumes,
asserting that the guest's wall clock has caught up to the host. This
exercises the clock catch-up that each architecture provides on restore:
kvmclock (KVM_CLOCK_REALTIME) on x86_64 today, and the CNTVCT advance on
aarch64 with later commits.
On x86_64 the guest is booted with clocksource=kvm-clock as the guest
clock is caught up after pause/resume only in that mode. A
tsc-clocksource guest's restored TSC freezes across the interval and
would never catch up.
Take this opportunity to improve the snapshot restore test as the
existing bare boolean mechanism was bit hard to read with new test.
Signed-off-by: Atish Patra <atishp@meta.com>
Currently, Cloud Hypervisor round-trips CNTVCT_EL0 through
KVM_GET_REG_LIST/SET_ONE_REG, which leaves a cold-restored or migrated
guest behind real UTC by the downtime. Same-host pause/resume
self-corrects (the physical counter keeps running across the pause), so
only restore and migration cases required the clock to catch up to wall
clock time.
Since ARM has no kernel helper, compute the difference in wall clock
time and compute the ticks so that it can advance the CNTVCT correctly.
It is set via vcpu0 only as it affects a single VM wide value after
Linux 6.4. For older kernels, it was a truly vcpu value which needs to
be invoked for every vcpu.
Gated on all(target_arch = "aarch64", feature = "kvm"); x86 is
unchanged.
Basic manual test case (aarch64 + KVM) verified both in intra host and
inter host snapshot save/restore:
1. Boot a Linux guest; in the guest, `date -u` tracks the host's UTC.
2. Pause and snapshot the VM (ch-remote pause; ch-remote snapshot
file:///<dir>).
3. Leave it down for several minutes (the off-host interval).
4. Restore and resume into a fresh VMM (ch-remote restore
source_url=file:///<dir>,resume=true).
5. In the guest, run `date -u` again and compare to the host: the guest
now tracks current UTC, having advanced by ~the time it spent down.
Before this change the restored guest reads behind real UTC by the
downtime; after it, the guest clock is back in sync (to within the
snapshot-to-restore sampling slop).
Signed-off-by: Atish Patra <atishp@meta.com>
Unlike x86, ARM64 has no kvmclock support to sync guest time upon
required. However, the guest reads the architected virtual timer
(CNTVCT_EL0) directly which can be modified by the VMM to update the
time after snapshot restore. Since the CNTVCT is in ticks, we also need
to read CNTFRQ (via mrs due to lack of ONEREG interface) to compute the
ticks from wall clock difference.
Because the counter is a vCPU register, the capture must run with the
vCPUs quiesced, so the VMM now captures the clock just after
cpu_manager.pause() through the boot vCPU. This is behaviorally
identical for x86, whose clock is VM-wide. There is no restore/advance
yet, so aarch64 guests still resume behind real time until the following
commit.
Signed-off-by: Atish Patra <atishp@meta.com>
Currently, VM pause/resume/snapshot paths invoke architecture specific
bits for guest clock udpates which ideally belongs to hypervisor layer.
Route it through the snapshot_clock()/restore_clock() pair added in the
previous commit instead, so the VMM no longer depends on an architecture
specific clock API and the upcoming aarch64 backend can hook the same
path without a parallel branch in vm.rs.
Signed-off-by: Atish Patra <atishp@meta.com>
Preserving the guest clock across pause/resume and snapshot/restore is
currently open-coded in the VMM against the x86-only
get_clock/set_clock. aarch64 needs the same correction but via a
different mechanism (i.e. the architected counter, CNTVCT). Having a
common backend-agnostic interface that VMM can drive uniformly allows us
to keep the architecture details behind the Hypervisor abstraction.
This commit only introduces the abstraction while the future commits
will actually move the implementation to use it.
Use this opportunity to fix the full path to get SystemTime as well.
Suggested-by: Sebastien Boeuf <sboeuf@meta.com>
Signed-off-by: Atish Patra <atishp@meta.com>
Import the std module used in the generated MSR-index file instead of
spelling the full paths at every use site.
The crate-level #![expect(clippy::absolute_paths)] is kept; the
remaining sites in the rest of the crate are handled by a follow-up.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
OpenBSD expects the ACPI _S5_ object to provide both sleep type values.
The single-value package made acpi_init_states() parse an invalid object
and fault during early ACPI setup with:
```
...
acpi0 at bios0: ACPI 6.3
acpi0: sleep statesfatal protection fault in supervisor mode
trap type 4 code 0 rip ffffffff814af264 cs 8 rflags 10282 cr2 0 cpl e rsp ffffffff81a06a30
gsbase 0xffffffff81755ff0 kgsbase 0x0
panic: trap type 4, code=0, pc=ffffffff814af264
Starting stack trace...
panic(ffffffff81a06980,4,ffffffff81a06a58,ffffffff81756ae0,ffffffff81a06960,ffffffff81a068e0) at panic+0x12e
kerntrap(ffffffff81a06aa0,ffff800000232400,ffffffff81261681,ffffffff81a06970,ffffffff81a06980,4) at kerntrap+0xe1
alltraps_kern_meltdown(4,ef0d316e102be1f4,ffff800000232480,0,ffffffff81a06aa0,ffff800000232400) at alltraps_kern_meltdown+0x7b
aml_val2int(ef0d316e102be1f4,0,10,ffffffff81a06a30,10282,8) at aml_val2int+0x24
acpi_init_states(1,ffff800000232400,ffff800000232470,0,ef0d316e102be1f4,5f35535f) at acpi_init_states+0xd5
acpi_attach_common(ffff800000235300,ffffffff81a06cf0,ffffffff81762a00,ffff800000232400,ffff800000232424,ef0d316e102be1f4) at acpi_attach_common+0x311
config_attach(ffffffff81a06d30,ffff800000235300,50,118,ffff80003158c004,ffffffff813e3270) at config_attach+0x1d2
bios_attach(ffff800000235280,ffffffff81a06e28,ffffffff8175ca50,ffff800000235300,ffff800000235324,ef0d316e102be1f4) at bios_attach+0x898
config_attach(ffffffff81a06e28,ffff800000235280,ffffffff81757e68,ffff800000235280,ffff8000002352a4,ffffffff8128c510) at config_attach+0x1d2
mainbus_attach(0,0,ef0d316e102be1f4,ffffffff81a06e50,ffffffff81a06ec0,3000000010) at mainbus_attach+0x70
config_attach(8,1001000,805f50,1000000,ffffffff81a00008,0) at config_attach+0x1d2
cpu_configure(8,1001000,ffffffff814f3859,ffffffff81a06f20,8,1001000) at cpu_configure+0x29
main(1001000,ef0d316e102be1f4,ffffffff812e8b2f,ffffffff81a06f40,8,1001000) at main+0x3af
end trace frame: 0x0, count: 244
End of stack trace.
```
Advertise S5 as the conventional four-element package as described in
the ACPI spec [1]. Cover the generated AML bytes with a unit test.
In AML, the package now looks like this:
```
Name (_S5, Package () {
0x05, 0x05, 0x00, 0x00
})
```
[1] https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/07_Power_and_Performance_Mgmt/oem-supplied-system-level-control-methods.html#sx-system-states
On-behalf-of: SAP thomas.prescher@sap.com
Signed-off-by: Thomas Prescher <thomas.prescher@cyberus-technology.de>
In #8053 it was agreed to use the ring backend in preference to
aws-lc-rs as it is pure Rust. However since aws-lc-rs is a default
feature of rustls the addition of ring was additive not a replacement.
Fix the features to exclude awc-lc-rs.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Closing an io_uring fd does not synchronously finish requests that
already reached the kernel. During block worker teardown this can let
an io-wq worker keep using retained guest-memory iovecs after reset.
Drain UringDataIo in Drop: retry any published SQEs and wait for CQEs
until no retained operation remains. If draining fails, leak retained
buffers. Drop QcowAsync's ring before its data fd so retrying
published SQEs still uses a valid descriptor.
To avoid a potential infinite loop when completions fail to be delivered
cap the number of iterations of the loop (2x the number of inflight
requests).
Assisted-by: Codex:GPT-5
Signed-off-by: Rob Bradford <rbradford@meta.com>
This is not a blocking job but the worker is no longer handling the jobs
so it remains pending indefinitely.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Import the modules used in the crate instead of spelling the full paths
at every use site, and drop the now-unnecessary crate-level
#![expect(clippy::absolute_paths)].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Extending the snapshot/restore documentation so that it explains what
are the goals behind this offloaded snapshot/restore feature, how to use
it in practice, and also by documenting the protocol used by the offload
daemon so that anyone could write its own daemon.
By relying on the existing local live migration support and reusing the
semantics and the protocol associated with it, we intend to provide a
way for snapshotting and restoring a VM to/from a dedicated process that
we can call the offload daemon.
By allowing an external process to perform the snapshot/restore actions
on behalf of Cloud Hypervisor, we give our users the opportunity to
implement their own offloaded daemon. The goal is to avoid bloating
Cloud Hypervisor with numerous features related to snapshot/restore, and
let the user decide how to perform the snapshot/restore actions. One
example is that we can decide to encrypt the guest RAM on the fly in
order to avoid writing an unencrypted version to local disk. Another
example is to be able to send guest RAM and associated state/config data
over the network without having to persist the data first to local
storage.
There might be other reasons to choose going with an offloaded daemon to
perform the snapshot/restore of the VM, but in every case, this empowers
the user to make their own choice.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Copy only populated extents when writing the snapshot file and when
filling the restore memfd, leaving unwritten ranges as holes. Both
the on-disk snapshot and the restored guest RAM stay sparse, so that
untouched guest pages cost no disk space or host memory.
This brings the offload daemon closer to be at feature parity with CH's
internal implementation of snapshot/restore. The only missing piece is
on-demand paging at this point.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Move next_data_extent and write_region_sparse out of memory_manager.rs
into a new vmm::sparse module so the snapshot writer, the restore
reader, and the offload daemon can share one implementation.
No functional change intended.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Adding a new dedicated binary that is meant to be used as a reference
implementation for validating that offloaded snapshot/restore works and
meant to be used through tests in general.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Expose VmMigrationConfig as a public facing structure that can be used
by an offload daemon to act as if it was the VM to migrate to, or the VM
to migrate from.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Import the std modules used in the crate instead of spelling the full
paths at every use site, and drop the now-unnecessary
#[cfg_attr(target_arch = "x86_64", expect(clippy::absolute_paths))].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import the std modules used in the crate instead of spelling the full
paths at every use site, and drop the now-unnecessary crate-level
#![expect(clippy::absolute_paths)].
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
