Document how live migration protocol versions are handled, including
the supported current/previous version window and the need to migrate
through an intermediate Cloud Hypervisor version for larger version
gaps.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Print the supported vm-migration protocol version range in
cloud-hypervisor --version as an extra line:
vm-migration protocol versions v0-v1
This makes the currently supported compatibility window
visible without having to inspect the migration code.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
The cloud-hypervisor binary prints vm-migration protocol constants
directly, so it needs its own dependency on the vm-migration crate
instead of relying on vmm's transitive dependency.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Validate the sender's migration protocol version when
handling the initial Start request.
Read the version from the Start command header, accept only
the supported version window n-1..=n, and reject unsupported
versions with Error. A rejected Start moves the receiver to
the aborted state.
This keeps compatibility one-way, from older protocol
versions to newer ones, and leaves later version-based
branching on the receiver side.
Log the protocol version on both sender and receiver to make
the active migration path visible.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Add protocol-side support for migration protocol versioning.
Use the existing 6-byte Start command header, which was
previously zero padding, to carry the sender's migration
protocol version without changing the wire layout.
Store the version as a little-endian u16 in the first two
bytes and ignore the remaining four bytes. A zeroed command
header continues to mean a legacy v0 sender.
This keeps the message flow unchanged for rollout:
Start is still followed by plain OK or Error (Aborted), and no new
command is needed.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Import the modules used in the crate instead of spelling the
fully-qualified paths at every use site, and collapse Result<T,
io::Error> into io::Result<T>. This covers the feature-gated modules
(fw_cfg, ivshmem, pvmemcontrol) as well, leaving the whole crate free
of clippy::absolute_paths warnings.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Drop stale #[allow]s whose lints no longer fire and convert the
rest to #[expect], which warns if they ever stop being needed.
Part of #8326.
Signed-off-by: Tushar Khatri <hello@tusharkhatri.in>
During migration send one final set of changed memory after capturing
the snapshot/state. This captures any memory changed as a side effect of
capturing that state. In particular with vhost-user capturing the device
state can lead to inflight requests being drained/flushed which could
change memory. As this is related to the snapshot account for this
memory transfer in the snapshot metrics.
No equivalent change is needed for snapshot as the memory is written
after the state is snapshotted.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Query and save the dirty log before shutting down the vhost-user device.
This allows any final dirty memory ranges to be recorded before it
becomes impossible to do that as the vhost-user handle has been closed.
This is required to ensure that all memory writes have been correctly
recorded that may be triggered by inflight I/O drains from vhost-user
device state capture.
One small implementation wrinkle: with local migrations there is no
dirty logging (since we just pass the memory FD over the socket) so
calling dirty_log() would generate an error. As there is no clean way to
query if dirty logging has been started add a boolean to track if its
active.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Rather than each device implementation holding a reference to the guest
memory move this to VhostUserCommon. This refactoring simplifies the
function signatures but also allows for methods that act on
VhostUserCommon that don't have the memory available to them.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Currently both kvm and mshv require an IGVM file to boot a SEV-SNP VM.
This is already configured in the top-level cloud-hypervisor
Cargo.toml where sev_snp depends on igvm.
Add a similar dependency in the vmm crate which helps simplify some of
the in-code cfg blocks by removing the ones that are within a sev_snp
cfg block.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Import the modules used in the crate instead of spelling the
fully-qualified paths at every use site. This covers std paths along
with a few crate-internal and external-crate paths, leaving pci free of
clippy::absolute_paths warnings.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Re-enable the fw_cfg integration tests for MSHV now that port string I/O
is handled by the hypervisor backend.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:GPT-5.5
Handle MSHV INS/OUTS port intercepts by translating the guest string
operand through MshvEmulatorContext and copying data between guest
memory and existing PIO callbacks.
Support REP counts, zero-count REP, and direction-flag based RSI/RDI
updates. Commit RIP plus RCX/RSI/RDI after the transfer completes.
This removes the fw_cfg/debug-port skip. OVMF can now use the real
string I/O path instead of relying on ignored ports.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:GPT-5.5
Decode MSHV port access size, direction, string, and REP state through
small helpers instead of open-coded bitfield reads.
Keep scalar I/O behavior unchanged and continue rejecting string I/O in
this step. Put the x86_64 port I/O helpers in their own module so the
string emulation path can grow without bloating the top-level MSHV code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:GPT-5.5
Move common string instruction bookkeeping into the x86 instruction
emulator so MOVS, STOS, and MSHV port string I/O use one implementation
for REP counts, direction-flag handling, and index advancement.
This keeps existing MOVS/STOS behavior unchanged while removing the need
for MSHV to open-code the same string-operation details.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:GPT-5.5
Move scalar HVMSG_X64_IO_PORT_INTERCEPT handling into a helper so the
string I/O implementation can build on the same dispatch path.
Keep the existing fw_cfg/debug-port skip plus string/REP assertions in
place. This is only code movement so later changes are easier to review.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:GPT-5.5
Turn reserve=on for the hugepage-backed memory zone in the UFFD
snapshot/restore integration test. Hugepages are the most likely place
to want reserve (an over-committed huge page pool is exactly the case
that otherwise SIGBUSes the guest), so this is the natural test to give
the option real coverage, as suggested in review.
It exercises the reserve mmap path twice: once on the source VM boot and
once on the demand-paged restore. The existing skip guard already
requires the 256 free 2MiB pages this zone needs, and the source VM is
killed before the restore VM is started, so reserving from the pool
never has to back two VMs at once.
Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Ian Klemm <hi@ianklemm.de>
Cloud Hypervisor maps guest RAM with MAP_NORESERVE, so the kernel never
reserves the backing pages at mmap time. On a host whose hugepage pool
cannot satisfy every guest, a VM is created successfully and then takes
a SIGBUS when the guest faults a page the pool can no longer back. This
is the failure mode reported in #5730 and #7387. As noted on #5730,
checking free pool headroom up front is not a reliable fix: another
process can consume pages between the check and the fault.
Add a reserve=on parameter to --memory and --memory-zone (default off,
preserving the current MAP_NORESERVE behaviour). When set, the region
is mapped without MAP_NORESERVE, so the kernel reserves the backing
pages (swap, or huge pages for hugepage-backed memory) at mmap time,
atomically with the mapping. An over-committed configuration then
fails cleanly at VM creation with an mmap ENOMEM instead of crashing
the guest later. Unlike prefault it does not fault the memory in, so
it does not slow down boot.
This mirrors QEMU's memory-backend reserve property, which has the same
name and meaning (reserve=off maps with MAP_NORESERVE). reserve is
threaded through the same mmap paths as the existing prefault option,
and is exposed in the OpenAPI schema, CLI help and docs. The top-level
--memory reserve=on path is unchanged: the default zone is synthesised
from MemoryConfig and inherits its reserve value.
Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Ian Klemm <hi@ianklemm.de>
remove_userspace_mapping tears down the KVM mapping but never returns
the slot id to the allocator's free list. Call `free_memory_slot` to
avoid the leak.
Signed-off-by: Dylan Reid <dgreid@fb.com>
Import the std modules used in the crate instead of spelling the full
paths at every use site.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Bring the confidential VM (CVM) integration tests up on the KVM SEV-SNP
backend in addition to MSHV. On KVM the IGVM is an Oak stage0 firmware
image and the guest kernel is supplied separately: stage0 reads the
kernel, cmdline and E820 over fw_cfg. The test harness selects this
model when a guest kernel is present at /igvm_files/bzImage, mirroring
how the stage0 IGVM is discovered; MSHV keeps using the monolithic IGVM
with the kernel baked in.
- test_infra: stage0 + direct-kernel + fw_cfg boot wiring (both the
command line and the HTTP/D-Bus API path) plus an on_kvm_sev_snp()
helper for tests to branch on.
- tests: the CVM tests that don't work on the KVM SEV-SNP path yet are
gated with #[cfg(not(feature = "kvm"))] inside the common_cvm module.
The MSHV build enables mshv,igvm,sev_snp (no kvm feature) while the
KVM build enables kvm,igvm,sev_snp,fw_cfg, so the cfg compiles these
tests into the MSHV binary only and drops them on KVM; both
hypervisors run the single common_cvm nextest profile. They all still
run on MSHV:
* test_pci_multiple_segments - stage0 places all 64-bit BARs in a
single global window, so a BAR allocated in a different
per-segment window is relocated cross-window and wedges boot.
* test_dmi_uuid / test_dmi_oem_strings /
test_dmi_system_and_chassis - SMBIOS is not delivered to SEV-SNP
guests on the KVM stage0 boot path, so the guest's DMI tables
read empty. VMM follow-up.
* test_vdpa_block - needs host vdpa_sim_blk setup, and vDPA DMA
into SEV-SNP-encrypted memory is unsupported (the guest hangs).
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
FwCfgConfig already has a Default impl (e820/kernel/cmdline/initramfs/
acpi_tables = true, items = None), but deserialization did not use it:
without serde default every field was mandatory, so any caller building
a payload config over the API had to spell out the whole object even to
flip a single flag.
Add a container-level #[serde(default)] so missing fields fall back to
FwCfgConfig::default(). The container form is required here because the
defaults are all true; a per-field #[serde(default)] would resolve bool
to false and contradict the Default impl.
This lets callers send only the fields that differ from the defaults.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
validate_memory() asserts that the guest's reported MemTotal exceeds the
expected size, but on failure printed nothing about either value.
Include both the actual and expected figures in the panic message so a
failing run is self-explanatory.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
The KVM SEV-SNP net-hotplug integration tests (supported added in
later commits) intermittently kill the VMM with SIGSYS on the
http-server thread. The thread is seen reading
/proc/sys/vm/overcommit_memory (openat + read).
Allow the syscalls there, gated on sev_snp+kvm. seccomp can't match
a path, so the open is restricted to O_RDONLY.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
On KVM, an SEV-SNP guest opens /dev/sev at VM creation and passes
its fd as kvm_sev_cmd.sev_fd. KVM uses it to issue the SNP_LAUNCH_*
commands. The open happens after the landlock ruleset is applied,
so grant rw access to /dev/sev when the sev_snp platform feature
is enabled.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
A confidential guest (e.g. SEV-SNP) requires every virtio device to
advertise VIRTIO_F_ACCESS_PLATFORM so the guest driver routes DMA
through the platform's bounce-buffer path; the driver refuses a device
that does not offer it.
Add VIRTIO_F_ACCESS_PLATFORM support to virtio-{watchdog,iommu} which
are exercised as part of the CVM integration tests.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Import the std modules used in the crate instead of spelling the full
paths at every use site.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import std::fmt and std::result instead of spelling the full paths at
every use site.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
The guest kernel build instructions in the README referenced the
ch-6.12.8 branch, which was inconsistent with the version used by the
test scripts (scripts/test-util.sh uses ch-6.16.9). Update the README
to point at ch-6.16.9 so documentation and CI stay in sync.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Delete the now unused QcowFile implementation it was only used for
creating disk images for the tests and for the performance-metrics. It
was not used for the virtio-block device.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Pull rebuild_refcounts out of QcowFile so QcowFile can be removed in a
follow up commit. This function is still required by parse_qcow().
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Port the internal/mod.rs tests from QcowFile to QcowDisk so
the surviving tests exercise the code paths that are hit when used via
QcowDisk (and thus virtio-block).
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Fuzzing QcowFile, which is not used by virtio-block, is less helpful
than using the abstraction that is. Ensure we cover at least the same
functionality as before.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch from QcowFile to QcowDisk taking advantage of QcowTempDisk where
appropriate.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Port tests over to QcowTempDisk and also over to QcowDisk rather than
QcowFile where necessary.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
A common pattern in the test code is to create a temporary file, format
it as QCOW2 and then open it as a QcowDisk. Create a helper struct that
can be used in those tests. This is marked as #[cfg(test)] as initially
it will only be used by the test suite.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add a method to format a file as a QCOW2 file which will mainly be used
by the test infrastructure. This copies the logic from QcowFile. It
doesn't refactor it as the removal of QcowFile is planned.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Rob Bradford <rbradford@meta.com>
With socket serial output now buffered and replayed on connect, a
late-connecting client receives the whole boot backlog. The pty
interaction test had three problems with that:
- pty_read() slept a second between 512-byte reads and the loop consumed
one chunk per two-second tick, far too slow to drain the backlog. Read
in larger chunks without the per-read sleep and drain everything
available each round; bound the loop so a missing marker can't run to
the harness timeout.
- it wrote the login keystrokes before reading, so the unread backlog
back-pressured the sender and the keystrokes never reached the prompt.
Start reading concurrently with typing instead.
- the socat pty was created with echo on, so the replayed backlog was
echoed back to the guest as serial input, flooding it (UART input
overrun, login never completing). Create the pty with echo=0.
Signed-off-by: Max Makarov <maxpain@linux.com>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
In Socket serial mode the device output sink was only installed once a
client connected, so output produced beforehand (kernel boot messages,
cloud-init) was dropped, and a client attaching after boot saw a blank
screen. Only PTY mode wrapped the sink in a SerialBuffer.
Install a persistent SerialBuffer as the Socket device's output sink at
SerialManager construction (discarding downstream via io::sink() until a
client connects), so output is captured into the 1 MiB ring even with no
client attached. On connect, retarget the buffer at the accepted client
and flush the backlog before live output resumes; on disconnect, keep
buffering so output produced while no client is attached is delivered to
the next one. The accepted socket is made non-blocking via
set_nonblocking() so a slow client cannot stall the vCPU thread
(SerialBuffer re-buffers on WouldBlock).
The serial-manager thread gains two syscalls under seccomp: sendto
(replaying the backlog is the first time it writes to the socket) and
ioctl restricted to FIONBIO, which is what set_nonblocking() issues.
Fixes: #7907
Signed-off-by: Max Makarov <maxpain@linux.com>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
SerialBuffer owns its downstream writer privately, with no way to
replace it. Buffering the Socket console requires keeping one buffer
alive across client connects and disconnects and pointing it at each
newly accepted client (or a discarding sink when none is connected)
without dropping bytes buffered while no client was attached.
Add set_out(), which swaps the writer while leaving the buffered
contents intact, plus unit tests covering accumulate-while-detached,
replay on connect, live pass-through, delivery of while-detached output
to the next client, and that bytes already drained by one client are not
resent to the next.
Signed-off-by: Max Makarov <maxpain@linux.com>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Import std::io instead of spelling the full paths at every use site.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import the std modules used across the crate instead of spelling the
full paths at every use site.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
KVM will emulate XSAVE for us, so we need only to skip XCRS
setting/retrieval if the respective CPU feature is not available.
Signed-off-by: Nikolas Kyx <55556836+nyx191@users.noreply.github.com>
eject_device calls shutdown, but Block doesn't implement it, so the
worker thread was never joined. Drop for Block does not help either,
VirtioPciDevice keeps an Arc<Mutex<Block>> past eject, so Drop never
runs. The async worker keeps completing in-flight I/O into the guest RAM
that backed it when issued.
Implement shutdown() to call wait_for_epoll_threads(), which drops the
WorkerThreads handle and ensures that worker's io completes.
Signed-off-by: Dylan Reid <dgreid@fb.com>
server_api_error_display_modifier deserialized the JSON error response
into a `Vec<&str>`. However, if the error message contained escaped
characters, it could not deserialize it into a borrowed string `&str`
because unescaping requires allocation. This resulted in a
deserialization error and a failure to print the error chain.
This change switches the deserialization target to `Vec<String>` to
allow allocation.
Signed-off-by: Andrei Vagin <avagin@google.com>
The snapshot and restore subcommands in ch-remote had optional
snapshot_config and restore_config arguments, but the implementation was
unconditionally unwrapping them.
This change marks these arguments as required to handle the missing
argument validation and report a proper error message instead of letting
the application panic.
Signed-off-by: Andrei Vagin <avagin@google.com>
Prevent `SkuNotAvailable` errors for mshv workflow by checking capacity
restrictions for each location. Enhance the VM provisioning logic to
validate resource availibility before deployment.
Signed-off-by: Aastha Rawat <aastharawat@microsoft.com>
