Advertise `VIRTIO_NET_F_GUEST_ANNOUNCE` on virtio-net devices, surface
`VIRTIO_NET_S_ANNOUNCE` through config status, and handle
`VIRTIO_NET_CTRL_ANNOUNCE_ACK` on the control queue.
This adds the guest-visible state needed for post-migration or
post-restore announce requests; the VMM side triggering is added in
follow-up commits.
The motivation is to reduce post-migration and post-restore
connectivity gap. After a live migration or after restoring, it can
take the guest several seconds to be reachable again over the network.
With these announcements, the network path should be refreshed within a
few milliseconds.
On-behalf-of: SAP sebastian.eydam@sap.com
Signed-off-by: Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
Add a Snapshot and Restore section to docs/vfio.md covering the
migration v2 requirements (Linux 5.18 kernel, variant VFIO
driver such as mlx5_vfio_pci) and the restore sequence for
devices that advertise migration v2.
The behavior description covers the full restore sequence. It
documents the RUNNING to RESUMING single transition (the kernel
walks the intermediate STOP arc), the post load PCI_COMMAND push
to the device, and the MSI or MSI-X eventfd rearm that the
kernel state does not carry. Behavior matches QEMU
vfio_pci_load_config().
It notes one limitation, the snapshot format stores the opaque
device blob as base64 inside the snapshot JSON, which may benefit
from a binary transport path for very large state.
docs/snapshot_restore.md replaces its VFIO out of scope
limitation with a short section that points to docs/vfio.md for
the requirements and behavior.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Broaden error!() to cover any user-initiated action that fails to do
what was expected (e.g. failed hotplug or live migration), not only
unrecoverable startup errors. Retarget info!() at operators and users,
clarify the warn!() and debug!() audiences, and document trace!().
Part of #8440.
Co-authored-by: Philipp Schuster <phip1611@gmail.com>
Signed-off-by: Tushar Khatri <hello@tusharkhatri.in>
Make the testing overview describe the dev_cli.sh workflow instead of
implying that every Cloud Hypervisor build must run in a container.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Wire up the source side of postcopy migration over TCP. When
`mode=postcopy` is requested on vm.send-migration, the source skips
the pre-copy dirty-tracking loop and lets the destination resume early,
then serves guest pages on demand over a dedicated connection.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Add an --ondemand flag to the offload daemon's restore subcommand to
support the post-copy mechanism from the live migration protocol.
In on-demand mode, the daemon creates empty memfds to back the guest
memory and sends them over to the VMM. This lets the VM start quickly,
right after the memfds are mapped into CH's address space.
At runtime, when the guest accesses a page (or the prefault handler
requests it), the daemon faults it in by copying the page content into
its shared memory mapping, then replies to the PageFault request so the
VMM can consider the page present.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Extending the snapshot/restore documentation so that it explains what
are the goals behind this offloaded snapshot/restore feature, how to use
it in practice, and also by documenting the protocol used by the offload
daemon so that anyone could write its own daemon.
By relying on the existing local live migration support and reusing the
semantics and the protocol associated with it, we intend to provide a
way for snapshotting and restoring a VM to/from a dedicated process that
we can call the offload daemon.
By allowing an external process to perform the snapshot/restore actions
on behalf of Cloud Hypervisor, we give our users the opportunity to
implement their own offloaded daemon. The goal is to avoid bloating
Cloud Hypervisor with numerous features related to snapshot/restore, and
let the user decide how to perform the snapshot/restore actions. One
example is that we can decide to encrypt the guest RAM on the fly in
order to avoid writing an unencrypted version to local disk. Another
example is to be able to send guest RAM and associated state/config data
over the network without having to persist the data first to local
storage.
There might be other reasons to choose going with an offloaded daemon to
perform the snapshot/restore of the VM, but in every case, this empowers
the user to make their own choice.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
Document how live migration protocol versions are handled, including
the supported current/previous version window and the need to migrate
through an intermediate Cloud Hypervisor version for larger version
gaps.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Cloud Hypervisor maps guest RAM with MAP_NORESERVE, so the kernel never
reserves the backing pages at mmap time. On a host whose hugepage pool
cannot satisfy every guest, a VM is created successfully and then takes
a SIGBUS when the guest faults a page the pool can no longer back. This
is the failure mode reported in #5730 and #7387. As noted on #5730,
checking free pool headroom up front is not a reliable fix: another
process can consume pages between the check and the fault.
Add a reserve=on parameter to --memory and --memory-zone (default off,
preserving the current MAP_NORESERVE behaviour). When set, the region
is mapped without MAP_NORESERVE, so the kernel reserves the backing
pages (swap, or huge pages for hugepage-backed memory) at mmap time,
atomically with the mapping. An over-committed configuration then
fails cleanly at VM creation with an mmap ENOMEM instead of crashing
the guest later. Unlike prefault it does not fault the memory in, so
it does not slow down boot.
This mirrors QEMU's memory-backend reserve property, which has the same
name and meaning (reserve=off maps with MAP_NORESERVE). reserve is
threaded through the same mmap paths as the existing prefault option,
and is exposed in the OpenAPI schema, CLI help and docs. The top-level
--memory reserve=on path is unchanged: the default zone is synthesised
from MemoryConfig and inherits its reserve value.
Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Ian Klemm <hi@ianklemm.de>
Bring the confidential VM (CVM) integration tests up on the KVM SEV-SNP
backend in addition to MSHV. On KVM the IGVM is an Oak stage0 firmware
image and the guest kernel is supplied separately: stage0 reads the
kernel, cmdline and E820 over fw_cfg. The test harness selects this
model when a guest kernel is present at /igvm_files/bzImage, mirroring
how the stage0 IGVM is discovered; MSHV keeps using the monolithic IGVM
with the kernel baked in.
- test_infra: stage0 + direct-kernel + fw_cfg boot wiring (both the
command line and the HTTP/D-Bus API path) plus an on_kvm_sev_snp()
helper for tests to branch on.
- tests: the CVM tests that don't work on the KVM SEV-SNP path yet are
gated with #[cfg(not(feature = "kvm"))] inside the common_cvm module.
The MSHV build enables mshv,igvm,sev_snp (no kvm feature) while the
KVM build enables kvm,igvm,sev_snp,fw_cfg, so the cfg compiles these
tests into the MSHV binary only and drops them on KVM; both
hypervisors run the single common_cvm nextest profile. They all still
run on MSHV:
* test_pci_multiple_segments - stage0 places all 64-bit BARs in a
single global window, so a BAR allocated in a different
per-segment window is relocated cross-window and wedges boot.
* test_dmi_uuid / test_dmi_oem_strings /
test_dmi_system_and_chassis - SMBIOS is not delivered to SEV-SNP
guests on the KVM stage0 boot path, so the guest's DMI tables
read empty. VMM follow-up.
* test_vdpa_block - needs host vdpa_sim_blk setup, and vDPA DMA
into SEV-SNP-encrypted memory is unsupported (the guest hangs).
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Describe --hypervisor as an optional override that defaults to
auto-detection from the host device node (/dev/mshv or /dev/kvm) for
both the build and tests commands and the shared test-script arguments.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add support for formatting logs following the popular glog crate style.
In particular this can use local or UTC time along with a single
character level.
As an implementation detail of that implement finegrained date time log
outputs including both local and UTC variants.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Assisted-by: Claude:Opus-4.6
cloud-hypervisor/edk2 publishes prebuilt CLOUDHV.fd (x86-64) and
CLOUDHV_EFI.fd (AArch64) as release assets. docs/uefi.md only
described the build from source, and the AArch64 firmware
customizations required for cloud-hypervisor were left undocumented.
Add a "Using Prebuilt UEFI Firmware" section to docs/uefi.md and
an "AArch64 Firmware Notes" section covering both customizations.
Updates to "Building UEFI Firmware for AArch64" section.
Switch the boot examples from --kernel to --firmware, which is the
direct UEFI load path on AArch64.
Minor README.md updates.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Explain how operators can exclude selected VFIO BARs from mmap
when they know their workloads do not require MMIO access to those
regions.
Documenting the option separately keeps the new device argument
discoverable and calls out the valid BAR index range.
Signed-off-by: Damian Barabonkov <dbctl@pm.me>
Assisted-by: OpenCode:gpt-5.5
Remove CI workflow description section as the yaml
files change often and the docs become stale quickly.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add docs/testing.md covering the dev_cli.sh interface, all
test types (unit, integration, VFIO, Windows, live migration,
rate limiter, CVM), custom kernel/firmware overrides via
environment variables, performance metrics, code coverage,
and the CI workflow matrix.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
QEMU supports passing inline string values to the guest via fw_cfg
(-fw_cfg name=...,string=...). Cloud Hypervisor previously only
supported file-backed fw_cfg items. This adds the 'string' option
so users can pass values like OVMF's X-PciMmio64Mb without creating
a temporary file on the host.
Each fw_cfg item now accepts exactly one of 'file' or 'string'.
The FwCfgInvalidItem invariant is validated in PayloadConfig::validate()
(via FwCfgConfig::validate()), covering both CLI and JSON API paths.
The populate_fw_cfg match arm uses unreachable!() since validation
guarantees the invariant holds at that point.
CLI syntax:
--fw-cfg-config items=[name=opt/ovmf/X-PciMmio64Mb,string=262144]
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Some of the documentation references PCI segment ID. For those documents
add a mention of the new PCI device ID.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add a `mergeable` field to `MemoryZoneConfig` so that KSM page merging
can be enabled selectively per memory zone rather than globally for all
guest RAM.
Previously, `MADV_MERGEABLE` was only controllable via the top-level
`--memory mergeable=on` flag, which applied uniformly to all regions.
With this change, users can leave boot memory unmerged while enabling
KSM only on hotplug zones:
--memory size=0,hotplug_method=virtio-mem
--memory-zone id=boot,size=512M,shared=on,mergeable=off
--memory-zone id=hotplug,size=256M,hotplug_size=1G,shared=off,mergeable=on
The `MemoryZone` runtime struct now carries the `mergeable` flag so
that both `allocate_address_space` and `add_ram_region` can apply
per-zone `MADV_MERGEABLE` instead of the global `self.mergeable`.
The top-level `--memory mergeable=on` path continues to work unchanged:
the default zone is synthesised from `MemoryConfig` and inherits its
`mergeable` value.
AI/LLM disclosure: this patch was co-authored with
GitHub Copilot and Claude Code (Opus 4.6).
Signed-off-by: JP Kobryn <inwardvessel@gmail.com>
Add an option that can be used when restoring to resume the VM. This is
particularly useful when restoring the VM via the direct VMM command
line, when you might not want/have an API socket configured.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add a MemoryRestoreMode enum (Copy | OnDemand) to RestoreConfig so
the restore path can be selected at restore time. Copy preserves the
existing eager read-copy behavior. OnDemand enables userfaultfd-based
demand paging and fails restore if the kernel does not support it.
Validate that prefault=on is not combined with OnDemand mode.
Update the OpenAPI spec with the new enum field.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Add docs/disk_locking.md explaining advisory OFD locking, the
lock_granularity parameter, byte-range vs whole-file semantics,
and fallback behavior.
Signed-off-by: Victor Vieux <vieux@repl.it>
Add a core_scheduling option to --cpus with three modes of operation.
This feature takes advantage of a kernel feature that restricts
scheduling of processes on the SMT threads on the same core. This is
useful for mitigating certain classes of side-channel attacks and has
better performance that disabling SMT on the CPU.
- vm (default): All vCPU threads share one core scheduling cookie.
They may be co-scheduled on SMT siblings while host threads are
excluded - this has minimal performance impact and can even
potentially improve performance from co-location.
- vcpu: Each vCPU gets a unique cookie preventing any two vCPUs from
sharing SMT siblings. This has the strongest isolation but at some
compromise of performance.
- off: No core scheduling applied (old behaviour).
This isolation is done by the kernel maintaining a "cookie" - threads
with the same cookie can share the same core.
In vCPU mode each vCPU thread the cookie is created when the thread
starts and each gets a unique cookie. For VM mode the first vCPU thread
(the leader) will create the cookie. All other vCPU threads started (via
hotplug or during boot) will have that cookie shared to it.
EINVAL/ENODEV from prctl is silently ignored so this works transparently
on kernels older than 5.14 that lack PR_SCHED_CORE or when SMT disabled.
Full details of this kernel feature can be found at:
https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html
This implementation was inspired by crosvm's implementation - in
particular the enable_core_scheduling() function.
This is challenging to test via integration testing but the logging of
the received cookie shows it working:
VM case:
cloud-hypervisor: 0.243102s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0x33e4c167
cloud-hypervisor: 0.243102s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x33e4c167
vCPU case:
cloud-hypervisor: 0.089356s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x13993ad6
cloud-hypervisor: 0.089380s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0xd48e86e
Signed-off-by: Rob Bradford <rbradford@meta.com>
Document device_id parameter in NumaConfig, automatic
guest_numa_id assignment, default NUMA distances and
restrictions on Generoc Initiator NUMA nodes
Add numa configuration examples with GPU device and distance
relationships.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
build_edk2() module in scripts/common-aarch64.sh
does not produce the UEFI firmware for aarch64 as
the commits used to assemble sources for acpica,
edk2-platforms and edk2 do not compile after GCC
version upgraded from 11.4.0 to 13.3.0 in the
developer container (ubuntu 22.04 to 24.04)
Apply minimum upgrade to EDK2_REPO and ACPICA_REPO
required to compile with GCC 13.3.0
while still assuring guest VM boot for all
integration tests
BaseTools: Brotli compression submodule that was
previously failing has been fixed following commit
bump
Developers can now produce UEFI firmware for
aarch64 using the following commands
```
./scripts/dev_cli.sh shell
source scripts/test-util.sh
source scripts/common-aarch64.sh
build_edk2
```
Update docs/uefi.md
Fixes#7608
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Minor modifications were made to make the sentences sound more natural.
Also fixed some parameter usage issues in bash code block.
Signed-off-by: Yi Wang <foxywang@tencent.com>
There are some syntax and format issues in tdx/sev documents.
Make some modification to make the description more natural.
And the link of SEV-SNP is invalid, fix it.
Signed-off-by: Yi Wang <foxywang@tencent.com>
There are some minor syntax and command issues in debug-port document.
As commit 5febdec81a (vmm: Enable `gdbstub` on AArch64) supported
aarch64, the docs should keep consistent.
Signed-off-by: Yi Wang <foxywang@tencent.com>
Fix some minor syntax issues in api/building document to make
the sentences more fluent and easy to read.
Signed-off-by: Yi Wang <foxywang@tencent.com>
Some description in the device document were inconsistent with the
source code. Also fix some syntax issues to make the sentences more
fluent.
Signed-off-by: Yi Wang <foxywang@tencent.com>
This patch updates the documentation to reflect the newly added
nested CPU feature option in the CLI.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The API documentation tells users to expect a message when Cloud
Hypervisor is launched, this message was removed in commit 13724db
This change updates the documentation to reflect how the program
actually functions, which is to say no message.
Signed-off-by: Ariel Chenet <achenet@fastmail.com>
This commit removes the SGX support from cloud hypervisor. SGX support
was deprecated in May as part of #7090.
Signed-off-by: Shubham Chakrawar <schakrawar@crusoe.ai>
Here we add the fw_cfg device as a legacy device to the device manager.
It is guarded behind a fw_cfg flag in vmm at creation of the
DeviceManager. In this cl we implement the fw_cfg device with one
function (signature).
Signed-off-by: Alex Orozco <alexorozco@google.com>
The current documentation only mentions Windows
server support. Since only minimal adjustments
are needed for Windows 11 support, these were
added where applicable.
Also contains whitespace fixes, adds missing `\`
in example code and adds more cross references.
Signed-off-by: Lucas Grosche <lucas.grosche@cyberus-technology.de>
