size_of is part of std::prelude as of Rust 1.80 (with size_of_val,
align_of, align_of_val), and the workspace MSRV is 1.89, so qualifying
it (mem::size_of, std::mem::size_of, core::mem::size_of) is unnecessary.
Convert every qualified size_of call-site to the bare prelude form and
drop the now-redundant `use std::mem::size_of;` imports, keeping
`use std::mem;` where it still serves non-prelude items (transmute,
swap, replace, take, zeroed, MaybeUninit, offset_of). size_of is the
only one of the four currently used in the tree.
Pure refactor, no behavioural change. Follow-up to the
clippy::absolute_paths cleanup (#7670), as discussed in #8444.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Import the modules used in the crate instead of spelling the full paths
at every use site, and drop the now-unnecessary crate-level
generated msr_index.rs was trimmed separately.
Signed-off-by: Henry Hrvoje Tonkovac <htonkovac@gmail.com>
Assisted-by: Claude:Opus-4.8
Restoring a snapshot (snapshot restore or live-migration receive) sets
each vCPU's TSC by writing MSR_IA32_TSC as the vCPU is created.
However because CpuManager creates and restores vCPUs one at a time, the
host TSC advances between the per-vCPU writes and KVM derives a slightly
different TSC offset for each vCPU.
KVM only engages its masterclock when every offset matches. This has a
side effect of breaking the HyperV TSC reference clock page resulting in
significantly reduced performance on Windows.
After restore synchronise all vCPU's TSC offset to the boot vCPU's via
the KVM_VCPU_TSC_CTRL device attribute group (Linux 5.16+) this allows
the KVM TSC masterclock to engage and mitigates performance issues with
the KVM HyperV emulation.
See: #8383
Signed-off-by: Rob Bradford <rbradford@meta.com>
Assisted-by: Claude <claude-opus-4-8>
Currently, Cloud Hypervisor round-trips CNTVCT_EL0 through
KVM_GET_REG_LIST/SET_ONE_REG, which leaves a cold-restored or migrated
guest behind real UTC by the downtime. Same-host pause/resume
self-corrects (the physical counter keeps running across the pause), so
only restore and migration cases required the clock to catch up to wall
clock time.
Since ARM has no kernel helper, compute the difference in wall clock
time and compute the ticks so that it can advance the CNTVCT correctly.
It is set via vcpu0 only as it affects a single VM wide value after
Linux 6.4. For older kernels, it was a truly vcpu value which needs to
be invoked for every vcpu.
Gated on all(target_arch = "aarch64", feature = "kvm"); x86 is
unchanged.
Basic manual test case (aarch64 + KVM) verified both in intra host and
inter host snapshot save/restore:
1. Boot a Linux guest; in the guest, `date -u` tracks the host's UTC.
2. Pause and snapshot the VM (ch-remote pause; ch-remote snapshot
file:///<dir>).
3. Leave it down for several minutes (the off-host interval).
4. Restore and resume into a fresh VMM (ch-remote restore
source_url=file:///<dir>,resume=true).
5. In the guest, run `date -u` again and compare to the host: the guest
now tracks current UTC, having advanced by ~the time it spent down.
Before this change the restored guest reads behind real UTC by the
downtime; after it, the guest clock is back in sync (to within the
snapshot-to-restore sampling slop).
Signed-off-by: Atish Patra <atishp@meta.com>
Unlike x86, ARM64 has no kvmclock support to sync guest time upon
required. However, the guest reads the architected virtual timer
(CNTVCT_EL0) directly which can be modified by the VMM to update the
time after snapshot restore. Since the CNTVCT is in ticks, we also need
to read CNTFRQ (via mrs due to lack of ONEREG interface) to compute the
ticks from wall clock difference.
Because the counter is a vCPU register, the capture must run with the
vCPUs quiesced, so the VMM now captures the clock just after
cpu_manager.pause() through the boot vCPU. This is behaviorally
identical for x86, whose clock is VM-wide. There is no restore/advance
yet, so aarch64 guests still resume behind real time until the following
commit.
Signed-off-by: Atish Patra <atishp@meta.com>
Preserving the guest clock across pause/resume and snapshot/restore is
currently open-coded in the VMM against the x86-only
get_clock/set_clock. aarch64 needs the same correction but via a
different mechanism (i.e. the architected counter, CNTVCT). Having a
common backend-agnostic interface that VMM can drive uniformly allows us
to keep the architecture details behind the Hypervisor abstraction.
This commit only introduces the abstraction while the future commits
will actually move the implementation to use it.
Use this opportunity to fix the full path to get SystemTime as well.
Suggested-by: Sebastien Boeuf <sboeuf@meta.com>
Signed-off-by: Atish Patra <atishp@meta.com>
The RISC-V device tree's timebase-frequency was hardcoded to 10 MHz
(0x989680). Actual hardware uses different frequencies.
Read the timebase frequency from KVM_GET_ONE_REG via
KVM_REG_RISCV_TIMER (offset 0, kvm_riscv_timer.frequency),
thread it through the VMM to arch to FDT layers, and fall back to
the 10 MHz default when KVM returns no value.
Signed-off-by: Meng Zhuo <mengzhuo@iscas.ac.cn>
The number of wired interrupt sources (SRCS) must be less than the
KVM device's maximum interrupt identities (kvm_riscv_aia_max_ids).
Platforms with smaller IMSIC capacity reject values that exceed this
limit.
Query KVM_DEV_RISCV_AIA_CONFIG_IDS before setting SRCS and clamp the
requested value to the reported nr_ids (which equals max_ids - 1).
Signed-off-by: Meng Zhuo <mengzhuo@iscas.ac.cn>
Remove stale #[allow]s whose lints no longer fire, convert the
unconditionally-firing ones to #[expect], and keep the conditional
ones as #[allow] (e.g. large_enum_variant only fires when both kvm
and mshv are enabled; a nonminimal_bool only on x86). The many
unreachable_patterns allows are feature-gated and left as #[allow].
Part of #8326.
Signed-off-by: Tushar Khatri <hello@tusharkhatri.in>
Since igvm is a required feature of sev_snp and also sev_snp is x86-64
only the cfg attributes at build time can be consolidated & simplified.
Signed-off-by: Rob Bradford <rbradford@meta.com>
KVM will emulate XSAVE for us, so we need only to skip XCRS
setting/retrieval if the respective CPU feature is not available.
Signed-off-by: Nikolas Kyx <55556836+nyx191@users.noreply.github.com>
The RISC-V AIA FDT node currently advertises a fixed riscv,num-ids
value. That can diverge from the interrupt identity count configured by
KVM, which matters for guests running with an emulated IMSIC.
Record the NR_IDS value reported by KVM and expose that value through
the generated device tree. Read back the KVM-selected AIA mode without
forcing an emulation mode.
Assisted-by: OpenAI-Codex:GPT-5
Signed-off-by: wangyf0611 <wangyufeng@iscas.ac.cn>
KVM initializes RISC-V vCPUs with sstateen0 cleared. When AIA is
exposed to the guest, Linux touches supervisor AIA CSRs while bringing
up the IMSIC path, and those accesses fail if the stateen bits remain
disabled.
Program sstateen0 for newly created vCPUs so the guest can use the
supervisor interrupt state needed by AIA.
Assisted-by: OpenAI-Codex:GPT-5
Signed-off-by: wangyf0611 <wangyufeng@iscas.ac.cn>
When SVE is enabled, KVM replaces the FPSIMD V-registers with wider
SVE Z-registers. Attempting to access the old FPSIMD offsets returns
EINVAL.
Fix by classifying each register from KVM_GET_REG_LIST as core, system,
or extended. Extended registers (currently SVE only) are saved as
generic `ExtendedReg` entries split into `pre_finalize_regs` (registers
like SVE VLS that must be written before `vcpu_finalize`) and
`extended_regs`. FPSIMD registers are only accessed when SVE is absent.
Unrecognized register families error immediately so future extensions
like SME fail clearly rather than silently losing state.
The snapshot is deserialized before vCPU init to make pre-finalize
register state available for the init -> VLS -> finalize ordering
required by KVM.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Move the inline FPSIMD register read/write code from `get_regs()` and
`set_regs()` into dedicated `get_fpsimd_regs()` and `set_fpsimd_regs()`
methods on `KvmVcpu`.
This helps keep the larger `{get,set}_regs()` easier to understand and
prepares them for a subsequent commit that needs to conditionally
skip FPSIMD access when SVE registers are used instead.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
When an SEV-SNP guest transitions pages from private to shared via
KVM_HC_MAP_GPA_RANGE, punch holes in the corresponding guest_memfd
backing it. Without this the balloon driver's `set_memory_decrypted()`
path transitions the page attributes but the physical memory stays
pinned in guest_memfd, making virtio-balloon ineffective for memory
overcommit with confidential VMs. Even without ballooning these pages
are unused by the guest so consume resources. This mirrors the hole
punching that the balloon device does for releasing pages.
The memory_slots Arc is cloned into each KvmVcpu at creation so the
punch can happen in the vcpu thread.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Rob Bradford <rbradford@meta.com>
Replace the bare OwnedFd map (guest_memfds) with a KvmMemorySlot struct
wrapped in an Arc so it can later be shared with KvmVcpu. This is a
pure refactor with no functional change; KvmMemorySlot currently holds
only the guest_memfd OwnedFd.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Rob Bradford <rbradford@meta.com>
When a triple fault happens [0], we now get at least a log message. This
helps to better understand the root cause of sudden reboots.
Broader context: We experience reboots caused by triple faults in
edk2 (6 months old as well as recent). They happen so early in the boot
that one doesn't really see them without looking at the VMM log. An
automatic system reset plus reboot often hides these situations - now
they are at least more visible in the log.
PS: Printing the registers to get more debugging help doesn't help, as
the guest already triple-faulted - the CPU state of the root cause
doesn't exist anymore.
[0] https://elixir.bootlin.com/linux/v6.18.6/source/arch/x86/kvm/x86.c#L11123
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add KvmSevSnpIdBlock and KvmSevSnpIdAuth structs matching the AMD
SEV-SNP Firmware ABI Spec (Rev 1.58), and build them from the IGVM
SNP ID block directive during launch finish. This properly populates
id_block_uaddr/id_auth_uaddr in KVM_SEV_SNP_LAUNCH_FINISH and derives
auth_key_en from the assembled author key, matching QEMU's behavior.
Thread the guest policy from sev_snp_init to launch_finish via an
atomic on KvmVm so the ID block gets the correct policy value.
Also track has_snp_id_block in IgvmLoadedInfo to enable the ID block
based on whether the IGVM file actually contains one, rather than
hardcoding it for KVM.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Adapt the IGVM loader to work with both MSHV and KVM backends, which
differ in page type constants, CPUID page layout, and VMSA handling.
Abstract page types into a PageTypeConfig struct populated at runtime
from the detected hypervisor, replacing hardcoded mshv_bindings constants.
Apply the VMSA register state to each vCPU via setup_sev_snp_regs(),
translating SevSelector attributes to KVM segment format using a bitfield
decoder.
KVM's SNP launch path sanitizes certain CPUID bits that could lead to
an insecure guest. If the VMM sets these bits, KVM rejects the CPUID
page import on the first attempt, requiring a retry with the
firmware-corrected values.
Pre-clear the known problematic bits before import to avoid the
reject-and-retry cycle:
- Leaf 0x1, ECX bit 24: TSC_DEADLINE (filtered by KVM)
- Leaf 0x7, EBX bit 1: SGX (filtered by KVM)
- Leaf 0x7, EDX: clear entirely (contains speculative features)
- Leaf 0x80000008, EBX bit 25: filtered by KVM
- Leaf 0x80000021, ECX: clear entirely
This keeps the CPUID page stable across launch updates and avoids
noisy error logs from the retry path.
Co-authored-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Alex Orozco <aorozco@google.com>
Co-authored-by: Dylan Reid <dgreid@fb.com>
Signed-off-by: Dylan Reid <dgreid@fb.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
During SNP boot all guest RAM is initially marked
KVM_MEMORY_ATTRIBUTE_PRIVATE. Pages imported via SNP_LAUNCH_UPDATE are
properly accepted by the guest, but generic RAM pages (e.g. the AP
trampoline at GPA 0xD000) are not. When stage0 on the BSP starts
secondary vCPUs via x2APIC, the APs try to execute from the trampoline
page through the shared mapping while KVM still has it marked private,
causing a KVM_EXIT_MEMORY_FAULT (flags=KVM_MEMORY_EXIT_FLAG_PRIVATE)
that previously fell through to the catch-all error, killing the VM.
Handle VcpuExit::MemoryFault by toggling the page's memory attribute
between private and shared based on the exit flags, allowing the vCPU
to retry the access.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
SEV-SNP guests will issue this hypercall to signal a change in the page
encryption status to the hypervisor.
Handle VcpuExit::Hypercall in the KVM vCPU run loop: decode the GPA,
page count, and private/shared attribute from the hypercall arguments,
then call KVM_SET_MEMORY_ATTRIBUTES to update the page state.
Co-authored-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Add the KVM_SEV_SNP_LAUNCH_FINISH ioctl, which finalizes the SNP
launch sequence and transitions the VM into a runnable encrypted
state.
Additionally, add KVM_SEV_SNP_LAUNCH_FINISH to the seccomp allowlist.
Co-authored-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Implement the KVM_SEV_SNP_LAUNCH_UPDATE ioctl.
Extend Vm::import_isolated_pages() with a uaddrs parameter carrying
host virtual addresses, which KVM needs, unlike MSHV. Compute uaddrs
from guest memory mappings in the IGVM loader.
Add KVM_SEV_SNP_LAUNCH_UPDATE to the seccomp allowlist.
Co-authored-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Introduce the SevFd abstraction that wraps /dev/sev and implements the
KVM_SEV_INIT2 and KVM_SEV_SNP_LAUNCH_START ioctls for SEV-SNP VM
initialization on KVM.
Key changes:
- Add sev.rs with KvmSevInit and KvmSevSnpLaunchStart ioctl structs
matching the kernel layout (linux/arch/x86/include/uapi/asm/kvm.h)
- Implement KVM_SEV_INIT2 and KVM_SEV_SNP_LAUNCH_START ioctls
- Set KVM_MEMORY_ATTRIBUTE_PRIVATE on newly created memory regions
when guest_memfd is supported
- Widen SevSnpPageAccessProxy cfg gates from mshv-only to all
sev_snp-enabled builds
- Make sev_snp_init a required trait method (remove default impl)
- Include KVM_SEV_SNP_LAUNCH_START in the seccomp allowlist
- Parse VMSA SEV features from IGVM and include them in the
KVM_SEV_INIT2 ioctl
Co-authored-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Alex Orozco <aorozco@google.com>
Co-authored-by: Rob Bradford <rbradford@meta.com>
Signed-off-by: Rob Bradford <rbradford@meta.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Add support for guest_memfd (available in Linux kernel v6.8+), which
enables private memory for confidential VMs.
Key changes:
- Introduce UserMemoryRegion abstraction with guest_memfd fields
- Add From impls between kvm_userspace_memory_region2 and UserMemoryRegion
- Convert all KVM memory region operations from kvm_userspace_memory_region
to kvm_userspace_memory_region2, with automatic fallback to v1 when
guest_memfd is not supported
- Add set_user_memory_region() wrapper that dispatches to v1/v2 based on
kvm_guest_memfd_supported capability
- Create guest_memfd via KVM_CREATE_GUEST_MEMFD ioctl when supported
- Extend KvmDirtyLogSlot to preserve region2 fields across dirty log
start/stop cycles
This is prerequisite infrastructure for KVM-based confidential computing
that requires private guest memory backed by guest_memfd.
Co-authored-by: Alex Orozco <aorozco@google.com>
Signed-off-by: Keith Adler <kadler@cloudflare.com>
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
KVM_GET_MSR_INDEX_LIST does not consistently include all
guest-programmable MTRR MSRs.
During save/restore while booting, the VMM initially sets only
MSR_MTRRdefType, then guest firmware or other early boot code can
program additional MTRR state before the snapshot is taken. If those
MSRs are missing from the vCPU MSR buffer, snapshot omits part of the
guest's MTRR configuration and restore resumes with an incomplete
MTRR map.
Add the guest-programmable MTRR MSRs to the KVM MSR index list used
to build the vCPU MSR buffer so the existing snapshot/restore path
preserves the guest's MTRR state.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Bump to the released versions that are compatible wherever possible but
for the vhost and vfio crates they are git hashes as no releases with
compatible versions have yet been made.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Nightly rustfmt now prefers `self` re-exports inline rather
than a separate 'pub use {kvm_bindings, kvm_ioctls}' line.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The TILE data state of AMX may require 8KB+ space, calling the legacy
KVM_GET_XSAVE will encounter an error since KVM_GET_XSAVE only can get
4KB space. This patch adds KVM_GET_XSAVE2 support to allow snapping more
data.
Fixes: #7533
Signed-off-by: Songqian Li <sionli@tencent.com>
The MSRs are constant at boot time so rather than creating a vector in
the boot_msr_entries() method instead reaturn a reference to static MSR
array data.
Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
This is a follow-up of [0].
# Advantages
- This saves dozens of unneeded clone()s across the whole code base
- Makes it much easier to reason about how parameters are used
(often we passed owned Arc/Rc versions without actually needing
ownership)
# Exceptions
For certain code paths, the alternatives would require awkward or overly
complex code, and in some cases the functions are the logical owners of
the values they take. In those cases, I've added
#[allow(clippy::needless_pass_by_value)].
This does not mean that one should not improve this in the future.
[0] 6a86c157af
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
As well as saving the MSRs as it is currently does ensure that the KVM
capability is enabled along with keeping the internal state updated.
Signed-off-by: Rob Bradford <rbradford@rivosinc.com>
Co-authored-by: Chengyu Fu <chengyu.fu@linux.alibaba.com>
It takes a pointer to a userspace address that it accesses, so it should
be marked unsafe. This was missed earlier.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
Also drop support for building the TDX code for 32-bit targets. All
CPUs with TDX support are 64-bit so supporting 32-bit targets is not
needed.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
To ensure that struct sizes are the same on 32-bit and 64-bit, various
kernel APIs use __u64 (Rust u64) to represent userspace pointers.
Userspace is expected to cast pointers to __u64 before passing them to
the kernel, and cast kernel-provided __u64 to a pointer before using
them. However, various safe APIs in Cloud Hypervisor took
caller-provided u64 values and passed them to syscalls that interpret
them as userspace addresses. Therefore, passing bad u64 values would
cause memory disclosure or corruption.
Fix the bug by using usize and pointer types as appropriate. To make
soundness of the code easier to reason about, the PCI code gains a new
MmapRegion abstraction that ensures the validity of pointers. The rest
of the code already has an MmapRegion abstraction it can use. To avoid
having to reason about whether something is keeping the MmapRegion
alive, reference counting is added. MmapRegion cannot hold references
to other objects, so the reference counting cannot introduce cycles.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
To ensure that struct sizes are the same on 32-bit and 64-bit, various
kernel APIs use __u64 (Rust u64) to represent userspace pointers.
Userspace is expected to cast pointers to __u64 before passing them to
the kernel, and cast kernel-provided __u64 to a pointer before using
them. However, various safe APIs in Cloud Hypervisor took
caller-provided u64 values and passed them to syscalls that treat them
as userspace addresses. Therefore, passing bad u64 values would cause
memory disclosure or corruption. The memory region APIs are one example
of this, so mark them as unsafe.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
This better aligns with the rest of the code and makes it clearer
that these tests can run "as is" in a normal hosted environments
without the special test environment.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Consuming `&Arc<T>` as argument is almost always an antipattern as it
hides whether the callee is going to take over (shared) ownership
(by .clone()) or not. Instead, it is better to consume `&dyn T` or
`Arc<dyn T>` to be more explicit. This commit cleans up the code.
The change is very mechanic and was very easy to implement across the
code base.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
