cloud-hypervisor/docs/testing.md

Testing

• Testing
  • Overview
  • Prerequisites
  • The dev_cli.sh entry point
    • Global flags
    • Building
    • Running tests
    • Argument passthrough
    • Custom kernel and firmware
  • Unit tests
  • Integration tests
    • x86_64
    • ARM64
    • VFIO
    • Windows guests
    • Rate limiter
    • Confidential VMs
  • Performance metrics
  • Code coverage
  • CI workflows

Overview

All Cloud Hypervisor builds and tests run inside a Docker container to provide a reproducible environment. The main entry point is scripts/dev_cli.sh, which manages the container lifecycle and forwards arguments to the appropriate test scripts.

The container image is published at ghcr.io/cloud-hypervisor/cloud-hypervisor and is automatically pulled on first use. A local build of the container can be triggered with scripts/dev_cli.sh build-container or by passing the --local flag.

Test workloads (guest images, kernels, firmware) are stored on the host under $HOME/workloads and bind-mounted into the container at /root/workloads. Most test scripts download missing workloads automatically on first run.

Prerequisites

A working Docker (or Podman) installation and access to /dev/kvm (or /dev/mshv for Microsoft Hypervisor tests) are required. The host must be running Linux on x86_64 or aarch64.

# Verify KVM is available
ls -l /dev/kvm

The container image bundles all build dependencies. No Rust toolchain is needed on the host.

The dev_cli.sh entry point

scripts/dev_cli.sh [flags] <command> [<command args>]

Global flags

Flag	Description
--local	Build and use a local container image instead of pulling from the registry.

Building

scripts/dev_cli.sh build [--debug|--release] [--libc musl|gnu] \
    [--hypervisor kvm|mshv] [--features <features>] \
    [--volumes /host:/ctr#...] [-- <cargo args>]

Flag	Default	Description
--debug	yes	Build debug binaries.
--release		Build release binaries.
--libc	gnu	C library to link against (musl/gnu).
--hypervisor	auto	Hypervisor backend (kvm/mshv). Auto-detected from the host device node when omitted.
--features		Additional cargo features.
--volumes		Extra host volumes (/a:/a#/b:/b).
--runtime	docker	Container runtime (docker/podman).

Arguments after -- are forwarded directly to cargo build.

Running tests

scripts/dev_cli.sh tests [<test type>] [--libc musl|gnu] \
    [--hypervisor kvm|mshv] [--volumes /host:/ctr#...] \
    [-- <script args> [-- <binary args>]]

Test type flags:

Flag	Description
--unit	Run unit tests.
--integration	Run integration tests (includes live migration).
--integration-vfio	Run VFIO integration tests.
--integration-windows	Run Windows guest integration tests.
--integration-rate-limiter	Run rate limiter integration tests.
--integration-cvm	Run confidential VM integration tests.
--metrics	Generate performance metrics.
--coverage	Generate code coverage.
--all	Run both unit and integration tests.

Configuration flags:

Flag	Default	Description
--libc	gnu	C library to link against (musl/gnu).
--hypervisor	auto	Hypervisor backend (kvm/mshv). Auto-detected from the host device node when omitted.
--volumes		Extra host volumes (/a:/a#/b:/b).

Argument passthrough

The -- separator creates layers of argument forwarding:

dev_cli.sh tests <flags> -- <script args> -- <binary args>

1. Everything before the first -- is consumed by dev_cli.sh.
2. Everything between the first and second -- is forwarded to the test script (e.g., run_integration_tests_x86_64.sh).
3. Everything after the second -- is forwarded to the test binary itself (e.g., performance-metrics).

The test scripts accept the following common arguments via process_common_args() in scripts/test-util.sh:

Argument	Description
--hypervisor kvm|mshv	Override the hypervisor (auto-detected from the host device node when omitted; also passed by dev_cli.sh).
--test-filter <name>	Run only tests matching the filter.
--test-exclude <name>	Exclude tests matching the pattern.
--build-guest-kernel	Build the guest kernel from source instead of downloading a prebuilt binary.

Example — run a single integration test:

scripts/dev_cli.sh tests --integration \
    -- --test-filter test_boot_from_virtio_pmem

Example — run metrics excluding micro-benchmarks:

scripts/dev_cli.sh tests --metrics \
    -- --test-exclude micro_ \
    -- --report-file /root/workloads/metrics.json

Custom kernel and firmware

The following environment variables allow overriding the default guest kernel or firmware binaries. Each variable is independent; set any combination without affecting the others.

Variable	Description
CH_CUSTOM_KERNEL	Path to a custom vmlinux (x86_64) or Image (aarch64) kernel binary.
CH_CUSTOM_FIRMWARE	Path to a custom hypervisor-fw firmware binary.
CH_CUSTOM_OVMF	Path to a custom OVMF binary (CLOUDHV.fd on x86_64, CLOUDHV_EFI.fd on aarch64).

The paths refer to locations on the host. Before launching the Docker container, dev_cli.sh copies the referenced files into $HOME/workloads at the default names the test scripts expect (e.g., vmlinux-x86_64, hypervisor-fw, CLOUDHV.fd). Because the workloads directory is bind-mounted into the container, the existing download-if-missing guards inside the test scripts automatically skip the network fetch.

# Use a custom kernel for integration tests
CH_CUSTOM_KERNEL=/path/to/vmlinux \
    scripts/dev_cli.sh tests --integration

# Override all three
CH_CUSTOM_KERNEL=/path/to/vmlinux \
CH_CUSTOM_FIRMWARE=/path/to/hypervisor-fw \
CH_CUSTOM_OVMF=/path/to/CLOUDHV.fd \
    scripts/dev_cli.sh tests --integration

Unit tests

scripts/dev_cli.sh tests --unit [--libc musl|gnu] [--hypervisor kvm|mshv]

Unit tests run cargo test on the entire workspace in release mode:

cargo test --lib --bins --target <target> --release --workspace
cargo test --doc --target <target> --release --workspace

The container is not fully privileged. It receives --device /dev/kvm (or /dev/mshv), --device /dev/net/tun, and --cap-add net_admin so that tests requiring a hypervisor device or TAP interfaces can run.

When the hypervisor is mshv, the feature flag --features mshv is passed to cargo. On x86_64 with KVM, the tdx feature is additionally enabled if needed (MSHV does not support TDX).

The hypervisor backend is auto-detected from the host device node: /dev/mshv selects MSHV and /dev/kvm selects KVM. Pass --hypervisor kvm|mshv to override the detection explicitly.

Integration tests

All integration test containers run with --privileged and full access to /dev. Workloads are bind-mounted from the host.

Test scripts use cargo nextest run with the --release profile. Tests are grouped by module name filter (e.g., common_parallel, common_sequential). Most groups support automatic retries (default 3). The nextest configuration in .config/nextest.toml sets a slow-timeout of 60 seconds per period with automatic termination after 10 periods (i.e., any single test running longer than 10 minutes is terminated).

The Rust test modules are defined in cloud-hypervisor/tests/integration.rs:

Module	Description
common_parallel	Main tests run in parallel, including live migration.
common_sequential	Tests requiring serial execution (snapshot/restore, OVS-DPDK live migration).
dbus_api	D-Bus API tests (requires dbus_api feature).
fw_cfg	Firmware configuration tests (requires fw_cfg feature).
ivshmem	Inter-VM shared memory tests (requires ivshmem feature).
aarch64_acpi	ARM64 ACPI-specific tests.
windows	Windows guest tests.
vfio	VFIO passthrough tests (x86_64 only).
rate_limiter	Network/block rate limiting tests.

Confidential VM tests are in a separate file (cloud-hypervisor/tests/integration_cvm.rs) under the common_cvm module.

x86_64

scripts/dev_cli.sh tests --integration [--libc musl|gnu]

Runs scripts/run_integration_tests_x86_64.sh. The script automatically downloads and prepares all required workloads:

• Kernel: vmlinux-x86_64 (prebuilt or built from source with --build-guest-kernel).
• Firmware: hypervisor-fw (rust-hypervisor-firmware).
• OVMF: CLOUDHV.fd.
• Guest images: Ubuntu Focal and Jammy cloud images in multiple formats (raw, qcow2, compressed, backing-file variants).
• Alpine: Minirootfs and initramfs for lightweight boot tests.
• virtiofsd: Built from source for virtio-fs tests.

System tuning applied inside the container:

• KSM (Kernel Same-page Merging) is enabled.
• 2 MiB hugepages are allocated for DPDK/VDPA tests.
• The open file descriptor limit is raised to 4096.

Test groups executed: common_parallel, common_sequential, dbus_api, fw_cfg, ivshmem.

The dbus_api, fw_cfg, and ivshmem groups are each built with their respective cargo feature enabled before running.

Live migration tests (test_live_migration_*, test_live_upgrade_*) run as part of the common_parallel and common_sequential groups.

ARM64

scripts/dev_cli.sh tests --integration --libc musl

Runs scripts/run_integration_tests_aarch64.sh. The workload setup mirrors x86_64 with aarch64-specific images:

• Kernel: Image-arm64.
• OVMF: CLOUDHV_EFI.fd.
• Guest images: Focal and Jammy aarch64 cloud images.

Test groups executed: common_parallel, common_sequential, aarch64_acpi, dbus_api, fw_cfg, ivshmem.

VFIO

scripts/dev_cli.sh tests --integration-vfio

Runs scripts/run_integration_tests_vfio.sh. Requires dedicated hardware with an Nvidia GPU (Tesla T4) for VFIO passthrough testing.

Test groups executed (single-threaded):

Group	Description
vfio::test_nvidia	Legacy VFIO with container/group interface.
vfio::test_iommufd	VFIO with cdev interface via iommufd.

Windows guests

scripts/dev_cli.sh tests --integration-windows [--libc musl|gnu]

Runs the architecture-appropriate script:

• x86_64: scripts/run_integration_tests_windows_x86_64.sh
• aarch64: scripts/run_integration_tests_windows_aarch64.sh

Both scripts require a pre-downloaded Windows guest image in the workloads directory. Device mapper snapshots are created to allow concurrent test runs without corrupting the base image.

Workloads:

Architecture	Image	Firmware
x86_64	windows-server-2025-amd64-1.raw	CLOUDHV.fd
aarch64	windows-11-iot-enterprise-aarch64.raw	CLOUDHV_EFI.fd

Test group: windows (single-threaded, retries 3).

Rate limiter

scripts/dev_cli.sh tests --integration-rate-limiter

Runs scripts/run_integration_tests_rate_limiter.sh. Downloads Jammy guest images and a prebuilt kernel.

Test group: rate_limiter (single-threaded, no retries).

Confidential VMs

scripts/dev_cli.sh tests --integration-cvm

Runs scripts/run_integration_tests_cvm.sh. Builds with --features mshv,igvm,sev_snp and requires IGVM files to be present at /usr/share/cloud-hypervisor/cvm on the host.

Test group: common_cvm (nproc / 4 threads, retries 3).

KVM SEV-SNP

scripts/dev_cli.sh tests --integration-cvm --hypervisor kvm

With --hypervisor kvm the script builds with --features kvm,igvm,sev_snp,fw_cfg. On KVM the IGVM is an Oak stage0 firmware image and the guest kernel is supplied separately (read by stage0 over fw_cfg); the harness selects this boot model when a guest kernel is present at /igvm_files/bzImage. When no --test-filter is given it runs the full common_cvm set.

Prerequisites:

• An AMD SEV-SNP-capable KVM host with /dev/kvm and /dev/sev present and SNP enabled.
• A KVM-bootable stage0 IGVM file at /usr/share/cloud-hypervisor/cvm and a guest kernel at /igvm_files/bzImage.

To scope the run to a single test explicitly:

scripts/dev_cli.sh tests --integration-cvm --hypervisor kvm \
    -- --test-filter test_jammy_simple_launch

Performance metrics

scripts/dev_cli.sh tests --metrics [-- <script args> [-- <binary args>]]

Runs scripts/run_metrics.sh, which builds and executes the performance-metrics binary. The binary produces a JSON report with boot time, throughput, and latency measurements.

Useful arguments:

# Exclude micro-benchmarks and write results to a file
scripts/dev_cli.sh tests --metrics \
    -- --test-exclude micro_ \
    -- --report-file /root/workloads/metrics.json

Code coverage

scripts/dev_cli.sh tests --coverage

Runs scripts/run_coverage.sh, which instruments the build with LLVM source-based code coverage, executes the test suite via dbus-run-session, and produces either an LCOV or HTML report. See coverage.md for details on collecting and viewing coverage data.