cloud-hypervisor/docs/testing.md

# Testing

- [Testing](#testing)
  - [Overview](#overview)
  - [Prerequisites](#prerequisites)
  - [The dev\_cli.sh entry point](#the-dev_clish-entry-point)
    - [Global flags](#global-flags)
    - [Building](#building)
    - [Running tests](#running-tests)
    - [Argument passthrough](#argument-passthrough)
    - [Custom kernel and firmware](#custom-kernel-and-firmware)
  - [Unit tests](#unit-tests)
  - [Integration tests](#integration-tests)
    - [x86\_64](#x86_64)
    - [ARM64](#arm64)
    - [VFIO](#vfio)
    - [Windows guests](#windows-guests)
    - [Rate limiter](#rate-limiter)
    - [Confidential VMs](#confidential-vms)
  - [Performance metrics](#performance-metrics)
  - [Code coverage](#code-coverage)

## Overview

Cloud Hypervisor's development script runs builds and tests inside a
Docker container to provide a reproducible environment. The main entry
point is `scripts/dev_cli.sh`, which manages the container lifecycle
and forwards arguments to the appropriate test scripts.

The container image is published at
`ghcr.io/cloud-hypervisor/cloud-hypervisor` and is automatically
pulled on first use. A local build of the container can be triggered
with `scripts/dev_cli.sh build-container` or by passing the `--local`
flag.

Test workloads (guest images, kernels, firmware) are stored on the host
under `$HOME/workloads` and bind-mounted into the container at
`/root/workloads`. Most test scripts download missing workloads
automatically on first run.

## Prerequisites

A working Docker (or Podman) installation and access to `/dev/kvm`
(or `/dev/mshv` for Microsoft Hypervisor tests) are required. The
host must be running Linux on x86_64 or aarch64.

```shell
# Verify KVM is available
ls -l /dev/kvm
```

The container image bundles all build dependencies. No Rust toolchain
is needed on the host.

## The dev_cli.sh entry point

```
scripts/dev_cli.sh [flags] <command> [<command args>]
```

### Global flags

| Flag      | Description                                      |
|-----------|--------------------------------------------------|
| `--local` | Build and use a local container image instead of pulling from the registry. |

### Building

```shell
scripts/dev_cli.sh build [--debug|--release] [--libc musl|gnu] \
    [--hypervisor kvm|mshv] [--features <features>] \
    [--volumes /host:/ctr#...] [-- <cargo args>]
```

| Flag           | Default | Description                              |
|----------------|---------|------------------------------------------|
| `--debug`      | yes     | Build debug binaries.                    |
| `--release`    |         | Build release binaries.                  |
| `--libc`       | `gnu`   | C library to link against (`musl`/`gnu`).|
| `--hypervisor` | auto    | Hypervisor backend (`kvm`/`mshv`). Auto-detected from the host device node when omitted. |
| `--features`   |         | Additional cargo features.               |
| `--volumes`    |         | Extra host volumes (`/a:/a#/b:/b`).      |
| `--runtime`    | `docker`| Container runtime (`docker`/`podman`).   |

Arguments after `--` are forwarded directly to `cargo build`.

### Running tests

```shell
scripts/dev_cli.sh tests [<test type>] [--libc musl|gnu] \
    [--hypervisor kvm|mshv] [--volumes /host:/ctr#...] \
    [-- <script args> [-- <binary args>]]
```

**Test type flags:**

| Flag                           | Description                                |
|--------------------------------|--------------------------------------------|
| `--unit`                       | Run unit tests.                            |
| `--integration`                | Run integration tests (includes live migration). |
| `--integration-vfio`           | Run VFIO integration tests.                |
| `--integration-windows`        | Run Windows guest integration tests.       |
| `--integration-rate-limiter`   | Run rate limiter integration tests.        |
| `--integration-cvm`            | Run confidential VM integration tests.     |
| `--metrics`                    | Generate performance metrics.              |
| `--coverage`                   | Generate code coverage.                    |
| `--all`                        | Run both unit and integration tests.       |

**Configuration flags:**

| Flag           | Default | Description                              |
|----------------|---------|------------------------------------------|
| `--libc`       | `gnu`   | C library to link against (`musl`/`gnu`).|
| `--hypervisor` | auto    | Hypervisor backend (`kvm`/`mshv`). Auto-detected from the host device node when omitted. |
| `--volumes`    |         | Extra host volumes (`/a:/a#/b:/b`).      |

### Argument passthrough

The `--` separator creates layers of argument forwarding:

```
dev_cli.sh tests <flags> -- <script args> -- <binary args>
```

1. Everything before the first `--` is consumed by `dev_cli.sh`.
2. Everything between the first and second `--` is forwarded to the
   test script (e.g., `run_integration_tests_x86_64.sh`).
3. Everything after the second `--` is forwarded to the test binary
   itself (e.g., `performance-metrics`).

The test scripts accept the following common arguments via
`process_common_args()` in `scripts/test-util.sh`:

| Argument              | Description                                  |
|-----------------------|----------------------------------------------|
| `--hypervisor kvm\|mshv` | Override the hypervisor (auto-detected from the host device node when omitted; also passed by dev_cli.sh). |
| `--test-filter <name>`| Run only tests matching the filter.           |
| `--test-exclude <name>`| Exclude tests matching the pattern.          |
| `--build-guest-kernel`| Build the guest kernel from source instead of downloading a prebuilt binary. |

**Example — run a single integration test:**

```shell
scripts/dev_cli.sh tests --integration \
    -- --test-filter test_boot_from_virtio_pmem
```

**Example — run metrics excluding micro-benchmarks:**

```shell
scripts/dev_cli.sh tests --metrics \
    -- --test-exclude micro_ \
    -- --report-file /root/workloads/metrics.json
```

### Custom kernel and firmware

The following environment variables allow overriding the default guest
kernel or firmware binaries. Each variable is independent; set any
combination without affecting the others.

| Variable             | Description                                |
|----------------------|--------------------------------------------|
| `CH_CUSTOM_KERNEL`   | Path to a custom `vmlinux` (x86_64) or `Image` (aarch64) kernel binary. |
| `CH_CUSTOM_FIRMWARE` | Path to a custom `hypervisor-fw` firmware binary. |
| `CH_CUSTOM_OVMF`    | Path to a custom OVMF binary (`CLOUDHV.fd` on x86_64, `CLOUDHV_EFI.fd` on aarch64). |

The paths refer to locations on the **host**. Before launching the
Docker container, `dev_cli.sh` copies the referenced files into
`$HOME/workloads` at the default names the test scripts expect
(e.g., `vmlinux-x86_64`, `hypervisor-fw`, `CLOUDHV.fd`). Because
the workloads directory is bind-mounted into the container, the
existing download-if-missing guards inside the test scripts
automatically skip the network fetch.

```shell
# Use a custom kernel for integration tests
CH_CUSTOM_KERNEL=/path/to/vmlinux \
    scripts/dev_cli.sh tests --integration

# Override all three
CH_CUSTOM_KERNEL=/path/to/vmlinux \
CH_CUSTOM_FIRMWARE=/path/to/hypervisor-fw \
CH_CUSTOM_OVMF=/path/to/CLOUDHV.fd \
    scripts/dev_cli.sh tests --integration
```

## Unit tests

```shell
scripts/dev_cli.sh tests --unit [--libc musl|gnu] [--hypervisor kvm|mshv]
```

Unit tests run `cargo test` on the entire workspace in release mode:

```
cargo test --lib --bins --target <target> --release --workspace
cargo test --doc --target <target> --release --workspace
```

The container is _not_ fully privileged. It receives `--device
/dev/kvm` (or `/dev/mshv`), `--device /dev/net/tun`, and `--cap-add
net_admin` so that tests requiring a hypervisor device or TAP
interfaces can run.

When the hypervisor is `mshv`, the feature flag `--features mshv` is
passed to cargo. On x86_64 with KVM, the `tdx` feature is additionally
enabled if needed (MSHV does not support TDX).

The hypervisor backend is auto-detected from the host device node:
`/dev/mshv` selects MSHV and `/dev/kvm` selects KVM. Pass
`--hypervisor kvm|mshv` to override the detection explicitly.

## Integration tests

All integration test containers run with `--privileged` and full
access to `/dev`. Workloads are bind-mounted from the host.

Test scripts use `cargo nextest run` with the `--release` profile.
Tests are grouped by module name filter (e.g., `common_parallel`,
`common_sequential`). Most groups support automatic retries (default
3). The nextest configuration in `.config/nextest.toml` sets a
slow-timeout of 60 seconds per period with automatic termination
after 10 periods (i.e., any single test running longer than 10
minutes is terminated).

The Rust test modules are defined in `cloud-hypervisor/tests/integration.rs`:

| Module               | Description                                        |
|----------------------|----------------------------------------------------|
| `common_parallel`    | Main tests run in parallel, including live migration. |
| `common_sequential`  | Tests requiring serial execution (snapshot/restore, OVS-DPDK live migration). |
| `dbus_api`           | D-Bus API tests (requires `dbus_api` feature).     |
| `fw_cfg`             | Firmware configuration tests (requires `fw_cfg` feature). |
| `ivshmem`            | Inter-VM shared memory tests (requires `ivshmem` feature). |
| `aarch64_acpi`       | ARM64 ACPI-specific tests.                         |
| `windows`            | Windows guest tests.                               |
| `vfio`               | VFIO passthrough tests (x86_64 only).              |
| `rate_limiter`       | Network/block rate limiting tests.                 |

Confidential VM tests are in a separate file
(`cloud-hypervisor/tests/integration_cvm.rs`) under the `common_cvm`
module.

### x86_64

```shell
scripts/dev_cli.sh tests --integration [--libc musl|gnu]
```

Runs `scripts/run_integration_tests_x86_64.sh`. The script
automatically downloads and prepares all required workloads:

- **Kernel:** `vmlinux-x86_64` (prebuilt or built from source with
  `--build-guest-kernel`).
- **Firmware:** `hypervisor-fw` (rust-hypervisor-firmware).
- **OVMF:** `CLOUDHV.fd`.
- **Guest images:** Ubuntu Focal and Jammy cloud images in multiple
  formats (raw, qcow2, compressed, backing-file variants).
- **Alpine:** Minirootfs and initramfs for lightweight boot tests.
- **virtiofsd:** Built from source for virtio-fs tests.

**System tuning applied inside the container:**

- KSM (Kernel Same-page Merging) is enabled.
- 2 MiB hugepages are allocated for DPDK/VDPA tests.
- The open file descriptor limit is raised to 4096.

**Test groups executed:** `common_parallel`, `common_sequential`,
`dbus_api`, `fw_cfg`, `ivshmem`.

The `dbus_api`, `fw_cfg`, and `ivshmem` groups are each built with
their respective cargo feature enabled before running.

Live migration tests (`test_live_migration_*`, `test_live_upgrade_*`)
run as part of the `common_parallel` and `common_sequential` groups.

### ARM64

```shell
scripts/dev_cli.sh tests --integration --libc musl
```

Runs `scripts/run_integration_tests_aarch64.sh`. The workload setup
mirrors x86_64 with aarch64-specific images:

- **Kernel:** `Image-arm64`.
- **OVMF:** `CLOUDHV_EFI.fd`.
- **Guest images:** Focal and Jammy aarch64 cloud images.

**Test groups executed:** `common_parallel`, `common_sequential`,
`aarch64_acpi`, `dbus_api`, `fw_cfg`, `ivshmem`.

### VFIO

```shell
scripts/dev_cli.sh tests --integration-vfio
```

Runs `scripts/run_integration_tests_vfio.sh`. Requires dedicated
hardware with an Nvidia GPU (Tesla T4) for VFIO passthrough testing.

**Test groups executed (single-threaded):**

| Group               | Description                               |
|----------------------|-------------------------------------------|
| `vfio::test_nvidia`  | Legacy VFIO with container/group interface.|
| `vfio::test_iommufd` | VFIO with cdev interface via iommufd.      |

### Windows guests

```shell
scripts/dev_cli.sh tests --integration-windows [--libc musl|gnu]
```

Runs the architecture-appropriate script:
- x86_64: `scripts/run_integration_tests_windows_x86_64.sh`
- aarch64: `scripts/run_integration_tests_windows_aarch64.sh`

Both scripts require a pre-downloaded Windows guest image in the
workloads directory. Device mapper snapshots are created to allow
concurrent test runs without corrupting the base image.

**Workloads:**

| Architecture | Image                                        | Firmware         |
|--------------|----------------------------------------------|------------------|
| x86_64       | `windows-server-2025-amd64-1.raw`            | `CLOUDHV.fd`     |
| aarch64      | `windows-11-iot-enterprise-aarch64.raw`       | `CLOUDHV_EFI.fd` |

**Test group:** `windows` (single-threaded, retries 3).

### Rate limiter

```shell
scripts/dev_cli.sh tests --integration-rate-limiter
```

Runs `scripts/run_integration_tests_rate_limiter.sh`. Downloads Jammy
guest images and a prebuilt kernel.

**Test group:** `rate_limiter` (single-threaded, no retries).

### Confidential VMs

```shell
scripts/dev_cli.sh tests --integration-cvm
```

Runs `scripts/run_integration_tests_cvm.sh`. Builds with `--features
mshv,igvm,sev_snp` and requires IGVM files to be present at
`/usr/share/cloud-hypervisor/cvm` on the host.

**Test group:** `common_cvm` (`nproc / 4` threads, retries 3).

#### KVM SEV-SNP

```shell
scripts/dev_cli.sh tests --integration-cvm --hypervisor kvm
```

With `--hypervisor kvm` the script builds with `--features
kvm,igvm,sev_snp,fw_cfg`. On KVM the IGVM is an Oak stage0 firmware image
and the guest kernel is supplied separately (read by stage0 over fw_cfg);
the harness selects this boot model when a guest kernel is present at
`/igvm_files/bzImage`. When no `--test-filter`
is given it runs the full `common_cvm` set.

Prerequisites:

- An AMD SEV-SNP-capable KVM host with `/dev/kvm` and `/dev/sev` present
  and SNP enabled.
- A KVM-bootable stage0 IGVM file at `/usr/share/cloud-hypervisor/cvm` and
  a guest kernel at `/igvm_files/bzImage`.

To scope the run to a single test explicitly:

```shell
scripts/dev_cli.sh tests --integration-cvm --hypervisor kvm \
    -- --test-filter test_jammy_simple_launch
```

## Performance metrics

```shell
scripts/dev_cli.sh tests --metrics [-- <script args> [-- <binary args>]]
```

Runs `scripts/run_metrics.sh`, which builds and executes the
`performance-metrics` binary. The binary produces a JSON report with
boot time, throughput, and latency measurements.

Useful arguments:

```shell
# Exclude micro-benchmarks and write results to a file
scripts/dev_cli.sh tests --metrics \
    -- --test-exclude micro_ \
    -- --report-file /root/workloads/metrics.json
```

## Code coverage

```shell
scripts/dev_cli.sh tests --coverage
```

Runs `scripts/run_coverage.sh`, which instruments the build with
LLVM source-based code coverage, executes the test suite via
`dbus-run-session`, and produces either an LCOV or HTML report. See
[coverage.md](coverage.md) for details on collecting and viewing
coverage data.