Each Windows integration test boots a full Windows Server guest using
several vCPUs and gigabytes of RAM. nextest runs up to one test per
logical CPU by default, so the whole suite started at once and thrashed
the host.
Add a dedicated 'windows' nextest profile that inherits from
integration and assigns the tests to a 'windows' test group capped at
four concurrent runs. Point the Windows test scripts at the profile and
drop the redundant retries flag, which now comes from the profile.
Running the tests in parallel is safe because every guest gets a unique
network via next_guest_id, so there is no shared address to conflict
over.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The Windows integration test scripts tore down their device-mapper
snapshot and loop devices with 'dmsetup remove_all -f' and 'losetup
-D'. Both operate on every such device on the host, not just the ones
the script created.
On a host whose root filesystem is device-mapper backed (for example an
LVM root), 'dmsetup remove_all -f' replaces the in-use root device's
table with an error target. The host is wedged until reboot. This is
harmless in CI, which runs in a disposable VM, but destroys a developer
machine.
It turns out windows-snapshot-base is not used at all. Drop it.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
The updated image is configured in a same way as before
SSH and RDP are enabled.
Includes latest stable virtio-win 0.1.285 drivers.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Adding a new dedicated binary that is meant to be used as a reference
implementation for validating that offloaded snapshot/restore works and
meant to be used through tests in general.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-7
The tests expect some generated images and assets so make sure that they
are there (directly copied from the x86-64 test).
Signed-off-by: Rob Bradford <rbradford@meta.com>
Bring the confidential VM (CVM) integration tests up on the KVM SEV-SNP
backend in addition to MSHV. On KVM the IGVM is an Oak stage0 firmware
image and the guest kernel is supplied separately: stage0 reads the
kernel, cmdline and E820 over fw_cfg. The test harness selects this
model when a guest kernel is present at /igvm_files/bzImage, mirroring
how the stage0 IGVM is discovered; MSHV keeps using the monolithic IGVM
with the kernel baked in.
- test_infra: stage0 + direct-kernel + fw_cfg boot wiring (both the
command line and the HTTP/D-Bus API path) plus an on_kvm_sev_snp()
helper for tests to branch on.
- tests: the CVM tests that don't work on the KVM SEV-SNP path yet are
gated with #[cfg(not(feature = "kvm"))] inside the common_cvm module.
The MSHV build enables mshv,igvm,sev_snp (no kvm feature) while the
KVM build enables kvm,igvm,sev_snp,fw_cfg, so the cfg compiles these
tests into the MSHV binary only and drops them on KVM; both
hypervisors run the single common_cvm nextest profile. They all still
run on MSHV:
* test_pci_multiple_segments - stage0 places all 64-bit BARs in a
single global window, so a BAR allocated in a different
per-segment window is relocated cross-window and wedges boot.
* test_dmi_uuid / test_dmi_oem_strings /
test_dmi_system_and_chassis - SMBIOS is not delivered to SEV-SNP
guests on the KVM stage0 boot path, so the guest's DMI tables
read empty. VMM follow-up.
* test_vdpa_block - needs host vdpa_sim_blk setup, and vDPA DMA
into SEV-SNP-encrypted memory is unsupported (the guest hangs).
Assisted-by: Claude:Opus-4.8
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Make the --hypervisor argument optional for the build and tests
commands. Add detect_hypervisor_device() and resolve_hypervisor_device()
helpers that map an explicit kvm/mshv name to its device node, or fall
back to probing /dev/mshv and /dev/kvm on the host when the flag is
omitted. The resolved device is validated before it is mounted into the
container, and an explicit --hypervisor is still forwarded to the test
scripts.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Make the --hypervisor argument optional in process_common_args().
When it is not supplied, detect_hypervisor() now selects the backend
from the host device node: /dev/mshv for MSHV, /dev/kvm for KVM, and
errors out when neither is present. An explicit --hypervisor still
overrides the detection.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This will allow us to work towards removing network access from the
container.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
This is structured such that if multiple test groups need the same asset
it is only listed once.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
This will allow us to remove the download steps from the build scripts
themselves.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
Centralize test configuration (filters, retries, fail-fast,
sequential/parallel scheduling) into nextest profiles, replacing
scattered flags across shell scripts. This simplifies the scripts
and provides a single source of truth for test behavior.
Enable JUnit XML output per profile, giving CI systems structured
test results for better reporting.
Not all test invocations are converted to profiles yet. Just the
ones that are repeated across scripts.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Update the container image tag to trigger a rebuild. The new image
pins cargo-nextest to version 0.9.128, which supports profile
inheritance, needed for the following nextest configuration changes.
Assisted-by: Copilot-CLI:Claude-Opus-4.7
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
The shared /tmp/cloud-hypervisor/ path is created by the first user to
run dev_cli.sh and owned by them, so other users on the same host fail
the +x+w check and cannot chmod it back. Move both tmp paths under
/tmp/cloud-hypervisor-${USER}/ so each user gets their own tree.
While here, fold the local BUILD_DIR in build_container() into the
existing (previously unused) CLH_CTR_BUILD_DIR, which ensure_build_dir()
already creates.
Assisted-by: Claude:Opus-4.7
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Focal has served us well for many years but is now beyond EOL. Remove
all remaining use of focal images from the CI.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Pass `-p cloud-hypervisor` to all `cargo nextest run` invocations in
the integration test scripts so test discovery and execution are
scoped to the cloud-hypervisor package only, avoiding running tests
from other workspace crates.
This avoids flooding the output messages like below
`test result: ok. 0 passed; 0 failed; 0 ignored;
0 measured; 0 filtered out; finished in 0.00s
Running unittests src/lib.rs`
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Use the VM_TYPE environment variable to conditionally enable
igvm and sev_snp features when building for confidential VMs.
Pass --vm-type confidential to the performance-metrics binary
so it can select the appropriate test configuration.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add a --vm-type argument to the tests command in dev_cli.sh to
allow specifying the type of VM (regular or confidential). The
value is passed to the container via the VM_TYPE environment
variable.
Also extract a prepare_igvm_files() helper function to reduce
duplication when copying IGVM files for confidential VM tests,
and call it from both integration_cvm and metrics paths.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add validation checks to prepare_linux() to catch invalid
kernel option combinations early:
- Error if --build-guest-kernel and CH_CUSTOM_KERNEL are
both provided, as they are mutually exclusive.
- On x86_64, error if only one of CH_CUSTOM_KERNEL or
CH_CUSTOM_BZIMAGE is set; both must be provided together.
- Fix kernel-already-present check: use per-architecture
branches with correct bash syntax (elif instead of
else-if, [[ ]] instead of [ && ]) so aarch64 and x86_64
are each handled properly.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add file-existence guards around firmware and OVMF download
calls in integration test scripts that were missing them.
Also guard prepare_linux() in test-util.sh so it returns
early when the kernel binary is already present.
This lets users pre-populate the workloads directory (e.g.
via CH_CUSTOM_KERNEL, CH_CUSTOM_FIRMWARE, CH_CUSTOM_OVMF)
and avoid redundant network fetches or source builds inside
the container.
Updated scripts:
- test-util.sh (prepare_linux early return)
- run_integration_tests_aarch64.sh
- run_integration_tests_vfio.sh
- run_integration_tests_windows_x86_64.sh
- run_integration_tests_windows_aarch64.sh
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Skip sha1sum verification for firmware files that were
provided via CH_CUSTOM_FIRMWARE or CH_CUSTOM_OVMF environment
variables. Custom files will not match the expected checksums,
so we filter them out of the sha1sums list before running
sha1sum --check.
Updated scripts:
- run_integration_tests_x86_64.sh
- run_integration_tests_aarch64.sh
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
When CH_CUSTOM_KERNEL, CH_CUSTOM_FIRMWARE, or CH_CUSTOM_OVMF
environment variables are set, copy the referenced files into
the host workloads directory before starting Docker. The files
land at the default paths the test scripts expect
(vmlinux-x86_64, Image-arm64, hypervisor-fw, CLOUDHV.fd,
CLOUDHV_EFI.fd), so the existing download-if-missing guards
inside the container skip the network fetch.
Each variable is independent; users can override any
combination without affecting the others.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Extract a common_env_args bash array with the environment
variables shared by both unit and integration test containers
(BUILD_TARGET, RUSTFLAGS, TARGET_CC). The unit test block
uses common_env_args plus its own LLVM_PROFILE_FILE. After
the unit block, common_env_args is extended with the USER
and AUTH_DOWNLOAD_TOKEN variables used by all integration
test groups.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Extract a common_args bash array with the runtime arguments
shared by both unit and integration test containers (name,
workdir, rm, seccomp, volumes). The unit test block uses
common_args plus its own device and cap-add flags. After
the unit block, common_args is extended with the privileged,
ipc, net, tmpfs, and workload-volume flags used by all
integration test groups.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Reduce `nr_hugepages` from 12 GB to 6 GB on both architectures. The
number if huge pages needed (if all the tests run at once) is 4GiB so
this gives 50% headroom.
This should reduce the number of tests that fail/flake out due to lack
of memory.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Move the live migration tests themselves into the common scopes allowing
the tests to now run interleaved together hopefully reducing CI time.
On MSHV the live migration tests are now not compiled in rather than
compiled in and skipped (as the helpers are not compiled in for MSHV.)
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Move the live migration test running from their own script into the
x86-64 script (on aarch64 they were already in the same script.) They
were historically separate as they were new. Now they are established it
makes sense for them to be combined.
The timeout in the GitHub workflow has been extended to accommodate the
extra work in the same step.
The Rust test scopes are unchanged - the running of the tests has been
moved.
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rob Bradford <rbradford@meta.com>
The aarch64 integration script hardcoded `LAST_RELEASE_VERSION="v39.0"`
for the live-upgrade binary download, while the live-migration runner
already accepts a `MIGRATABLE_VERSION` env override with a `vxx.0`
regex check. Standardise the aarch64 script on the same env-override
block so both arches honour the same knob with the same validation.
Default is unchanged (v39.0).
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Increase the number of parallel integration tests in CI to save ~3-5
minutes per x86_64 run. The thread limit is driven by RAM and disk
space constraints, not CPU availability.
A new `PARALLEL_INTEGRATION_TESTS_NUM` environment variable controls
the thread count. In CI it is set explicitly (12 for x86_64, 25 for
ARM64); locally it falls back to `nproc / 4`, preserving the previous
behavior.
Only the first test group (`common_parallel`, `live_migration_parallel`)
uses the overridden value - subsequent groups (dbus_api, fw_cfg,
ivshmem, aarch64_acpi) continue to use the `nproc / 4` default.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
During development, it is common to cancel a running test and
rerun it after making changes. However, pressing Ctrl+C while
dev_cli.sh runs long-running container commands (wget, qemu-img,
cargo build, etc.) does not reliably terminate the process.
Bash defers signal handling while a foreground process is
running, so the trap only fires after the docker run command
returns. This makes it difficult to cancel and restart quickly.
Fix this by introducing a run_container() wrapper that runs
docker in the background and uses 'wait', which is immediately
interruptible by signals. A cleanup() trap handler is set for
SIGINT and SIGTERM that kills the named container, the tracked
background PID, and any remaining child processes.
Each docker run invocation is assigned a unique --name based
on the script PID (clh-dev-$$) to allow targeted cleanup.
The interactive shell (cmd_shell) is left unwrapped since it
needs foreground terminal I/O.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The unit test container runs with Docker default seccomp
profile which blocks io_uring_setup, io_uring_enter and
io_uring_register. This causes all qcow_async unit tests to
fail with EPERM when creating an io_uring instance.
Add --security-opt seccomp=unconfined to the unit test docker
run invocation. The container already has --device access and
cap_net_admin, so this does not materially change the security
posture.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The updated image is configured in a same way as the
previously used 2022.
SAC, SSH, and RDP are configured.
All Windows updates to the curent date are installed.
Includes latest stable virtio-win 0.1.285 drivers.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Rename download_ovmf to download_amd64_ovmf and add a separate
download_aarch64_ovmf for CLOUDHV_EFI.fd
Replace build_edk2 with download_aarch64_ovmf in aarch64 scripts
Update OVMF firmware tag to ch-1e1b96f126
Update CLOUDHV.fd sha1sum in sha1sums-x86_64
Add CLOUDHV_EFI.fd sha1sum in sha1sums-aarch64-common
Fixes: #7622
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Add an `iommufd` flag to existing VFIO integration tests. When false,
tests use the legacy vfio container/group backend (existing behavior).
When true, tests use vfio cdev with iommufd and vfio_p2p_dma=off.
vfio_p2p_dma=off is required because the VFIO test runner uses a stock
Ubuntu 24.04 kernel (v6.8) which does not support mapping device MMIO
pages (VM_PFNMAP) through iommufd, causing IOMMU_IOAS_MAP to fail with
-EFAULT on MMIO BAR regions.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
The aarch64 dbus, fw_cfg & ivshmem tests don't build the mshv feature
causing them to fail when run on MSHV. Fix by building the mshv feature
too just like the x86 version of the script does.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Use IFS-based splitting instead of parameter expansion to
correctly separate '#'-delimited volume paths in
process_volumes_args(). The previous approach placed all
volumes into a single array element, causing Docker to
receive malformed --volume arguments.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Switch to a git hash of virtiofsd as unfortunately the last release is
some time ago and does not incorporate a dependency bump that enables it
to work with snapshot-restore.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add --test-exclude to process_common_args in test-util.sh and forward
it to the performance-metrics binary from run_metrics.sh.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Rename the aarch64 sha1sums file to sha1sums-aarch64-common to follow
the same naming convention as sha1sums-x86_64-common. This allows
run_metrics.sh to use the generic sha1sums-${TEST_ARCH}-common
pattern for all architectures, removing the need for aarch64-specific
conditionals.
Update run_integration_tests_aarch64.sh to reference the renamed file.
Signed-off-by: Souradeep <schakrabarti@microsoft.com>
Split the vDPA preparation flow into helper functions for
building modules, validating availability, loading modules,
and creating devices.
Build the vdpa_sim modules only on Ubuntu, where the script
installs dependencies and compiles them from the matching
kernel source. On other distributions, reuse the installed
kernel modules and verify that they are available before
continuing.
This makes the script easier to follow and supports systems
such as Azure Linux, where the modules are provided by the
kernel package.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
