This will allow us to work towards removing network access from the
container.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
This is structured such that if multiple test groups need the same asset
it is only listed once.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
This will allow us to remove the download steps from the build scripts
themselves.
Assisted-by: Claude:Opus-4.6
Signed-off-by: Rob Bradford <rbradford@meta.com>
Centralize test configuration (filters, retries, fail-fast,
sequential/parallel scheduling) into nextest profiles, replacing
scattered flags across shell scripts. This simplifies the scripts
and provides a single source of truth for test behavior.
Enable JUnit XML output per profile, giving CI systems structured
test results for better reporting.
Not all test invocations are converted to profiles yet. Just the
ones that are repeated across scripts.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Update the container image tag to trigger a rebuild. The new image
pins cargo-nextest to version 0.9.128, which supports profile
inheritance, needed for the following nextest configuration changes.
Assisted-by: Copilot-CLI:Claude-Opus-4.7
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
The shared /tmp/cloud-hypervisor/ path is created by the first user to
run dev_cli.sh and owned by them, so other users on the same host fail
the +x+w check and cannot chmod it back. Move both tmp paths under
/tmp/cloud-hypervisor-${USER}/ so each user gets their own tree.
While here, fold the local BUILD_DIR in build_container() into the
existing (previously unused) CLH_CTR_BUILD_DIR, which ensure_build_dir()
already creates.
Assisted-by: Claude:Opus-4.7
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Focal has served us well for many years but is now beyond EOL. Remove
all remaining use of focal images from the CI.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Pass `-p cloud-hypervisor` to all `cargo nextest run` invocations in
the integration test scripts so test discovery and execution are
scoped to the cloud-hypervisor package only, avoiding running tests
from other workspace crates.
This avoids flooding the output messages like below
`test result: ok. 0 passed; 0 failed; 0 ignored;
0 measured; 0 filtered out; finished in 0.00s
Running unittests src/lib.rs`
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Use the VM_TYPE environment variable to conditionally enable
igvm and sev_snp features when building for confidential VMs.
Pass --vm-type confidential to the performance-metrics binary
so it can select the appropriate test configuration.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add a --vm-type argument to the tests command in dev_cli.sh to
allow specifying the type of VM (regular or confidential). The
value is passed to the container via the VM_TYPE environment
variable.
Also extract a prepare_igvm_files() helper function to reduce
duplication when copying IGVM files for confidential VM tests,
and call it from both integration_cvm and metrics paths.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add validation checks to prepare_linux() to catch invalid
kernel option combinations early:
- Error if --build-guest-kernel and CH_CUSTOM_KERNEL are
both provided, as they are mutually exclusive.
- On x86_64, error if only one of CH_CUSTOM_KERNEL or
CH_CUSTOM_BZIMAGE is set; both must be provided together.
- Fix kernel-already-present check: use per-architecture
branches with correct bash syntax (elif instead of
else-if, [[ ]] instead of [ && ]) so aarch64 and x86_64
are each handled properly.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add file-existence guards around firmware and OVMF download
calls in integration test scripts that were missing them.
Also guard prepare_linux() in test-util.sh so it returns
early when the kernel binary is already present.
This lets users pre-populate the workloads directory (e.g.
via CH_CUSTOM_KERNEL, CH_CUSTOM_FIRMWARE, CH_CUSTOM_OVMF)
and avoid redundant network fetches or source builds inside
the container.
Updated scripts:
- test-util.sh (prepare_linux early return)
- run_integration_tests_aarch64.sh
- run_integration_tests_vfio.sh
- run_integration_tests_windows_x86_64.sh
- run_integration_tests_windows_aarch64.sh
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Skip sha1sum verification for firmware files that were
provided via CH_CUSTOM_FIRMWARE or CH_CUSTOM_OVMF environment
variables. Custom files will not match the expected checksums,
so we filter them out of the sha1sums list before running
sha1sum --check.
Updated scripts:
- run_integration_tests_x86_64.sh
- run_integration_tests_aarch64.sh
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
When CH_CUSTOM_KERNEL, CH_CUSTOM_FIRMWARE, or CH_CUSTOM_OVMF
environment variables are set, copy the referenced files into
the host workloads directory before starting Docker. The files
land at the default paths the test scripts expect
(vmlinux-x86_64, Image-arm64, hypervisor-fw, CLOUDHV.fd,
CLOUDHV_EFI.fd), so the existing download-if-missing guards
inside the container skip the network fetch.
Each variable is independent; users can override any
combination without affecting the others.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Extract a common_env_args bash array with the environment
variables shared by both unit and integration test containers
(BUILD_TARGET, RUSTFLAGS, TARGET_CC). The unit test block
uses common_env_args plus its own LLVM_PROFILE_FILE. After
the unit block, common_env_args is extended with the USER
and AUTH_DOWNLOAD_TOKEN variables used by all integration
test groups.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Extract a common_args bash array with the runtime arguments
shared by both unit and integration test containers (name,
workdir, rm, seccomp, volumes). The unit test block uses
common_args plus its own device and cap-add flags. After
the unit block, common_args is extended with the privileged,
ipc, net, tmpfs, and workload-volume flags used by all
integration test groups.
Assisted-by: GitHub Copilot:Claude-Opus-4.6
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Reduce `nr_hugepages` from 12 GB to 6 GB on both architectures. The
number if huge pages needed (if all the tests run at once) is 4GiB so
this gives 50% headroom.
This should reduce the number of tests that fail/flake out due to lack
of memory.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Move the live migration tests themselves into the common scopes allowing
the tests to now run interleaved together hopefully reducing CI time.
On MSHV the live migration tests are now not compiled in rather than
compiled in and skipped (as the helpers are not compiled in for MSHV.)
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Move the live migration test running from their own script into the
x86-64 script (on aarch64 they were already in the same script.) They
were historically separate as they were new. Now they are established it
makes sense for them to be combined.
The timeout in the GitHub workflow has been extended to accommodate the
extra work in the same step.
The Rust test scopes are unchanged - the running of the tests has been
moved.
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rob Bradford <rbradford@meta.com>
The aarch64 integration script hardcoded `LAST_RELEASE_VERSION="v39.0"`
for the live-upgrade binary download, while the live-migration runner
already accepts a `MIGRATABLE_VERSION` env override with a `vxx.0`
regex check. Standardise the aarch64 script on the same env-override
block so both arches honour the same knob with the same validation.
Default is unchanged (v39.0).
Assisted-by: Claude:Opus-4.7
Signed-off-by: Rob Bradford <rbradford@meta.com>
Increase the number of parallel integration tests in CI to save ~3-5
minutes per x86_64 run. The thread limit is driven by RAM and disk
space constraints, not CPU availability.
A new `PARALLEL_INTEGRATION_TESTS_NUM` environment variable controls
the thread count. In CI it is set explicitly (12 for x86_64, 25 for
ARM64); locally it falls back to `nproc / 4`, preserving the previous
behavior.
Only the first test group (`common_parallel`, `live_migration_parallel`)
uses the overridden value - subsequent groups (dbus_api, fw_cfg,
ivshmem, aarch64_acpi) continue to use the `nproc / 4` default.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
During development, it is common to cancel a running test and
rerun it after making changes. However, pressing Ctrl+C while
dev_cli.sh runs long-running container commands (wget, qemu-img,
cargo build, etc.) does not reliably terminate the process.
Bash defers signal handling while a foreground process is
running, so the trap only fires after the docker run command
returns. This makes it difficult to cancel and restart quickly.
Fix this by introducing a run_container() wrapper that runs
docker in the background and uses 'wait', which is immediately
interruptible by signals. A cleanup() trap handler is set for
SIGINT and SIGTERM that kills the named container, the tracked
background PID, and any remaining child processes.
Each docker run invocation is assigned a unique --name based
on the script PID (clh-dev-$$) to allow targeted cleanup.
The interactive shell (cmd_shell) is left unwrapped since it
needs foreground terminal I/O.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The unit test container runs with Docker default seccomp
profile which blocks io_uring_setup, io_uring_enter and
io_uring_register. This causes all qcow_async unit tests to
fail with EPERM when creating an io_uring instance.
Add --security-opt seccomp=unconfined to the unit test docker
run invocation. The container already has --device access and
cap_net_admin, so this does not materially change the security
posture.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The updated image is configured in a same way as the
previously used 2022.
SAC, SSH, and RDP are configured.
All Windows updates to the curent date are installed.
Includes latest stable virtio-win 0.1.285 drivers.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Rename download_ovmf to download_amd64_ovmf and add a separate
download_aarch64_ovmf for CLOUDHV_EFI.fd
Replace build_edk2 with download_aarch64_ovmf in aarch64 scripts
Update OVMF firmware tag to ch-1e1b96f126
Update CLOUDHV.fd sha1sum in sha1sums-x86_64
Add CLOUDHV_EFI.fd sha1sum in sha1sums-aarch64-common
Fixes: #7622
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Add an `iommufd` flag to existing VFIO integration tests. When false,
tests use the legacy vfio container/group backend (existing behavior).
When true, tests use vfio cdev with iommufd and vfio_p2p_dma=off.
vfio_p2p_dma=off is required because the VFIO test runner uses a stock
Ubuntu 24.04 kernel (v6.8) which does not support mapping device MMIO
pages (VM_PFNMAP) through iommufd, causing IOMMU_IOAS_MAP to fail with
-EFAULT on MMIO BAR regions.
Signed-off-by: Bo Chen <bchen@crusoe.ai>
The aarch64 dbus, fw_cfg & ivshmem tests don't build the mshv feature
causing them to fail when run on MSHV. Fix by building the mshv feature
too just like the x86 version of the script does.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Use IFS-based splitting instead of parameter expansion to
correctly separate '#'-delimited volume paths in
process_volumes_args(). The previous approach placed all
volumes into a single array element, causing Docker to
receive malformed --volume arguments.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Switch to a git hash of virtiofsd as unfortunately the last release is
some time ago and does not incorporate a dependency bump that enables it
to work with snapshot-restore.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add --test-exclude to process_common_args in test-util.sh and forward
it to the performance-metrics binary from run_metrics.sh.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Rename the aarch64 sha1sums file to sha1sums-aarch64-common to follow
the same naming convention as sha1sums-x86_64-common. This allows
run_metrics.sh to use the generic sha1sums-${TEST_ARCH}-common
pattern for all architectures, removing the need for aarch64-specific
conditionals.
Update run_integration_tests_aarch64.sh to reference the renamed file.
Signed-off-by: Souradeep <schakrabarti@microsoft.com>
Split the vDPA preparation flow into helper functions for
building modules, validating availability, loading modules,
and creating devices.
Build the vdpa_sim modules only on Ubuntu, where the script
installs dependencies and compiles them from the matching
kernel source. On other distributions, reuse the installed
kernel modules and verify that they are available before
continuing.
This makes the script easier to follow and supports systems
such as Azure Linux, where the modules are provided by the
kernel package.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
- Modified the integration test script to support CVM test threads
- Add more parameters to cargo nextest to match other files
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Accept arguments after -- in 'dev_cli.sh shell' and forward them
to 'bash -c' inside the container. When no arguments are given,
an interactive shell is started as before. This enables running
one-off commands in the CI container without an interactive session,
for example:
./scripts/dev_cli.sh shell -- rustup toolchain install nightly \&\& cargo +nightly fmt --all -- --check
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The focal image checksums have been moved in the -common
sha1sums file. Use the correct file for metrics.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
When running manual tests locally, it is sometimes necessary to
generate a cloud-init file at a custom path instead of defaulting
to /tmp. This is useful for developers and higher-level management
layers where files in /tmp may be cleaned up automatically.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This patch adds the skeleton of the CVM test
support and modify existing scripts and test framework
to enable such scenario. Split the sha1sum to support both
regular and CVM guest. Add one test case for CVM. Will further
add more test cases.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
X64_64 image download steps is being used for both
regular and CVM guest. Keeping the steps withing a function
in the test-util.sh
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Testing generic vhost-user devices will require virtiofsd to support the
--tag option, which v1.8.0 does not support.
Signed-off-by: Demi Marie Obenour <demiobenour@gmail.com>
build_edk2 was leaving behind .built markers
even when compilation failed.
Gate creation of.built marker to occur only on
successful build
Modify build_edk2() to exit with error code
when arm64 firmware artifact : CLOUDHV_EFI.fd
is not produced
Fixes#7608
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
build_edk2() module in scripts/common-aarch64.sh
does not produce the UEFI firmware for aarch64 as
the commits used to assemble sources for acpica,
edk2-platforms and edk2 do not compile after GCC
version upgraded from 11.4.0 to 13.3.0 in the
developer container (ubuntu 22.04 to 24.04)
Apply minimum upgrade to EDK2_REPO and ACPICA_REPO
required to compile with GCC 13.3.0
while still assuring guest VM boot for all
integration tests
BaseTools: Brotli compression submodule that was
previously failing has been fixed following commit
bump
Developers can now produce UEFI firmware for
aarch64 using the following commands
```
./scripts/dev_cli.sh shell
source scripts/test-util.sh
source scripts/common-aarch64.sh
build_edk2
```
Update docs/uefi.md
Fixes#7608
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
CI reports:
In scripts/test-util.sh line 216:
cleanup() {
^-- SC2329 (info): This function is never invoked. Check usage (or ignored if invoked indirectly).
The shellcheck can't trace calling in trap, so we need add hint
to make it happy.
Signed-off-by: Yi Wang <foxywang@tencent.com>
