ProcessRegistry::cleanup sent SIGKILL to a test's process group and
returned at once. SIGKILL is asynchronous, so a killed process could
still hold its boot listener socket when the next test started, causing
an EADDRINUSE bind failure.
Reap the group after the signal and wait for it to exit before
returning. Return an error if it does not exit within the timeout so a
caller can react to a guest that survived cleanup.
Since cleanup now reaps the whole group, the unit tests drop their Child
handles without waiting. They carry expect(clippy::zombie_processes),
since cleanup already reaps them.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The qcow2 backend caches L2 table and refcount updates in memory and
only writes them back on a guest-initiated flush, clean shutdown or
drop. A paused VM therefore leaves the on-disk image without the
mappings for any cluster allocated since the last guest flush: the
data clusters are present in the file, but nothing references them.
Anything that reads the image while the VM is paused sees those
writes as missing. Copying the disk alongside vm.snapshot (the
documented snapshot workflow) captures a stale image, and live
migration releases the disk locks after pausing so the destination
reopens the file with the same stale metadata. In both cases writes
the guest has completed, and may later read back, silently disappear.
Add a MetadataSync capability trait with a no-op default, fold it
into FullDiskFile, implement it for the qcow2 backend as a metadata
cache flush, and call it from the virtio-block pause path after
in-flight requests have drained. Pause is the quiesce point both
flows rely on, and it is a cold path, so the extra flush does not
affect runtime I/O.
Reproduced by writing to a qcow2 disk from the guest with O_DIRECT
and no explicit flush, pausing the VM and copying the image: qemu-img
map on the copy shows no mapped clusters and reads return zeros. With
this change the copy contains every completed write. A unit test
covers the same sequence at the format level: a completed write is
invisible to a fresh reader until sync_metadata, and visible after.
Signed-off-by: CMGS <ilskdw@gmail.com>
Each Windows integration test boots a full Windows Server guest using
several vCPUs and gigabytes of RAM. nextest runs up to one test per
logical CPU by default, so the whole suite started at once and thrashed
the host.
Add a dedicated 'windows' nextest profile that inherits from
integration and assigns the tests to a 'windows' test group capped at
four concurrent runs. Point the Windows test scripts at the profile and
drop the redundant retries flag, which now comes from the profile.
Running the tests in parallel is safe because every guest gets a unique
network via next_guest_id, so there is no shared address to conflict
over.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
When reconnecting to a vhost-user backend, Cloud Hypervisor keeps the
same queues and currently restarts them from avail_idx.
That can skip descriptors that were made available by the guest but not
completed before the old backend crashed. For queues where
VIRTIO_F_IN_ORDER was negotiated, used_idx is a safe completion
boundary, so reconnect can resume from used_idx and kick the queue if
work remains.
Do not do this when inflight tracking is active, because the backend
inflight state is the more precise recovery mechanism. Also leave queues
without VIRTIO_F_IN_ORDER on the existing avail_idx path, since used_idx
does not identify which descriptors completed for out-of-order devices.
Signed-off-by: Peter Delevoryas <pdel@meta.com>
Client connect retried for a full minute on every failure and did not
watch kill_evt for early exit, unlike reconnect, so teardown and
permanent connect errors both stalled for CONNECT_TIMEOUT.
Update the client path to watch kill_evt for early abort, fail
non-retryable errors immediately, and surface timeout with a dedicated
error.
Partially Fixes#8052
Signed-off-by: Chinmoy <daschinmoyy21@gmail.com>
Assisted-by: GLM 5.2
Exercise the EOPNOTSUPP fallback path using memfd, which shares
the same backing as tmpfs and triggers the fallback without
requiring privileges or a specific filesystem.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The WriteZeroesAt fallback means write_zeroes no longer fails on
filesystems that lack fallocate support. The skip is now dead
code.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Filesystems such as tmpfs do not support fallocate with
FALLOC_FL_ZERO_RANGE or FALLOC_FL_PUNCH_HOLE and return EOPNOTSUPP.
When a raw disk image lives on such a filesystem, virtio write zeroes
and discard requests fail with IOERR.
Use the WriteZeroesAt trait from vmm_sys_util through AlignedFile,
which already bundles fallocate with a positional write fallback.
For punch_hole, catch EOPNOTSUPP and fall back to the same trait.
The io_uring engine previously submitted fallocate directly through
the ring, where the async EOPNOTSUPP completion had no retry path.
Route it through the same sync helpers that the other engines
already use.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
In case the host MAC address associated with a TAP device wasn't
explicitly provided by the user, Cloud Hypervisor would get the host MAC
associated by default with this TAP device and store it through the
network config. Problem is, in the context of a snapshot/restore, that
meant the network config provided by the user was different on the
destination host compared to the source host. This was causing an issue
when Cloud Hypervisor wasn't started with CAP_NET_ADMIN permissions as
it couldn't set the host MAC address on the destination, while the
source never needed these permissions since the MAC was automatically
allocated by the kernel.
We're fixing this issue by setting the host MAC address when it's
explicitly requested by the user through the network config, and making
the host MAC immutable so that it can't be changed at runtime.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Further stability improvements in the villain harness keep printed
verdicts when a batch times out, stop batch timeouts from wedging
benign tests, and capture VMM stderr separately so it cannot
corrupt verdict markers, reducing spurious WEDGED results.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Add sparse region support to remote live migration as it will save some
of the network bandwidth and therefore speed up the migration to the
destination VM.
Instead of sending the entire guest RAM over the network, only send
ranges which have been populated during the first iteration. Pages full
of holes are simply skipped.
Signed-off-by: Sebastien Boeuf <sboeuf@meta.com>
Assisted-by: Claude:claude-opus-4-8
A PCI BAR stores its address in the upper bits and type information in
the lower bits. To determine the BAR size, an operating system disables
memory decoding, writes ones to the BAR's address bits, and reads the
value back. Bits that remain zero reveal the required alignment and
therefore the size.
We currently recognize only 0xffffffff as a size probe. OpenBSD instead
writes the memory BAR address mask 0xfffffff0, without setting the lower
four non-address bits [0][1].
We misinterpret that probe as a real BAR relocation. Because OpenBSD has
disabled memory space, the relocation remains pending. We apply it when
OpenBSD enables memory space again, moving the virtio block BAR away
from its mapped address and preventing the guest from booting.
Recognize probes using the address mask for each BAR type and exclude
them from BAR reprogramming. Add a test for the OpenBSD memory BAR
probe.
[0]: 54a2bc1e54/sys/dev/pci/pci_map.c (L154-L180)
[1]: 54a2bc1e54/sys/dev/pci/pcireg.h (L429-L443)
Co-authored-by: Thomas Prescher <thomas.prescher@cyberus-technology.de>
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
Both tests intermittently report WEDGED because the villain harness
can lose the guest verdict marker when the VM exits before the VMM
drains the console virtqueue, so the host reads a console with no
marker under load. This is a harness race, not a CH defect, and it
lands on a random test each run. Exclude the two most affected
until the harness drains the console on shutdown.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
After churning a single L2 region with an fsync per write, close and
reopen the image. parse_qcow rebuilds the free list from the on disk
refcounts, so a reopen that recovers many clusters means the runtime
allocator stranded them. Without the fix the reopen recovered every
relocated cluster. Now it finds none.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
After churning a single L2 region with an fsync per write, scan the file
for refcount==0 clusters and assert every one is on the runtime free
list. The reuse bug left relocated refcount block clusters free on disk
but off the list, so this fails without the fix, reporting the exact
count of stranded clusters.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Add a regression test that churns one L2 region with an fsync per write,
relocating the L2 table and its refcount block every iteration, and
asserts committed host blocks track live data instead of doubling. Data
written across the churn is read back to confirm integrity.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Metadata tables use relocate on write. The first modification after a
flush writes the table to a new cluster and frees the old one. The L2
path returns the old cluster to the free list, but the refcount block
path did not. QcowState::set_cluster_refcount set the dropped refcount
block cluster to refcount 0 yet never pushed it onto unref_clusters, so
it stayed committed on the host and was invisible to the allocator.
Every refcount block relocation stranded one cluster, and
get_new_cluster extended the file instead of reusing it.
Under a sync heavy workload each fsync that dirties a table relocates
it, so the physical footprint grew at roughly twice the live data
without bound. qemu-img check stayed clean because the refcounts are
correct, and a reopen, which rebuilds the free list from refcounts,
recovered the space.
Push the dropped refcount block cluster onto unref_clusters so the
running allocator reuses it, matching the L2 table path.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Add a live migration section to the VFIO documentation covering the
requirements beyond snapshot and restore, the dirty tracking behavior,
the destination file descriptor substitution with an example, and the
failure recovery behavior. Link it from the live migration and
snapshot sections.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
The VmConfig that arrives over the migration stream carries the
source's device paths and stale FDs, none of which are usable on the
destination. A received VFIO device therefore needs fresh descriptors
supplied with the request.
VmReceiveMigrationData gains vfio_fds, pairing each device id with a
cdev FD, and iommufd_fd for the backing iommufd, both arriving over
SCM_RIGHTS. Once the received VmConfig is available, each named
device's path or FD is swapped for the received one and the iommufd is
installed before the VM is built, so the device comes up FD backed.
The request is rejected when a substituted device lacks the iommufd
backend, when an id is unknown or repeated, or when a device names no
replacement in vfio_fds. These checks run against the migrated VmConfig
once it has been received.
ch-remote gains the vfio_fds and iommufd_fd options and forwards the
descriptors over SCM_RIGHTS. The D-Bus transport cannot carry file
descriptors and drops them.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
A VFIO device restored onto a different host has a device path and
iommufd that are meaningless there, and an FD backed device cannot
serialize a live descriptor into the snapshot at all. Restoring one
therefore needs fresh descriptors supplied with the request.
RestoreConfig gains vfio_fds, pairing each device id with a cdev FD,
and iommufd_fd for the backing iommufd. Both arrive over SCM_RIGHTS on
the restore request. vm_restore swaps each named device's stale path
or FD for the received one and installs the iommufd before the VM is
built, so the device comes up FD backed.
The request is rejected when a substituted device lacks the iommufd
backend, when an id is unknown or repeated, or when an FD backed device
names no replacement.
ch-remote gains the vfio_fds and iommufd_fd options and forwards the
descriptors through the SCM_RIGHTS pool.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
Wire the previously empty Migratable implementation on VfioPciDevice to
VFIO DMA logging so the pages the device writes to guest memory are
tracked during a live migration. The Vfio trait gains the DMA logging
operations with defaults that report no migration support.
Without a virtual IOMMU the device sees an identity mapping of guest
memory, so the tracked ranges are the guest memory regions with iova
equal to gpa. The requested page size is only a hint, so the device
reports back the granularity it actually applied.
A device that does not support migration, or one behind a virtual
IOMMU, is refused at migration start rather than silently skipping
dirty tracking and producing an incorrect log. Under a virtual IOMMU
the reported IOVAs are not guest addresses and the logged range set
would have to track the mapping changes a guest makes mid migration,
so that configuration is out of scope.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
A failed VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE set can leave the device
in its original state, midway along a combination transition, or in
ERROR. The uAPI leaves recovery to the user, and from ERROR only a
VFIO_DEVICE_RESET brings the device back to RUNNING.
Previously a failed transition left the device wherever it landed, so
one that reached ERROR stayed wedged for the life of the guest even
though a reset would have recovered it.
On a failed transition the caller now names an optional recovery state
to attempt first, falling back to a device reset. Pause, resume, and
the return to STOP after STOP_COPY name no recovery state, so their
failure resets directly. The original transition error is always
propagated and recovery failures are logged.
Signed-off-by: Saravanan D <saravanand@crusoe.ai>
The S0094 state test could be reported as WEDGED when the host per
test timeout killed the VM before the guest finished its wait,
dropping the verdict marker. v0.5.2 accepts that timeout case as a
pass since the device does not consume the buffer before DRIVER_OK.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Replace the QEMU debugger VM approach with a simpler setup that runs
WinDbg directly on a Windows host. Cloud Hypervisor exposes the guest
serial port on a UNIX socket, socat turns it into a TCP listener, and
the convey tool bridges the TCP endpoint to a named pipe that WinDbg
attaches to.
This drops the need for a second Windows VM and the socat SYSTEM shell
script wrapping.
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Add a GitHub Actions job that runs the virtio-villain suite against
cloud-hypervisor, giving continuous coverage of how the virtio device
model responds to guest input that violates the driver side rules of
the virtio specification. The suite drives the device model from the
guest side with out of spec virtqueue input, malformed descriptor
chains, transport register abuse, and device specific requests, then
checks that each violation is handled without crashing the device or
leaving it wedged. This makes a class of guest triggered failures a
signal that shows up on every relevant change rather than found by
chance.
The job builds cloud-hypervisor with the kvm feature, clones
virtio-villain at a pinned tag, builds its initramfs, and runs the
tests in short lived VMs, rerunning a wedged batch in isolation so a
failure is attributed to the offending test. Results reach the run
summary page, and the JUnit report plus per test logs upload as an
artifact. The compiled harness, the initramfs, and the fetched guest
kernel are cached under the resolved villain commit, so an unchanged
pin skips the rebuild.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Assisted-by: Claude:Opus-4.8
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Replace the hard-coded KVM SNP page type constants with the
KVM_SEV_SNP_PAGE_TYPE_* values from kvm-bindings.
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
KVM builds the VMSA internally, so the logic in igvm-loader passing a
VMSA page for KVM is misleading. The page is silently dropped later
in import_isolated_pages(). Only MSHV actually imports it.
Make PageTypeConfig::vmsa an Option that is None on KVM and push the
VMSA page only when it is set. As KVM no longer produces a VMSA page,
remove the now-dead skip in import_isolated_pages().
Signed-off-by: Ruben Hakobyan <hruben@meta.com>
Whitespace around `TupleList` elements can lead to errors while parsing
the contained `Tuple` keys and values. In some cases, whitespace in
input can lead to different parsing results for semantically identical
keys or values, e.g. "id1" and "id1 ". We therefore trim whitespace
from `TupleList` elements.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
The coding style in `CONTRIBUTING.md` under
`Coding Style & Code Comments` states that `thiserror` error messages
should start with a capital letter.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
We split the existing `Tuple` type into two types, one that represents
a single `Tuple` and one that represents a list of tuples. This allows
us to test tuple parsing and the parsing of tuple lists via distinct
unit tests in follow-up commits.
The renaming also brings consistency by adapting the naming scheme of
the other list types defined in `option_parser`, e.g. `IntegerList`
and `StringList`.
Signed-off-by: Pascal Scholz <pascal.scholz@cyberus-technology.de>
On-behalf-of: SAP pascal.scholz@sap.com
Add a Windows integration test that verifies kernel network debugging
(KDNET) works over a Cloud Hypervisor virtio-net device.
The test boots a Windows guest with a dedicated second virtio-net NIC,
enables KDNET on it via bcdedit (selecting the adapter by the PCI bus
params discovered over SSH), reboots, and then listens on the debugger
host address. Receiving a KDNET poll datagram from the debuggee proves
the whole virtio-net device path works: discovery, feature negotiation,
virtqueue setup and the TX doorbell. No debugger is needed because KDNET
connections are initiated by the target.
Gated to x86-64, where the Windows image ships the virtio-net KDNET
module. The test exercises only the generic virtio-net doorbell path,
so it runs under both KVM and MSHV.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
Windows can run its KDNET kernel-debugging transport over a Cloud
Hypervisor virtio-net device. Add a guide describing how it works.
Link to the new guide from the Windows support document.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
The Windows serial (COM/KDCOM) kernel debugging instructions lived in a
long subsection of the Windows support document. Move them verbatim
into a dedicated docs/windows-kdcom-debugging.md and leave a short
pointer in windows.md, so the debugging methods can be documented and
extended independently.
Drop the useless disclaimer. Things should work the same across
different Windows versions.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
A virtqueue notification (doorbell) is normally delivered to the device
through an ioeventfd registered on the notify address, so a plain MMIO
write to the notify register is consumed by the hypervisor and never
reaches write_bar().
It does reach write_bar() when the driver rings the doorbell through the
VIRTIO_PCI_CAP_PCI_CFG window (write_cap_pci_cfg -> write_bar) instead
of a mapped BAR, or on backends that deliver the write to the VMM such
as SEV-SNP.
The notification arm of write_bar() only re-signalled the matching
queue eventfd under the sev_snp feature and logged an error
otherwise, so a doorbell delivered through the PCI_CFG window was
silently dropped on standard builds and the queue was never
processed.
The virtio spec allows driving the device purely through the PCI_CFG
window, so signal the matching queue eventfd for any doorbell that
reaches write_bar() on every build.
Add unit tests that ring a queue's doorbell via write_bar() and
assert only the addressed queue's eventfd is signalled.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
The Windows integration test scripts tore down their device-mapper
snapshot and loop devices with 'dmsetup remove_all -f' and 'losetup
-D'. Both operate on every such device on the host, not just the ones
the script created.
On a host whose root filesystem is device-mapper backed (for example an
LVM root), 'dmsetup remove_all -f' replaces the in-use root device's
table with an error target. The host is wedged until reboot. This is
harmless in CI, which runs in a disposable VM, but destroys a developer
machine.
It turns out windows-snapshot-base is not used at all. Drop it.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Assisted-by: Copilot:Opus-4.8
Use std::thread::available_parallelism() when sizing the prefault
worker threads so the number of threads scales with the host CPU
count instead of relying on unsafe sysconf.
Fixes: #8495
Signed-off-by: Pulak Kanti Bhowmick <pkbhowmick007@gmail.com>
The generic vhost-user device took its virtio device type on the
command line via the `virtio_id` parameter, but the same value is
called `device_type` in the API and the resulting config struct. This
irregularity was due to churn during the review process, `device_type`
was the intended name.
Accept `device_type` on the command line and keep `virtio_id` as a
deprecated alias that logs a warning. The alias will then be removed in
a later release.
Fixes: #8545
Assisted-by: Claude:Opus-4.8
Signed-off-by: Rob Bradford <rbradford@meta.com>
On Intel MSHV the memory-intercept guest_physical_address and the
MSHV_VP_TRANSLATE_GVA ioctl both return a page-aligned GPA, while
guest_virtual_address is byte-exact. Returning the cached/translated
GPA unchanged made byte-sized MMIO land at BAR offset 0: virtio
device_status writes (BAR+0x14) hit device_feature_select, so
VIRTIO_F_VERSION_1 was never acked and virtio_blk/net/rng probes
failed with -EINVAL, leaving the guest unable to mount rootfs.
Splice gva & 0xfff into the returned GPA on both the intercept fast
path and the translate_gva fallback, and relax the cached-GVA match
to page granularity so it still hits for other byte offsets in the
same page.
This issue is reproducible on Intel machine, launching
Cloud-Hypervisor on nested scenario, using the Linux Dom0
image as the guest image to turn on nested hypervisor
into the guest.
Assisted-by: Claude:Opus-4.7
Signed-off-by: Muminul Islam <muislam@microsoft.com>
