This can then be used with the OptionParser::add_all() API to reduce the
number of locations the same options are added to the parser. The only
quirk is that some devices do not support an IOMMU (because they are
vhost-user / vfio-user based). There are two different versions of the
array to support that.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The IommuNotSupportedOnSegment variant is no longer needed as the common
PciDeviceCommonConfig::validate() handles this case with the
OnIommuSegment variant along with more use of the IommuNotSupported
error variant.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch VsockConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch VdpaConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch UserDeviceConfig over to using the newly extracted struct members
as used by all PCI based devices. The use of #[serde(flatten)] means
that this change has no impact on the JSON format that the data is
stored as.
As VFIO user devices do not support being placed behind an IOMMU an
error is now raised if iommu is set. This can't happen via the CLI but
could via the JSON/API.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch DeviceConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch PmemConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch GenericVhostUserConfig over to using the newly extracted struct
members as used by all PCI based devices. The use of #[serde(flatten)]
means that this change has no impact on the JSON format that the data is
stored as.
As generic vhost-user devices do not support being placed behind an
IOMMU an error is now raised if iommu is set. This can't happen via the
CLI but could via the JSON/API.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch FsConfig over to using the newly extracted struct members as used
by all PCI based devices. The use of #[serde(flatten)] means that this
change has no impact on the JSON format that the data is stored as.
As virtio-fs does not support being placed behind an IOMMU an error is
now raised if iommu is set. This option is not exposed via the CLI but
could happen with a miscontructed JSON/API call.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch NetConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Switch DiskConfig over to using the newly extracted struct members as
used by all PCI based devices. The use of #[serde(flatten)] means that
this change has no impact on the JSON format that the data is stored as.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Implement some common PCI segment validation. This can be used to reduce
duplication across the different validation methods.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Introduce a common struct that can encompass all the config fields
used by devices that are PCI based. The use of `skip_serializing_if`
means that the iommu field will only be included if set (otherwise
falling back to default false). This neatly handles the devices that
don't support an iommu.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This unit test was trying to test with extra "="s in the input but was
instead testing using an unknown option. Add the option to the parser to
not hit that incorrect error.
Signed-off-by: Rob Bradford <rbradford@meta.com>
The error message for the InvalidSyntax was copied from UnknownOption.
Correct it to "invalid syntax".
Signed-off-by: Rob Bradford <rbradford@meta.com>
Add an OptionParser::add_all method that takes a slice of option names
and use that to add to the set of parameters that the parser works on.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Refactor parse() into a version that can control whether to tolerate
unknown options. This can then be used to then parse a subset of the
options.
Signed-off-by: Rob Bradford <rbradford@meta.com>
Previously the UNIX socket file was only removed on the success path
(start_vmm returned Ok(Some(path))). If start_vmm failed after the
HTTP API had bound a path-based socket, the file could be left on disk.
Parse --api-socket in parse_api_socket(), call start_vmm with the
result, then unlink the path in main after start_vmm returns for both
success and failure (fd= mode unchanged: no path to remove).
Signed-off-by: Nguyen Dinh Phi <phind.uet@gmail.com>
Cross-check each UFFDIO_* constant against the Linux _IOC(dir, type,
nr, size) encoding formula at compile time so that transposed direction
bits or struct sizes are caught immediately rather than silently
producing wrong ioctl numbers at runtime.
Signed-off-by: Shayon Mukherjee <shayonj@gmail.com>
Now on the generic restore path the worker thread is notified on the
events and also the guest is notified via the interrupt. This avoids the
same "livelock" situation that required this "driver_awake" workaround
when restoring the net device.
Signed-off-by: Rob Bradford <rbradford@meta.com>
This will wake up the guest and avoid a livelock situation by ensuring
that it will process any pending queues on its side.
Signed-off-by: Rob Bradford <rbradford@meta.com>
A restored virtqueue can already contain pending descriptors when the VM
resumes. Before this change, the worker thread was unparked and then
waited for a fresh queue eventfd signal. That is normally fine, but not
when the queue was already non-empty at snapshot time. The virtqueue
state lives in guest memory and is restored, but the original host-side
queue eventfd signal is not persistent snapshot state. If the guest
already notified the queue before the snapshot, it may not notify it
again after resume.
That can leave the worker idle while the guest is still waiting for the
pending request to complete. In one observed case, this stalled a
virtio-blk flush during early boot after snapshot/restore.
We mitigate this in the shared `VirtioCommon` resume path.
`VirtioCommon` retains cloned queue eventfds for activated virtqueues
and signals each of them once on resume after unparking the worker
threads.
Keep virtio-net on its existing special-case path: it resumes worker
threads without signaling queue eventfds so the `driver_awake`
workaround remains intact until the guest performs a real notify.
On-behalf-of: SAP leander.kohler@sap.com
Signed-off-by: Leander Kohler <leander.kohler@cyberus-technology.de>
UFFDIO_WAKE was 0x4010_aa02 (_IOW) but should be 0x8010_aa02,
causing every wake call to silently fail with -EINVAL.
UFFD_FEATURE_MISSING_HUGETLBFS was (1<<6) but should be (1<<4),
colliding with UFFD_FEATURE_EVENT_UNMAP.
Signed-off-by: Zhiheng Tao <junchuan.tzh@antgroup.com>
This reduces pressure on CI and enables to run more tests locally on
developer machines (with 16GB of RAM or less).
No functional changes.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Replace fixed sleeps in Windows integration tests with polling
helpers that wait for boot, snapshot readiness, and device
enumeration.
This keeps the same test intent while avoiding long fixed delays on
the fast path.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Replace fixed sleeps in Linux snapshot and restore integration tests
with event monitor and API readiness checks.
This updates ivshmem and common_sequential snapshot paths to wait for
concrete restore and snapshot completion signals instead of sleeping
for an assumed amount of time.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Use polling helpers in common integration tests instead of fixed
sleeps where the tests already know the expected ready state.
This updates CPU and memory hotplug checks as well as a few
device- and restore-related waits in common_parallel to stop
oversleeping on the fast path while keeping the same assertions.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Treat missing or still-short event monitor files as a retryable state
in integration test helpers.
This keeps polling-based restore and snapshot checks from failing early
with file-not-found or short-file assertions while the monitor output
is still being written.
In the following, we can gracefully wait for the corresponding
conditions to become true.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Allow one-shot SSH commands to install a libssh2 session timeout and
use that path from wait_for_ssh.
This keeps SSH readiness probes from blocking far beyond their caller
provided timeout when the guest network is slow or broken.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Split the SSH helpers into a one-shot execution path and a retrying
wrapper with linear backoff.
This makes it possible to use a single bounded SSH attempt when tests
need a direct readiness probe while preserving the existing retrying
behavior for callers that expect it.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add generic polling helpers for integration tests and build the
SSH wait helpers on top of them.
This lets follow-up test changes replace fixed sleeps with
condition-based waits without duplicating retry logic at each call
site.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
The MMIO64 allocator size is computed with alignment truncation:
size = (range / alignment) * alignment
This loses up to one alignment unit (4 GiB) at the top of the
address space. When a guest (Windows with virtio-win 0.1.285)
programs a BAR near the top of the physical address space, the
allocation fails because the address falls in the truncated gap.
Give the last PCI segment allocator all remaining space up to
the end of the device area, so no addresses are lost.
The `end` parameter of create_mmio_allocators() is an inclusive
address (the last valid byte). Fix the 32-bit caller and tests
to pass inclusive values, consistent with the 64-bit caller
which already uses the inclusive end_of_device_area().
Signed-off-by: CMGS <ilskdw@gmail.com>
When BAR reprogramming is detected, detect_bar_reprogramming()
eagerly updates the BAR address in config space before the actual
MMIO remapping occurs. If the subsequent move_bar() fails (e.g.
the new address falls outside the allocator range), the config
register retains the new address while the MMIO bus still uses
the old one, leaving the device broken.
Add restore_bar_addr() to undo the config space update when
move_bar() fails, so the device remains functional at its
original address.
For 64-bit BARs, restore both the low and high BAR slots as well
as the corresponding config registers, mirroring the two-slot
update logic in detect_bar_reprogramming().
Implement restore_bar_addr() for all PciDevice implementations
(VirtioPciDevice, VfioPciDevice, VfioUserPciDevice, IvshmemDevice,
PvPanicDevice, and PvmemcontrolPciDevice) by delegating to their
respective PciConfiguration::restore_bar_addr().
Signed-off-by: CMGS <ilskdw@gmail.com>
Add a --continue-on-failure CLI flag that allows the test harness to
continue executing remaining tests after encountering a failure, instead
of aborting immediately. When set, failed tests are recorded with zeroed
metrics and a "FAILED" status, the report file is always generated, and
the process exits with a non-zero code if any test failed.
Without the flag, the existing fail-fast behavior is preserved.
Also add a "status" field ("PASSED"/"FAILED") to PerformanceTestResult
so report consumers can distinguish successful tests from failed ones.
Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>
Instrument the two main downtime-phase operations on the destination
side - receiving state and resuming the VM - so their costs are visible
in logs and can be iterated on.
The new log messages may look like this:
```text
cloud-hypervisor: 7.283424s: <vmm> DEBUG:vmm/src/lib.rs:948 -- Migration (incoming): recv_snapshot:3ms restore:10ms
cloud-hypervisor: 7.284824s: <vmm> DEBUG:vmm/src/lib.rs:967 -- Migration (incoming): resume:1ms
cloud-hypervisor: 7.284842s: <vmm> DEBUG:vmm/src/lib.rs:977 -- Migration (incoming): Receiving final state and resuming the VM took 15ms
```
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Use OngoingMigrationContext to measure and log the effective VM downtime
(pause to remote resume) and the cost of each non-trivial step in the
downtime window: snapshotting, sending the snapshot, and awaiting
completion. This makes it straightforward to identify and reduce
downtime as live migration matures.
Example:
```
cloud-hypervisor: 7.703402s: <vmm> INFO:vmm/src/lib.rs:1494 -- Migration completed after 2.2s with a downtime of 298ms (goal was 300ms)
cloud-hypervisor: 7.703453s: <vmm> DEBUG:vmm/src/lib.rs:1500 -- Downtime breakdown: 298ms (final_iter:269ms state:7ms send_state:19ms complete:1ms)
```
Note: downtime is measured on the source only; cross-host clock skew
may cause unreliable results.
# Terminology
At first glance, the use of "state" and "[VM] snapshot" may seem
confusing. As discussed in [0], we use "state" consistently in the
migration code. On the VM side, "snapshotting" is merely the mechanism
used to obtain the VM state.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7979#discussion_r3061359899
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add a small helper that returns both the successful result of an
operation and the time it took to complete.
Subsequent migration instrumentation uses this to keep timing code
compact and consistent.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add migration-level context types that extend the existing memory-only
metrics with overall migration duration and downtime breakdown.
OngoingMigrationContext models the sender-side migration progress until
all inputs needed for final downtime accounting are available.
CompletedMigrationContext then stores the finalized migration metrics,
including the final memory iteration, snapshotting, snapshot transfer,
and completion phase.
This provides the data needed to log effective downtime in the VMM and
lays the groundwork for future migration statistics reporting.
# Terminology
At first glance, the use of "state" and "[VM] snapshot" may seem
confusing. As discussed in [0], we use "state" consistently in the
migration code. On the VM side, "snapshotting" is merely the mechanism
used to obtain the VM state.
[0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7979#discussion_r3061359899
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Expose the finalized per-iteration timing fields needed by higher-level
migration metrics and factor the iteration-overhead calculation into a
small helper.
This keeps the existing MemoryMigrationContext behavior intact while
making the timing data easier to consume from migration-level context
in the following commits.
On-behalf-of: SAP philipp.schuster@sap.com
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Extend test_pci_multiple_segments_numa_node to assert that every
PNP0A08 host bridge in the guest DSDT exposes a unique _UID
matching its PCI segment id. Linux surfaces the evaluated _UID
value through /sys/bus/acpi/devices/PNP0A08:*/uid, so the check
is a single additional ssh command on top of the existing test
plumbing.
This test is used (rather than test_pci_multiple_segments) so
that the assertion runs on both x86_64 and aarch64: the numa_node
variant boots through edk2 firmware on aarch64, making ACPI (and
PNP0A08 host bridges) available, whereas the non-firmware variant
uses FDT on aarch64 and exposes no PNP0A08 nodes.
Without a per-segment _UID, two PNP0A08 nodes share _UID=0 which
violates ACPI 6.5 section 6.1.12 and triggers BSOD 0xA5 on
Windows guests. This assertion would catch any future regression
of that kind.
Signed-off-by: Max Makarov <maxpain@linux.com>
The ACPI specification requires _UID to be unique across devices
sharing the same _HID (ACPI 6.5 section 6.1.12). Currently every
PciSegment emits _UID=0 for its PNP0A08 host bridge, which violates
the spec when num_pci_segments > 1.
Windows guests detect this during ACPI namespace enumeration and
abort boot with BSOD 0xA5 ACPI_BIOS_ERROR, pointing at the _UID
object of the second PNP0A08 node. Linux guests are lenient and
silently accept the collision, so the issue has gone unnoticed.
Use self.id as _UID, matching what _SEG does on the line above.
For single-segment VMs (id == 0) this is a no-op at runtime.
Signed-off-by: Max Makarov <maxpain@linux.com>
