And along with virtio-queue, we must also bump vhost-user-backend from
0.3.0 to 0.5.0 (since it relies on virtio-queue 0.4.0).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The vhost-user backend was always provided the maximum queue size but
this is incorrect. Instead it must be informed of the actual queue size
that has been negotiated with the virtio driver running in the guest.
This ensures proper functioning of vhost-user-block with the Rust
Hypervisor Firmware, which uses a hardcoded queue size of 16.
Partially fixes#4285
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The latest vhost-user specification describes VHOST_USER_RESET_OWNER
command as deprecated with the following explanation:
This is no longer used. Used to be sent to request disabling all
rings, but some back-ends interpreted it to also discard connection
state (this interpretation would lead to bugs). It is recommended that
back-ends either ignore this message, or use it to disable all rings.
Also, it's been observed that when using either Rust Hypervisor Firmware
or EDK2 OVMF firmware with SPDK (using the block device as the boot
disk), the virtio reset that happens when the firmware no longer needs
to access the block device caused a failure by triggering the command
VHOST_USER_RESET_OWNER.
For all these reasons, this patch simplifies the virtio reset
implementation by simply disabling the virtqueues and no longer calling
into VHOST_USER_RESET_OWNER.
Partially fixes#4285
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Enabling the TAP interface is only about the IFF_UP flag, meaning we
shouldn't be testing the presence of IFF_RUNNING, and therefore we
shouldn't set it when not present.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The Linux kernel now checks for this before marking CPUs as
hotpluggable:
commit aa06e20f1be628186f0c2dcec09ea0009eb69778
Author: Mario Limonciello <mario.limonciello@amd.com>
Date: Wed Sep 8 16:41:46 2021 -0500
x86/ACPI: Don't add CPUs that are not online capable
A number of systems are showing "hotplug capable" CPUs when they
are not really hotpluggable. This is because the MADT has extra
CPU entries to support different CPUs that may be inserted into
the socket with different numbers of cores.
Starting with ACPI 6.3 the spec has an Online Capable bit in the
MADT used to determine whether or not a CPU is hotplug capable
when the enabled bit is not set.
Link: https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html?#local-apic-flags
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This check is new in the beta version of clippy and exists to avoid
potential deadlocks by highlighting when the test in an if or for loop
is something that holds a lock. In many cases we would need to make
significant refactorings to be able to pass this check so disable in the
affected crates.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: accessing first element with `data.get(0)`
--> virtio-devices/src/transport/pci_device.rs:1055:34
|
1055 | if let Some(v) = data.get(0) {
| ^^^^^^^^^^^ help: try: `data.first()`
|
= note: `#[warn(clippy::get_first)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#get_first
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Tested:
1. SIGTERM based
2. VM shutdown/poweroff
3. Injected VM boot failure after calling Vm::setup_tty()
Fixes: #4248
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The snapshots are stored in a BTree which is ordered however as the ids
are strings lexical ordering places "11" ahead of "2". So encode the
vCPU id with zero padding so it is lexically sorted.
This fixes issues with CPU restore on aarch64.
See: #4239
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Git will refuse to generate information for the report if the repository
is owned by a different user (root in the container vs the real owner
outside).
e.g:
Error generating human readable git reference: fatal: unsafe repository ('/cloud-hypervisor' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /cloud-hypervisor
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is reasonable for these to fail as it the tool could be run outside
of a git repository but by not giving any error message we cannot see
issues when we expect the report to have the git details.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By default Microsoft Hypervisor send a GP to the guest if it tries
read/write an unimplemented MSR from the hypervisor prospective. Instead
change this behavior to ignore read/write operations for unimplemented
MSRs
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
When restoring a VM, the restore codepath will take care of mapping the
MMIO regions based on the information from the snapshot, rather than
having the mapping being performed during device creation.
When the device is created, information such as which BARs contain the
MSI-X tables are missing, preventing to perform the mapping of the MMIO
regions.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the VfioCommon implementation, the VfioUserPciDevice now
implements the Migratable trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the VfioCommon implementation, the VfioPciDevice now implements
the Migratable trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduces the common code to handle one aspect of the migration
support. Particularly, the ability to store VMM internal states related
to such device. The internal state of the device will happen later in a
dedicated patchset that will implement the VFIO migration API.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
So that we can save and restore the whole structure through snapshot and
restore operations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This reverts commit 0d0013c46e.
Grovvy shell script execution engine does not like backslash as the
escape character. So we need to put another backslash to escape the
backslash character. This would most likely fix the issue that we saw
with the CI.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
In order to conserve resources it is better to not run CI, whenever there
are changes only in fuzz/Cargo.toml or fuzz/Cargo.lock.
Fixes#4148
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
Based on recent KVM host patches (merged in Linux 5.16), it's forbidden
to call into KVM_SET_CPUID2 after the first successful KVM_RUN returned.
That means saving CPU states during the pause sequence, and restoring
these states during the resume sequence will not work with the current
design starting with kernel version 5.16.
In order to solve this problem, let's simply move the save/restore logic
to the snapshot/restore sequences rather than the pause/resume ones.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The vCPU is created and set after all the devices on a VM's boot.
There's no reason to follow a different order on the restore codepath as
this could cause some unexpected behaviors.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Combined the `GicDevice` struct in `arch` crate and the `Gic` struct in
`devices` crate.
After moving the KVM specific code for GIC in `arch`, a very thin wapper
layer `GicDevice` was left in `arch` crate. It is easy to combine it
with the `Gic` in `devices` crate.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
In order to ensure that the virtio device thread is spawned from the vmm
thread we use an asynchronous activation mechanism for the virtio
devices. This change optimises that code so that we do not need to
iterate through all virtio devices on the platform in order to find the
one that requires activation. We solve this by creating a separate short
lived VirtioPciDeviceActivator that holds the required state for the
activation (e.g. the clones of the queues) this can then be stored onto
the device manager ready for asynchronous activation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As coredump function is to make a vmcore for crash tool to analyze,
in order not to introduce a big thing in integration, we just check
if ch-remote command runs no error report here.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Based on the newly added guest_debug feature, this patch adds http
endpoint support.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Co-authored-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The crash tool use a special note segment which named 'QEMU' to
analyze kaslr info and so on. If we don't add the 'QEMU' note
segment, crash tool can't find linux version to move on.
For now, the most convenient way is to add 'QEMU' note segment to
make crash tool happy.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Guest memory is needed for analysis in crash tool, so save it
for coredump.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Co-authored-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's useful to dump the guest, which named coredump so that crash
tool can be used to analysize it when guest hung up.
Let's add GuestDebuggable trait and Coredumpxxx error to support
coredump firstly.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Co-authored-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The error message incorrectly said that the user was trying to combine
cache_size without dax whereas it is only usuable with dax.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the code from the DeviceManager that prepares the DAX cache since
the functionality has now been removed.
Fixes: #3889
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`GicDevice` trait was defined for the common part of GicV3 and ITS.
Now that the standalone GicV3 do not exist, `GicDevice` is not needed.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Why combine:
- GicV3 is not required alone
- GicV3 and GicV3Its has separate snapshot/pause code. But the code of
GicV3 was never used.
- Reduce the code complexity of GIC related traits and structs.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
From the logs it appears that booting the VM to the point at which it
can signal to the host can sometimes take longer than then 30 seconds
specified.
Fixes: #4136
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactoring and features in this patch:
- Add support for aarch64 build
- Add offline build configuration using vendored crates
- Add script to generate vendored sources
- Fix openssl-sys dependency
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
This reverts commit 6a099257e8.
It is now clear that pinning the toolchain for cross compilation is not
necessary since we only use it for building to aarch64:
https://github.com/rust-lang/rust/issues/95926#issue-1199547707
"This problem doesn't currently show up when cross-compiling from x86_64
to aarch64, since aarch64 doesn't use static-pie by default, but
enabling PIE with -C relocation-model=pie triggers the same bug and
makes the resulting binaries segfault."
Fixes: #3962
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the capturing of the debug port event without requiring more
verbose logging which itself has a performance impact.
Unless users are using our custom kernel or their own custom patches
they are unlikely to see the impact.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit f160572f9d.
There has been increased flakiness around the live migration tests since
this was merged. Speculatively reverting to see if there is increased
stability.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to ensure that the virtio device thread is spawned from the vmm
thread we use an asynchronous activation mechanism for the virtio
devices. This change optimises that code so that we do not need to
iterate through all virtio devices on the platform in order to find the
one that requires activation. We solve this by creating a separate short
lived VirtioPciDeviceActivator that holds the required state for the
activation (e.g. the clones of the queues) this can then be stored onto
the device manager ready for asynchronous activation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The current patch fixes the following error that was raised by clippy:
error: this let-binding has unit value
--> tests/integration.rs:6538:13
|
6538 | / let _ = stdin
6539 | | .write_all("type=7".as_bytes())
6540 | | .expect("failed to write stdin");
| |_________________________________________________^
|
= note: `-D clippy::let-unit-value` implied by `-D warnings`
= help: for further information visit
https://rust-lang.github.io/rust-clippy/master/index.html#let_unit_value
help: omit the `let` binding
|
6538 ~ stdin
6539 + .write_all("type=7".as_bytes())
6540 + .expect("failed to write stdin");
|
error: could not compile `cloud-hypervisor` due to previous error
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Latest cargo beta version raises warnings about unused macro rules.
Simply remove them to fix the beta build.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Cut and paste doesn't quite work. Not that cut and pasters will get
this far along, but the docs should be correct in any regards.
Signed-off-by: Steven Dake <steven.dake@gmail.com>
The qemu-img tool is used as part of the tutorial but is not installed
by default on a fresh ubuntu 22.04 server.
Signed-off-by: Steven Dake <steven.dake@gmail.com>
There is no need to include serde_derive separately,
as it can be specified as serde feature instead.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
In this way, we can cover a broad range of events from the event monitor
while avoiding code duplication.
Fixes: #4054
Signed-off-by: Bo Chen <chen.bo@intel.com>
Rely on the newly added helper from vm-virtio crate to keep cloning the
list of Queue structures.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the QueueState structure has been updated by not implementing
Clone anymore, we need a way to keep cloning the Queue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This variable name is residual from when these functions acted directly
on the vCPU fd rather than the hypervisor wrapper.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Explicitly re-export types from the hypervisor specific modules. This
makes it much clearer what the common functionality that is exposed is.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All the required functionality is already exported from the hypervisor
crate so for consistency make this module private.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
And thus only export what is necessary through a `pub use`. This is
consistent with some of the other modules and makes it easier to
understand what the external interface of the hypervisor crate is.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move this enum from vm-device to hypervisor crate so that hypervisor
crate does not gain an extra dependency.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By taking advantage of the fact that IrqRoutingEntry is exported by the
hypervisor crate (that is typedef'ed to the hypervisor specific version)
then the code for handling the MsiInterruptManager can be simplified.
This is particularly useful if in this future it is not a typedef but
rather a wrapper type.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the requirement to leak as many datastructures from the
hypervisor crate into the vmm crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The trait and functionality is about operations on the VM rather than
the VMM so should be named appropriately. This clashed with with
existing struct for the concrete implementation that was renamed
appropriately.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Whenever going through the codepath of loading a RAW firmware, we always
add an extra RAM region to the guest memory through the memory manager.
But we must be careful to use the updated guest memory rather than a
previous reference that wasn't containing the new region, as this can
lead to the following error:
VmBoot(FirmwareLoad(InvalidGuestAddress(GuestAddress(4290772992))))
This is fixed by the current patch, getting the latest reference onto
the guest memory from the memory manager right after the new region has
been added.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We can return prematurely from 'map_mmio_regions()' (e.g. when a mmap call
failed for vfio or 'create_user_memory_region()' failed for vfio-user)
without updating the 'MmioRegion::user_memory_regions' with the
information of previous successful mmaps, which in turn would cause mmap
leaks particularly for the case of hotplug where the 'vmm' thread will
keep running. To fix the issue, let's keep 'MmioRegion::user_memory_regions'
updated right after successful mmap calls.
Fixes: #4068
Signed-off-by: Bo Chen <chen.bo@intel.com>
This is required when hot-removing a vfio-user device. Details code path
below:
Thread 6 "vcpu0" received signal SIGSYS, Bad system call.
[Switching to Thread 0x7f8196889700 (LWP 2358305)]
0x00007f8196dae7ab in shutdown () at ../sysdeps/unix/syscall-template.S:78
78 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
0x00007f8196dae7ab in shutdown () at ../sysdeps/unix/syscall-template.S:78
0x000056189240737d in std::sys::unix::net::Socket::shutdown ()
at library/std/src/sys/unix/net.rs:383
std::os::unix::net::stream::UnixStream::shutdown () at library/std/src/os/unix/net/stream.rs:479
0x000056189210e23d in vfio_user::Client::shutdown (self=0x7f8190014300)
at vfio_user/src/lib.rs:787
0x00005618920b9d02 in <pci::vfio_user::VfioUserPciDevice as core::ops::drop::Drop>::drop (
self=0x7f819002d7c0) at pci/src/vfio_user.rs:551
0x00005618920b8787 in core::ptr::drop_in_place<pci::vfio_user::VfioUserPciDevice> ()
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ptr/mod.rs:188
0x00005618920b92e3 in core::ptr::drop_in_place<core::cell::UnsafeCell<dyn pci::device::PciDevice>>
() at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ptr/mod.rs:188
0x00005618920b9362 in core::ptr::drop_in_place<std::sync::mutex::Mutex<dyn pci::device::PciDevice>> () at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ptr/mod.rs:188
0x00005618920d8a3e in alloc::sync::Arc<T>::drop_slow (self=0x7f81968852b8)
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/sync.rs:1092
0x00005618920ba273 in <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop (self=0x7f81968852b8)
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/sync.rs:1688
0x00005618920b76fb in core::ptr::drop_in_place<alloc::sync::Arc<std::sync::mutex::Mutex<dyn pci::device::PciDevice>>> ()
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ptr/mod.rs:188
0x0000561891b5e47d in vmm::device_manager::DeviceManager::eject_device (self=0x7f8190009600,
pci_segment_id=0, device_id=3) at vmm/src/device_manager.rs:4000
0x0000561891b674bc in <vmm::device_manager::DeviceManager as vm_device::bus::BusDevice>::write (
self=0x7f8190009600, base=70368744108032, offset=8, data=&[u8](size=4) = {...})
at vmm/src/device_manager.rs:4625
0x00005618921927d5 in vm_device::bus::Bus::write (self=0x7f8190006e00, addr=70368744108040,
data=&[u8](size=4) = {...}) at vm-device/src/bus.rs:235
0x0000561891b72e10 in <vmm::vm::VmOps as hypervisor::vm::VmmOps>::mmio_write (
self=0x7f81900097b0, gpa=70368744108040, data=&[u8](size=4) = {...}) at vmm/src/vm.rs:378
0x0000561892133ae2 in <hypervisor::kvm::KvmVcpu as hypervisor::cpu::Vcpu>::run (
self=0x7f8190013c90) at hypervisor/src/kvm/mod.rs:1114
0x0000561891914e85 in vmm::cpu::Vcpu::run (self=0x7f819001b230) at vmm/src/cpu.rs:348
0x000056189189f2cb in vmm::cpu::CpuManager::start_vcpu::{{closure}}::{{closure}} ()
at vmm/src/cpu.rs:953
Signed-off-by: Bo Chen <chen.bo@intel.com>
Since both Net and vhost_user::Net implement the Migratable trait, we
can factorize the common part to simplify the code related to the net
creation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since both Block and vhost_user::Blk implement the Migratable trait, we
can factorize the common part to simplify the code related to the disk
creation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the validate() function for both DiskConfig and NetConfig so that
we return an error if a vhost-user-block or vhost-user-net device is
expected to be placed behind the virtual IOMMU. Since these devices
don't support this feature, we can't allow iommu to be set to true in
these cases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Reorganizing the code to leverage the same mechanics implemented for
vfio-user and aimed at supporting sparse memory mappings for a single
region.
Relying on the capabilities returned by the vfio-ioctls crate, we create
a list of sparse areas depending if we get SPARSE_MMAP or MSIX_MAPPABLE
capability, or a single sparse area in case we couldn't find any
capability.
The list of sparse areas is then used to create both the memory mappings
in the Cloud Hypervisor address space and the hypervisor user memory
regions.
This allowed for the simplification of the MmioRegion structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This is a cleaner approach to handling the I/O port write to 0x80.
Whilst doing this also use generate the timestamp at the start of the VM
creation. For consistency use the same timestamp for the ARM equivalent.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We don't use the VmmOps trait directly for manipulating memory in the
core of the VMM as it's really designed for the MSHV crate to handle
instruction decoding. As I plan to make this trait MSHV specific to
allow reduced locking for MMIO and PIO handling when running on KVM this
use should be removed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of always creating a single large mmap for the MMIO region of a
BAR, we create multiple mmaps for the BARs that need multiple kvm user
memory regions. In this way, we can simplify 'unmap_mmio_regions()' (by
reusing information kept from 'MmioRegion::user_memory_region').
Signed-off-by: Bo Chen <chen.bo@intel.com>
To correctly map MMIO regions to the guest, we will need to wait for valid
MMIO region information which is generated from 'PciDevice::allocate_bars()'
(as a part of 'DeviceManager::add_pci_device()').
Signed-off-by: Bo Chen <chen.bo@intel.com>
To prepare the support of region capabilities, this commit factorizes
the `VFIO_USER_DEVICE_GET_REGION_INFO` operation in its own function. In
this process, it also removes the need of the hard-coded arbitrary value
for `vfio_region_info.argsz` (the maximum size of the reply payload).
Signed-off-by: Bo Chen <chen.bo@intel.com>
Similar to what's being supported for vfio devices, vfio-user devices
may also have BARs that need multiple kvm user memory regions,
e.g. device regions with `VFIO_REGION_INFO_CAP_SPARSE_MMAP`.
Signed-off-by: Bo Chen <chen.bo@intel.com>
For devices that cannot be named by the user use the "__" prefix to
identify them as internal devices. Check that any identifiers provided
in the config do not clash with those internal names. This prevents the
user from creating a disk such as "__serial" which would then cause a
failure in unpredictable manner.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Whenever a device (virtio, vfio, vfio-user or vdpa) is hotplugged, we
must verify the provided identifier is unique, otherwise we must return
an error.
Particularly, this will prevent issues with identifiers for serial,
console, IOAPIC, balloon, rng, watchdog, iommu and gpio since all of
these are hardcoded by the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
All hotpluggable devices were properly removed from the VmConfig when a
remove-device command was issued, except for the "fs" type. Fix this
lack of support as it is causing the integration tests to fail with the
recent addition of verifying that identifiers are unique.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The device identifiers generated from the DeviceManager were not
guaranteed to be unique since they were not taking the list of
identifiers provided through the configuration.
By returning the list of unique identifiers from the configuration, and
by providing it to the DeviceManager, the generation of new identifiers
can rely both on the DeviceTree and the list of IDs from the
configuration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This prevents a conflict since the old API socket will not have been
cleaned up (due to the use of SIGKILL.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The socket will safely deleted on shutdown and so it is not necessary to
delete the API socket when starting the HTTP server.
Fixes: #4026
Signed-off-by: LiHui <andrewli@kubesphere.io>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Start loading the kernel as possible in the VM in a separate thread.
Whilst it is loading other work can be carried out such as initialising
the devices.
The biggest performance improvement is seen with a more complex set of
devices. If using e.g. four virtio-net devices then the time to start the
kernel improves by 20-30ms. With the simplest configuration the
improvement was of the order of 2-3ms.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the same code for generating the kernel command line to be
used on both aarch64 and x86_64 when the latter starts loading the
kernel in asynchronously.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is not required for x86_64 and maintains a tight coupling between
kernel loading and the DeviceManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 87eed369cd.
The reason we're reverting this is that OpenAPI Specification[0] doesn't
know how to deal with unsigned types. :-/
Right now the best to do is keep it as it's, as an int64, and try to fix
OpenAPI, or even switch to swagger, as the latter knows how to properly
deal with those. However, switching to swagger is far from being an 1:1
transition and will require time to experiment, thus reverting this for
now seems the best approach.
[0]: https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md#data-types
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The Token Bucket fields are, on the Cloud Hypervisor side, u64.
However, we expose those as int64 in the OpenAPI YAML file.
With that in mind, let's adjust the yaml file to expose those as uint64.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This means that the automatic enabling of the virtio-iommu will also be
applied to VMs creates via the API as well as the CLI.
Fixes: #4016
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Improve the request parsing/handling code by allowing an error status to
be returned back to the guest driver before we return an error
internally.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the Domain structure to store the information about each domain
being in bypass mode or not. Based on this new information, the address
translation of the virtio devices is performed according to the bypass
mode of each domain. And both MAP/UNMAP requests are generating errors
in case the domain has been previously set to bypass mode.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for associating more than mappings with a domain, we
factorize the list of mappings associated with a domain behind a
dedicated Domain structure. We also update the field name so that it
reads better in the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Exposing the VIRTIO_IOMMU_F_BYPASS_CONFIG feature to the guest, which
allows to update the bypass global knob through virtio configuration.
Based on the value of this global knob, the address translations for
endpoints that have not been added to a domain is allowed with a simple
identity mapping.
By default, we enable the bypass mode for all endpoints that are not
attached to any domain.
Fixes#3987
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the VIRTIO specification, we must be able to support multiple
endpoints per domain. This is fixed along with the introduction of some
simplification regarding how we can retrieve the external mapping
directly based on the endpoint.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If using the ACPI based hotplug only memory can be added so if the
hotplug RAM size is the same as the boot RAM size then do not include
the memory manager DSDT entries.
Also: this change simplifies the code marginally by making the
HotplugMethod enum Copyable.
This was identified from the following perf output:
1.78% 0.00% vmm cloud-hypervisor [.] <vmm::memory_manager::MemorySlots as acpi_tables::aml::Aml>::append_aml_bytes
|
---<vmm::memory_manager::MemorySlots as acpi_tables::aml::Aml>::append_aml_bytes
<vmm::memory_manager::MemorySlot as acpi_tables::aml::Aml>::append_aml_bytes
acpi_tables::aml::Name::new
<acpi_tables::aml::Path as acpi_tables::aml::Aml>::append_aml_bytes
__libc_malloc
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
No further changes are necessary that adding a #[derive(Error)] as there
is a manual implementation of Display.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend VfioCommon to simplify the overall code, and also in preparation
for supporting the restore code path.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend VfioCommon structure to own the legacy interrupt manager. This
will be useful for implementing the restore code path.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend VfioCommon structure to own the MSI interrupt manager. This will
be useful for implementing the restore code path.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need to split the parsing functions into one function dedicated to
the actual parsing and a second function for initializing the interrupt
type. This will be useful on the restore path as the parsing won't be
needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding support for snapshot and restore to the MsiConfig structure, as
it will be needed part of VFIO migration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This carries a string that is exposed via DMI/SMBIOS and is particularly
useful for cloud-init initialisation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is particularly helpful as the cloud-init DMI serial number
encoding scheme uses "=" and ";" for key value pairs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the guest has not activated the virtio-mem device then reject an
attempt to resize using it.
Fixes: #4001
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Live upgrade is currently not guaranteed during this development cycle
and we will try to enable these tests after the next release.
Signed-off-by: Bo Chen <chen.bo@intel.com>
By augmenting existing set of tests, this patch added a set of
tests for live-upgrade that covers use cases with NUMA,
vhost-user (OVS-DPDK), and local-migration.
Fixes: #3949
Signed-off-by: Bo Chen <chen.bo@intel.com>
Use a single enum member for representing errors from the internal API.
This avoids the ugly duplication of the API call name in the error
message:
e.g.
$ target/debug/ch-remote --api-socket /tmp/api resize --cpus 2
Error running command: Server responded with an error: InternalServerError: VmResize(VmResize(CpuManager(DesiredVCpuCountExceedsMax)))
Becomes:
$ target/debug/ch-remote --api-socket /tmp/api resize --cpus 2
Error running command: Server responded with an error: InternalServerError: ApiError(VmResize(CpuManager(DesiredVCpuCountExceedsMax)))
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case a list of resources is provided to allocate_bars(), it directly
means we're restoring some existing BARs. That's why we shouldn't share
the codepath that creates BARs from scratch as we don't need to interact
with the device to retrieve the information.
Whenever resources are provided, we simply iterate over the list of
possible BAR indexes and create the BARs if the resource could be found.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of defining some very generic resources as PioAddressRange or
MmioAddressRange for each PCI BAR, let's move to the new Resource type
PciBar in order to make things clearer. This allows the code for being
more readable, but also removes the need for hard assumptions about the
MMIO and PIO ranges. PioAddressRange and MmioAddressRange types can be
used to describe everything except PCI BARs. BARs are very special as
they can be relocated and have special information we want to carry
along with them.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to make the code more consistent and easier to read, we remove
the former tuple that was used to describe a BAR, replacing it with the
existing structure PciBarConfiguration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The code was quite unclear regarding the type of index that was being
used regarding a BAR. This is improved by differenciating register
indexes and BAR indexes more clearly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By factorizing the algorithm untangling TDVF sections from guest RAM
into a dedicated function, we can write some unit tests to validate it
properly achieves what we expect.
Adding the "tdx" feature to the unit tests, otherwise it wouldn't get
tested.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The I/O operations knobs are prefixed `ops_` rather than `bw_`, as `bw_`
refers to the "bandwidth" knobs.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
By adding a new method id() to the PciDevice trait, we allow the caller
to retrieve a unique identifier. This is used in the context of BAR
relocation to identify the device being relocated, so that we can update
the DeviceTree resources for all PCI devices (and not only
VirtioPciDevice).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the list of resources, VfioCommon is now able to allocate the
BARs at specific addresses. This will be useful for restoring VFIO and
vfio-user devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By returning the new PCI resources from add_pci_device(), we allow the
factorization of the code translating the BARs into resources. This
allows VIRTIO, VFIO and vfio-user to add the resources to the DeviceTree
node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the function introduced recently to get the PCI resources and
handle the restore case, both VFIO and vfio-user device creation paths
now have access to PCI resources, which can be provided to the function
add_pci_device().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create a dedicated function for getting the PCI segment, b/d/f and
optional resources. This is meant for handling the potential case of a
restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the way of restoring BAR addresses for virtio-pci by providing
a more generic approach that will be reused for other PciDevice
implementations (i.e VfioPcidevice and VfioUserPciDevice).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The calls to these functions are always preceded by a call to
InterruptSourceGroup::update(). By adding a masked boolean to that
function call it possible to remove 50% of the calls to the
KVM_SET_GSI_ROUTING ioctl as the the update will correctly handle the
masked or unmasked case.
This causes the ioctl to disappear from the perf report for a boot of
the VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It suffered from the same issue like the main program after switching to
Rust 2021 edition, but the issue was not caught by the CI.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
EDK2 execution requires a flash device at address 0.
The new added device is not a fully functional flash. It doesn't
implement any spec of a flash device. Instead, a piece of memory is used
to simulate the flash simply.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
We modified a test case to workaround the RAM calculation error caused
by hidding 4MiB memory for UEFI. Now change it back to normal.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Size of `memory` node in FDT was reduced by 4MiB. Now it is returned.
In memory regions, a `Ram` region of 4MiB was created at address 0. Now
it is removed.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Rust 2021 edition has a few improvements over the 2018 edition. Migrate
the project to 2021 edition by following recommended migration steps.
Luckily, the code itself doesn't require fixing.
Bump MSRV to 1.56 as it is required by the 2021 edition. Also fix the
clap build dependency to make Cloud Hypervisor build again.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This is a refactoring commit to simplify source code.
Removed some functions that only return a layout const.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Some addresses defined in `layout.rs` were of type `GuestAddress`, and
are `u64`. Now align the types of all the `*_START` definitions to
`GuestAddress`.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Now that address translations performed by virtio-iommu can error out if
the address can't be translated, we uncovered an issue in integration
test aarch64_acpi::test_virtio_iommu.
We disable the test until we can investigate and fix the root cause.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It doesn't matter if we're trying to translate a GVA or a GPA address,
but in both cases we must error out if the address couldn't be
translated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Remove EOLed distributions and clarify the stability guarantees and what
the experimental features are.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The reserved space is for devices.
Some devices (like TPM) require arbitrary addresses close to 4GiB.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
`RAM_64BIT_START` was set to 1 GiB, not a real 64-bit address. Now
rename it `RAM_START` to avoid confusion.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The extra vDPA device in the test is hotplugged behind the vIOMMU, which
covers the use case of placing a vDPA device behind a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever a virtio device is placed behind a vIOMMU, we have some code in
pci_common_config.rs to translate the queue addresses (descriptor table,
available ring and used ring) from GVA to GPA, so that they can be used
correctly.
But in case of vDPA, we also need to provide the queue addresses to the
vhost backend. And since the vhost backend deals with consistent IOVAs,
all addresses being provided should be GVAs if the device is placed
being a vIOMMU. For that reason, we perform a translation of the queue
addresses back from GPA to GVA if necessary, and only to be provided to
the vhost backend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new iommu parameter to VdpaConfig in order to place the vDPA
device behind a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case an external mapping would have been added after the virtio-iommu
device has been activated, it would have simply be ignored because the
code wasn't using a shared object between the vmm thread and the iommu
thread. This behavior is only triggered on the hotplug codepath, and
only if the hotplugged device is placed behind the virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for the vDPA need to translate a GPA back into a GVA, we
extend the existing trait DmaRemapping and AccessPlatform to perform
such operation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Renaming translate() to translate_gva() to clarify we want to translate
a GVA address into a GPA.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
According to latest official images based on latest TDX guest kernel, we
update the documentation to reflect the change regarding serial support.
New guest kernels have been updated to disable usage of serial ports,
meaning adding console=ttyS0 to the kernel boot parameters will have no
effect.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The list of memory resources provided through the HOB wasn't accurate
because of the broken logic. The fix provides correct ranges to the
firmware.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on latest QEMU patches from branch tdx-qemu-2022.03.29-v7.0.0-rc1
we should only report as memory resources the TempMem sections from TDVF
sections.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on latest QEMU patches from branch tdx-qemu-2022.03.29-v7.0.0-rc1
we don't need EFI_RESOURCE_ATTRIBUTE_ENCRYPTED as part of the attributes
we must enable with EFI_RESOURCE_SYSTEM_MEMORY and
EFI_RESOURCE_MEMORY_RESERVED resource types.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The introduction of a error if live resizing is not possible is a
regression compared to the original behaviour where the new size would
be stored in the config and reflected in the next boot. This behaviour
was also inconsistent with the effect of resizing with no VM booted.
Instead of generating an error allow the code to go ahead and update the
config so that the new size will be available upon the reboot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As reported by the periodic CI runs, it may take more time for the NVMe
device to present in the guest after being hotplugged as a VFIO user
device on `aarch64` (especially under high load). Let's increase the
timeout after device hotplug from `1s` to `10s` to increase the test
stability.
Fixes: #3495
Signed-off-by: Bo Chen <chen.bo@intel.com>
Similarly to the previous commit restricting the cpu resizing error only
to the situations where the vcpu amount has changed, let's do the same
with the memory and be consistent throughout our code base.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
188078467d made clear that resize should
only happen when dealing with a "dynamic" CpuManager. Although this is
very much correct, it causes a regression on Kata Containers (and on any
other consumer of Cloud Hypervisor) in cases where a resize would be
triggered but the vCPUs values wouldn't be changed.
There's no doubt Kata Containers could do better and do not call a
resize in such situations, and that's something that should **also** be
solved there. However, we should also work this around on Cloud
Hypervisor side as it introduces a regression with the current Kata
Containers code.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Extend the Hypervisor API in order to retrieve the TDX capabilities from
the underlying hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By enabling the VIRTIO feature VIRTIO_F_IOMMU_PLATFORM for all
vhost-user devices when needed, we force the guest to use the DMA API,
making these devices compatible with TDX. By using DMA API, the guest
triggers the TDX codepath to share some of the guest memory, in
particular the virtqueues and associated buffers so that the VMM and
vhost-user backends/processes can access this memory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If EFI reset fails on the Linux kernel then it will fallthrough to CMOS
reset. Implement this as one of our reset solutions.
Fixes: #3912
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Compile this feature in by default as it's well supported on both
aarch64 and x86_64 and we only officially support using it (no non-acpi
binaries are available.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These were erroneously skipping features for the unit tests and the
"build" target for dev_cli.sh
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These are never run by the CI and is inconsistent with the way we build
test which is specified inside the .github workflows.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This includes the removal of testing without the "acpi" feature. The
command have been reordered to reduce the amount of recompilation
required.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It seems the vdpa_sim_block isn't behaving properly after the vhost
device is closed, as it sometimes returns EBUSY when we try to open it
again. The easiest way to deal with this issue is by simplifying the
integration test, avoid to plug the same device after it's been
unplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit unifies the custom linux kernel build in x86, Arm, and
performance metrics to the same function. Therefore, when bumping
the kernel version, we can make sure we only need to make the change
in one place.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
With the addition of the amx feature, add a new build workflow to
enable the feature and a clippy quality check.
Signed-off-by: William Douglas <william.douglas@intel.com>
AMX is an x86 extension adding hardware units for matrix
operations (int and float dot products). The goal of the extension is
to provide performance enhancements for these common operations.
On Linux, AMX requires requesting the permission from the kernel prior
to use. Guests wanting to make use of the feature need to have the
request made prior to starting the vm.
This change then adds the first --cpus features option amx that when
passed will enable AMX usage for guests (needs a 5.17+ kernel) or
exits with failure.
The activation is done in the CpuManager of the VMM thread as it
allows migration and snapshot/restore to work fairly painlessly for
AMX enabled workloads.
Signed-off-by: William Douglas <william.douglas@intel.com>
Now that we rely on vhost v0.4.0, which contains the fix for
get_iova_range(), we don't need the workaround anymore, and we can
actually call into the dedicated function.
Fixes#3861
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rely on newly released versions of the vhost and vhost-user-backend
crates from rust-vmm.
The new vhost version includes the fixes needed for vDPA.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit makes sure that the custom linux kernel is always
rebuilt when running the performance metrics tests, and therefore
changes to the kernel config file is always caught.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Explain the reason why using vDPA might be interesting and how to use it
with Cloud Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Disable the DAX feature from the virtio-fs implementation as the feature
is still not stable. The feature is deprecated, meaning the 'dax'
parameter will be removed in about 2 releases cycles.
In the meantime, the parameter value is ignored and forced to be
disabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The test is sporadically failing whenever we try to hotplug the vDPA
device we've just unplugged. This is causing the kernel to complain with
EBUSY because the device hasn't been released yet. This is happening
because the CI system is under very high load, therefore taking quite
some time to the host to update the state of this device.
The easy way to fix such issue is by increasing the sleep time between
the unplug and the replug.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When running non-dynamic or with virtio-mem for hotplug the ACPI
functionality should not be included on the DSDT nor does the
MemoryManager need to be placed on the MMIO bus.
Fixes: #3883
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is now consistent with not supplying the _CRS for the device when
CpuManager is not dynamic.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The file descriptor provided to fs_slave_map() and fs_slave_io() is
passed as a AsRawFd trait, meaning the caller owns it. For that reason,
there's no need for these functions to close the file descriptor as it
will be closed later on anyway.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As ethr is not stable on AArch64, disable it until we find other tool.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
fio test depends on hot-add disk which needs uefi on AArch64, that
will lead to lots of performance issues. So, it's better to use
cold-plug disk.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
On Arm64, the fio binary that comes from ubuntu distribution cannot
support io_uring, which is a required engine for the block
performance tests in performance metrics. Therefore this commit
build the fio from a relatively new version, and replace the native
ubuntu fio with the newly build one.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
To emulate debug I/O port on arm64, we need trap from guest to VMM.
Generally, there are 3 ways to go:
1) execute a privilege intruction like "wfi";
2) access a system register;
3) access device MMIO region;
However, 1) and 2) often handled in kvm, thus 3) it's the choice.
There maybe lots of repeated work and enlarge the size of clh to develop
a new device, so it's better to reuse the current device. Luckily, I
find that pl011 is the eariest device initialized in kernel and, there
is reserved region in it which we can reuse for emulating debug I/O.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Rather than just printing a message return an error back through the API
if the user attempts to hotplug a device that supports being behind an
IOMMU where that device isn't placed on an IOMMU segment.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ensure devices that are specified to be on a PCI segment that is behind
the IOMMU are IOMMU enabled if possible or error out for those devices
that do not support it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previously it was not possible to enable vIOMMU for a virtio device.
However with the ability to place an entire PCI segment behind the
IOMMU the IOMMU mapping needs to be setup for the virtio device if it is
behind the IOMMU.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This can already be calculated by the summing the tables reported by the
Linux kernel but this is more convenient.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Changes in commit "scripts: aarch64: Reduce source code build in CI"
for the Linux kernel will lead to the kernel cannot be rebuilt if
the config file is changed. Therefore should be reverted.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As Linux kernel, edk2, virtiofd, SPDK are all build from source, we
need to find a way to reduce the number of source code build.
To address this issue, this commit adds a stage that touching a
".built" file to each repo of the source code after a successfully
build. By checking if the ".built" file exists, we can determine
if there is already an old build. Since the source code repo will
be removed and cloned again when we bump the repo hash, the CI will
rebuild the repo.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Whenever a vfio-user device is dropped, the communication between the
VMM and the backend should be shutdown.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Separate the destruction and cleanup of original VM and the creation of
the new one. In particular have a clear hand off point for resources
(e.g. reset EventFd) used by the new VM from the original. In the
situation where vm.shutdown() generates an error this also avoids the
Vmm reference to the Vm (self.vm) from being maintained.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding two new integration tests for vDPA, relying on both block and net
simulators from the host kernel.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the newly added Vdpa device along with the new vdpa parameter,
this patch enables the support for vDPA devices.
It's important to note this the only virtio device for which we provide
an ExternalDmaMapping instance. This will allow for the right DMA ranges
to be mapped/unmapped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduce a new --vdpa parameter associated with a VdpaConfig for the
future creation of a Vdpa device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
vDPA is a kernel framework introduced fairly recently in order to handle
devices complying with virtio specification on their datapath, while the
control path is vendor specific. For the datapath, that means the
virtqueues are handled through DMA directly between the hardware and the
guest, while the control path goes through the vDPA framework,
eventually exposed through a vhost-vdpa device.
vDPA, like VFIO, aims at achieving baremetal performance for devices
that are passed into a VM. But unlike VFIO, it provides a simpler/better
framework for achieving migration. Because the DMA accesses between the
device and the guest are going through virtio queues, migration can be
achieved way more easily, and doesn't require each device driver to
implement the migration support. In the VFIO case, each vendor is
expected to provide an implementation of the VFIO migration framework,
which makes things harder as it must be done for each and every device.
So to summarize the point is to support migration for hardware devices
through which we can achieve baremetal performances.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Update the content of the documentation to clarify how the cache option
from virtiofsd has an impact on the host page cache, while the dax
option for the filesystem in the guest has an impact on the guest page
cache.
The document is also updated to not have lines exceeding 80 characters.
Fixes#3826
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Document the reason why someone might need to use the balloon device and
detail the different parameters associated with it.
Fixes#3807
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will significantly reduce the size of the DSDT and the effort
required to parse them if there is no requirement to support
hotplug/unplug.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the CpuManager is dynamic it devices CPUs can be
hotplugged/unplugged.
Since TDX does not support CPU hotplug this is currently the only
determinator as to whether the CpuManager is dynamic.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since the code has been adapted to support VIRTIO_F_EVENT_IDX we use
Queue::needs_notification() to determine whether to signal the guest so
it is no longer necessary to check if there are any used descriptors. If
the feature is not negotiated then Queue::needs_notification() will
return true triggering an interrupt of the guest. Theoretically this
could be a spurious interrupt of the guest if there were no used used
descriptors but this is unlikely as we only generate used descriptors
for the control queue as a result of an interrupt of the VMM by the
guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This indicates if anything has been added to the available queue since
it was last iterated through. If it returns true then it is necessary to
iterate through the queue again otherwise it is appropriate to break out
from the loop.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
vmm.ping/vm.info will hang for PUT method, vm.create/vmm.shutdonw hang for GET method.
Because these four APIs do not write the response body when the HTTP method does not match.
Signed-off-by: LiHui <andrewli@kubesphere.io>
In case the virtio device which requires DMA mapping is placed behind a
virtual IOMMU, we shouldn't map/unmap any region manually. Instead, we
provide the DMA handler to the virtio-iommu device so that it can
trigger the proper mappings.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If a virtio device is associated with a DMA handler, the DMA mapping and
unmapping is performed from the device manager through the handler.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given that some virtio device might need some DMA handling, we provide a
way to store this through the VirtioPciDevice layer, so that it can be
accessed when the PCI device is removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for handling potential DMA mapping/unmapping operations for a
virtio device, we extend the MetaVirtioDevice with an additional field
that holds an optional DMA handler.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The tuple of information related to each virtio device is too big, and
it's better to factorize it through a dedicated structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the VIRTIO_F_EVENT_IDX handling now conducted inside the
virtio-queue crate it is necessary to activate the functionality on
every queue if it is negotiatated. Otherwise this leads to a failure of
the guest to signal to the host that there is something in the available
queue as the queue's internal state has not been configured correctly.
Fixes: #3829
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When mask a msi irq, we set the entry.masked to be true, so kvm
hypervisor will not pass the gsi to kernel through KVM_SET_GSI_ROUTING
ioctl which update kvm->irq_routing. This will trigger kernel
panic on AMD platform when the gsi is the largest one in kernel
kvm->irqfds.items:
crash> bt
PID: 22218 TASK: ffff951a6ad74980 CPU: 73 COMMAND: "vcpu8"
#0 [ffffb1ba6707fa40] machine_kexec at ffffffff8565b397
#1 [ffffb1ba6707fa90] __crash_kexec at ffffffff85788a6d
#2 [ffffb1ba6707fb58] crash_kexec at ffffffff8578995d
#3 [ffffb1ba6707fb70] oops_end at ffffffff85623c0d
#4 [ffffb1ba6707fb90] no_context at ffffffff856692c9
#5 [ffffb1ba6707fbf8] exc_page_fault at ffffffff85f95b51
#6 [ffffb1ba6707fc50] asm_exc_page_fault at ffffffff86000ace
[exception RIP: svm_update_pi_irte+227]
RIP: ffffffffc0761b53 RSP: ffffb1ba6707fd08 RFLAGS: 00010086
RAX: ffffb1ba6707fd78 RBX: ffffb1ba66d91000 RCX: 0000000000000001
RDX: 00003c803f63f1c0 RSI: 000000000000019a RDI: ffffb1ba66db2ab8
RBP: 000000000000019a R8: 0000000000000040 R9: ffff94ca41b82200
R10: ffffffffffffffcf R11: 0000000000000001 R12: 0000000000000001
R13: 0000000000000001 R14: ffffffffffffffcf R15: 000000000000005f
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffffb1ba6707fdb8] kvm_irq_routing_update at ffffffffc09f19a1 [kvm]
#8 [ffffb1ba6707fde0] kvm_set_irq_routing at ffffffffc09f2133 [kvm]
#9 [ffffb1ba6707fe18] kvm_vm_ioctl at ffffffffc09ef544 [kvm]
RIP: 00007f143c36488b RSP: 00007f143a4e04b8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007f05780041d0 RCX: 00007f143c36488b
RDX: 00007f05780041d0 RSI: 000000004008ae6a RDI: 0000000000000020
RBP: 00000000000004e8 R8: 0000000000000008 R9: 00007f05780041e0
R10: 00007f0578004560 R11: 0000000000000246 R12: 00000000000004e0
R13: 000000000000001a R14: 00007f1424001c60 R15: 00007f0578003bc0
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
To solve this problem, move route.disable() before set_gsi_routes() to
remove the gsi from irqfds.items first.
This problem only exists on AMD platform, 'cause on Intel platform
kernel just return when update irte while it only prints a warning on
AMD.
Also, this patch adjusts the order of enable() and set_gsi_routes() in
unmask(), which should do no harm.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
When mask a msi irq, we set the entry.masked to be true, so kvm
hypervisor will not pass the gsi to kernel through KVM_SET_GSI_ROUTING
ioctl which update kvm->irq_routing. This will trigger kernel
panic on AMD platform when the gsi is the largest one in kernel
kvm->irqfds.items:
crash> bt
PID: 22218 TASK: ffff951a6ad74980 CPU: 73 COMMAND: "vcpu8"
#0 [ffffb1ba6707fa40] machine_kexec at ffffffff8565b397
#1 [ffffb1ba6707fa90] __crash_kexec at ffffffff85788a6d
#2 [ffffb1ba6707fb58] crash_kexec at ffffffff8578995d
#3 [ffffb1ba6707fb70] oops_end at ffffffff85623c0d
#4 [ffffb1ba6707fb90] no_context at ffffffff856692c9
#5 [ffffb1ba6707fbf8] exc_page_fault at ffffffff85f95b51
#6 [ffffb1ba6707fc50] asm_exc_page_fault at ffffffff86000ace
[exception RIP: svm_update_pi_irte+227]
RIP: ffffffffc0761b53 RSP: ffffb1ba6707fd08 RFLAGS: 00010086
RAX: ffffb1ba6707fd78 RBX: ffffb1ba66d91000 RCX: 0000000000000001
RDX: 00003c803f63f1c0 RSI: 000000000000019a RDI: ffffb1ba66db2ab8
RBP: 000000000000019a R8: 0000000000000040 R9: ffff94ca41b82200
R10: ffffffffffffffcf R11: 0000000000000001 R12: 0000000000000001
R13: 0000000000000001 R14: ffffffffffffffcf R15: 000000000000005f
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffffb1ba6707fdb8] kvm_irq_routing_update at ffffffffc09f19a1 [kvm]
#8 [ffffb1ba6707fde0] kvm_set_irq_routing at ffffffffc09f2133 [kvm]
#9 [ffffb1ba6707fe18] kvm_vm_ioctl at ffffffffc09ef544 [kvm]
RIP: 00007f143c36488b RSP: 00007f143a4e04b8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007f05780041d0 RCX: 00007f143c36488b
RDX: 00007f05780041d0 RSI: 000000004008ae6a RDI: 0000000000000020
RBP: 00000000000004e8 R8: 0000000000000008 R9: 00007f05780041e0
R10: 00007f0578004560 R11: 0000000000000246 R12: 00000000000004e0
R13: 000000000000001a R14: 00007f1424001c60 R15: 00007f0578003bc0
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
To solve this problem, move route.disable() before set_gsi_routes() to
remove the gsi from irqfds.items first.
This problem only exists on AMD platform, 'cause on Intel platform
kernel just return when update irte while it only prints a warning on
AMD.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Reduce the number of open pull requests dependabot will create to avoid
excessive rebase and rebuilds of PRs. This avoids a situation with a 2nd
PR being rebased after a 1st is merged resulting in a wasteful rebuild.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move to release version v0.2.0 for both vm-virtio and vhost-user-backend
crates rather than relying on their main branch, as they might be
subject to breaking changes.
Fixes#3800
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Replace the thread for handling SIGSYS with a simple signal handler.
This resolves inconsistent delivery of signals to the SIGSYS thread due
to other threads manipulating the signals.
Tested by removing key syscalls from vCPU and virtio device filters and
observing correct notice.
Fixes: #3811
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To be consistently with the integration scripts, we now pass through the
`--test-filter` option of the test scripts to the `performance-metrics`
binary. Now the following two commands achieve the same results:
`$./scripts/dev_cli.sh tests --metrics -- --test-filter boot_time_ -- --test-filter latency`
`$./scripts/dev_cli.sh tests --metrics -- -- --test-filter boot_time_ --test-filter latency`
Note that, the test scripts support only single occurrence of the
`--test-filter` option while the `performance-metrics` binary allows
multiple occurrences for filtering based on different keywords.
Fixes: #3787
Signed-off-by: Bo Chen <chen.bo@intel.com>
Add a field for its length and fix up users.
Things work just because all hardcoded values agree with each other.
This is prone to breakage.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Needed since:
commit 0ab4097606
Author: Bo Chen <chen.bo@intel.com>
Date: Thu Feb 10 18:29:53 2022 -0800
test_infra: Enable cross-build for "musl" and "aarch64" targets
With enabling the `vendered-openssl` feature, we can now cross-build the
`test_infra` crate for "musl" and "aarch64" targets. In this way, we
can remove the `test_infra` crate from the "exclude" list, so that this
crate can be checked and processed by "cargo clippy" and "cargo fmt".
More details can be found: https://docs.rs/openssl/latest/openssl/#vendored
As 'musl-gcc' is required, this commit also installs the `musl-tools`
package for our "build" github action on the musl target [1].
[1] https://github.com/actions-rs/toolchain/issues/102
Signed-off-by: Bo Chen <chen.bo@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Some host kernels are IPv6-only, and they cannot make an AF_INET socket.
The tun ioctls such as SIOCGIFHWADDR work with an AF_UNIX socket, which
all host kernels should have.
Signed-off-by: Barret Rhoden <brho@google.com>
Adding the missing TARGET_CC environment variable to get the build to
complete correctly.
Fixes#3776
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After writing to an address, Windows 11 on ARM64 unconditionally reads
it back. It is harmless. Drop the error message to avoid spamming.
Fixes: #3732
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This looks like it was copy and pasted from the network test which
required 2 vCPUs per queue pair but has since been resolved.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Using the default queue sizes provides more realistic data about what
our users are testing. Extra metrics can be added later that also
modify the queue size but overloading the existing metrics is confusing.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Don't treat it as the number of pairs. Instead calculate the number of
pairs later when necessary.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Also be clear that the block tests are in bytes per second (Bps) vs
network that is in bits per second (bps).
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The former is ambiguous as it could be considered the ordinal (and is
used elsewhere in the codebase for that).
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests already amortize their results over a longer time
period/sample so it is not necessary to run 30 iterations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Let's officially have a way to pass the features used to build
cloud-hypervisor to the dev_cli.sh script.
This doesn't invalidate the previous commit, as we still don't what the
features_build variable to be quoted, otherwise we face the following
issue:
```
error: Found argument '--features tdx' which wasn't expected, or isn't valid in this context
Did you mean --features?
USAGE:
cargo build --all --features <FEATURES>...
```
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
2805e7b1dc quoted enclose variables to
prevent globbing or incorrect splitting. However, by doing with with
$features_build it broke the capability to call the script as:
```
$ ./scripts/dev_cli.sh build --release --libc musl -- --features tdx
```
Before 2805e7b1dc it simply worked, after,
the result is:
```
docker run --user 1000:1000 --workdir /cloud-hypervisor --rm --volume /dev/kvm --volume /home/ffidenci/go/src/github.com/cloud-hypervisor/cloud-hypervisor:/cloud-hypervisor --env RUSTFLAGS= cloudhypervisor/dev:20220223-0 cargo build --all '' --target-dir /cloud-hypervisor/build/cargo_target --features tdx --release --target x86_64-unknown-linux-musl
error: Found argument '' which wasn't expected, or isn't valid in this context
USAGE:
cargo build --all
For more information try --help
```
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
For now only generate the boot time related tests as the full metrics
test suite needs some more time to bed in.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The dev container interface script (e.g. 'dev_cli.sh') now supports the
following arguments syntax for running tests:
`tests [--unit|--cargo|--all] [--libc musl|gnu] [-- [<test scripts args>] [-- [<test binary args>]]] `
In this way, we can pass custom arguments to the test binary (either
"cargo test" or "performance-metrics") with our dev container script.
For example:
`$ ./dev_cli.sh tests --metrics -- -- --report-file /tmp/metrics.json --test-filter latency`
`$ ./dev_cli.sh tests --integration -- --test-filter "test_serial" -- --nocapture --test-threads=1`
Fixes: #3739
Signed-off-by: Bo Chen <chen.bo@intel.com>
We don't need to install docker inside the container. Add missing
depenencies that docker was pulling in.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The same way we mask the writes coming from the guest to the message
control register related to MSI-X capability, let's do the same for MSI.
The point is to prevent the guest from writing to read-only bits.
The correct writable bits for MSI are only bits 0, 4, 5 and 6 of 2nd
16-bit word.
Those are:
* MSI Enable: 0
* Multiple Message Enable: 6-4
See "Table 7-39 Message Control Register for MSI" from
"NCB-PCI_Express_Base_5.0r1.0-2019-05-22.pdf".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Starting from Rust 1.56 Cargo supports specifying the minimum supported
rust version (MSRV) via "rust-version". If the compiler version is not
satisfied, Cargo prints an error and exits early.
MSRV is useful information to packagers. Using this field also saves us
from adding another file to the tree.
The version is currently set to 1.54, which is tested to build Cloud
Hypervisor successfully. Although anyone who uses 1.54 will see a
warning because "rust-version" is only introduced in 1.56. The warning
can be safely ignored.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This commit adds event fds and the event handler to send/receive
requests and responses from the GDB thread. It also adds `--gdb` flag to
enable GDB stub feature.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds `stop_on_boot` to `Vm` so that the VM stops before
starting on boot requested. This change is required to keep the target
VM stopped before a debugger attached as the user expected.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds `Vm::debug_request` to handle `GdbRequestPayload`,
which will be sent from the GDB thread.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds initial gdb.rs implementation for `Debuggable` trait to
describe a debuggable component. Some part of the trait bound
implementations is based on the crosvm GDB stub code [1].
[1] https://github.com/google/crosvm/blob/main/src/gdb.rs
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds `KVM_SET_GUEST_DEBUG` and `KVM_TRANSLATE` ioctls to
seccomp filter to enable guest debugging without `--seccomp=false`.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds `VmState::BreakPoint` to handle hardware breakpoint.
The VM will enter this state when a breakpoint hits or a debugger
interrupts the execution.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds `VmExit::Debug` for x86/KVM. When the guest hits a
hardware breakpoint, `VcpuExit::Debug` vm exit occurs. This vm exit
will be handled with code implemented in the following commits.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
This commit adds `translate_gva` for x86/KVM. The same name function is
already implemented for MSHV, but the implementation differs as
KVM_TRANSLATE does not take the flag argument and does not return status
code. This change requires the newer version of kvm-ioctls [1].
[1]
97ff779b6e
Signed-off-by: Akira Moroo <retrage01@gmail.com>
`translate_gva` returns mshv-specific type `hv_translate_gva_result`.
This return type is not a problem since this function is implemented
only for mshv, but we need to remove the type as the same function will
be implemented for KVM in PR #3575. This commit replaces the
mshv-specific type with `u32`.
Signed-off-by: Akira Moroo <retrage01@gmail.com>
error: writing `&mut Vec` instead of `&mut [_]` involves a new object
where a slice will do
--> virtio-devices/src/transport/pci_common_config.rs:93:17
|
93 | queues: &mut
Vec<Queue<GuestMemoryAtomic<GuestMemoryMmap>>>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: change this to: `&mut [Queue<GuestMemoryAtomic<GuestMemoryMmap>>]`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit
https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
Signed-off-by: Akira Moroo <retrage01@gmail.com>
I incorrectly used the MSI message control register values for the mask
not the the MSI-X control registers.
The correct writable fields for MSI-X are only bits 14 and 15 of 2nd
16-bit word.
Those are:
* Function Mask: 14
* MSI-X Enable: 15
See "Table 7-47 Message Control Register for MSI-X" from
"NCB-PCI_Express_Base_5.0r1.0-2019-05-22.pdf"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
42b5d4a2f7 has changed how the PciBdf
field of a DeviceNode is represented (from an int32 to its own struct).
To avoid marshelling / demarshelling issues for the projects relying on
the openapi auto generated code, let's propagate the change, updating
the yaml file accordingly.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
`Dies per package` setting of VCPU topology doesnot apply on AArch64.
Now we only accept `1` value. This way we can make the `dies` field
transparent, avoid it from impacting the topology setting.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The logic of determining VCPU index in creating `cpu-map` node of FDT
can be optimized.
The code is invoked when VCPU topology is specified.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
In Flattened Device Tree (FDT) on AArch64, the VCPU topology is
represented by `cpu-map` node. The source code of creating the node
can be simplified.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
If `--local` is provided or if the version is not available then build
the container before use. This allows combining updates to the
Dockerfile with a full CI run.
Drop the "--dev" parameter as we only support one container type for
simplicity.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For vhost-user devices, we don't want to loose the vhost-user protocol
feature through the negotiation between guest and device. Since we know
VIRTIO has no knowledge of the vhost-user protocol feature, there is no
way it would ever be acknowledged by the guest. For that reason, we
create each vhost-user device with the set of acked features containing
the vhost-user protocol feature is this one was part of the available
list.
Having the set of acked features containing this bit allows for solving
a bug that was happening through the migration process since the
vhost-user protocol feature wasn't explicitely enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In this way, we can cover local-migration with dpdk in our regular CI,
to prevent similar regressions reported and fixed by #3657.
Fixes: #3659
Signed-off-by: Bo Chen <chen.bo@intel.com>
Based on the helpers from the hypervisor crate, the VMM can identify
what type of hypercall has been issued through the KVM_EXIT_TDX reason.
For now, we only log warnings and set the status to INVALID_OPERAND
since these hypercalls aren't supported. The proper handling will be
implemented later.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the recent additions to the kvm-ioctls crate, this commit
implements the support for providing the exit reason details to the
caller, which allows the identification of the type of hypercall that
was issued. It also introduces a way for the consumer to set the status
code that must be sent back to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the object returned from CpuManager.create_vcpu() is never used,
we can avoid the cloning of this object.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rely on latest version of rust-vmm/kvm-ioctls to be able to invoke the
new method get_kvm_run() that returns a mutable reference of the kvm_run
structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A new fork of the kvm-bindings crate has been submitted to the
ch-v0.5.0-tdx branch. It contains updated bindings for x86 to support
TDX.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will allow easier transitioning between versions of the container.
A later PR will update the CI to use this new tag.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 98bbfa9738.
Some tests are continuing to fail even after reverting
d27316dab6. This is the only other
relevant change.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit d27316dab6.
Jenkins testing showed up issues that didn't appear during manual
testing even after using "dev_cli.sh build-container" before running
integration tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove unused packages or more specific dependencies. In particular use
the packaged virtiofsd to avoid compiling it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To run unit tests correctly on musl target, We don't need to provide
specific "CFLAGS" or "TARGET_CC", as long as we use the correct build
target `*-linux-musl` with the "cargo test" command.
Signed-off-by: Bo Chen <chen.bo@intel.com>
It provides a CLI optoin to list and filter tests, e.g.:
`$ performance-metrics --test-filter boot_time --test-filter net_latency`
Signed-off-by: Bo Chen <chen.bo@intel.com>
Performance tests are expected to be executed sequentially, so we can
start VM guests with the same IP while putting them on a different
private network. The default constructor "Guest::new()" does not work
well, as we can easily create more than 256 VMs from repeating various
performance tests dozens times in a single run.
Signed-off-by: Bo Chen <chen.bo@intel.com>
In this way, we can avoid create/delete large temporary files during
every iteration of block I/O performance tests. Also, we can reuse the
'init/clean_tests()' interface in the future for other setup/cleanup.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This commit adds unit tests for parsing iperf3/ethr/boot_time/fio, e.g.:
`$ cargo test --package performance_metrics `
Signed-off-by: Bo Chen <chen.bo@intel.com>
This commit creates a new binary target for generating performance metrics,
and also adds a set of performance tests for network throughput/latency,
block I/O performance, and boot time.
Signed-off-by: Bo Chen <chen.bo@intel.com>
error: unused variable: `fw_path`
--> test_infra/src/lib.rs:837:9
|
837 | fw_path: &str,
| ^^^^^^^ help: if this is intentional, prefix it with an underscore: `_fw_path`
|
= note: `-D unused-variables` implied by `-D warnings`
Signed-off-by: Bo Chen <chen.bo@intel.com>
With enabling the `vendered-openssl` feature, we can now cross-build the
`test_infra` crate for "musl" and "aarch64" targets. In this way, we
can remove the `test_infra` crate from the "exclude" list, so that this
crate can be checked and processed by "cargo clippy" and "cargo fmt".
More details can be found: https://docs.rs/openssl/latest/openssl/#vendored
As 'musl-gcc' is required, this commit also installs the `musl-tools`
package for our "build" github action on the musl target [1].
[1] https://github.com/actions-rs/toolchain/issues/102
Signed-off-by: Bo Chen <chen.bo@intel.com>
keep it as default to print out commandline when spawning guest in favor
of integration tests for debugging purpose, while allowing to disable
it when needed (e.g. for performance tests).
Signed-off-by: Bo Chen <chen.bo@intel.com>
The PCI spec specifies that only the following bits are writable:
16: MSI Enable
20,21,22: Multiple Message Enable
26: Extended Message Data Enable
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By having the DeviceNode storing a PciBdf, we simplify the internal code
as well as allow for custom Serialize/Deserialize implementation for the
PciBdf structure. These custom implementations let us display the PCI
s/b/d/f in a human readable format.
Fixes#3711
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As we've added support for cold adding devices to a VM that was created
but not already started, we should propagate the `204` response
generated on those cases to the yaml file, so openapi-generator can
produce the correct client code on the go side, to handle both `200` and
`204` successful results.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Instead of erroring out when trying to change the configuration of the
VM somewhere between the VM was created but not yet booted, let's allow
users to change that without any issue, as long as the VM has already
been created.
Fixes: #3639
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's add very basic unit for the vm_add_$device() functions, so we can
easily expand those when changing its behaviour in the coming commits.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Instead of doing the validation of the configuration change as part of
the vm, let's do this in the uper layer, in the Vmm.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's move add_to_config to config.rs so it can be used from both inside
and outside of the vm.rs file.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The GitHub security reporting link only works for those who are
repository owners.
Fixes: #3701
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
TDx support is already present on the project for quite some time, but
the TDx configuration was not yet exposed to the ones using CH via the
OpenAPI auto generated code.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Since the devices behind the IOMMU cannot be changed at runtime we offer
the ability to place all devices on user chosen segments behind the
IOMMU. This allows the hotplugging of devices behind the IOMMU provided
that they are assigned to a segment that is located behind the iommu.
Fixes: #911
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introducing a new integration test relying on the virtio-balloon ability
to free host pages that have been reported as freed by the guest.
This test checks that after consuming a lot of RAM in the guest, the VMM
process is able to releases the pages reported by the guest. Simply done
by checking the RSS associated with the VMM's process follows the memory
trend in the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement the VIRTIO_BALLOON_F_REPORTING feature, indicating to the
guest it can report set of free pages. A new virtqueue dedicated for
receiving the information about the free pages is created. The VMM
releases the memory by punching holes with fallocate() if the guest
memory is backed by a file, and madvise() the host about the ranges of
memory that shouldn't be needed anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a new parameter free_page_reporting=on|off to the balloon device
so that we can enable the corresponding feature from virtio-balloon.
Running a VM with a balloon device where this feature is enabled allows
the guest to report pages that are free from guest's perspective. This
information is used by the VMM to release the corresponding pages on the
host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Improving the existing code for better readability and in anticipation
for adding an additional virtqueue for the free page reporting feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
No need to run the Jenkins CI on pull request for which exclusively
markdown files have been modified.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since AZURE_CONNECTION_STRING is only useful for the Windows build,
let's remove it from other builds where it's not invoked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a bit of documentation and links about the TDShim and how to run
it with Cloud Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This should not occur as ioeventfd is used for notification. Such an
error message would have made the discovery of the underlying cause of
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to allow for human readable output for the VM configuration, we
pull it out of the snapshot, which becomes effectively the list of
states from the VM. The configuration is stored through a dedicated file
in JSON format (not including any binary output).
Having the ability to read and modify the VM configuration manually
between the snapshot and restore phases makes debugging easier, as well
as empowers users for extending the use cases relying on the
snapshot/restore feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As per this kernel documentation:
For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR, KVM_EXIT_XEN,
KVM_EXIT_EPR, KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR the corresponding
operations are complete (and guest state is consistent) only after userspace
has re-entered the kernel with KVM_RUN. The kernel side will first finish
incomplete operations and then check for pending signals.
The pending state of the operation is not preserved in state which is
visible to userspace, thus userspace should ensure that the operation is
completed before performing a live migration. Userspace can re-enter the
guest with an unmasked signal pending or with the immediate_exit field set
to complete pending operations without allowing any further instructions
to be executed.
Since we capture the state as part of the pause and override it as part
of the resume we must ensure the state is consistent otherwise we will
lose the results of the MMIO or PIO operation that caused the exit from
which we paused.
Fixes: #3658
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the bits about this feature being experimental, as well as
limitations around virtio-mem and vhost-user devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The patch mechanism in Cargo.toml doesn't support features to be
specified, and was leading to the following warning:
warning: patch for `kvm-bindings` uses the features mechanism.
default-features and features will not take effect because the patch
dependency does not support this mechanism
Removing the "features" parameter from the patch fixes this warning
without breaking the build.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In this way, we allows to reuse the struct `Guest` with kernel paths and
kernel commands (e.g. hardcoded constants) that are tests-specific.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Instead of using hardcoded firmware paths inside the `Guest` struct
constructor, this commit removes `fw_path` related code paths from the
`Guest` struct and asks each test constructs its firmware path
explicitly. This allows better flexibility for the `Guest` struct so
that it can be reused for the performance tests we are adding soon.
Signed-off-by: Bo Chen <chen.bo@intel.com>
If a payload is found in the TDVF section, and after it's been copied to
the guest memory, make sure to create the corresponding TdPayload
structure and insert it through the HOB.
Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a new method to the TdHob structure so that we can easily insert
a HOB_PAYLOAD_INFO_TABLE into the HOB.
Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case of TDX, if a kernel and/or a command line are provided by the
user, they can't be treated the same way as for the non-TDX case. That
is why this patch ensures the function load_kernel() is only invoked for
the non-TDX case.
For the TDX case, whenever TDVF contains a Payload and/or PayloadParam
sections, the file provided through --kernel and the parameters provided
through --cmdline are copied at the locations specified by each TDVF
section.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the recent updates of the TDVF specification introducing new
types of TDVF sections, let's extend the enum in our code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The TDVF specification has been updated with the ability to provide a
specific payload, which means we will be able to achieve direct kernel
boot.
For that reason, let's not prevent the user from using --kernel
parameter when running with TDX.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make sure Cloud Hypervisor relies on upstream and actively maintained
vfio-ioctls crate from the rust-vmm/vfio repository instead of the
deprecated version coming from rust-vmm/vfio-ioctls repository.
Fixes#3673
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we introduced a separate method to indicate when the migration
is started, both start_dirty_log() and stop_dirty_log() don't have to
carry an implicit meaning as they can focus entirely on the dirty log
being started or stopped.
For that reason, we can now safely move stop_dirty_log() to the code
section performing non-local migration. It makes only sense to stop
logging dirty pages if this has been started before.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to clearly decouple when the migration is started compared to
when the dirty logging is started, we introduce a new method to the
Migratable trait. This clarifies the semantics as we don't end up using
start_dirty_log() for identifying when the migration has been started.
And similarly, we rely on the already existing complete_migration()
method to know when the migration has been ended.
A bug was reported when running a local migration with a vhost-user-net
device in server mode. The reason was because the migration_started
variable was never set to "true", since the start_dirty_log() function
was never invoked.
Signed-off-by: lizhaoxin1 <Lxiaoyouling@163.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since both vhost-user-block and vhost-user-net backend now rely on the
upstream vhost-user-backend crate from rust-vmm, we can simply remove
the local version of the crate from our repository.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on the local version of vhost-user-backend, this
patch allows the block backend implementation to rely on the upstream
version of the crate from rust-vmm.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since virtio-iommu is now part of the upstream Linux kernel, the main
reason for having our own Linux kernel fork is to carry the patches for
virtio-watchdog support.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we moved to the CloudHvX64 target from EDK2, we can't use the same
OVMF firmware for both Cloud Hypervisor and QEMU. The documentation
needed some updates.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on the local version of vhost-user-backend, this
patch allows the block backend implementation to rely on the upstream
version of the crate from rust-vmm.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The Linux Foundation have recommended the use of the this license for
documentation so this change included the license in the documentation
directory.
The file "networking.md" was remove as the copyright holder did not give
their consent.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
While cloud-hypervisor does support receiving the file descriptors of a
tuntap device, advertising the fds structure via the openAPI can lead to
misinterpretations of what can and what should be done.
An unadvertised consumer will think that they could rather just set the
file descriptors there directly, or even pass them as a byte array.
However, the proper way to go in those cases would be actually sending
those via send_msg(), together with the request.
As hacking the openAPI auto-generated code to properly do this is not
*that* trivial, and as doing so during a `create VM` request is not
supported, we better not advertising those.
Please, for more details, also check:
https://github.com/cloud-hypervisor/cloud-hypervisor/pull/3607#issuecomment-1020935523
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Now that all the preliminary work has been merged to make Cloud
Hypervisor work with the upstream crate virtio-queue from
rust-vmm/vm-virtio repository, we can move the whole codebase and remove
the local copy of the virtio-queue crate.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that multiple file descriptors can be provided along with add-net,
that means we can hotplug a multiqueue macvtap interface to the VM.
The common macvtap test is updated, meaning that both coldplug and
hotplug codepath now use multiqueue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the latest code from the micro-http crate, this patch adds the
support for multiple file descriptors to be sent along with the add-net
request. This means we can now hotplug multiqueue network interface to
the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new trait simplifies the address translation of a GuestAddress by
having GuestAddress implementing it.
The three crates virtio-devices, block_util and net_util have been
updated accordingly to rely on this new trait, helping with code
readability and limiting the amount of duplicated code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Descriptor addresses are now translated from the virtio devices directly
and the definition of the AccessPlatform trait has been moved to
vm-virtio crate. For these reasons, the virtio-queue crate can be
simplified, which makes it very close to the upstream version.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Moving the whole codebase to rely on the AccessPlatform definition from
vm-virtio so that we can fully remove it from virtio-queue crate.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Moving away from the virtio-queue mechanism for descriptor address
translation. Instead, we enable the new mechanism added to every
VirtioDevice implementation, by setting the AccessPlatform trait if one
can be found.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we're trying to move away from the translation happening in the
virtio-queue crate, the device itself is performing the address
translation when needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we're trying to move away from the translation happening in the
virtio-queue crate, the device itself is performing the address
translation when needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we're trying to move away from the translation happening in the
virtio-queue crate, the device itself is performing the address
translation when needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we're trying to move away from the translation happening in the
virtio-queue crate, the device itself is performing the address
translation when needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we're trying to move away from the translation happening in the
virtio-queue crate, the device itself is performing the address
translation when needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we're trying to move away from the translation happening in the
virtio-queue crate, the device itself is performing the address
translation when needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new method set_access_platform() to the VirtioDevice trait in
order to allow an AccessPlatform trait to be setup on any virtio device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the virtio-devices crate can take care of the queue addresses
when placed behind a vIOMMU, we can remove the corresponding code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Upon the enablement of the queue by the guest, we perform a translation
of the descriptor table, the available ring and used ring addresses
prior to enabling the device itself. This only applies to the case where
the device is placed behind a vIOMMU, which is the reason why the
translation is needed. Indeed, the addresses allocated by the guest are
IOVAs which must be translated into GPAs before we can access the queue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on the virtio-queue crate to store the information
about the MSI-X vectors for each queue, we handle this directly from the
PCI transport layer.
This is the first step in getting closer to the upstream version of
virtio-queue so that we can eventually move fully to the upstream
version.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When freeing memory sometimes glibc will attempt to read
"/proc/sys/vm/overcommit_memory" to find out how it should release the
blocks. This happens sporadically with Cloud Hypervisor but has been
seen in use. It is not necessary to add the read() syscall to the list
as it is already included in the virtio devices common set. Similarly
the vCPU and vmm threads already have both these in the allowed list.
Fixes: #3609
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add integration tests for "pmu=on". It depends on checking if there
is "arm-pmu" item in "/proc/interrupts". As PMU info has not been added
to ACPI, the tests are only for dt.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
If a PMU is enabled in a VM, we also need to initialize the PMU
when the VM is restored. Otherwise, vCPUs cannot be started after
the VM is restored.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
When enable PMU on arm64, ioctl with group KVM_HAS_DEVICE_ATTR will be
blocked by seccomp, add it to authorized list.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Like devicefd, vcpufd also has ability to set/has attribute through kvm
ioctl. These traits are used when enable PMU on arm64, so add it here.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
As it might take more time for the VM to boot (especially under high
load) when using the firmware, let's increase the timeout waiting for
the VM to be reachable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on helpers for creating the ACPI tables and to add each table to
the HOB, this patch connects the dot to provide the set of ACPI tables
to the TD firmware.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a new method to the TdHob structure so that we can easily insert
a ACPI_TABLE_HOB into the HOB.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The way to create ACPI tables for TDX is different as each table must be
passed through the HOB. This means the XSDT table is not required since
the firmware will take care of creating it. Same for RSDP, this is
firmware responsibility to provide it to the guest.
That's why this patch creates a TDX dedicated function, returning a list
of Sdt objects, which will let the calling code copy the content of each
table through the HOB.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case of TDX, we don't want to create the ACPI tables the same way we
do for all the other use cases. That's because the ACPI tables don't
need to be written to guest memory at a specific address, instead they
are passed directly through the HOB.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's been decided the ACPI tables will be passed to the firmware in a
different way, rather than using TD_VMM_DATA. Since TD_VMM_DATA was
introduced for this purpose, there's no reason to keep it in our
codebase.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both OVMF and RHF firmwares triggered an error when O_DIRECT was used
because they didn't align the buffers to the block sector size.
In order to prevent regressions, we're adding a new test validating the
VM can properly boot when the OS disk is opened with O_DIRECT and booted
from the rust-hypervisor-fw.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the backing file of our virtio-block device is opened with
O_DIRECT, there's a requirement about the buffer address and size to be
aligned to the sector size.
We know virtio-block requests are sector aligned in terms of size, but
we must still check if the buffer address is. In case it's not, we
create an intermediate buffer that will be passed through the system
call. In case of a write operation, the content of the non-aligned
buffer must be copied beforehand, and in case of a read operation, the
content of the aligned buffer must be copied to the non-aligned one
after the operation has been completed.
Fixes#3587
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This table is listed as required in the ARM Base Boot Requirements
document. The particular need arises to make the serial debugging of
Windows guest functional.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
If the guest hasn't initialised a PV clock then the KVM_KVMCLOCK_CTRL
ioctl will return -EINVAL. Therefore if running in the firmware or an OS
that doesn't use the PV clock then we should ignore that error
Tested by migrating a VM that has not yet booted into the Linux kernel
(just in firmware) by specifying no disk image:
e.g. target/debug/cloud-hypervisor --kernel ~/workloads/hypervisor-fw --api-socket /tmp/api --serial tty --console off
Fixes: #3586
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
From 15358ef79d: Device Mapper Multipath
config can avoid systemd errors related to Device Mapper multipath while
guest booting.
From 46672c384c: CONFIG_NVME_MULTIPATH is
needed to fix the observed guest hanging issue cased by systemd crash
while booting.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The SPDK-NVMe is needed for the integration test for vfio_user.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Signed-off-by: Bo Chen <chen.bo@intel.com>
AArch64 does not use IOBAR, and current code of panics the whole VMM if
we need to allocate the IOBAR.
This commit checks if IOBAR is enabled before the arch conditional code
of IOBAR allocation and if the IOBAR is not enabled, we can just skip
the IOBAR allocation and do nothing.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/3479
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Ensure all pending virtio activations (as triggered by MMIO write on the
vCPU threads leading to a barrier wait) are completed before pausing the
vCPUs as otherwise there will a deadlock with the VMM waiting for the
vCPU to acknowledge it's pause and the vCPU waiting for the VMM to
activate the device and release the barrier.
Fixes: #3585
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The code can be written in a better form and the clippy warning
suppression can be dropped.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
These are the leftovers from the commit 8155be2:
arch: aarch64: vm_memory is not required when configuring vcpu
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Update documentation and CI to rely on the new CLOUDHV.fd firmware built
from the newly introduced target CloudHvX64.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By pinning the OVMF version, we will be able to update the EDK2 fork
with a new version without potentially breaking our Cloud Hypervisor CI.
Once the new version is ready on the EDK2 fork, we'll be able to update
Cloud Hypervisor codebase, replacing the fixed version with the latest,
as well as replacing OVMF.fd with CLOUDHV.fd. This is because we'll
start building from the new target CloudHvX64.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When in local migration mode send the FDs for the guest memory over the
socket along with the slot that the FD is associated with. This removes
the requirement for copying the guest RAM and gives significantly faster
live migration performance (of the order of 3s to 60ms).
Fixes: #3566
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Create the VM using the FDs (wrapped in Files) that have been received
during the migration process.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If this FD (wrapped in a File) is supplied when the RAM region is being
created use that over creating a new one.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This function is used to open an FD (wrapped in a File) that points to
guest memory from memfd_create() or backed on the filesystem.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently cloud-hypervisor supports hotplug of CPUs, passthrough devices, virtio devices and memory resizing,
but in hotplug.md it still mentiones: "Currently Cloud Hypervisor only supports hot plugging of CPU devices."
We need to remove the incorrect information from hotplug.md to reflect the true situation.
Fixes: #3504
Signed-off-by: Liang Zhou <zhoul110@chinatelecom.cn>
Drop the unused parameter throughout the code base.
Also take the chance to drop a needless clone.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
error: unneeded late initalization
Error: --> devices/src/legacy/rtc_pl031.rs:294:9
|
294 | let v;
| ^^^^^^
|
= note: `-D clippy::needless-late-init` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `v` here
|
297 | let v = if (AMBA_ID_LOW..AMBA_ID_HIGH).contains(&offset) {
| +++++++
help: remove the assignments from the branches
|
299 ~ u32::from(PL031_ID[index])
300 | } else {
301 ~ match offset {
302 | RTCDR => self.get_time(),
303 | RTCMR => {
304 | // Even though we are not implementing RTC alarm we return the last value
...
help: add a semicolon after the `if` expression
|
317 | };
| +
error: unneeded late initalization
Error: --> devices/src/legacy/uart_pl011.rs:297:9
|
297 | let v;
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `v` here
|
299 | let v = if (AMBA_ID_LOW..AMBA_ID_HIGH).contains(&(offset >> 2)) {
| +++++++
help: remove the assignments from the branches
|
301 ~ u32::from(PL011_ID[index])
302 | } else {
303 ~ match offset >> 2 {
304 | UARTDR => {
305 | let c: u32;
306 | let r: u32;
...
help: add a semicolon after the `if` expression
|
340 | };
| +
error: unneeded late initalization
Error: --> devices/src/legacy/uart_pl011.rs:305:21
|
305 | let c: u32;
| ^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `c` here
|
309 | let c: u32 = self.read_fifo.pop_front().unwrap_or_default().into();
| ~~~~~~~~~~
error: unneeded late initalization
Error: --> devices/src/legacy/uart_pl011.rs:306:21
|
306 | let r: u32;
| ^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `r` here
|
320 | let r: u32 = c;
| ~~~~~~~~~~
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: this boolean expression can be simplified
--> tests/integration.rs:3755:33
|
3755 | assert!(!(empty > 5), "No login on pty");
| ^^^^^^^^^^^^ help: try: `empty <= 5`
|
= note: `-D clippy::nonminimal-bool` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#nonminimal_bool
error: unneeded late initalization
--> tests/integration.rs:7619:13
|
7619 | let mut success;
| ^^^^^^^^^^^^^^^^
|
= note: `-D clippy::needless-late-init` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `success` here
|
7621 | let mut success = if let Some(status) = send_migration
| +++++++++++++++++
help: remove the assignments from the branches
|
7625 ~ status.success()
7626 | } else {
7627 ~ false
|
help: add a semicolon after the `if` expression
|
7628 | };
| +
error: unneeded late initalization
--> tests/integration.rs:7838:13
|
7838 | let mut success;
| ^^^^^^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `success` here
|
7840 | let mut success = if let Some(status) = send_migration
| +++++++++++++++++
help: remove the assignments from the branches
|
7844 ~ status.success()
7845 | } else {
7846 ~ false
|
help: add a semicolon after the `if` expression
|
7847 | };
| +
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: unneeded late initalization
Error: --> arch/src/aarch64/gic/gicv3_its.rs:127:9
|
127 | let attr: u64;
| ^^^^^^^^^^^^^^
|
= note: `-D clippy::needless-late-init` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `attr` here
|
128 | let attr: u64 = if save {
| +++++++++++++++
help: remove the assignments from the branches
|
129 ~ u64::from(kvm_bindings::KVM_DEV_ARM_ITS_SAVE_TABLES)
130 | } else {
131 ~ u64::from(kvm_bindings::KVM_DEV_ARM_ITS_RESTORE_TABLES)
|
help: add a semicolon after the `if` expression
|
132 | };
| +
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We've been currently using "fd" as the field name, but it should be
called "fds" since 6664e5a6e7 introduced
the name change on the structure field.
Fixes: #3560
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Passing no boot related parameters (e.g. no --kernel) is used for e.g.
receiving a live migration or an API based boot.
marvin:~/src/cloud-hypervisor (2022-01-11-live-migration-with-fds *)$ target/debug/cloud-hypervisor --api-socket /tmp/api2
thread 'main' panicked at '`tdx` is not a name of an argument or a group.
Make sure you're using the name of the argument itself and not the name of short or long flags.', /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.0.6/src/parse/matches/arg_matches.rs:598:14
stack backtrace:
0: rust_begin_unwind
at /rustc/7d6f948173ccb18822bab13d548c65632db5f0aa/library/std/src/panicking.rs:498:5
1: core::panicking::panic_fmt
at /rustc/7d6f948173ccb18822bab13d548c65632db5f0aa/library/core/src/panicking.rs:107:14
2: clap::parse::matches::arg_matches::ArgMatches::get_arg
at /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.0.6/src/parse/matches/arg_matches.rs:1052:17
3: clap::parse::matches::arg_matches::ArgMatches::is_present
at /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.0.6/src/parse/matches/arg_matches.rs:598:9
4: cloud_hypervisor::start_vmm
at ./src/main.rs:530:46
5: cloud_hypervisor::main
at ./src/main.rs:566:27
6: core::ops::function::FnOnce::call_once
at /rustc/7d6f948173ccb18822bab13d548c65632db5f0aa/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`tap` has its default value set to `None`, but in the openapi yaml file
we've been setting it to `""`.
When using this code on the Kata Containers side we'd be hit by a non
expected behaviour of cloud-hypervisor, as even when using a different
method to initialise the `tuntap` device the code would be treated as if
using `--net tap` (which is a valid use-case).
Related: #3554
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
When enabling the `mshv` feature, we skip quite some tests and
hence have known dead-code. This annotation silences dead-code
related warnings for our quality workflow to pass.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Given integration tests don't have special annotations, all our
integration test scripts do not need to carry special flags when running
`cargo test`. This commit also removed the "test::" prefix for
pattern-matching different set of integration tests.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Given integration tests are placed in a dedicate directory, they don't
need annotations (e.g. `#[cfg(integration_test)]` and `#[cfg(test)]`) or
defining `test mod` to exclude themselves from the common compilation
process.
Signed-off-by: Bo Chen <chen.bo@intel.com>
warning: unneeded late initalization
--> src/main.rs:134:5
|
134 | let mut app: App;
| ^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::needless_late_init)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `app` here
|
138 | let mut app: App = App::new("cloud-hypervisor")
| ~~~~~~~~~~~~~~~~
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: unneeded late initalization
--> vmm/src/acpi.rs:525:5
|
525 | let mut prev_tbl_len: u64;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::needless_late_init)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `prev_tbl_len` here
|
552 | let mut prev_tbl_len: u64 = madt.len() as u64;
| ~~~~~~~~~~~~~~~~~~~~~~~~~
warning: unneeded late initalization
--> vmm/src/acpi.rs:526:5
|
526 | let mut prev_tbl_off: GuestAddress;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `prev_tbl_off` here
|
553 | let mut prev_tbl_off: GuestAddress = madt_offset;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: unneeded late initalization
--> virtio-queue/src/chain.rs:381:13
|
381 | let desc: Descriptor;
| ^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::needless_late_init)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `desc` here
|
382 | let desc: Descriptor = if i < 3 {
| ++++++++++++++++++++++
help: remove the assignments from the branches
|
383 ~ Descriptor::new(0x1000 * i as u64, 0x1000, VIRTQ_DESC_F_NEXT, i + 1)
384 | } else {
385 ~ Descriptor::new(0x1000 * i as u64, 0x1000, 0, 0)
|
help: add a semicolon after the `if` expression
|
386 | };
| +
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Current command to run cloud-hypervisor binary has
wrong --rng argument in the README file.
This commit fixes all the references of --rng argument
by removing the --rng option to use default.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This crate contains up to date definition of the Queue, AvailIter,
DescriptorChain and Descriptor structures forked from the upstream
crate rust-vmm/vm-virtio 27b18af01ee2d9564626e084a758a2b496d2c618.
The following patches have been applied on top of this base in order to
make it work correctly with Cloud Hypervisor requirements:
- Add MSI vector field to the Queue
In order to help with MSI/MSI-X support, it is convenient to store the
value of the interrupt vector inside the Queue directly.
- Handle address translations
For devices with access to data in memory being translated, we add to
the Queue the ability to translate the address stored in the
descriptor.
It is very helpful as it performs the translation right after the
untranslated address is read from memory, avoiding any errors from
happening from the consumer's crate perspective. It also allows the
consumer to reduce greatly the amount of duplicated code for applying
the translation in many different places.
- Add helpers for Queue structure
They are meant to help crate's consumers getting/setting information
about the Queue.
These patches can be found on the 'ch' branch from the Cloud Hypervisor
fork: https://github.com/cloud-hypervisor/vm-virtio.git
This patch takes care of updating the Cloud Hypervisor code in
virtio-devices and vm-virtio to build correctly with the latest version
of virtio-queue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This crate was used in the integration tests to allow the tests to
continue and clean up after a failure. This isn't necessary in the unit
tests and adds a large build dependency chain including an unmaintained
crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This updated tree now includes a fix for virtio-net regression that was
present in 5.14:
commit 732b74d647048668f0f8dc0c848f0746c69e2e2f
Author: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Date: Sat Oct 9 05:17:53 2021 -0400
virtio-net: fix for skb_over_panic inside big mode
commit 126285651b7f ("Merge ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net")
accidentally reverted the effect of
commit 1a8024239da ("virtio-net: fix for skb_over_panic inside big mode")
on drivers/net/virtio_net.c
As a result, users of crosvm (which is using large packet mode)
are experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13.
Crash trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-rc1 linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).
For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.
This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).
When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.
This is an unfortunate naming mistake that caused a lot of problems.
Fixes#3497
Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the underlying kernel is old PTY resize is disabled and this is
represented by the use of None in the provided Option<File> type. In the
virtio-console PTY path don't blindly unwrap() the value that will be
preserved across a reboot.
Fixes: #3496
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The test test_virtio_block_topology is flaky on aarch64, let's disable
it until we find the right way to fix it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is a corner case of using the Cloud Hypervisor on AArch64:
If the VM is started on a device where RAM is limited, and if the user
allocates nearly as much memory for the guest as is still free on the
host, we need to enable the swap memory.
This commit documented this corner case with explanation.
Also, this commit corrects the hardware requirement of the GIC interrupt
controller for running the Cloud Hypervisor on AArch64, accroding to
[1].
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/3419
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
[1] https://www.kernel.org/doc/html/latest/virt/kvm/devices/arm-vgic-its.html
It's important to maintain the ability to run in a non-TDX environment
a Cloud Hypervisor binary with the 'tdx' feature enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This test relies on using losetup with a block size to create a block
device from a file that has a specific block size for the topology
detection code to pick up and passthrough to the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the disk is backed by a block device on the host a non-default
topology will be available and that topology can be advertised by virtio
block.
Fixes: #3262
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For simplicity this trait implements a default version that has a
topology with 512 byte (i.e. sector) recommended sizes.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Setting up the SIGWINCH handler requires at least Linux 5.7. However
this functionality is not required for basic PTY operation.
Fixes: #3456
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 58d25b3ccc.
This change introduced a regression when running iperf with the guest
running as the server:
marvin:~/src/cloud-hypervisor ((58d25b3c...))$ iperf -c 192.168.249.2
------------------------------------------------------------
Client connecting to 192.168.249.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.249.1 port 47078 connected with 192.168.249.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.40 sec 14.0 MBytes 11.3 Mbits/sec
marvin:~/src/cloud-hypervisor ((58d25b3c...))$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.249.1 port 5001 connected with 192.168.249.2 port 42866
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.01 sec 51.2 GBytes 44.0 Gbits/sec
Fixes: #3450
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The sendmsg() syscall is limited in the number of fds it can handle.
This number matches that used by the vfio-user library and is
conservative (since we've seen it work with 64 fds.)
Fixes: #3401
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This time we use the Rust Hypervisor Firmware for test_vfio_user() in
order to fix the systemd issues we've seen so far.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Advertise the PCI MMIO config spaces here so that the MMIO config space
is correctly recognised.
Tested by: --platform num_pci_segments=1 or 16 hotplug NVMe vfio-user device
works correctly with hypervisor-fw & OVMF and direct kernel boot.
Fixes: #3432
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When restoring replace the internal value of the device tree rather than
replacing the Arc<Mutex<DeviceTree>> itself. This is fixes an issue
where the AddressManager has a copy of the the original
Arc<Mutex<DeviceTree>> from when the DeviceManager was created. The
original restore path only replaced the DeviceManager's version of the
Arc<Mutex<DeviceTree>>. Instead replace the contents of the
Arc<Mutex<DeviceTree>> so all users see the updated version.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The observation is that the code in question was used to bridge
synchronized and asynchronized code.
We can group the functions for that purpose under an adaptor trait. To
limit the scope of locking, the users of the trait are required to
implement a method to return a MutexGuard for the underlying file.
This then allows us to use concrete types (QcowFile and Vhdx) in code,
which is easier to read than a bunch of traits.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Saving the KVM clock and restoring it is key for correct behaviour of
the VM when doing snapshot/restore or live migration. The clock is
restored to the KVM state as part of the Vm::resume() method prior to
that it must be extracted from the state object and stored for later use
by this method. This change simplifies the extraction and storage part
so that it is done in the same way for both snapshot/restore and live
migration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit enhances the integration test for multiple PCI segments
by:
(1) Enables the `test_virtio_fs_multi_segment` on AArch64.
(2) Adds a new integration test case for both x86_64 and AArch64 using
the direct kernel boot to test virtio-disk multiple PCI segments.
The test case does:
- Start a VM using direct kernel boot with 16 PCI segments and assign
the last PCI segment with a virtio-disk device.
- Check if the number of PCI host bridges equals to 16 after VM boots.
- Mount the virtio-disk device on the last PCI segment to the rootfs
and write/read data to the virtio-disk device.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Each PCI device under a root complex is uniquely identified by its
Requester ID (AKA RID). A Requester ID is a triplet of a Bus number,
Device number, and Function number.
MSIs may be distinguished in part through the use of sideband data
accompanying writes. In the case of PCI devices, this sideband data
may be derived from the Requester ID. A mechanism is required to
associate a device with both the MSI controllers it can address,
and the sideband data that will be associated with its writes to
those controllers.
This commit adds the `msi-map` property for PCI nodes, therefore
creating MSI mapping for each PCI device.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit rewrites the `create_pci_node` in the FDT creator to
create multiple PCI nodes based on the vector of `PciSpaceInfo`,
and each PCI node in FDT reflects a PCI segment.
- The PCI MMIO config space, 32 bits PCI device space and 64 bits
PCI device space is re-calculated based on the `PciSpaceInfo` for
each PCI segment.
- A new FDT property `linux,pci-domain` is added.
- The virtio-iommu node is only created for the first PCI segment.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The constant `PCI_MMIO_CONFIG_SIZE` defined in `vmm/pci_segment.rs`
describes the MMIO configuation size for each PCI segment. However,
this name conflicts with the `PCI_MMCONFIG_SIZE` defined in `layout.rs`
in the `arch` crate, which describes the memory size of the PCI MMIO
configuration region.
Therefore, this commit renames the `PCI_MMIO_CONFIG_SIZE` to
`PCI_MMIO_CONFIG_SIZE_PER_SEGMENT` and moves this constant from `vmm`
crate to `arch` crate.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently, a tuple containing PCI space start address and PCI space
size is used to pass the PCI space information to the FDT creator.
In order to support the multiple PCI segment for FDT, more information
such as the PCI segment ID should be passed to the FDT creator. If we
still use a tuple to store these information, the code flexibility and
readablity will be harmed.
To address this issue, this commit replaces the tuple containing the
PCI space information to a structure `PciSpaceInfo` and uses a vector
of `PciSpaceInfo` to store PCI space information for each segment, so
that multiple PCI segment information can be passed to the FDT together.
Note that the scope of this commit will only contain the refactor of
original code, the actual multiple PCI segments support will be in
following series, and for now `--platform num_pci_segments` should only
be 1.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Previously mutex (semaphore) and file were separated. The code needed to
create artificial scopes to use mutex to protect file.
Rewrite the code to be idiomatic. The file itself is turned into a trait
object and placed inside the mutex. This requires providing a new
ReadWriteSeekFile trait to unify all helper functions.
The rewrite further simplified vhdx_sync code. The original code
contained two mutex'es for no apparent reason.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Extending the test_simple_launch() integration test to validate Cloud
Hypervisor boots correctly with both rust-hypervisor-fw and OVMF on
x86_64 platforms.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Bumping the OVMF binary version along with UEFI documentation to
reflect the latest set of patches on top of tianocore/edk2 'master'
branch, which can be found on the Cloud Hypervisor fork on 'ch' branch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to avoid the identity map region to conflict with a possible
firmware being placed in the last 4MiB of the 4GiB range, we must set
the address to a chosen location. And it makes the most sense to have
this region placed right after the TSS region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extending the Vm trait with set_identity_map_address() in order to
expose this ioctl to the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Commit ac25172176 bumps the rust
version of virtiofsd named `virtiofsd-rs`, which causes a warning
```
warning: use of deprecated parameter '--socket':
Please use the '--socket-path' option instead.
```
This commit updates the cmdline parameter accordingly.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Place the 3 page TSS at an explicit location in the 32-bit address space
to avoid conflicting with the loaded raw firmware.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Reduce the size of the reserved 32-bit address space to the range used
by both the PCI MMIO config data and the 32-bit PCI device space.
This avoids issues when using firmware that is loaded into the very top
of the 32-bit address space as the RAM conflicts with the reserved
memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Current `Getting Started` section only contains steps for the x86_64
platform, as we have a documentation doing the same thing for AArch64,
we can point users to the correct documentation.
Also, this commit modifies the `docs/arm64.md` to fit the documentation
style within the project.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This test is flaky (#3400) while we are experiencing a bug of using the latest
SPDK/NVMe backend as VFIO user device (#3401). Let's disable this test
before we fix the above two issues.
Signed-off-by: Bo Chen <chen.bo@intel.com>
For now we only enable the vfio-user test on x86_64 platform, as we have
a known hanging issue to resovle on the aarch64 platform.
Fixes: #3098
Signed-off-by: Bo Chen <chen.bo@intel.com>
Enabling these configs can avoid systemd errors related to Device Mapper
multipath while guest booting. Especially, the guest can hang when being
used with an NVMe backend without these configs (#3352).
Signed-off-by: Bo Chen <chen.bo@intel.com>
This kernel config is needed to fix the observed guest hanging issue
cased by systemd crash while booting.
Fixes: #3352
Signed-off-by: Bo Chen <chen.bo@intel.com>
Added fields:
- `Memory address size limit`: the missing of this field triggered
warnings in guest kernel
- `Node ID`
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
After introducing multiple PCI segments, the `devid` value in
`kvm_irq_routing_entry` exceeds the maximum supported range on AArch64.
This commit restructed the `devid` to the allowed range.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The register D has only one bit that is not reserved, and its purpose is
to report if the RTC/CMOS device is powered or not.
The OVMF firmware was failing to boot as it was getting the information
that the device was powered off from the register D.
The simple way to fix this issue is by always returning the bit 7 from
register D as 1, indicating the device is always powered.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the provided binary isn't an ELF binary assume that it is a firmware
to be loaded in directly. In this case we shouldn't program any of the
registers as KVM starts in that state.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Function alloc_zeroed can fail. Check its return in read and write
functions. Its return value in is_valid_alignment is not checked because
handling error in that case does not give us much benefit. Instead, an
assertion is added.
Add safety comments to all `unsafe`s.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Simple aggregate types are Sync by default. There is no need to use
`impl Sync` for MockVmm (a simple struct).
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That function call can return -1 when it fails. Wrapping -1 into File
causes the code to panic when the File is dropped.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
I encountered some trouble trying to use a virtio-console hooked up to
a PTY. Reading from the PTY would produce stuff like this
"\n\nsh-5.1# \n\nsh-5.1# " (where I'm just pressing enter at a shell
prompt), and a terminal would render that like this:
----------------------------------------------------------------
sh-5.1#
sh-5.1#
----------------------------------------------------------------
This was because we weren't disabling the ICRNL termios iflag, which
turns carriage returns (\r) into line feeds (\n). Other raw mode
implementations (like QEMU's) set this flag, and don't have this
problem.
Instead of fixing our raw mode implementation to just disable ICRNL,
or copy the flags from QEMU's, though, here I've changed it to use the
raw mode implementation in libc. It seems to work correctly in my
testing, and means we don't have to worry about what exactly raw mode
looks like under the hood any more.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Creating some brief documentation for TDX, summarizing the links on
where to find more information about TDX, as well as how to run Cloud
Hypervisor on it.
Fixes#3318
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Fix seccomp violation when trying to add the out FD to the epoll loop
when the serial buffer needs to be flushed.
0x00007ffff7dc093e in epoll_ctl () at ../sysdeps/unix/syscall-template.S:120
0x0000555555db9b6d in epoll::ctl (epfd=56, op=epoll::ControlOptions::EPOLL_CTL_MOD, fd=55, event=...)
at /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/epoll-4.3.1/src/lib.rs:155
0x00005555556f5127 in vmm::serial_buffer::SerialBuffer::add_out_poll (self=0x7fffe800b5d0) at vmm/src/serial_buffer.rs:101
0x00005555556f583d in vmm::serial_buffer::{impl#1}::write (self=0x7fffe800b5d0, buf=...) at vmm/src/serial_buffer.rs:139
0x0000555555a30b10 in std::io::Write::write_all<vmm::serial_buffer::SerialBuffer> (self=0x7fffe800b5d0, buf=...)
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1527
0x0000555555ab82fb in devices::legacy::serial::Serial::handle_write (self=0x7fffe800b520, offset=0, v=13) at devices/src/legacy/serial.rs:217
0x0000555555ab897f in devices::legacy::serial::{impl#2}::write (self=0x7fffe800b520, _base=1016, offset=0, data=...) at devices/src/legacy/serial.rs:295
0x0000555555f30e95 in vm_device::bus::Bus::write (self=0x7fffe8006ce0, addr=1016, data=...) at vm-device/src/bus.rs:235
0x00005555559406d4 in vmm::vm::{impl#4}::pio_write (self=0x7fffe8009640, port=1016, data=...) at vmm/src/vm.rs:459
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When running with `--serial pty --console pty --seccomp=false` the
SIGWICH listener thread would panic as the seccomp filter was empty.
Adopt the mechanism used in the rest of the code and check for non-empty
filter before trying to apply it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This test generates an array of random numbers and then applies the same
trivial algorithm twice -- once in set_apic_delivery_mode and another
time in an anonymous function.
Its usefulness is limited. Drop it to remove one unsafe in code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Fixing a few inconsistencies and extending the document to tackle
multiple devices use case, as well as having multiple devices under the
same IOMMU group.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The variable tmp was never initialized. Calling assume_init when the
content is not yet initialized causes immediate undefined behaviour.
We also cannot create any intermediate references because they will be
subject to the same requirements for references -- the referred object
must be valid.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The proper way to refer to the project is "Cloud Hypervisor" without the
hyphen in the middle. On the other hand, if one refers to the binary
name, it is "cloud-hypervisor".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new document describes the available options for the parameter
`--cpus`, how to use them and what is their usage.
Fixes#3317
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing list of options available for the 'cpus' parameter
with the newly added option 'affinity'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
OSTree based systems (such as Fedora CoreOS) behave differently than yum
based systems (such as Fedora) with regarding to the RPM installations.
While the latter will install the RPM on the running system, modifying
it; the former will be layered (installed in a "chroot") and only will
be present on the system after the user boots this layer up.
While in theory there shouldn't be much differences on the installation
itself, when installing cloud-hypervisor on RHCOS, the following issue
would be seen:
```
$ sudo sudo rpm-ostree install cloud-hypervisor
Checking out tree 9c1cdf9... done
Enabled rpm-md repositories: copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor
Updating metadata for 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor'... done
rpm-md repo 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor'; generated: 2021-11-08T19:02:32Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 1 package (3.1 MB)
Downloading from 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
error: Running %post for cloud-hypervisor: Executing bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(cloud-hypervisor.post)'` for more information
$ sudo journalctl -t 'rpm-ostree(cloud-hypervisor.post)'
-- Logs begin at Fri 2021-11-12 12:18:33 UTC, end at Fri 2021-11-12 12:22:55 UTC. --
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Failed to set capabilities on file `/usr/bin/cloud-hypervisor' (Operation not supported)
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: usage: setcap [-q] [-v] [-n <rootid>] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Note <filename> must be a regular (non-symlink) file.
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Failed to set capabilities on file `/usr/lib64/cloud-hypervisor/vhost_user_net' (Operation not supported)
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: usage: setcap [-q] [-v] [-n <rootid>] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Note <filename> must be a regular (non-symlink) file.
```
It seems to happen as the "chroot" where the package is installed does
not have enough privileges to set the binaries capabilities during the
`%post` step.
Considering this, and also considering that calling `setcap` during
`%post` may not be the most elegant solution when speaking RPM spec
files, let's take advantage of the already existent `%caps()` directive
for the `%files` list to do so.
With that change applied, cloud-hypervisor could be successfully
installed on RHCOS, as shown below:
```
$ sudo rpm-ostree install cloud-hypervisor-19.0-2.el8.x86_64.rpm
Checking out tree 9c1cdf9... done
Enabled rpm-md repositories: copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor
rpm-md repo 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor' (cached); generated: 2021-11-08T19:02:32Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 18.2 MB (pkgcache branches: 2)
Added:
cloud-hypervisor-19.0-2.el8.x86_64
Run "systemctl reboot" to start a reboot
...
$ getcap /usr/bin/cloud-hypervisor
/usr/bin/cloud-hypervisor = cap_net_admin+ep
$ getcap /usr/lib64/cloud-hypervisor/vhost_user_net
/usr/lib64/cloud-hypervisor/vhost_user_net = cap_net_admin+ep
```
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Although this not strictly needed, considering cloud-hypervisor installs
only 8 binaries in total (4 per target), and that having that list of
files expanded will help us in the very near future, let's just do it
now.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
As part of checking if io_uring is supported various functionality is
tested. The test for whether io_uring supports EventFds is very time
consuming (~10ms) however this test can be removed as a later test will
test for functionality added after this one.
The support for register_eventfd() was released in Linux 5.1 but the
support for register_probe() was released in Linux 5.4. So if the latter
is present the former also is.
Before:
cloud-hypervisor: 4.880411ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/home/rob/workloads/focal-server-cloudimg-amd64-custom-20210609-0.raw"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk0"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 14.105123ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
cloud-hypervisor: 14.134837ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/tmp/disk"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk1"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 14.221869ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
After:
cloud-hypervisor: 3.140716ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/home/rob/workloads/focal-server-cloudimg-amd64-custom-20210609-0.raw"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk0"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 3.376027ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
cloud-hypervisor: 3.40446ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/tmp/disk"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk1"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 3.513969ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This new integration test validates the vCPUs are running on the
expected set of CPUs on the host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the introduction of a new option `affinity` to the `cpus`
parameter, Cloud Hypervisor can now let the user choose the set
of host CPUs where to run each vCPU.
This is useful when trying to achieve CPU pinning, as well as making
sure the VM runs on a specific NUMA node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Give the option parser the ability to handle tuples with inner brackets
containing list of integers. The following example can now be handled
correctly "option=[key@[v1-v2,v3,v4]]" which means the option is
assigned a tuple with a key associated with a list of integers between
the range v1 - v2, as well as v3 and v4.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The purpose is to add more info while running
spdk vhost tgt. For example, adding more info
(total_size, block_size) to describe the meaning
on creating malloc bdev inside spdk vhost tgt.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Because anyhow version 1.0.46 has been yanked, let's move back to the
previous version 1.0.45.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Always properly initialize vectors so that we don't run in undefined
behaviors when the vector gets dropped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Creates a new generic type Tuple so that the same implementation of
FromStr trait can be reused for both parsing a list of two integers and
parsing a list of one integer associated with a list of integers.
This anticipates the need for retrieving sublists, which will be needed
when trying to describe the host CPU affinity for every vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case we want to implement a type that would hold a list of lists, we
need the option parser to be able to ignore the commas for multiple
layers of brackets.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The elements of a list should be using commas as the correct delimiter
now that it is supported. Deprecate use of colons as delimiter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While parsing each parameter for getting the list of option/value, we
check if the value is a list separated between brackets. If that's the
case, we reconstruct the list so that it can be parsed afterwards, even
though it uses commas for separating each value.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While some distros support `musl` libc as a rust target, some others
(such as RHEL) definitely don't (at least not installing distro
packages).
Knowing this, let's have a simple way to not have a hard dependency on
musl on our spec file.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
`using_rustup` is needed as distros (such as Fedora or RHEL) don't
provide the users a way to install`rustup` without conflicting with the
distro provided `rust` package.
In order to minimize the troubles for those who want to build
cloud-hypervisor using the distros packages, let's allow the users to
change that variable and then simply rely on their system packages.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's document the most basic assumptions about the RPM spec file we
maintain for cloud-hypervisor, being those:
* You must have access to the internet during build, otherwise pulling
the vendored code down won't work.
* You must have rustup installed on your system, otherwise basic checks
performed during the build target will fail.
* You must have both gnu and musl libc targets installed, otherwise
either the dynamically or the statically link build will fail.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Avoid removing from the source vector by using Vec::extend_from_slice().
The primitive values (bytes) will be copied from the source in either
case.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the predicates as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the predicates as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As it relies on primitive types that have already been ported
Aml::append_aml_bytes can be used for the child values.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the child as not all
structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since we know all the numerical types now have implementations of
Aml::append_aml_bytes() we can use that directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As this is a DWord and we know that DWord::append_aml_bytes() is
implemented we may call it directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a naive implementation and there is scope to improve this
without extra copies but that requires addressing the users to ensure
there are no lifetime issues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As an optimisation to avoid allocating byte vectors add a trait method
that will append to an existing vector. Further to support the
transition add a default implementation of Aml::to_aml_bytes() that uses
the newly added Aml::append_aml_bytes()
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vfio-ioctls correctly exposes the list of capabilities related
to each region, Cloud Hypervisor can decide to mmap a region based on
the presence or absence of MSIX_MAPPABLE. Instead of blindly mmap'ing
the region, we check if the MSI-X table or PBA is present on the BAR,
and if that's the case, we look for MSIX_MAPPABLE.
If MSIX_MAPPABLE is present, we can go ahead and mmap the entire region.
If MSIX_MAPPABLE is not present, we simply ignore the mmap'ing of this
region as it wouldn't be supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As v19.0 has been release almost a month ago, let's also update the
project's spec file accordingly.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
By merging receive buffers through the VIRTIO_NET_F_MRG_RXBUF feature,
as well as enabling the use of indirect descriptors through
VIRTIO_RING_F_INDIRECT_DESC feature, we achieve better throughput for
the virtio-net device without hurting its latency.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allocator allocates 64-bit MMIO addresses for use with platform
devices e.g. ACPI control devices and ensures there is no overlap with
PCI address space ranges which can cause issues with PCI device
remapping.
Use this allocator the ACPI platform devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than use the system MMIO allocator for RAM use an allocator that
covers the full RAM range.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is because the SGX region will be placed between the end of ram and
the start of the device area.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With the segment id now encoded in the bdf it is not necessary to have
the separate field for it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the existing virtio fs test to support controlling the PCI
segment the device should be added to and use this for a multiple
segment test.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the existing net hotplug test to support controlling the PCI
segment the device should be added to and use this for a multiple
segment test.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the existing pmem hotplug test to support controlling the PCI
segment the device should be added to and use this for a multiple
segment test.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In particular use the accessor for getting the device id from the bdf.
As a side effect the VIOT table is now segment aware.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will allow making the code that handles bdf parsing simpler and
remove the need for manual shifting.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since each segment must have a non-overlapping memory range associated
with it the device memory must be equally divided amongst all segments.
A new allocator is used for each segment to ensure that BARs are
allocated from the correct address ranges. This requires changes to
PciDevice::allocate/free_bars to take that allocator and when
reallocating BARs the correct allocator must be identified from the
ranges.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For all the devices that support being hotplugged (disk, net, pmem, fs
and vsock) add "pci_segment" option and propagate that through to the
addition onto the PCI busses.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the decision on whether to use a 64-bit bar up to the DeviceManager
so that it can use both the device type (e.g. block) and the PCI segment
ID to decide what size bar should be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Generate a set of 8 IRQs and round-robin distribute those over all the
slots for a bus. This same set of IRQs is then used for all PCI
segments.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The platform config may specify a number of PCI segments to use, if this
greater than 1 then we add supplemental PCI segments as well as the
default segment.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This currently contains only the number over PCI segments to create.
This is limited to 16 at the moment which should allow 496 user specified
PCI devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For the bus scanning the GED AML code now calls into a PSCN method that
scans all buses. This approach was chosen since it handles the case
correctly where one GED interrupt is services for two hotplugs on
distinct segments.
The PCIU and PCID field values are now determined by the PSEG field that
is uses to select which segment those values should be used for.
Similarly _EJ0 will notify based on the value of _SEG.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Replace the hardcoded zero PCI segment id when adding devices to the bus
and extend the DeviceTree to hold the PCI segment id.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since each segment must have disjoint address spaces only advertise
address space in the 32-bit range and the PIO address space on the
default (zero) segment.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split PciSegment::new_default_segment() into a separate
PciSegment::new() and those parts required only for the default segment
(PIO PCI config device.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now this still contains just one segment but is expanding in
preparation for more segments.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit moves the code that generates the DSDT data for the PCI bus
into PciSegment making no functional changes to the generated AML.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to get proper performance out of hardware that might be passed
through the VM with VFIO, we decide to try mapping any region marked as
mappable, ignoring the failure and moving on to the next region. When
the mapping succeeds, we establish a list of user memory regions so that
we enable only subparts of the global mapping through the hypervisor.
This allows MSI-X table and PBA to keep trapping accesses from the guest
so that the VMM can properly emulate MSI-X.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Variables that start with underscore are used to silence rustc.
Normally those variables are not used in code.
This patch drops the underscore from variables that are used. This is
less confusing to readers.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This makes it much easier to use since the info!() level produces far
fewer messages and thus has less overhead.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Create a source archive with vendored sources as part of the release
workflow. This is to enable building the release offline for distros.
Note: The use of realpath and CARGO_HOME are to work around a cargo
vendor bug: https://github.com/rust-lang/cargo/issues/8443.
Signed-off-by: William Douglas <william.douglas@intel.com>
Relying on the vm-virtio/virtio-queue crate from rust-vmm which has been
copied inside the Cloud Hypervisor tree, the entire codebase is moved to
the new definition of a Queue and other related structures.
The reason for this move is to follow the upstream until we get some
agreement for the patches that we need on top of that to make it
properly work with Cloud Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This crate contains a new definition of the Queue, AvailIter,
DescriptorChain and Descriptor structures forked from the upstream
crate rust-vmm/vm-virtio d62f2246568d4f544e848b23c025b268effac5ca.
The following patches have been applied on top of this base in order to
make it work correctly with Cloud Hypervisor requirements:
- Add MSI vector field to the Queue
In order to help with MSI/MSI-X support, it is convenient to store the
value of the interrupt vector inside the Queue directly.
- Handle address translations
For devices with access to data in memory being translated, we add to
the Queue the ability to translate the address stored in the
descriptor.
It is very helpful as it performs the translation right after the
untranslated address is read from memory, avoiding any errors from
happening from the consumer's crate perspective. It also allows the
consumer to reduce greatly the amount of duplicated code for applying
the translation in many different places.
- Add helpers for Queue structure
They are meant to help crate's consumers getting/setting information
about the Queue.
These patches can be found on the 'ch' branch from the Cloud Hypervisor
fork: https://github.com/cloud-hypervisor/vm-virtio.git
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This was causing some issues because of the use of 2 different versions
for the vm-memmory crate. We'll wait for all dependencies to be properly
resolved before we move to 0.7.0.
This reverts commit 76b6c62d07.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We now build SPDK-NVMe inside the container only for the x86_64
platform, as the cross-platform build with 'docker buildx' does not work
for SPDK. For aarch64 the platform, we will build it as a part of the CI
workflow (which is running on the bare-metal machine).
Signed-off-by: Bo Chen <chen.bo@intel.com>
PRs from dependabot are failing to meet the check from DCO as the
Signed-Off-By is now a GitHub support email address.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These fields are needed for correctly accessing the datastructures but
are not read by the VHDX implementation itself.
As identified by the new beta clippy.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These structs are not read on the VMM side but are used in communication
with the guest.
As identified by the new beta clippy.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When allocating PCI MMIO BARs they should always be naturally aligned
(i.e. aligned to the size of the BAR itself.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Hypercall register needs to be saved and restored for
TLB flush and IPI synthetic features enablement.
Enabling these two synthetic features improves
guest performance.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The allocate_bars method has a side effect which collates the BARs used
for the device and stores them internally. Ensure that any use of this
internal state is after the state is created otherwise no MMIO regions
will be seen and so none will be mapped.
Fixes: #3237
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When the synthetic interrupt controller is enabled, an extra set of MSRs
must be stored in case of migration. There was one MSR missing in the
list, HV_X64_MSR_SINT14 corresponding to the 15th interrupt source from
the synthetic interrupt controller.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to patch the vhost crate anymore since the fixes we were
looking for have been released as part of 0.2.0 on crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of creating a MemoryManager from scratch, let's reuse the same
code path used by snapshot/restore, so that memory regions are created
identically to what they were on the source VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that all the pieces are in place, we can restore a VM with the new
codepath that restores properly all memory regions, allowing for ACPI
memory hotplug to work properly with snapshot/restore feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extending the MemoryManager::new() function to be able to create a
MemoryManager from data that have been previously stored instead of
always creating everything from scratch.
This change brings real added value as it allows a VM to be restored
respecting the proper memory layout instead of hoping the regions will
be created the way they were before.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Storing multiple data coming from the MemoryManager in order to be able
to restore without creating everything from scratch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new function will be able to restore memory regions and memory
zones based on the GuestMemoryMapping list that will be provided through
snapshot/restore and migration phases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This can help identifying which zone relates to which memory range.
This is going to be useful when recreating GuestMemory regions from
the previous layout instead of having to recreate everything from
scratch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create a dedicated function to factorize the allocation of the memory
ranges, and helping with the simplification of MemoryManager::new()
function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By updating the list of GuestMemory regions with the virtio-mem ones
before the creation of the MemoryManager, we know the GuestMemory is up
to date and the allocation of memory ranges is simplified afterwards.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to simplify MemoryManager::new() function. let's move the
memory configuration validation to its own function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This script checks if an image is compatible with Cloud Hypervisor.
At first, it detects the image type(raw or qcow2),
partition type whether it is DOS or GPT.
Then it mounts the image and checks if VIRTIO Configs
are enabled in the kernel config. In the end, it provides
a message about the compatibility of the image.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Move the PciSegment struct and the associated code to a new file. This
will allow some clearer separation between the core DeviceManager and
PCI handling.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the PCI related state from the DeviceManager struct to a PciSegment
struct inside the DeviceManager. This is in preparation for multiple
segment support. Currently this state is just the bus itself, the MMIO
and PIO config devices and hotplug related state.
The main change that this required is using the Arc<Mutex<PciBus>> in
the device addition logic in order to ensure that
the bus could be created earlier.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When using PVH for booting (which we use for all firmwares and direct
kernel boot) the Linux kernel does not configure LA57 correctly. As such
we need to limit the address space to the maximum 4-level paging address
space.
If the user knows that their guest image can take advantage of the
5-level addressing and they need it for their workload then they can
increase the physical address space appropriately.
This PR removes the TDX specific handling as the new address space limit
is below the one that that code specified.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Whenever running TDX, we must pass the ACPI tables to the TDVF firmware
running in the guest. The proper way to do this is by adding the tables
to the TdHob as a TdVmmData type, so that TDVF will know how to access
these tables and expose them to the guest OS.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of having the ACPI tables being created both in x86_64 and
aarch64 implementations of configure_system(), we can remove the
duplicated code by moving the ACPI tables creation in vm.rs inside the
boot() function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On AArch64, device hotplug can be enabled with ACPI. Therefore,
this commit enables the hotplug test case for following devices:
- PCI bar reprogramming
- virtio-disk
- virtio-net
- macvtap
- virtio-vsock
- virtio-pmem: Works with the latest reference kernel
- virtio-fs: Works with the latest reference kernel
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently vfio and nested virtualization is not used on AArch64,
and SGX is a x86_64 only feature. Therefore this commit adds the
architecture gates for helper functions related to vfio, SGX, and
nested virtualization to mute warnings when building tests on the
AArch64 platform.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Update introduction of option `prefault`.
In addition, this commit also did the following:
- Rearrange options, synchronize order with `config.rs`.
- Break long lines in `hugepages`.
- Update old example of `hugepages` in memory zone.
Signed-off-by: Li Yu <liyu.yukiteru@bytedance.com>
Memory hotplug and virtio_balloon works on arm64 with:
- memory hotplug: An updated kernel using ACPI
- virtio balloon: `stress` installed in the cloud image
Therefore, we can enable test cases for them in integration test.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
On MSHV some of the integration test cases are not supported yet
or still in progress. This patch disables all those test cases.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The argument `prefault` is provided in MemoryManager, but it can
only be used by SGX and restore.
With prefault (MAP_POPULATE) been set, subsequent page faults will
decrease during running, although it will make boot slower.
This commit adds `prefault` in MemoryConfig and MemoryZoneConfig.
To resolve conflict between memory and restore, argument
`prefault` has been changed from `bool` to `Option<bool>`, when
its value is None, config from memory will be used, otherwise
argument in Option will be used.
Signed-off-by: Yu Li <liyu.yukiteru@bytedance.com>
Setting the reply_ack should depend on the set of acknowledged features
containing the REPLY_ACK flag.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding some bits to the existing live migration test with NUMA in order
to properly validate virtio-mem works with live migration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By using a single file for storing the memory ranges, we simplify the
way snapshot/restore works by avoiding multiples files, but the main and
more important point is that we have now a way to save only the ranges
that matter. In particular, the ranges related to virtio-mem regions are
not always fully hotplugged, meaning we don't want to save the entire
region. That's where the usage of memory ranges is interesting as it
lets us optimize the snapshot/restore process when one or multiple
virtio-mem regions are involved.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
During snapshot/restore we will need to store this structure, which is
why it must derive the Versionize trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The function memory_range_table() will be reused by the MemoryManager in
a following patch to describe all the ranges that we should snapshot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Copy only the memory ranges that have been plugged through virtio-mem,
allowing for an interesting optimization regarding the time it takes to
migrate a large virtio-mem device. Even if the hotpluggable space is
very large (say 64GiB), if only 1GiB has been previously added to the
VM, only 1GiB will be sent to the destination VM, avoiding the transfer
of the remaining 63GiB which are unused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support correctly the snapshot/restore and migration use
cases, we must be careful with the ranges that we discard by punching
holes. On restore, there might be some ranges already plugged in,
meaning they should not be discarded. That's why we loop over the list
of blocks to discard only the ranges that are marked as unplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By creating the BlocksState object in the MemoryManager, we can directly
provide it to the virtio-mem device when being created. This will allow
the MemoryManager through each VirtioMemZone to have a handle onto the
blocks that are plugged at any point in time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This is going to be useful to let virtio-mem report the list of ranges
that are currently plugged, so that both snapshot/restore and migration
will copy only what is needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be helpful to support the creation of a MemoryRangeTable from
virtio-mem, as it uses 2M pages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding the snapshot/restore support along with migration as well,
allowing a VM with virtio-mem devices attached to be properly
migrated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The amount of memory plugged in the virtio-mem region should always be
kept up to date in the hotplugged_size field from VirtioMemZone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to duplicate the GuestMemory for snapshot purpose, as we
always have a handle onto the GuestMemory through the guest_memory
field.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we only support a single PCI bus right now advertise only a single
bus in the ACPI tables. This reduces the number of VM exits from probing
substantially.
Number of PCI config I/O port exits: 17871 -> 1551 (91% reduction) with
direct kernel boot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use a simpler method for extracting the affected slot on the eject
command. Also update the terminology to reflect that this a slot rather
than a bdf (which is what device id refers to elsewhere.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the serial buffer handling in order to write the serial
buffer's output to a PTY connected after the serial device stops being
written to by the guest.
This change moves the serial buffer initialization inside the serial
manager. That is done to allow the serial buffer to be made aware of
the PTY and epoll fds needed in order to modify the
EpollDispatch::File trigger. These are then used by the serial buffer
to trigger an epoll event when the PTY fd is writable and the buffer
has content in it. They are also used to remove the trigger when the
buffer is emptied in order to avoid unnecessary wake-ups.
Signed-off-by: William Douglas <william.douglas@intel.com>
In preparation for reorganizing how the serial output is constructed
add methods to the serial devices for setting the out buffer after the
device is created.
Also add a method to enable flushing the output buffer to be used to
write the buffer to the PTY fd once the PTY is writable.
Signed-off-by: William Douglas <william.douglas@intel.com>
In integration test, we fetch latest EDK2 code on its master branch and
build. While the update on EDK2 master is frequent. And the building is
time consuming. It takes a lot of time in CI and local test. Floating on
top of a busy master branch also bring potential risk in tracking and
debugging.
Now that Cloud Hypervisor support in EDK2 has been steady, we can pin
the EDK2 software versions to avoid unnecessary updating and building.
We can update the versions manually every after several months.
The commit also optimizes the build process by applying multi-threaded
compiling.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Added a bash function in integration test script to checkout source code
of a GIT repo with specified branch and commit.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
These packages will be used to compile `stress` from source, and
the `stress` will be used by the virtio-balloon integration test.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Both read_exact_from() and write_all_to() functions from the GuestMemory
trait implementation in vm-memory are buggy. They should retry until
they wrote or read the amount of data that was expected, but instead
they simply return an error when this happens. This causes the migration
to fail when trying to send important amount of data through the
migration socket, due to large memory regions.
This should be eventually fixed in vm-memory, and here is the link to
follow up on the issue: https://github.com/rust-vmm/vm-memory/issues/174
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Refactored the test case `test_virtio_iommu` to adapt architectures and
different choices among ACPI and FDT. In the case of ACPI, a Focal image
with modified kernel is tested.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, ACPI must work with UEFI (EDK2). This way, the kernel is
always loaded from the disk image. We can not specify a direct custom
kernel while using ACPI.
To use a custom kernel, we have to replace the kernel file in the disk
image by:
- Making a copy of the Focal `raw` image
- Mounting the rootfs with `libguestfs-tools`
- Replacing the compressed kernel file
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Installed `libguestfs-tools` to replace kernel file in cloud image.
Installed a kernel as `libguestfs-tools` requires.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Implement the infrastructure that lets a virtio-mem device map the guest
memory into the device. This is necessary since with virtio-mem zones
memory can be added or removed and the vfio-user device must be
informed.
Fixes: #3025
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving this from the VfioUserPciDevice to DeviceManager the client
can be reused for handling DMA mapping behind an IOMMU.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For vfio-user the mapping handler is per device and needs to be removed
when the device in unplugged.
For VFIO the mapping handler is for the default VFIO container (used
when no vIOMMU is used - using a vIOMMU does not require mappings with
virtio-mem)
To represent these two use cases use an enum for the handlers that are
stored.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Adding the snapshot/restore support along with migration as well,
allowing a VM with a virtio-balloon device attached to be properly
migrated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given the 'virtiofsd' executable is used in multiple CI workers,
installing them directly to the docker image is more efficient and can
save CI time.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Looking up devices on the port I/O bus is time consuming during the
boot at there is an O(lg n) tree lookup and the overhead from taking a
lock on the bus contents.
Avoid this by adding a fast path uses the hardcoded port address and
size and directs PCI config requests directly to the device.
Command line:
target/release/cloud-hypervisor --kernel ~/src/linux/vmlinux --cmdline "root=/dev/vda1 console=ttyS0" --serial tty --console off --disk path=~/workloads/focal-server-cloudimg-amd64-custom-20210609-0.raw --api-socket /tmp/api
PIO exit: 17913
PCI fast path: 17871
Percentage on fast path: 99.8%
perf before:
marvin:~/src/cloud-hypervisor (main *)$ perf report -g | grep resolve
6.20% 6.20% vcpu0 cloud-hypervisor [.] vm_device::bus::Bus::resolve
perf after:
marvin:~/src/cloud-hypervisor (2021-09-17-ioapic-fast-path *)$ perf report -g | grep resolve
0.08% 0.08% vcpu0 cloud-hypervisor [.] vm_device::bus::Bus::resolve
The compromise required to implement this fast path is bringing the
creation of the PciConfigIo device into the DeviceManager::new() so that
it can be used in the VmmOps struct which is created before
DeviceManager::create_devices() is called.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Added a section in "Usage" chapter of "iommu.md" to introduce the
special behavior when virtio-iommu is working with FDT on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
For AArch64, now virtual IOMMU is only tested on FDT, not ACPI.
In the case of FDT, the behavior of IOMMU is a bit different with ACPI.
All the devices on the PCI bus will be attached to the virtual IOMMU,
except the virtio-iommu device itself. So these devices will all be
added to IOMMU groups, and appear in folder '/sys/kernel/iommu_groups/'.
The result is, on AArch64 IOMMU group '0' contains "0000:00:01.0" which
is the console device. But on X86, console device is not attached to
IOMMU. So the IOMMU group '0' contains "0000:00:02.0" which is the first
disk.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The MSI IOVA address on X86 and AArch64 is different.
This commit refactored the code to receive the MSI IOVA address and size
from device_manager, which provides the actual IOVA space data for both
architectures.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Move the definition of MSI space to layout.rs, so other crates can
reference it. Now it is needed by virtio-iommu.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Add a virtio-iommu node into FDT if iommu option is turned on. Now we
support only one virtio-iommu device.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This change switches from handling serial input in the VMM thread to
its own thread controlled by the SerialManager.
The motivation for this change is to avoid the VMM thread being unable
to process events while serial input is happening and vice versa.
The change also makes future work flushing the serial buffer on PTY
connections easier.
Signed-off-by: William Douglas <william.douglas@intel.com>
This change adds a SerialManager with its own epoll handling that
should be created and run by the DeviceManager when creating an
appropriately configured console (serial tty or pty).
Both stdin and pty input are handled by the SerialManager. The stdin
and pty specific methods used by the VMM should be removed in a future
commit.
Signed-off-by: William Douglas <william.douglas@intel.com>
The clone method for PtyPair should have been an impl of the Clone
trait but the method ended up not being used. Future work will make
use of the trait however so correct the missing trait implementation.
Signed-off-by: William Douglas <william.douglas@intel.com>
libc::getrandom need to be called inside unsafe and it is not
cross-platform friendly.
Change it to getrandom::getrandom that is safe and cross-platform
friendly.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Updating kvm-ioctls from 0.9.0 to 0.10.0 now that Cloud Hypervisor
relies on kvm-bindings 0.5.0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For most use cases, there is no need to create multiple VFIO containers
as it causes unwanted behaviors. Especially when passing multiple
devices from the same IOMMU group, we need to use the same container so
that it can properly list the groups that have been already opened. The
correct logic was already there in vfio-ioctls, but it was incorrectly
used from our VMM implementation.
For the special case where we put a VFIO device behind a vIOMMU, we must
create one container per device, as we need to control the DMA mappings
per device, which is performed at the container level. Because we must
keep one container per device, the vIOMMU use case prevents multiple
devices attached to the same IOMMU group to be passed through the VM.
But this is a limitation that we are fine with, especially since the
vIOMMU doesn't let us group multiple devices in the same group from a
guest perspective.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allows Cloud Hypervisor to be run under `perf` as some of the
signals will already be blocked in the child process.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the kvm-bindings dependency so that Cloud Hypervisor now depends
on the version 0.5.0, which is based on Linux kernel v5.13.0. We still
have to rely on a forked version to be able to serialize all the KVM
structures we need.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Validate the size of I/O reads and check that no request is made to an
out of bounds index (which would cause a panic.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check the size of data buffer for reading on the ApciPmTimer device to
avoid a potential panic if the guest uses non-DWORD access.
Simplify the zeroring of the buffer for AcpiShutdownDevice.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a pty is resized (using the TIOCSWINSZ ioctl -- see ioctl_tty(2)),
the kernel will send a SIGWINCH signal to the pty's foreground process
group to notify it of the resize. This is the only way to be notified
by the kernel of a pty resize.
We can't just make the cloud-hypervisor process's process group the
foreground process group though, because a process can only set the
foreground process group of its controlling terminal, and
cloud-hypervisor's controlling terminal will often be the terminal the
user is running it in. To work around this, we fork a subprocess in a
new process group, and set its process group to be the foreground
process group of the pty. The subprocess additionally must be running
in a new session so that it can have a different controlling
terminal. This subprocess writes a byte to a pipe every time the pty
is resized, and the virtio-console device can listen for this in its
epoll loop.
Alternatives I considered were to have the subprocess just send
SIGWINCH to its parent, and to use an eventfd instead of a pipe.
I decided against the signal approach because re-purposing a signal
that has a very specific meaning (even if this use was only slightly
different to its normal meaning) felt unclean, and because it would
have required using pidfds to avoid race conditions if
cloud-hypervisor had terminated, which added complexity. I decided
against using an eventfd because using a pipe instead allows the child
to be notified (via poll(2)) when nothing is reading from the pipe any
more, meaning it can be reliably notified of parent death and
terminate itself immediately.
I used clone3(2) instead of fork(2) because without
CLONE_CLEAR_SIGHAND the subprocess would inherit signal-hook's signal
handlers, and there's no other straightforward way to restore all signal
handlers to their defaults in the child process. The only way to do
it would be to iterate through all possible signals, or maintain a
global list of monitored signals ourselves (vmm:vm::HANDLED_SIGNALS is
insufficient because it doesn't take into account e.g. the SIGSYS
signal handler that catches seccomp violations).
Signed-off-by: Alyssa Ross <hi@alyssa.is>
This prepares us to be able to handle console resizes in the console
device's epoll loop, which we'll have to do if the output is a pty,
since we won't get SIGWINCH from it.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Musl often uses mmap to allocate memory where Glibc would use brk.
This has caused seccomp violations for me on the API and signal
handling threads.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
As well as reducing the amount of code this also improves the binary
size slightly:
cargo bloat --release -n 2000 --bin cloud-hypervisor | grep virtio_devices::seccomp_filters::get_seccomp_rules
Before:
0.1% 0.2% 7.8KiB virtio_devices virtio_devices::seccomp_filters::get_seccomp_rules
After:
0.0% 0.1% 3.0KiB virtio_devices virtio_devices::seccomp_filters::get_seccomp_rules
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch adds a separate function to launch two guest VMs and ensure
they are connected through ovs-dpdk, so that we can reuse this function
in other tests, e.g. the test for live-migration with ovs-dpdk.
Signed-off-by: Bo Chen <chen.bo@intel.com>
2021-09-10 07:41:15 +01:00
244 changed files with 33115 additions and 19666 deletions
Cloud Hypervisor is an open source project licensed under the [Apache v2 License](https://opensource.org/licenses/Apache-2.0) and the [BSD 3 Clause](https://opensource.org/licenses/BSD-3-Clause) license.
Cloud Hypervisor is an open source project licensed under the [Apache v2
License](https://opensource.org/licenses/Apache-2.0) and the [BSD 3
sh 'scripts/dev_cli.sh tests --metrics -- -- --report-file /root/workloads/metrics.json'
}
}
stage ('Upload metrics report') {
steps {
sh 'curl -X PUT https://cloud-hypervisor-metrics.azurewebsites.net/api/publishmetrics -H "x-functions-key: $METRICS_PUBLISH_KEY" -T ~/workloads/metrics.json'
}
}
}
}
}
}
}
@@ -276,3 +396,33 @@ def cancelPreviousBuilds() {
}
}
}
def installAzureCli() {
sh "sudo apt install -y ca-certificates curl apt-transport-https lsb-release gnupg"
sh "curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null"
sh "echo \"deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ focal main\" | sudo tee /etc/apt/sources.list.d/azure-cli.list"
- [Containerized builds and tests](#containerized-builds-and-tests)
@@ -26,12 +27,19 @@
# 1. What is Cloud Hypervisor?
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt) and the MSHV hypervisors .
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on
top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt)
hypervisor and Microsoft Hypervisor (MSHV).
The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern operating systems with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 64-bit CPUs.
The project focuses on exclusively running modern, cloud workloads, on top of
a limited set of hardware architectures and platforms. Cloud workloads refers
to those that are usually run by customers inside a cloud provider. For our
purposes this means modern operating systems with most I/O handled by
paravirtualised devices (i.e. virtio), no requirement for legacy devices, and
64-bit CPUs.
Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is based on the [rust-vmm](https://github.com/rust-vmm) crates.
Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is
based on the [rust-vmm](https://github.com/rust-vmm) crates.
## Objectives
@@ -50,7 +58,9 @@ Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is bas
### Architectures
Cloud Hypervisor supports the `x86-64` and `AArch64` architectures. There are some small differences in functionality between the two architectures (see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)).
Cloud Hypervisor supports the `x86-64` and `AArch64` architectures. There are
some small differences in functionality between the two architectures
(see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)).
### Guest OS
@@ -58,6 +68,12 @@ Cloud Hypervisor supports `64-bit Linux` and Windows 10/Windows Server 2019.
# 2. Getting Started
Below sections describe how to build and run Cloud Hypervisor on the `x86_64`
platform. For getting started on the `AArch64` platform, please refer to the
[Arm64 documentation](docs/arm64.md).
## Preparation
We create a folder to build and run `cloud-hypervisor` at `$HOME/cloud-hypervisor`
```shell
@@ -67,17 +83,15 @@ $ mkdir $CLOUDH
## Install prerequisites
You need to install some prerequisite packages in order to build and test Cloud Hypervisor.
Here, all the steps are based on Ubuntu, for other Linux distributions please replace the
package manager and package name.
You need to install some prerequisite packages in order to build and test Cloud
Hypervisor. Here, all the steps are based on Ubuntu, for other Linux
distributions please replace the package manager and package name.
```shell
# Install git
$ sudo apt install git
# Install build-essential, git, and qemu-img
$ sudo apt install git build-essential qemu-img
# Install rust tool chain
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install build-essential
$ sudo apt install build-essential
# If you want to build statically linked binary please add musl target
@@ -172,7 +188,10 @@ Multiple arguments can be given to the `--disk` parameter.
#### Building your kernel
Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel. In order to support virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.
Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel.
In order to support virtio-watchdog we have our own development branch. You are
of course able to use your own kernel but these instructions will continue with
Direct kernel boot to userspace should work with a rootfs from most distributions.
Direct kernel boot to userspace should work with a rootfs from most
distributions.
Further details can be found in the [release documentation](docs/releases.md).
## Hot Plug
Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO),`virtio-{net,block,pmem,fs,vsock}` and memory resizing. This [document](docs/hotplug.md) details how to add devices to
a running VM.
Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO),
`virtio-{net,block,pmem,fs,vsock}` and memory resizing. This
[document](docs/hotplug.md) details how to add devices to a running VM.
## Device Model
Details of the device model can be found in this [documentation](docs/device_model.md).
Details of the device model can be found in this
[documentation](docs/device_model.md).
## TODO
We are not tracking the Cloud Hypervisor TODO list from a specific git tracked file but through
# How to build and test Cloud Hypervisor on AArch64
This document introduces how to build and test Cloud Hypervisor on AArch64 servers. Currently Cloud Hypervisor cannot be tested on Raspberry PI. Because on AArch64, Cloud Hypervisor requires GICv3-ITS device for PCIe MSI interrupt handling. But GICv3-ITS has not been equipped on any Raspberry PI product so far.
This document introduces how to build and test Cloud Hypervisor on AArch64.
Currently, Cloud Hypervisor supports 2 methods of booting on AArch64: UEFI
booting and direct-kernel booting. The document covers both methods.
Now Cloud Hypervisor supports 2 ways of booting on AArch64: UEFI booting and direct-kernel booting. The document covers both of the ways.
All the steps are based on Ubuntu. We use the Ubuntu cloud image for guest VM
disk.
All the steps are based on Ubuntu. We use the Ubuntu cloud image for guest VM disk.
## Hardware requirements
- AArch64 servers (recommended) or development boards equipped with the GICv3
interrupt controller.
- On development boards that have constrained RAM resources, if the creation of
a VM consumes a large portion of the free memory on the host, it may be required
to enable swap. For example, this was required on a board with 3 GB of RAM
booting a 2 GB VM at a point in time when 2.8 GB were free. Without enabling
swap the `cloud-hypervisor` process was terminated by the OOM killer. In this
situation memory was allocated for the virtual machine using memfd while the
page cache was filled, leading to a situation where the kernel could not even
drop caches. Making a small section of swap available (observably, 1 to 15 MB),
this situation can be resolved and the resulting memory footprint of
In the context of virtualization, it is always convenient to be able to share a directory from the host with the guest.
In the context of virtualization, it is always convenient to be able to share a
directory from the host with the guest.
__virtio-fs__, also known as __vhost-user-fs__ is a virtual device defined by the VIRTIO specification which allows any VMM to perform filesystem sharing.
__virtio-fs__, also known as __vhost-user-fs__ is a virtual device defined by
the VIRTIO specification which allows any VMM to perform filesystem sharing.
## Pre-requisites
### The daemon
This virtual device relies on the _vhost-user_ protocol, which assumes the backend (device emulation) is handled by a dedicated process running on the host. This daemon is called __virtiofsd__ and needs to be present on the host.
This virtual device relies on the _vhost-user_ protocol, which assumes the
backend (device emulation) is handled by a dedicated process running on the
host. This daemon is called __virtiofsd__ and needs to be present on the host.
The `cache=none` option should be the default when using `virtiofsd` with the __cloud-hypervisor__ VMM. This prevents from using the guest page cache, which reduces the memory footprint of the guest. When running multiple virtual machines on the same host, this will let the host deal with page cache, which will increase the density of virtual machines which can be launched.
The `cache=never` option is the default when using `virtiofsd` with
Cloud Hypervisor. This prevents from using the host page cache, reducing the
overall footprint on host memory. This increases the maximum density of virtual
machines that can be launched on a single host.
The `cache=always` option will allow for the guest page cache to be used, which will increase the memory footprint of the guest. This option should be used only for specific use cases where a single VM is going to be running on a host.
The `cache=always` option will allow the host page cache to be used, which can
result in better performance for the guest's workload at the cost of increasing
the footprint on host memory.
### Kernel support
Modern Linux kernels starting (at least v5.10) have support for virtio-fs. Use
of older kernels, with additional patches, are not supported.
Modern Linux kernels (at least v5.10) have support for virtio-fs. Use of older
kernels, with additional patches, are not supported.
## How to share directories with cloud-hypervisor
### Start the VM
Once the daemon is running, the option `--fs` from __cloud-hypervisor__ needs to be used.
Direct kernel boot is the preferred option, but we can boot from an EFI cloud image if it contains a recent enough kernel.
Once the daemon is running, the option `--fs` from Cloud Hypervisor needs
to be used.
Because _vhost-user_ expects a dedicated process (__virtiofsd__ in this case) to be able to access the guest RAM to communicate through the _virtqueues_ with the driver running in the guest, `--memory` option needs to be slightly modified. It must specify `shared=on` to share the memory pages so that an external process can access them.
Both direct kernel boot and EFI firmware can be used to boot a VM with
virtio-fs, given that the cloud image contains a recent enough kernel.
Assuming you have `focal-server-cloudimg-amd64.raw` and `vmlinux` on your system, here is the __cloud-hypervisor__ command you need to run:
Correct functioning of `--fs` requires `--memory shared=on` to facilitate
interprocess memory sharing.
Assuming you have `focal-server-cloudimg-amd64.raw` and `vmlinux` on your
system, here is the Cloud Hypervisor command you need to run:
```bash
./cloud-hypervisor \
--cpus boot=1\
@@ -60,26 +74,20 @@ Assuming you have `focal-server-cloudimg-amd64.raw` and `vmlinux` on your system
By default, DAX is enabled with a cache window of 8GiB. You can specify a custom size (let's say 4GiB for this example) for the cache by explicitly setting DAX and the cache size:
In case you don't want to use a shared window of cache to pass the shared files content, this means you will have to explicitly disable DAX with `dax=off`. Note that in this case, the `cache_size` parameter will be ignored.
The last step is to mount the shared directory inside the guest, using the `virtiofs` filesystem type.
The last step is to mount the shared directory inside the guest, using the
`virtiofs` filesystem type.
```bash
mkdir mount_dir
mount -t virtiofs -o dax myfs mount_dir/
mount -t virtiofs myfs mount_dir/
```
The `tag` needs to be consistent with what has been provided through the __cloud-hypervisor__ command line, which happens to be `myfs` in this example.
The `-o dax` option must be removed in case the shared cache region is not enabled from the VMM.
The `tag` needs to be consistent with what has been provided through the
Cloud Hypervisor command line, which happens to be `myfs` in this example.
## DAX feature
Given the DAX feature is not stable yet from a daemon standpoint, it is not
Currently Cloud Hypervisor only support hot plugging of CPU devices.
Currently Cloud Hypervisor supports hot plugging of CPUs devices (x86 only), PCI devices and memory resizing.
## Kernel support
@@ -9,7 +9,7 @@ or by using this kernel patch (available in 5.5-rc1 and later): https://git.kern
## CPU Hot Plug
Extra vCPUs can be added and removed from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
Extra vCPUs can be added and removed from a running `cloud-hypervisor` instance. This is controlled by two mechanisms:
1. Specifying a number of maximum potential vCPUs that is greater than the number of default (boot) vCPUs.
2. Making a HTTP API request to the VMM to ask for the additional vCPUs to be added.
@@ -65,7 +65,7 @@ As per adding CPUs to the guest, after a reboot the VM will be running with the
### ACPI method
Extra memory can be added from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
Extra memory can be added from a running `cloud-hypervisor` instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM. In the case of expanding the memory for the VM the new memory will be hotplugged into the running VM, if reducing the size of the memory then change will take effect after the next reboot.
@@ -157,7 +157,9 @@ The same API can also be used to reduce the desired RAM for a VM. It is importan
## PCI Device Hot Plug
Extra PCI devices can be added and removed from a running Cloud Hypervisor instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
Extra PCI devices can be added and removed from a running `cloud-hypervisor` instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
Note: On AArch64 platform, PCI device hotplug can only be achieved using ACPI. Please refer to the [documentation](arm64.md#uefi-booting) for more information.
To use PCI device hotplug start the VM with the HTTP server.
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size. If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the specified size for the VMM to use.
Failure to do so may result in strange VMM behaviour, e.g. error with `ReadKernelImage` is common.
If there is a strange error with `hugepages` enabled, just disable it or check whether there are enough huge pages.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `hotplug_method`
Selects the way of adding and/or removing memory to/from a booted VM.
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size.
If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the
specified size for the VMM to use. Failure to do so may result in strange VMM
behaviour, e.g. error with `ReadKernelImage` is common. If there is a strange
error with `hugepages` enabled, just disable it or check whether there are enough
huge pages.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `prefault`
Specifies if the memory must be `mmap(2)` with `MAP_POPULATE` flag.
By triggering prefault, one can allocate all required physical memory and create
its page tables while calling `mmap`. With physical memory allocated, the number
of page faults will decrease during running, and performance will also improve.
Note that boot of VM will be slower with `prefault` enabled because of allocating
physical memory and creating page tables in advance, and physical memory of the
specified size will be consumed quickly.
This option only takes effect at boot of VM. There is also a `prefault` option in
restore and its choice will overwrite `prefault` in memory.
By default this option is turned off.
_Example_
```
--memory size=1G,prefault=on
```
## Advanced Parameters
`MemoryZoneConfig` or what is known as `--memory-zone` from the CLI perspective
@@ -157,14 +184,16 @@ struct MemoryZoneConfig {
file: Option<PathBuf>,
shared: bool,
hugepages: bool,
hugepage_size: Option<u64>,
host_numa_node: Option<u32>,
hotplug_size: Option<u64>,
hotplugged_size: Option<u64>,
prefault: bool,
}
```
```
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>"
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off"
```
This parameter expects one or more occurences, allowing for a list of memory
@@ -252,24 +281,30 @@ _Example_
--memory-zone id=mem0,size=1G,shared=on
```
### `hugepages`
### `hugepages` and `hugepage_size`
Specifies if the memory zone must be `mmap(2)` with `MAP_HUGETLB` and
`MAP_HUGE_2MB`flags. This performs a memory zone mapping relying on 2MiB
pages instead of the default 4kiB pages.
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size.
If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the
specified size for the VMM to use. Failure to do so may result in strange VMM
behaviour, e.g. error with `ReadKernelImage` is common. If there is a strange
error with `hugepages` enabled, just disable it or check whether there are enough
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypervisor.
## Multiple queue support for net devices
While multiple vcpus defined for guest, to gain the benefit of vcpu scalable to improve performance, it suggests to define multiple queue pairs for net devices, one Tx/Rx queue pair per one vcpu, that means the number of queue pairs at least is equal to the vcpu count. In that case, after virtnet driver set cpu affinity for virtqueues in guest kernel, vcpus could handle interrupt from different virtqueue pairs in parallel.
It will gain better performance for guest that has multiple queues defined for net devices while it has multiple net sessions running in userspace.
To enable multiple queue support in cloud-hypervisor, multiple queue pairs will be defined, while multiple tap fds will be opened for the same tap device, it will also have multiple threads started, each thread will monitor and handle the events from each virtqueue pairs and the associated tap fd.
Note:
- Currently, it does not support to use ethtool to change the combined queue numbers in guest.
- Multiple queue is enabled for vhost-user-net backend in cloud-hypervisor, however, multiple thread is not added to handle mq, thus, the performance for vhost-user-net backend is not supposed to be improved. The multiple thread will be added for backend later.
- Performance test for vhost-user-net will be covered once vhost-user-net backend has multiple thread supported.
- Performance test for virtio-net is done by comparing 2 queue pairs with 1 queue pairs, that to run 2 iperf3 sessions in the same test environments, throughput is improved about 37%.
## Start cloud-hypervisor with net devices
Use one `--net` command-line argument from cloud-hypervisor to specify the emulation of one or more virtual NIC's. The example below instructs cloud-hypervisor to emulate for instance 2 virtual NIC's:
num_queues is the total number of tx and rx queues, the default value is 2, and it could be increased by multiples of 2. Additionally, num_queues is suggested to be as 2 times of vcpu count. The default value for queue_size is 256.
If the tap device is pre-created on host before guest boot up. To use multiple queue support for net device in guest, the tap device should be opened like this from host.
```bash
[root@localhost ~]# ip tuntap add name ich0 mode tap multi_queue
```
And the `--net` device should specify support for multiple queues. `num_queues` must be a multiple of 2 starting at least from 4 since multiple queues really means multiple queue pairs. We need at least 2 pairs for this configuration to be correct:
Different networking models can be used to provide external connectivity. In this example we will
use 2 linux bridges emulating 2 different networks. The integration bridge (ich-int) in this example will also be used
for external connectivity.
Create the bridges and connect the cloud-hypervisor tap devices to the bridges:
```bash
root@host:~# brctl addbr ich-int
root@host:~# brctl addbr ich-dpl
root@host:~# ip link set up ich-int
root@host:~# ip link set up ich-dpl
root@host:~# brctl addif ich-int ich0
root@host:~# brctl addif ich-dpl ich1
root@host:~# brctl show
bridge name bridge id STP enabled interfaces
ich-dpl 8000.067afc1b9a67 no ich1
ich-int 8000.725412ffce6f no ich0
```
This completes the layer 2 wiring: The cloud-hypervisor is now connected to the hypervisor host via the 2 linux bridges.
## IP (Layer 3) provisioning
### Hypervisor host
On the hypervisor host add the network gateway IP address of each network to the 2 linux bridges:
```bash
root@host:~# ip addr add 192.168.4.1/24 dev ich-int
root@host:~# ip addr add 10.0.1.1/24 dev ich-dpl
```
The routing table of the hypervisor host should now also have corresponding routing entries:
```bash
root@host:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.178.1 0.0.0.0 UG 60000 wlan1
10.0.1.0 0.0.0.0 255.255.255.0 U 000 ich-dpl
192.168.4.0 0.0.0.0 255.255.255.0 U 000 ich-int
192.168.178.0 0.0.0.0 255.255.255.0 U 60000 wlan1
```
### Virtual Machine
Within the virtual machine set the vNIC's to up state and provision the corresponding IP addresses on the 2 vNIC's. The steps outlined below use the ip command as an example. Alternative distribution specific procedures can also apply.
```bash
root@guest:~# ip link set up enp0s2
root@guest:~# ip link set up enp0s3
root@guest:~# ip addr add 192.168.4.2/24 dev enp0s2
root@guest:~# ip addr add 10.0.1.2/24 dev enp0s3
```
IP connectivity between the virtual machine and the hypervisor-host can be verified by sending
ICMP requests to the hypervisor-host for the gateway IP address from within the virtual machine:
```bash
root@guest:~# ping 192.168.4.1
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1ttl=64time=0.456 ms
64 bytes from 192.168.4.1: icmp_seq=2ttl=64time=0.226 ms
root@guest:~# ping 10.0.1.1
PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
64 bytes from 10.0.1.1: icmp_seq=1ttl=64time=0.449 ms
64 bytes from 10.0.1.1: icmp_seq=2ttl=64time=0.393 ms
```
The connection can now be used for instance to log into the virtual machine with
ssh under the precondition that the machine has an ssh daemon provisioned:
```bash
root@host:~# ssh root@192.168.4.2
The authenticity of host '192.168.4.2 (192.168.4.2)' can't be established.
ECDSA key fingerprint is SHA256:qNAUmTtDMW9pNuZARkpLQhfw+Yc1tqUDBrQp7aZGSjw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.4.2' (ECDSA) to the list of known hosts.
root@192.168.4.2's password:
Linux cloud-hypervisor 5.2.0 #2 SMP Thu Jul 11 08:08:16 CEST 2019 x86_64
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Jul 12 13:27:56 2019 from 192.168.4.1
root@guest:~#
```
## Internet connectivity
To enable internet connectivity a default gw and a nameserver has to be set within
the virtual machine:
```bash
root@guest:~# ip route add default via 192.168.4.1
root@guest:~# cat /etc/resolv.conf
options timeout:2
domain vallis.nl
search vallis.nl
nameserver 192.168.178.1
```
make sure that the default gateway of the hypervisor host (in this example host 192.168.178.1 which is an adsl router) has an entry in the routing table for the 192.168.4.0/24 network otherwise IP connectivity will not work.
If profiling with a network device attached either the TAP device must be already created and configured or the profiling must be done as root so that the TAP device can be created.
## Userspace only profiling with LBR
The use of LBR (Last Branch Record; available since Haswell) offers lower
overhead if only userspace profiling is required. This lower overhead can allow
a higher frequency of sampling. This also removes the requirement to compile
with custom `RUSTFLAGS` however debug symbols should still be included:
e.g.
```
$ perf record --call-graph lbr --all-user --user-callchains -g target/release/cloud-hypervisor \
This document provides guidance to users, downstream maintainers and
any other consumers of the Cloud Hypervisor project, this document
describes the release process, release cadence, stability expectations and
related topics.
## Basic Terms
### Stability
For Cloud Hypervisor the following areas are subject to stability guarantees:
- [REST API](api.md#rest-api)
- [Command line options](api.md#command-line-interface)
- [Device Model](device_model.md)
- Device tree, device list, ACPI, Hyper-V enlightenments and any other
features exposed to guest
- KVM compatibility
- Rust edition compatibility
This list is incomplete but this document serves as a best effort guide to stability
across releases.
### Experimental features
Experimental features are under active development and no guarantees are made about their stability.
List of experimental features:
- TDX
- vfio-user
- vDPA
### Security
Security fixes should be included in a new point release.
For security issues an advisory will be published via the GitHub security advisory process along with the release. Watching the project on GitHub will notify you of those issues.
## Releases
### Versioning
The versioning scheme uses `MAJOR.POINT` pattern:
-`MAJOR` can introduce incompatible changes along with support for new features. Changes to the [API](api.md#rest-api),
[CLI options](api.md#command-line-interface) and [device model](device_model.md)
require a notice at least 2 releases in advance for the actual change to take
Cloud Hypervisor is under active development. A new major release is issued approximately
every 6 weeks. Point releases are issued on demand, when important bug fixes are in
the queue. A major release would receive bug fixes for the next two cycles (~12 weeks)
and then be considered EOL.
```
+ - Active release support
E - EOL
2021 2022 2023
| | | | | | | | |
18.0 | | | ++++++++E
19.0 | | | |++++++++E
20.0 | | | | ++++++++E
21.0 | | | | | ++++++++E
22.0 | | | | | +++++++++E
23.0 | | | | | | +++++++++E
```
### Major Release Stability Considerations
Snapshot/restore support is not compatible across `MAJOR` versions.
Live migration support is not compatible across `MAJOR` versions.
### LTS Release Cadence
A regular release is promoted to LTS every 12 months. An LTS release is supported for 18 months. This gives a 6 months window for users to move to the new LTS.
After the successful build, the resulting firmware binaries are available under `Build/OvmfCh/DEBUG_GCC5/FV` underneath the edk2 checkout.
After the successful build, the resulting firmware binaries are available under `Build/CloudHvX64/DEBUG_GCC5/FV` underneath the edk2 checkout.
## Using OVMF Binaries
Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there.
To make Cloud Hypervisor use UEFI boot, pass the `OVMF.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
The same firmware can be used with Cloud Hypervisor or with QEMU. This is particularly useful if using QEMU for the preparation phase.
## Building UEFI Firmware with Compatibility Support Module (CSM)
CSM is a module that allows to boot legacy operating systems using the OVMF firmware. OVMF can embed a CSM build of SeaBIOS. To build the SeaBIOS with CSM support, add `CONFIG_CSM=y` to `.config` before the build. The outcome `out/Csm16.bin` is to be moved into `OvmfPkg/Csm/Csm16/Csm16.bin` before OVMF is built. Then, the OVMF build will have to be passed the `-D CSM_ENABLE` option in order to generate a legacy aware UEFI firmware. At the current stage, all the necessary patches are included in the Cloud Hypervisor specific [SeaBIOS branch](https://github.com/cloud-hypervisor/seabios/tree/ch). Taking into account the previous instructions, the modified command sequence to compile an OVMF binary with CSM support is the following one:
Please note, that the CSM support has currently only been tested with Linux guests. There are no plans to provide legacy support for other OSes (e.g. Windows).
To make Cloud Hypervisor use UEFI boot, pass the `CLOUDHV.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
- Suitable firmware for Cloud Hypervisor (`CLOUDHV.fd`) and for QEMU (`OVMF.fd`)
- With the suggested image size of 30G, there should be enough free disk space to hold the installation ISO and any other necessary files
This step currently requires QEMU to install Windows onto the guest. QEMU is only used at the preparation stage, the resulting image is then fully functional with Cloud Hypervisor.
@@ -67,7 +67,7 @@ The basic command to boot a Windows image. The configuration section should be c
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
