With the VIRTIO_F_EVENT_IDX handling now conducted inside the
virtio-queue crate it is necessary to activate the functionality on
every queue if it is negotiatated. Otherwise this leads to a failure of
the guest to signal to the host that there is something in the available
queue as the queue's internal state has not been configured correctly.
Fixes: #3829
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 223d0cf787)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When mask a msi irq, we set the entry.masked to be true, so kvm
hypervisor will not pass the gsi to kernel through KVM_SET_GSI_ROUTING
ioctl which update kvm->irq_routing. This will trigger kernel
panic on AMD platform when the gsi is the largest one in kernel
kvm->irqfds.items:
crash> bt
PID: 22218 TASK: ffff951a6ad74980 CPU: 73 COMMAND: "vcpu8"
#0 [ffffb1ba6707fa40] machine_kexec at ffffffff8565b397
#1 [ffffb1ba6707fa90] __crash_kexec at ffffffff85788a6d
#2 [ffffb1ba6707fb58] crash_kexec at ffffffff8578995d
#3 [ffffb1ba6707fb70] oops_end at ffffffff85623c0d
#4 [ffffb1ba6707fb90] no_context at ffffffff856692c9
#5 [ffffb1ba6707fbf8] exc_page_fault at ffffffff85f95b51
#6 [ffffb1ba6707fc50] asm_exc_page_fault at ffffffff86000ace
[exception RIP: svm_update_pi_irte+227]
RIP: ffffffffc0761b53 RSP: ffffb1ba6707fd08 RFLAGS: 00010086
RAX: ffffb1ba6707fd78 RBX: ffffb1ba66d91000 RCX: 0000000000000001
RDX: 00003c803f63f1c0 RSI: 000000000000019a RDI: ffffb1ba66db2ab8
RBP: 000000000000019a R8: 0000000000000040 R9: ffff94ca41b82200
R10: ffffffffffffffcf R11: 0000000000000001 R12: 0000000000000001
R13: 0000000000000001 R14: ffffffffffffffcf R15: 000000000000005f
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffffb1ba6707fdb8] kvm_irq_routing_update at ffffffffc09f19a1 [kvm]
#8 [ffffb1ba6707fde0] kvm_set_irq_routing at ffffffffc09f2133 [kvm]
#9 [ffffb1ba6707fe18] kvm_vm_ioctl at ffffffffc09ef544 [kvm]
RIP: 00007f143c36488b RSP: 00007f143a4e04b8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007f05780041d0 RCX: 00007f143c36488b
RDX: 00007f05780041d0 RSI: 000000004008ae6a RDI: 0000000000000020
RBP: 00000000000004e8 R8: 0000000000000008 R9: 00007f05780041e0
R10: 00007f0578004560 R11: 0000000000000246 R12: 00000000000004e0
R13: 000000000000001a R14: 00007f1424001c60 R15: 00007f0578003bc0
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
To solve this problem, move route.disable() before set_gsi_routes() to
remove the gsi from irqfds.items first.
This problem only exists on AMD platform, 'cause on Intel platform
kernel just return when update irte while it only prints a warning on
AMD.
Also, this patch adjusts the order of enable() and set_gsi_routes() in
unmask(), which should do no harm.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
(cherry picked from commit 5375b84e3b)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When mask a msi irq, we set the entry.masked to be true, so kvm
hypervisor will not pass the gsi to kernel through KVM_SET_GSI_ROUTING
ioctl which update kvm->irq_routing. This will trigger kernel
panic on AMD platform when the gsi is the largest one in kernel
kvm->irqfds.items:
crash> bt
PID: 22218 TASK: ffff951a6ad74980 CPU: 73 COMMAND: "vcpu8"
#0 [ffffb1ba6707fa40] machine_kexec at ffffffff8565b397
#1 [ffffb1ba6707fa90] __crash_kexec at ffffffff85788a6d
#2 [ffffb1ba6707fb58] crash_kexec at ffffffff8578995d
#3 [ffffb1ba6707fb70] oops_end at ffffffff85623c0d
#4 [ffffb1ba6707fb90] no_context at ffffffff856692c9
#5 [ffffb1ba6707fbf8] exc_page_fault at ffffffff85f95b51
#6 [ffffb1ba6707fc50] asm_exc_page_fault at ffffffff86000ace
[exception RIP: svm_update_pi_irte+227]
RIP: ffffffffc0761b53 RSP: ffffb1ba6707fd08 RFLAGS: 00010086
RAX: ffffb1ba6707fd78 RBX: ffffb1ba66d91000 RCX: 0000000000000001
RDX: 00003c803f63f1c0 RSI: 000000000000019a RDI: ffffb1ba66db2ab8
RBP: 000000000000019a R8: 0000000000000040 R9: ffff94ca41b82200
R10: ffffffffffffffcf R11: 0000000000000001 R12: 0000000000000001
R13: 0000000000000001 R14: ffffffffffffffcf R15: 000000000000005f
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffffb1ba6707fdb8] kvm_irq_routing_update at ffffffffc09f19a1 [kvm]
#8 [ffffb1ba6707fde0] kvm_set_irq_routing at ffffffffc09f2133 [kvm]
#9 [ffffb1ba6707fe18] kvm_vm_ioctl at ffffffffc09ef544 [kvm]
RIP: 00007f143c36488b RSP: 00007f143a4e04b8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007f05780041d0 RCX: 00007f143c36488b
RDX: 00007f05780041d0 RSI: 000000004008ae6a RDI: 0000000000000020
RBP: 00000000000004e8 R8: 0000000000000008 R9: 00007f05780041e0
R10: 00007f0578004560 R11: 0000000000000246 R12: 00000000000004e0
R13: 000000000000001a R14: 00007f1424001c60 R15: 00007f0578003bc0
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
To solve this problem, move route.disable() before set_gsi_routes() to
remove the gsi from irqfds.items first.
This problem only exists on AMD platform, 'cause on Intel platform
kernel just return when update irte while it only prints a warning on
AMD.
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
(cherry picked from commit db9e5e5a87)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As per this kernel documentation:
For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR, KVM_EXIT_XEN,
KVM_EXIT_EPR, KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR the corresponding
operations are complete (and guest state is consistent) only after userspace
has re-entered the kernel with KVM_RUN. The kernel side will first finish
incomplete operations and then check for pending signals.
The pending state of the operation is not preserved in state which is
visible to userspace, thus userspace should ensure that the operation is
completed before performing a live migration. Userspace can re-enter the
guest with an unmasked signal pending or with the immediate_exit field set
to complete pending operations without allowing any further instructions
to be executed.
Since we capture the state as part of the pause and override it as part
of the resume we must ensure the state is consistent otherwise we will
lose the results of the MMIO or PIO operation that caused the exit from
which we paused.
Fixes: #3658
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 507912385a)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When freeing memory sometimes glibc will attempt to read
"/proc/sys/vm/overcommit_memory" to find out how it should release the
blocks. This happens sporadically with Cloud Hypervisor but has been
seen in use. It is not necessary to add the read() syscall to the list
as it is already included in the virtio devices common set. Similarly
the vCPU and vmm threads already have both these in the allowed list.
Fixes: #3609
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 53caa565bb)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Both OVMF and RHF firmwares triggered an error when O_DIRECT was used
because they didn't align the buffers to the block sector size.
In order to prevent regressions, we're adding a new test validating the
VM can properly boot when the OS disk is opened with O_DIRECT and booted
from the rust-hypervisor-fw.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the backing file of our virtio-block device is opened with
O_DIRECT, there's a requirement about the buffer address and size to be
aligned to the sector size.
We know virtio-block requests are sector aligned in terms of size, but
we must still check if the buffer address is. In case it's not, we
create an intermediate buffer that will be passed through the system
call. In case of a write operation, the content of the non-aligned
buffer must be copied beforehand, and in case of a read operation, the
content of the aligned buffer must be copied to the non-aligned one
after the operation has been completed.
Fixes#3587
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This table is listed as required in the ARM Base Boot Requirements
document. The particular need arises to make the serial debugging of
Windows guest functional.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
If the guest hasn't initialised a PV clock then the KVM_KVMCLOCK_CTRL
ioctl will return -EINVAL. Therefore if running in the firmware or an OS
that doesn't use the PV clock then we should ignore that error
Tested by migrating a VM that has not yet booted into the Linux kernel
(just in firmware) by specifying no disk image:
e.g. target/debug/cloud-hypervisor --kernel ~/workloads/hypervisor-fw --api-socket /tmp/api --serial tty --console off
Fixes: #3586
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
From 15358ef79d: Device Mapper Multipath
config can avoid systemd errors related to Device Mapper multipath while
guest booting.
From 46672c384c: CONFIG_NVME_MULTIPATH is
needed to fix the observed guest hanging issue cased by systemd crash
while booting.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The SPDK-NVMe is needed for the integration test for vfio_user.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Signed-off-by: Bo Chen <chen.bo@intel.com>
AArch64 does not use IOBAR, and current code of panics the whole VMM if
we need to allocate the IOBAR.
This commit checks if IOBAR is enabled before the arch conditional code
of IOBAR allocation and if the IOBAR is not enabled, we can just skip
the IOBAR allocation and do nothing.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/3479
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Ensure all pending virtio activations (as triggered by MMIO write on the
vCPU threads leading to a barrier wait) are completed before pausing the
vCPUs as otherwise there will a deadlock with the VMM waiting for the
vCPU to acknowledge it's pause and the vCPU waiting for the VMM to
activate the device and release the barrier.
Fixes: #3585
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The code can be written in a better form and the clippy warning
suppression can be dropped.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
These are the leftovers from the commit 8155be2:
arch: aarch64: vm_memory is not required when configuring vcpu
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Update documentation and CI to rely on the new CLOUDHV.fd firmware built
from the newly introduced target CloudHvX64.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By pinning the OVMF version, we will be able to update the EDK2 fork
with a new version without potentially breaking our Cloud Hypervisor CI.
Once the new version is ready on the EDK2 fork, we'll be able to update
Cloud Hypervisor codebase, replacing the fixed version with the latest,
as well as replacing OVMF.fd with CLOUDHV.fd. This is because we'll
start building from the new target CloudHvX64.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When in local migration mode send the FDs for the guest memory over the
socket along with the slot that the FD is associated with. This removes
the requirement for copying the guest RAM and gives significantly faster
live migration performance (of the order of 3s to 60ms).
Fixes: #3566
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Create the VM using the FDs (wrapped in Files) that have been received
during the migration process.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If this FD (wrapped in a File) is supplied when the RAM region is being
created use that over creating a new one.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This function is used to open an FD (wrapped in a File) that points to
guest memory from memfd_create() or backed on the filesystem.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently cloud-hypervisor supports hotplug of CPUs, passthrough devices, virtio devices and memory resizing,
but in hotplug.md it still mentiones: "Currently Cloud Hypervisor only supports hot plugging of CPU devices."
We need to remove the incorrect information from hotplug.md to reflect the true situation.
Fixes: #3504
Signed-off-by: Liang Zhou <zhoul110@chinatelecom.cn>
Drop the unused parameter throughout the code base.
Also take the chance to drop a needless clone.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
error: unneeded late initalization
Error: --> devices/src/legacy/rtc_pl031.rs:294:9
|
294 | let v;
| ^^^^^^
|
= note: `-D clippy::needless-late-init` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `v` here
|
297 | let v = if (AMBA_ID_LOW..AMBA_ID_HIGH).contains(&offset) {
| +++++++
help: remove the assignments from the branches
|
299 ~ u32::from(PL031_ID[index])
300 | } else {
301 ~ match offset {
302 | RTCDR => self.get_time(),
303 | RTCMR => {
304 | // Even though we are not implementing RTC alarm we return the last value
...
help: add a semicolon after the `if` expression
|
317 | };
| +
error: unneeded late initalization
Error: --> devices/src/legacy/uart_pl011.rs:297:9
|
297 | let v;
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `v` here
|
299 | let v = if (AMBA_ID_LOW..AMBA_ID_HIGH).contains(&(offset >> 2)) {
| +++++++
help: remove the assignments from the branches
|
301 ~ u32::from(PL011_ID[index])
302 | } else {
303 ~ match offset >> 2 {
304 | UARTDR => {
305 | let c: u32;
306 | let r: u32;
...
help: add a semicolon after the `if` expression
|
340 | };
| +
error: unneeded late initalization
Error: --> devices/src/legacy/uart_pl011.rs:305:21
|
305 | let c: u32;
| ^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `c` here
|
309 | let c: u32 = self.read_fifo.pop_front().unwrap_or_default().into();
| ~~~~~~~~~~
error: unneeded late initalization
Error: --> devices/src/legacy/uart_pl011.rs:306:21
|
306 | let r: u32;
| ^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `r` here
|
320 | let r: u32 = c;
| ~~~~~~~~~~
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: this boolean expression can be simplified
--> tests/integration.rs:3755:33
|
3755 | assert!(!(empty > 5), "No login on pty");
| ^^^^^^^^^^^^ help: try: `empty <= 5`
|
= note: `-D clippy::nonminimal-bool` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#nonminimal_bool
error: unneeded late initalization
--> tests/integration.rs:7619:13
|
7619 | let mut success;
| ^^^^^^^^^^^^^^^^
|
= note: `-D clippy::needless-late-init` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `success` here
|
7621 | let mut success = if let Some(status) = send_migration
| +++++++++++++++++
help: remove the assignments from the branches
|
7625 ~ status.success()
7626 | } else {
7627 ~ false
|
help: add a semicolon after the `if` expression
|
7628 | };
| +
error: unneeded late initalization
--> tests/integration.rs:7838:13
|
7838 | let mut success;
| ^^^^^^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `success` here
|
7840 | let mut success = if let Some(status) = send_migration
| +++++++++++++++++
help: remove the assignments from the branches
|
7844 ~ status.success()
7845 | } else {
7846 ~ false
|
help: add a semicolon after the `if` expression
|
7847 | };
| +
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: unneeded late initalization
Error: --> arch/src/aarch64/gic/gicv3_its.rs:127:9
|
127 | let attr: u64;
| ^^^^^^^^^^^^^^
|
= note: `-D clippy::needless-late-init` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `attr` here
|
128 | let attr: u64 = if save {
| +++++++++++++++
help: remove the assignments from the branches
|
129 ~ u64::from(kvm_bindings::KVM_DEV_ARM_ITS_SAVE_TABLES)
130 | } else {
131 ~ u64::from(kvm_bindings::KVM_DEV_ARM_ITS_RESTORE_TABLES)
|
help: add a semicolon after the `if` expression
|
132 | };
| +
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We've been currently using "fd" as the field name, but it should be
called "fds" since 6664e5a6e7 introduced
the name change on the structure field.
Fixes: #3560
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Passing no boot related parameters (e.g. no --kernel) is used for e.g.
receiving a live migration or an API based boot.
marvin:~/src/cloud-hypervisor (2022-01-11-live-migration-with-fds *)$ target/debug/cloud-hypervisor --api-socket /tmp/api2
thread 'main' panicked at '`tdx` is not a name of an argument or a group.
Make sure you're using the name of the argument itself and not the name of short or long flags.', /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.0.6/src/parse/matches/arg_matches.rs:598:14
stack backtrace:
0: rust_begin_unwind
at /rustc/7d6f948173ccb18822bab13d548c65632db5f0aa/library/std/src/panicking.rs:498:5
1: core::panicking::panic_fmt
at /rustc/7d6f948173ccb18822bab13d548c65632db5f0aa/library/core/src/panicking.rs:107:14
2: clap::parse::matches::arg_matches::ArgMatches::get_arg
at /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.0.6/src/parse/matches/arg_matches.rs:1052:17
3: clap::parse::matches::arg_matches::ArgMatches::is_present
at /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.0.6/src/parse/matches/arg_matches.rs:598:9
4: cloud_hypervisor::start_vmm
at ./src/main.rs:530:46
5: cloud_hypervisor::main
at ./src/main.rs:566:27
6: core::ops::function::FnOnce::call_once
at /rustc/7d6f948173ccb18822bab13d548c65632db5f0aa/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`tap` has its default value set to `None`, but in the openapi yaml file
we've been setting it to `""`.
When using this code on the Kata Containers side we'd be hit by a non
expected behaviour of cloud-hypervisor, as even when using a different
method to initialise the `tuntap` device the code would be treated as if
using `--net tap` (which is a valid use-case).
Related: #3554
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
When enabling the `mshv` feature, we skip quite some tests and
hence have known dead-code. This annotation silences dead-code
related warnings for our quality workflow to pass.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Given integration tests don't have special annotations, all our
integration test scripts do not need to carry special flags when running
`cargo test`. This commit also removed the "test::" prefix for
pattern-matching different set of integration tests.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Given integration tests are placed in a dedicate directory, they don't
need annotations (e.g. `#[cfg(integration_test)]` and `#[cfg(test)]`) or
defining `test mod` to exclude themselves from the common compilation
process.
Signed-off-by: Bo Chen <chen.bo@intel.com>
warning: unneeded late initalization
--> src/main.rs:134:5
|
134 | let mut app: App;
| ^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::needless_late_init)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `app` here
|
138 | let mut app: App = App::new("cloud-hypervisor")
| ~~~~~~~~~~~~~~~~
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: unneeded late initalization
--> vmm/src/acpi.rs:525:5
|
525 | let mut prev_tbl_len: u64;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::needless_late_init)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `prev_tbl_len` here
|
552 | let mut prev_tbl_len: u64 = madt.len() as u64;
| ~~~~~~~~~~~~~~~~~~~~~~~~~
warning: unneeded late initalization
--> vmm/src/acpi.rs:526:5
|
526 | let mut prev_tbl_off: GuestAddress;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `prev_tbl_off` here
|
553 | let mut prev_tbl_off: GuestAddress = madt_offset;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: unneeded late initalization
--> virtio-queue/src/chain.rs:381:13
|
381 | let desc: Descriptor;
| ^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::needless_late_init)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_late_init
help: declare `desc` here
|
382 | let desc: Descriptor = if i < 3 {
| ++++++++++++++++++++++
help: remove the assignments from the branches
|
383 ~ Descriptor::new(0x1000 * i as u64, 0x1000, VIRTQ_DESC_F_NEXT, i + 1)
384 | } else {
385 ~ Descriptor::new(0x1000 * i as u64, 0x1000, 0, 0)
|
help: add a semicolon after the `if` expression
|
386 | };
| +
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Current command to run cloud-hypervisor binary has
wrong --rng argument in the README file.
This commit fixes all the references of --rng argument
by removing the --rng option to use default.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This crate contains up to date definition of the Queue, AvailIter,
DescriptorChain and Descriptor structures forked from the upstream
crate rust-vmm/vm-virtio 27b18af01ee2d9564626e084a758a2b496d2c618.
The following patches have been applied on top of this base in order to
make it work correctly with Cloud Hypervisor requirements:
- Add MSI vector field to the Queue
In order to help with MSI/MSI-X support, it is convenient to store the
value of the interrupt vector inside the Queue directly.
- Handle address translations
For devices with access to data in memory being translated, we add to
the Queue the ability to translate the address stored in the
descriptor.
It is very helpful as it performs the translation right after the
untranslated address is read from memory, avoiding any errors from
happening from the consumer's crate perspective. It also allows the
consumer to reduce greatly the amount of duplicated code for applying
the translation in many different places.
- Add helpers for Queue structure
They are meant to help crate's consumers getting/setting information
about the Queue.
These patches can be found on the 'ch' branch from the Cloud Hypervisor
fork: https://github.com/cloud-hypervisor/vm-virtio.git
This patch takes care of updating the Cloud Hypervisor code in
virtio-devices and vm-virtio to build correctly with the latest version
of virtio-queue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This crate was used in the integration tests to allow the tests to
continue and clean up after a failure. This isn't necessary in the unit
tests and adds a large build dependency chain including an unmaintained
crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This updated tree now includes a fix for virtio-net regression that was
present in 5.14:
commit 732b74d647048668f0f8dc0c848f0746c69e2e2f
Author: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Date: Sat Oct 9 05:17:53 2021 -0400
virtio-net: fix for skb_over_panic inside big mode
commit 126285651b7f ("Merge ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net")
accidentally reverted the effect of
commit 1a8024239da ("virtio-net: fix for skb_over_panic inside big mode")
on drivers/net/virtio_net.c
As a result, users of crosvm (which is using large packet mode)
are experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13.
Crash trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-rc1 linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When forwarding an epoll event from the unix muxer to the
targeted connection event handler, the eventset the connection
registered is forwarded instead of the actual epoll
operation (IN/OUT).
For example, if the connection was registered for EPOLLIN,
and receives an EPOLLOUT, the connection will actually handle
an EPOLLOUT.
This is the root cause of previous bug, which caused the
introduction of some workarounds (i.e: handling ewouldblock
when reading after receiving EPOLLIN, which should never happen).
When matching the connection, we retrieve and use the evset of
the connection instead of the one passed as a parameter.
The compiler does not complain for an unused variable because
it was first logged in a debug! statement.
This is an unfortunate naming mistake that caused a lot of problems.
Fixes#3497
Signed-off-by: Eduard Kyvenko <eduard.kyvenko@gmail.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the underlying kernel is old PTY resize is disabled and this is
represented by the use of None in the provided Option<File> type. In the
virtio-console PTY path don't blindly unwrap() the value that will be
preserved across a reboot.
Fixes: #3496
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The test test_virtio_block_topology is flaky on aarch64, let's disable
it until we find the right way to fix it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is a corner case of using the Cloud Hypervisor on AArch64:
If the VM is started on a device where RAM is limited, and if the user
allocates nearly as much memory for the guest as is still free on the
host, we need to enable the swap memory.
This commit documented this corner case with explanation.
Also, this commit corrects the hardware requirement of the GIC interrupt
controller for running the Cloud Hypervisor on AArch64, accroding to
[1].
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/3419
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
[1] https://www.kernel.org/doc/html/latest/virt/kvm/devices/arm-vgic-its.html
It's important to maintain the ability to run in a non-TDX environment
a Cloud Hypervisor binary with the 'tdx' feature enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This test relies on using losetup with a block size to create a block
device from a file that has a specific block size for the topology
detection code to pick up and passthrough to the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the disk is backed by a block device on the host a non-default
topology will be available and that topology can be advertised by virtio
block.
Fixes: #3262
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For simplicity this trait implements a default version that has a
topology with 512 byte (i.e. sector) recommended sizes.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Setting up the SIGWINCH handler requires at least Linux 5.7. However
this functionality is not required for basic PTY operation.
Fixes: #3456
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 58d25b3ccc.
This change introduced a regression when running iperf with the guest
running as the server:
marvin:~/src/cloud-hypervisor ((58d25b3c...))$ iperf -c 192.168.249.2
------------------------------------------------------------
Client connecting to 192.168.249.2, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.249.1 port 47078 connected with 192.168.249.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.40 sec 14.0 MBytes 11.3 Mbits/sec
marvin:~/src/cloud-hypervisor ((58d25b3c...))$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.249.1 port 5001 connected with 192.168.249.2 port 42866
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.01 sec 51.2 GBytes 44.0 Gbits/sec
Fixes: #3450
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The sendmsg() syscall is limited in the number of fds it can handle.
This number matches that used by the vfio-user library and is
conservative (since we've seen it work with 64 fds.)
Fixes: #3401
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This time we use the Rust Hypervisor Firmware for test_vfio_user() in
order to fix the systemd issues we've seen so far.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Advertise the PCI MMIO config spaces here so that the MMIO config space
is correctly recognised.
Tested by: --platform num_pci_segments=1 or 16 hotplug NVMe vfio-user device
works correctly with hypervisor-fw & OVMF and direct kernel boot.
Fixes: #3432
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When restoring replace the internal value of the device tree rather than
replacing the Arc<Mutex<DeviceTree>> itself. This is fixes an issue
where the AddressManager has a copy of the the original
Arc<Mutex<DeviceTree>> from when the DeviceManager was created. The
original restore path only replaced the DeviceManager's version of the
Arc<Mutex<DeviceTree>>. Instead replace the contents of the
Arc<Mutex<DeviceTree>> so all users see the updated version.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The observation is that the code in question was used to bridge
synchronized and asynchronized code.
We can group the functions for that purpose under an adaptor trait. To
limit the scope of locking, the users of the trait are required to
implement a method to return a MutexGuard for the underlying file.
This then allows us to use concrete types (QcowFile and Vhdx) in code,
which is easier to read than a bunch of traits.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Saving the KVM clock and restoring it is key for correct behaviour of
the VM when doing snapshot/restore or live migration. The clock is
restored to the KVM state as part of the Vm::resume() method prior to
that it must be extracted from the state object and stored for later use
by this method. This change simplifies the extraction and storage part
so that it is done in the same way for both snapshot/restore and live
migration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit enhances the integration test for multiple PCI segments
by:
(1) Enables the `test_virtio_fs_multi_segment` on AArch64.
(2) Adds a new integration test case for both x86_64 and AArch64 using
the direct kernel boot to test virtio-disk multiple PCI segments.
The test case does:
- Start a VM using direct kernel boot with 16 PCI segments and assign
the last PCI segment with a virtio-disk device.
- Check if the number of PCI host bridges equals to 16 after VM boots.
- Mount the virtio-disk device on the last PCI segment to the rootfs
and write/read data to the virtio-disk device.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Each PCI device under a root complex is uniquely identified by its
Requester ID (AKA RID). A Requester ID is a triplet of a Bus number,
Device number, and Function number.
MSIs may be distinguished in part through the use of sideband data
accompanying writes. In the case of PCI devices, this sideband data
may be derived from the Requester ID. A mechanism is required to
associate a device with both the MSI controllers it can address,
and the sideband data that will be associated with its writes to
those controllers.
This commit adds the `msi-map` property for PCI nodes, therefore
creating MSI mapping for each PCI device.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit rewrites the `create_pci_node` in the FDT creator to
create multiple PCI nodes based on the vector of `PciSpaceInfo`,
and each PCI node in FDT reflects a PCI segment.
- The PCI MMIO config space, 32 bits PCI device space and 64 bits
PCI device space is re-calculated based on the `PciSpaceInfo` for
each PCI segment.
- A new FDT property `linux,pci-domain` is added.
- The virtio-iommu node is only created for the first PCI segment.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The constant `PCI_MMIO_CONFIG_SIZE` defined in `vmm/pci_segment.rs`
describes the MMIO configuation size for each PCI segment. However,
this name conflicts with the `PCI_MMCONFIG_SIZE` defined in `layout.rs`
in the `arch` crate, which describes the memory size of the PCI MMIO
configuration region.
Therefore, this commit renames the `PCI_MMIO_CONFIG_SIZE` to
`PCI_MMIO_CONFIG_SIZE_PER_SEGMENT` and moves this constant from `vmm`
crate to `arch` crate.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently, a tuple containing PCI space start address and PCI space
size is used to pass the PCI space information to the FDT creator.
In order to support the multiple PCI segment for FDT, more information
such as the PCI segment ID should be passed to the FDT creator. If we
still use a tuple to store these information, the code flexibility and
readablity will be harmed.
To address this issue, this commit replaces the tuple containing the
PCI space information to a structure `PciSpaceInfo` and uses a vector
of `PciSpaceInfo` to store PCI space information for each segment, so
that multiple PCI segment information can be passed to the FDT together.
Note that the scope of this commit will only contain the refactor of
original code, the actual multiple PCI segments support will be in
following series, and for now `--platform num_pci_segments` should only
be 1.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Previously mutex (semaphore) and file were separated. The code needed to
create artificial scopes to use mutex to protect file.
Rewrite the code to be idiomatic. The file itself is turned into a trait
object and placed inside the mutex. This requires providing a new
ReadWriteSeekFile trait to unify all helper functions.
The rewrite further simplified vhdx_sync code. The original code
contained two mutex'es for no apparent reason.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Extending the test_simple_launch() integration test to validate Cloud
Hypervisor boots correctly with both rust-hypervisor-fw and OVMF on
x86_64 platforms.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Bumping the OVMF binary version along with UEFI documentation to
reflect the latest set of patches on top of tianocore/edk2 'master'
branch, which can be found on the Cloud Hypervisor fork on 'ch' branch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to avoid the identity map region to conflict with a possible
firmware being placed in the last 4MiB of the 4GiB range, we must set
the address to a chosen location. And it makes the most sense to have
this region placed right after the TSS region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extending the Vm trait with set_identity_map_address() in order to
expose this ioctl to the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Commit ac25172176 bumps the rust
version of virtiofsd named `virtiofsd-rs`, which causes a warning
```
warning: use of deprecated parameter '--socket':
Please use the '--socket-path' option instead.
```
This commit updates the cmdline parameter accordingly.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Place the 3 page TSS at an explicit location in the 32-bit address space
to avoid conflicting with the loaded raw firmware.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Reduce the size of the reserved 32-bit address space to the range used
by both the PCI MMIO config data and the 32-bit PCI device space.
This avoids issues when using firmware that is loaded into the very top
of the 32-bit address space as the RAM conflicts with the reserved
memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Current `Getting Started` section only contains steps for the x86_64
platform, as we have a documentation doing the same thing for AArch64,
we can point users to the correct documentation.
Also, this commit modifies the `docs/arm64.md` to fit the documentation
style within the project.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This test is flaky (#3400) while we are experiencing a bug of using the latest
SPDK/NVMe backend as VFIO user device (#3401). Let's disable this test
before we fix the above two issues.
Signed-off-by: Bo Chen <chen.bo@intel.com>
For now we only enable the vfio-user test on x86_64 platform, as we have
a known hanging issue to resovle on the aarch64 platform.
Fixes: #3098
Signed-off-by: Bo Chen <chen.bo@intel.com>
Enabling these configs can avoid systemd errors related to Device Mapper
multipath while guest booting. Especially, the guest can hang when being
used with an NVMe backend without these configs (#3352).
Signed-off-by: Bo Chen <chen.bo@intel.com>
This kernel config is needed to fix the observed guest hanging issue
cased by systemd crash while booting.
Fixes: #3352
Signed-off-by: Bo Chen <chen.bo@intel.com>
Added fields:
- `Memory address size limit`: the missing of this field triggered
warnings in guest kernel
- `Node ID`
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
After introducing multiple PCI segments, the `devid` value in
`kvm_irq_routing_entry` exceeds the maximum supported range on AArch64.
This commit restructed the `devid` to the allowed range.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The register D has only one bit that is not reserved, and its purpose is
to report if the RTC/CMOS device is powered or not.
The OVMF firmware was failing to boot as it was getting the information
that the device was powered off from the register D.
The simple way to fix this issue is by always returning the bit 7 from
register D as 1, indicating the device is always powered.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the provided binary isn't an ELF binary assume that it is a firmware
to be loaded in directly. In this case we shouldn't program any of the
registers as KVM starts in that state.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Function alloc_zeroed can fail. Check its return in read and write
functions. Its return value in is_valid_alignment is not checked because
handling error in that case does not give us much benefit. Instead, an
assertion is added.
Add safety comments to all `unsafe`s.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Simple aggregate types are Sync by default. There is no need to use
`impl Sync` for MockVmm (a simple struct).
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That function call can return -1 when it fails. Wrapping -1 into File
causes the code to panic when the File is dropped.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
I encountered some trouble trying to use a virtio-console hooked up to
a PTY. Reading from the PTY would produce stuff like this
"\n\nsh-5.1# \n\nsh-5.1# " (where I'm just pressing enter at a shell
prompt), and a terminal would render that like this:
----------------------------------------------------------------
sh-5.1#
sh-5.1#
----------------------------------------------------------------
This was because we weren't disabling the ICRNL termios iflag, which
turns carriage returns (\r) into line feeds (\n). Other raw mode
implementations (like QEMU's) set this flag, and don't have this
problem.
Instead of fixing our raw mode implementation to just disable ICRNL,
or copy the flags from QEMU's, though, here I've changed it to use the
raw mode implementation in libc. It seems to work correctly in my
testing, and means we don't have to worry about what exactly raw mode
looks like under the hood any more.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Creating some brief documentation for TDX, summarizing the links on
where to find more information about TDX, as well as how to run Cloud
Hypervisor on it.
Fixes#3318
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Fix seccomp violation when trying to add the out FD to the epoll loop
when the serial buffer needs to be flushed.
0x00007ffff7dc093e in epoll_ctl () at ../sysdeps/unix/syscall-template.S:120
0x0000555555db9b6d in epoll::ctl (epfd=56, op=epoll::ControlOptions::EPOLL_CTL_MOD, fd=55, event=...)
at /home/rob/.cargo/registry/src/github.com-1ecc6299db9ec823/epoll-4.3.1/src/lib.rs:155
0x00005555556f5127 in vmm::serial_buffer::SerialBuffer::add_out_poll (self=0x7fffe800b5d0) at vmm/src/serial_buffer.rs:101
0x00005555556f583d in vmm::serial_buffer::{impl#1}::write (self=0x7fffe800b5d0, buf=...) at vmm/src/serial_buffer.rs:139
0x0000555555a30b10 in std::io::Write::write_all<vmm::serial_buffer::SerialBuffer> (self=0x7fffe800b5d0, buf=...)
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1527
0x0000555555ab82fb in devices::legacy::serial::Serial::handle_write (self=0x7fffe800b520, offset=0, v=13) at devices/src/legacy/serial.rs:217
0x0000555555ab897f in devices::legacy::serial::{impl#2}::write (self=0x7fffe800b520, _base=1016, offset=0, data=...) at devices/src/legacy/serial.rs:295
0x0000555555f30e95 in vm_device::bus::Bus::write (self=0x7fffe8006ce0, addr=1016, data=...) at vm-device/src/bus.rs:235
0x00005555559406d4 in vmm::vm::{impl#4}::pio_write (self=0x7fffe8009640, port=1016, data=...) at vmm/src/vm.rs:459
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When running with `--serial pty --console pty --seccomp=false` the
SIGWICH listener thread would panic as the seccomp filter was empty.
Adopt the mechanism used in the rest of the code and check for non-empty
filter before trying to apply it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This test generates an array of random numbers and then applies the same
trivial algorithm twice -- once in set_apic_delivery_mode and another
time in an anonymous function.
Its usefulness is limited. Drop it to remove one unsafe in code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Fixing a few inconsistencies and extending the document to tackle
multiple devices use case, as well as having multiple devices under the
same IOMMU group.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The variable tmp was never initialized. Calling assume_init when the
content is not yet initialized causes immediate undefined behaviour.
We also cannot create any intermediate references because they will be
subject to the same requirements for references -- the referred object
must be valid.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The proper way to refer to the project is "Cloud Hypervisor" without the
hyphen in the middle. On the other hand, if one refers to the binary
name, it is "cloud-hypervisor".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new document describes the available options for the parameter
`--cpus`, how to use them and what is their usage.
Fixes#3317
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing list of options available for the 'cpus' parameter
with the newly added option 'affinity'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
OSTree based systems (such as Fedora CoreOS) behave differently than yum
based systems (such as Fedora) with regarding to the RPM installations.
While the latter will install the RPM on the running system, modifying
it; the former will be layered (installed in a "chroot") and only will
be present on the system after the user boots this layer up.
While in theory there shouldn't be much differences on the installation
itself, when installing cloud-hypervisor on RHCOS, the following issue
would be seen:
```
$ sudo sudo rpm-ostree install cloud-hypervisor
Checking out tree 9c1cdf9... done
Enabled rpm-md repositories: copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor
Updating metadata for 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor'... done
rpm-md repo 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor'; generated: 2021-11-08T19:02:32Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 1 package (3.1 MB)
Downloading from 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
error: Running %post for cloud-hypervisor: Executing bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(cloud-hypervisor.post)'` for more information
$ sudo journalctl -t 'rpm-ostree(cloud-hypervisor.post)'
-- Logs begin at Fri 2021-11-12 12:18:33 UTC, end at Fri 2021-11-12 12:22:55 UTC. --
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Failed to set capabilities on file `/usr/bin/cloud-hypervisor' (Operation not supported)
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: usage: setcap [-q] [-v] [-n <rootid>] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Note <filename> must be a regular (non-symlink) file.
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Failed to set capabilities on file `/usr/lib64/cloud-hypervisor/vhost_user_net' (Operation not supported)
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: usage: setcap [-q] [-v] [-n <rootid>] (-r|-|<caps>) <filename> [ ... (-r|-|<capsN>) <filenameN> ]
Nov 12 12:22:40 rhcoslatest rpm-ostree(cloud-hypervisor.post)[1637]: Note <filename> must be a regular (non-symlink) file.
```
It seems to happen as the "chroot" where the package is installed does
not have enough privileges to set the binaries capabilities during the
`%post` step.
Considering this, and also considering that calling `setcap` during
`%post` may not be the most elegant solution when speaking RPM spec
files, let's take advantage of the already existent `%caps()` directive
for the `%files` list to do so.
With that change applied, cloud-hypervisor could be successfully
installed on RHCOS, as shown below:
```
$ sudo rpm-ostree install cloud-hypervisor-19.0-2.el8.x86_64.rpm
Checking out tree 9c1cdf9... done
Enabled rpm-md repositories: copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor
rpm-md repo 'copr:copr.fedorainfracloud.org:fidencio:cloud-hypervisor' (cached); generated: 2021-11-08T19:02:32Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 18.2 MB (pkgcache branches: 2)
Added:
cloud-hypervisor-19.0-2.el8.x86_64
Run "systemctl reboot" to start a reboot
...
$ getcap /usr/bin/cloud-hypervisor
/usr/bin/cloud-hypervisor = cap_net_admin+ep
$ getcap /usr/lib64/cloud-hypervisor/vhost_user_net
/usr/lib64/cloud-hypervisor/vhost_user_net = cap_net_admin+ep
```
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Although this not strictly needed, considering cloud-hypervisor installs
only 8 binaries in total (4 per target), and that having that list of
files expanded will help us in the very near future, let's just do it
now.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
As part of checking if io_uring is supported various functionality is
tested. The test for whether io_uring supports EventFds is very time
consuming (~10ms) however this test can be removed as a later test will
test for functionality added after this one.
The support for register_eventfd() was released in Linux 5.1 but the
support for register_probe() was released in Linux 5.4. So if the latter
is present the former also is.
Before:
cloud-hypervisor: 4.880411ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/home/rob/workloads/focal-server-cloudimg-amd64-custom-20210609-0.raw"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk0"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 14.105123ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
cloud-hypervisor: 14.134837ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/tmp/disk"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk1"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 14.221869ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
After:
cloud-hypervisor: 3.140716ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/home/rob/workloads/focal-server-cloudimg-amd64-custom-20210609-0.raw"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk0"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 3.376027ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
cloud-hypervisor: 3.40446ms: <vmm> INFO:vmm/src/device_manager.rs:1916 -- Creating virtio-block device: DiskConfig { path: Some("/tmp/disk"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_socket: None, poll_queue: true, rate_limiter_config: None, id: Some("_disk1"), disable_io_uring: false, pci_segment: 0 }
cloud-hypervisor: 3.513969ms: <vmm> INFO:vmm/src/device_manager.rs:1998 -- Using asynchronous RAW disk file (io_uring)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This new integration test validates the vCPUs are running on the
expected set of CPUs on the host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the introduction of a new option `affinity` to the `cpus`
parameter, Cloud Hypervisor can now let the user choose the set
of host CPUs where to run each vCPU.
This is useful when trying to achieve CPU pinning, as well as making
sure the VM runs on a specific NUMA node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Give the option parser the ability to handle tuples with inner brackets
containing list of integers. The following example can now be handled
correctly "option=[key@[v1-v2,v3,v4]]" which means the option is
assigned a tuple with a key associated with a list of integers between
the range v1 - v2, as well as v3 and v4.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The purpose is to add more info while running
spdk vhost tgt. For example, adding more info
(total_size, block_size) to describe the meaning
on creating malloc bdev inside spdk vhost tgt.
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Because anyhow version 1.0.46 has been yanked, let's move back to the
previous version 1.0.45.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Always properly initialize vectors so that we don't run in undefined
behaviors when the vector gets dropped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Creates a new generic type Tuple so that the same implementation of
FromStr trait can be reused for both parsing a list of two integers and
parsing a list of one integer associated with a list of integers.
This anticipates the need for retrieving sublists, which will be needed
when trying to describe the host CPU affinity for every vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case we want to implement a type that would hold a list of lists, we
need the option parser to be able to ignore the commas for multiple
layers of brackets.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The elements of a list should be using commas as the correct delimiter
now that it is supported. Deprecate use of colons as delimiter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While parsing each parameter for getting the list of option/value, we
check if the value is a list separated between brackets. If that's the
case, we reconstruct the list so that it can be parsed afterwards, even
though it uses commas for separating each value.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While some distros support `musl` libc as a rust target, some others
(such as RHEL) definitely don't (at least not installing distro
packages).
Knowing this, let's have a simple way to not have a hard dependency on
musl on our spec file.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
`using_rustup` is needed as distros (such as Fedora or RHEL) don't
provide the users a way to install`rustup` without conflicting with the
distro provided `rust` package.
In order to minimize the troubles for those who want to build
cloud-hypervisor using the distros packages, let's allow the users to
change that variable and then simply rely on their system packages.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's document the most basic assumptions about the RPM spec file we
maintain for cloud-hypervisor, being those:
* You must have access to the internet during build, otherwise pulling
the vendored code down won't work.
* You must have rustup installed on your system, otherwise basic checks
performed during the build target will fail.
* You must have both gnu and musl libc targets installed, otherwise
either the dynamically or the statically link build will fail.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Avoid removing from the source vector by using Vec::extend_from_slice().
The primitive values (bytes) will be copied from the source in either
case.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the predicates as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the predicates as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As it relies on primitive types that have already been ported
Aml::append_aml_bytes can be used for the child values.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the child as not all
structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now it still relies on Aml::to_aml_bytes() for the children as not
all structures have been ported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since we know all the numerical types now have implementations of
Aml::append_aml_bytes() we can use that directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As this is a DWord and we know that DWord::append_aml_bytes() is
implemented we may call it directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a naive implementation and there is scope to improve this
without extra copies but that requires addressing the users to ensure
there are no lifetime issues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As an optimisation to avoid allocating byte vectors add a trait method
that will append to an existing vector. Further to support the
transition add a default implementation of Aml::to_aml_bytes() that uses
the newly added Aml::append_aml_bytes()
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vfio-ioctls correctly exposes the list of capabilities related
to each region, Cloud Hypervisor can decide to mmap a region based on
the presence or absence of MSIX_MAPPABLE. Instead of blindly mmap'ing
the region, we check if the MSI-X table or PBA is present on the BAR,
and if that's the case, we look for MSIX_MAPPABLE.
If MSIX_MAPPABLE is present, we can go ahead and mmap the entire region.
If MSIX_MAPPABLE is not present, we simply ignore the mmap'ing of this
region as it wouldn't be supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As v19.0 has been release almost a month ago, let's also update the
project's spec file accordingly.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
By merging receive buffers through the VIRTIO_NET_F_MRG_RXBUF feature,
as well as enabling the use of indirect descriptors through
VIRTIO_RING_F_INDIRECT_DESC feature, we achieve better throughput for
the virtio-net device without hurting its latency.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allocator allocates 64-bit MMIO addresses for use with platform
devices e.g. ACPI control devices and ensures there is no overlap with
PCI address space ranges which can cause issues with PCI device
remapping.
Use this allocator the ACPI platform devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than use the system MMIO allocator for RAM use an allocator that
covers the full RAM range.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is because the SGX region will be placed between the end of ram and
the start of the device area.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With the segment id now encoded in the bdf it is not necessary to have
the separate field for it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the existing virtio fs test to support controlling the PCI
segment the device should be added to and use this for a multiple
segment test.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the existing net hotplug test to support controlling the PCI
segment the device should be added to and use this for a multiple
segment test.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the existing pmem hotplug test to support controlling the PCI
segment the device should be added to and use this for a multiple
segment test.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In particular use the accessor for getting the device id from the bdf.
As a side effect the VIOT table is now segment aware.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will allow making the code that handles bdf parsing simpler and
remove the need for manual shifting.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since each segment must have a non-overlapping memory range associated
with it the device memory must be equally divided amongst all segments.
A new allocator is used for each segment to ensure that BARs are
allocated from the correct address ranges. This requires changes to
PciDevice::allocate/free_bars to take that allocator and when
reallocating BARs the correct allocator must be identified from the
ranges.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For all the devices that support being hotplugged (disk, net, pmem, fs
and vsock) add "pci_segment" option and propagate that through to the
addition onto the PCI busses.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the decision on whether to use a 64-bit bar up to the DeviceManager
so that it can use both the device type (e.g. block) and the PCI segment
ID to decide what size bar should be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Generate a set of 8 IRQs and round-robin distribute those over all the
slots for a bus. This same set of IRQs is then used for all PCI
segments.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The platform config may specify a number of PCI segments to use, if this
greater than 1 then we add supplemental PCI segments as well as the
default segment.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This currently contains only the number over PCI segments to create.
This is limited to 16 at the moment which should allow 496 user specified
PCI devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For the bus scanning the GED AML code now calls into a PSCN method that
scans all buses. This approach was chosen since it handles the case
correctly where one GED interrupt is services for two hotplugs on
distinct segments.
The PCIU and PCID field values are now determined by the PSEG field that
is uses to select which segment those values should be used for.
Similarly _EJ0 will notify based on the value of _SEG.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Replace the hardcoded zero PCI segment id when adding devices to the bus
and extend the DeviceTree to hold the PCI segment id.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since each segment must have disjoint address spaces only advertise
address space in the 32-bit range and the PIO address space on the
default (zero) segment.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split PciSegment::new_default_segment() into a separate
PciSegment::new() and those parts required only for the default segment
(PIO PCI config device.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now this still contains just one segment but is expanding in
preparation for more segments.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit moves the code that generates the DSDT data for the PCI bus
into PciSegment making no functional changes to the generated AML.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to get proper performance out of hardware that might be passed
through the VM with VFIO, we decide to try mapping any region marked as
mappable, ignoring the failure and moving on to the next region. When
the mapping succeeds, we establish a list of user memory regions so that
we enable only subparts of the global mapping through the hypervisor.
This allows MSI-X table and PBA to keep trapping accesses from the guest
so that the VMM can properly emulate MSI-X.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Variables that start with underscore are used to silence rustc.
Normally those variables are not used in code.
This patch drops the underscore from variables that are used. This is
less confusing to readers.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This makes it much easier to use since the info!() level produces far
fewer messages and thus has less overhead.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Create a source archive with vendored sources as part of the release
workflow. This is to enable building the release offline for distros.
Note: The use of realpath and CARGO_HOME are to work around a cargo
vendor bug: https://github.com/rust-lang/cargo/issues/8443.
Signed-off-by: William Douglas <william.douglas@intel.com>
Relying on the vm-virtio/virtio-queue crate from rust-vmm which has been
copied inside the Cloud Hypervisor tree, the entire codebase is moved to
the new definition of a Queue and other related structures.
The reason for this move is to follow the upstream until we get some
agreement for the patches that we need on top of that to make it
properly work with Cloud Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This crate contains a new definition of the Queue, AvailIter,
DescriptorChain and Descriptor structures forked from the upstream
crate rust-vmm/vm-virtio d62f2246568d4f544e848b23c025b268effac5ca.
The following patches have been applied on top of this base in order to
make it work correctly with Cloud Hypervisor requirements:
- Add MSI vector field to the Queue
In order to help with MSI/MSI-X support, it is convenient to store the
value of the interrupt vector inside the Queue directly.
- Handle address translations
For devices with access to data in memory being translated, we add to
the Queue the ability to translate the address stored in the
descriptor.
It is very helpful as it performs the translation right after the
untranslated address is read from memory, avoiding any errors from
happening from the consumer's crate perspective. It also allows the
consumer to reduce greatly the amount of duplicated code for applying
the translation in many different places.
- Add helpers for Queue structure
They are meant to help crate's consumers getting/setting information
about the Queue.
These patches can be found on the 'ch' branch from the Cloud Hypervisor
fork: https://github.com/cloud-hypervisor/vm-virtio.git
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This was causing some issues because of the use of 2 different versions
for the vm-memmory crate. We'll wait for all dependencies to be properly
resolved before we move to 0.7.0.
This reverts commit 76b6c62d07.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We now build SPDK-NVMe inside the container only for the x86_64
platform, as the cross-platform build with 'docker buildx' does not work
for SPDK. For aarch64 the platform, we will build it as a part of the CI
workflow (which is running on the bare-metal machine).
Signed-off-by: Bo Chen <chen.bo@intel.com>
PRs from dependabot are failing to meet the check from DCO as the
Signed-Off-By is now a GitHub support email address.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These fields are needed for correctly accessing the datastructures but
are not read by the VHDX implementation itself.
As identified by the new beta clippy.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These structs are not read on the VMM side but are used in communication
with the guest.
As identified by the new beta clippy.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When allocating PCI MMIO BARs they should always be naturally aligned
(i.e. aligned to the size of the BAR itself.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Hypercall register needs to be saved and restored for
TLB flush and IPI synthetic features enablement.
Enabling these two synthetic features improves
guest performance.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The allocate_bars method has a side effect which collates the BARs used
for the device and stores them internally. Ensure that any use of this
internal state is after the state is created otherwise no MMIO regions
will be seen and so none will be mapped.
Fixes: #3237
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When the synthetic interrupt controller is enabled, an extra set of MSRs
must be stored in case of migration. There was one MSR missing in the
list, HV_X64_MSR_SINT14 corresponding to the 15th interrupt source from
the synthetic interrupt controller.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to patch the vhost crate anymore since the fixes we were
looking for have been released as part of 0.2.0 on crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of creating a MemoryManager from scratch, let's reuse the same
code path used by snapshot/restore, so that memory regions are created
identically to what they were on the source VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that all the pieces are in place, we can restore a VM with the new
codepath that restores properly all memory regions, allowing for ACPI
memory hotplug to work properly with snapshot/restore feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extending the MemoryManager::new() function to be able to create a
MemoryManager from data that have been previously stored instead of
always creating everything from scratch.
This change brings real added value as it allows a VM to be restored
respecting the proper memory layout instead of hoping the regions will
be created the way they were before.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Storing multiple data coming from the MemoryManager in order to be able
to restore without creating everything from scratch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new function will be able to restore memory regions and memory
zones based on the GuestMemoryMapping list that will be provided through
snapshot/restore and migration phases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This can help identifying which zone relates to which memory range.
This is going to be useful when recreating GuestMemory regions from
the previous layout instead of having to recreate everything from
scratch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create a dedicated function to factorize the allocation of the memory
ranges, and helping with the simplification of MemoryManager::new()
function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By updating the list of GuestMemory regions with the virtio-mem ones
before the creation of the MemoryManager, we know the GuestMemory is up
to date and the allocation of memory ranges is simplified afterwards.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to simplify MemoryManager::new() function. let's move the
memory configuration validation to its own function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This script checks if an image is compatible with Cloud Hypervisor.
At first, it detects the image type(raw or qcow2),
partition type whether it is DOS or GPT.
Then it mounts the image and checks if VIRTIO Configs
are enabled in the kernel config. In the end, it provides
a message about the compatibility of the image.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Move the PciSegment struct and the associated code to a new file. This
will allow some clearer separation between the core DeviceManager and
PCI handling.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the PCI related state from the DeviceManager struct to a PciSegment
struct inside the DeviceManager. This is in preparation for multiple
segment support. Currently this state is just the bus itself, the MMIO
and PIO config devices and hotplug related state.
The main change that this required is using the Arc<Mutex<PciBus>> in
the device addition logic in order to ensure that
the bus could be created earlier.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When using PVH for booting (which we use for all firmwares and direct
kernel boot) the Linux kernel does not configure LA57 correctly. As such
we need to limit the address space to the maximum 4-level paging address
space.
If the user knows that their guest image can take advantage of the
5-level addressing and they need it for their workload then they can
increase the physical address space appropriately.
This PR removes the TDX specific handling as the new address space limit
is below the one that that code specified.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Whenever running TDX, we must pass the ACPI tables to the TDVF firmware
running in the guest. The proper way to do this is by adding the tables
to the TdHob as a TdVmmData type, so that TDVF will know how to access
these tables and expose them to the guest OS.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of having the ACPI tables being created both in x86_64 and
aarch64 implementations of configure_system(), we can remove the
duplicated code by moving the ACPI tables creation in vm.rs inside the
boot() function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On AArch64, device hotplug can be enabled with ACPI. Therefore,
this commit enables the hotplug test case for following devices:
- PCI bar reprogramming
- virtio-disk
- virtio-net
- macvtap
- virtio-vsock
- virtio-pmem: Works with the latest reference kernel
- virtio-fs: Works with the latest reference kernel
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently vfio and nested virtualization is not used on AArch64,
and SGX is a x86_64 only feature. Therefore this commit adds the
architecture gates for helper functions related to vfio, SGX, and
nested virtualization to mute warnings when building tests on the
AArch64 platform.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Update introduction of option `prefault`.
In addition, this commit also did the following:
- Rearrange options, synchronize order with `config.rs`.
- Break long lines in `hugepages`.
- Update old example of `hugepages` in memory zone.
Signed-off-by: Li Yu <liyu.yukiteru@bytedance.com>
Memory hotplug and virtio_balloon works on arm64 with:
- memory hotplug: An updated kernel using ACPI
- virtio balloon: `stress` installed in the cloud image
Therefore, we can enable test cases for them in integration test.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
On MSHV some of the integration test cases are not supported yet
or still in progress. This patch disables all those test cases.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The argument `prefault` is provided in MemoryManager, but it can
only be used by SGX and restore.
With prefault (MAP_POPULATE) been set, subsequent page faults will
decrease during running, although it will make boot slower.
This commit adds `prefault` in MemoryConfig and MemoryZoneConfig.
To resolve conflict between memory and restore, argument
`prefault` has been changed from `bool` to `Option<bool>`, when
its value is None, config from memory will be used, otherwise
argument in Option will be used.
Signed-off-by: Yu Li <liyu.yukiteru@bytedance.com>
Setting the reply_ack should depend on the set of acknowledged features
containing the REPLY_ACK flag.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding some bits to the existing live migration test with NUMA in order
to properly validate virtio-mem works with live migration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By using a single file for storing the memory ranges, we simplify the
way snapshot/restore works by avoiding multiples files, but the main and
more important point is that we have now a way to save only the ranges
that matter. In particular, the ranges related to virtio-mem regions are
not always fully hotplugged, meaning we don't want to save the entire
region. That's where the usage of memory ranges is interesting as it
lets us optimize the snapshot/restore process when one or multiple
virtio-mem regions are involved.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
During snapshot/restore we will need to store this structure, which is
why it must derive the Versionize trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The function memory_range_table() will be reused by the MemoryManager in
a following patch to describe all the ranges that we should snapshot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Copy only the memory ranges that have been plugged through virtio-mem,
allowing for an interesting optimization regarding the time it takes to
migrate a large virtio-mem device. Even if the hotpluggable space is
very large (say 64GiB), if only 1GiB has been previously added to the
VM, only 1GiB will be sent to the destination VM, avoiding the transfer
of the remaining 63GiB which are unused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support correctly the snapshot/restore and migration use
cases, we must be careful with the ranges that we discard by punching
holes. On restore, there might be some ranges already plugged in,
meaning they should not be discarded. That's why we loop over the list
of blocks to discard only the ranges that are marked as unplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By creating the BlocksState object in the MemoryManager, we can directly
provide it to the virtio-mem device when being created. This will allow
the MemoryManager through each VirtioMemZone to have a handle onto the
blocks that are plugged at any point in time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This is going to be useful to let virtio-mem report the list of ranges
that are currently plugged, so that both snapshot/restore and migration
will copy only what is needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be helpful to support the creation of a MemoryRangeTable from
virtio-mem, as it uses 2M pages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding the snapshot/restore support along with migration as well,
allowing a VM with virtio-mem devices attached to be properly
migrated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The amount of memory plugged in the virtio-mem region should always be
kept up to date in the hotplugged_size field from VirtioMemZone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to duplicate the GuestMemory for snapshot purpose, as we
always have a handle onto the GuestMemory through the guest_memory
field.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we only support a single PCI bus right now advertise only a single
bus in the ACPI tables. This reduces the number of VM exits from probing
substantially.
Number of PCI config I/O port exits: 17871 -> 1551 (91% reduction) with
direct kernel boot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use a simpler method for extracting the affected slot on the eject
command. Also update the terminology to reflect that this a slot rather
than a bdf (which is what device id refers to elsewhere.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the serial buffer handling in order to write the serial
buffer's output to a PTY connected after the serial device stops being
written to by the guest.
This change moves the serial buffer initialization inside the serial
manager. That is done to allow the serial buffer to be made aware of
the PTY and epoll fds needed in order to modify the
EpollDispatch::File trigger. These are then used by the serial buffer
to trigger an epoll event when the PTY fd is writable and the buffer
has content in it. They are also used to remove the trigger when the
buffer is emptied in order to avoid unnecessary wake-ups.
Signed-off-by: William Douglas <william.douglas@intel.com>
In preparation for reorganizing how the serial output is constructed
add methods to the serial devices for setting the out buffer after the
device is created.
Also add a method to enable flushing the output buffer to be used to
write the buffer to the PTY fd once the PTY is writable.
Signed-off-by: William Douglas <william.douglas@intel.com>
In integration test, we fetch latest EDK2 code on its master branch and
build. While the update on EDK2 master is frequent. And the building is
time consuming. It takes a lot of time in CI and local test. Floating on
top of a busy master branch also bring potential risk in tracking and
debugging.
Now that Cloud Hypervisor support in EDK2 has been steady, we can pin
the EDK2 software versions to avoid unnecessary updating and building.
We can update the versions manually every after several months.
The commit also optimizes the build process by applying multi-threaded
compiling.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Added a bash function in integration test script to checkout source code
of a GIT repo with specified branch and commit.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
These packages will be used to compile `stress` from source, and
the `stress` will be used by the virtio-balloon integration test.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Both read_exact_from() and write_all_to() functions from the GuestMemory
trait implementation in vm-memory are buggy. They should retry until
they wrote or read the amount of data that was expected, but instead
they simply return an error when this happens. This causes the migration
to fail when trying to send important amount of data through the
migration socket, due to large memory regions.
This should be eventually fixed in vm-memory, and here is the link to
follow up on the issue: https://github.com/rust-vmm/vm-memory/issues/174
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Refactored the test case `test_virtio_iommu` to adapt architectures and
different choices among ACPI and FDT. In the case of ACPI, a Focal image
with modified kernel is tested.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, ACPI must work with UEFI (EDK2). This way, the kernel is
always loaded from the disk image. We can not specify a direct custom
kernel while using ACPI.
To use a custom kernel, we have to replace the kernel file in the disk
image by:
- Making a copy of the Focal `raw` image
- Mounting the rootfs with `libguestfs-tools`
- Replacing the compressed kernel file
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Installed `libguestfs-tools` to replace kernel file in cloud image.
Installed a kernel as `libguestfs-tools` requires.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Implement the infrastructure that lets a virtio-mem device map the guest
memory into the device. This is necessary since with virtio-mem zones
memory can be added or removed and the vfio-user device must be
informed.
Fixes: #3025
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving this from the VfioUserPciDevice to DeviceManager the client
can be reused for handling DMA mapping behind an IOMMU.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For vfio-user the mapping handler is per device and needs to be removed
when the device in unplugged.
For VFIO the mapping handler is for the default VFIO container (used
when no vIOMMU is used - using a vIOMMU does not require mappings with
virtio-mem)
To represent these two use cases use an enum for the handlers that are
stored.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Adding the snapshot/restore support along with migration as well,
allowing a VM with a virtio-balloon device attached to be properly
migrated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given the 'virtiofsd' executable is used in multiple CI workers,
installing them directly to the docker image is more efficient and can
save CI time.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Looking up devices on the port I/O bus is time consuming during the
boot at there is an O(lg n) tree lookup and the overhead from taking a
lock on the bus contents.
Avoid this by adding a fast path uses the hardcoded port address and
size and directs PCI config requests directly to the device.
Command line:
target/release/cloud-hypervisor --kernel ~/src/linux/vmlinux --cmdline "root=/dev/vda1 console=ttyS0" --serial tty --console off --disk path=~/workloads/focal-server-cloudimg-amd64-custom-20210609-0.raw --api-socket /tmp/api
PIO exit: 17913
PCI fast path: 17871
Percentage on fast path: 99.8%
perf before:
marvin:~/src/cloud-hypervisor (main *)$ perf report -g | grep resolve
6.20% 6.20% vcpu0 cloud-hypervisor [.] vm_device::bus::Bus::resolve
perf after:
marvin:~/src/cloud-hypervisor (2021-09-17-ioapic-fast-path *)$ perf report -g | grep resolve
0.08% 0.08% vcpu0 cloud-hypervisor [.] vm_device::bus::Bus::resolve
The compromise required to implement this fast path is bringing the
creation of the PciConfigIo device into the DeviceManager::new() so that
it can be used in the VmmOps struct which is created before
DeviceManager::create_devices() is called.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Added a section in "Usage" chapter of "iommu.md" to introduce the
special behavior when virtio-iommu is working with FDT on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
For AArch64, now virtual IOMMU is only tested on FDT, not ACPI.
In the case of FDT, the behavior of IOMMU is a bit different with ACPI.
All the devices on the PCI bus will be attached to the virtual IOMMU,
except the virtio-iommu device itself. So these devices will all be
added to IOMMU groups, and appear in folder '/sys/kernel/iommu_groups/'.
The result is, on AArch64 IOMMU group '0' contains "0000:00:01.0" which
is the console device. But on X86, console device is not attached to
IOMMU. So the IOMMU group '0' contains "0000:00:02.0" which is the first
disk.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The MSI IOVA address on X86 and AArch64 is different.
This commit refactored the code to receive the MSI IOVA address and size
from device_manager, which provides the actual IOVA space data for both
architectures.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Move the definition of MSI space to layout.rs, so other crates can
reference it. Now it is needed by virtio-iommu.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Add a virtio-iommu node into FDT if iommu option is turned on. Now we
support only one virtio-iommu device.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This change switches from handling serial input in the VMM thread to
its own thread controlled by the SerialManager.
The motivation for this change is to avoid the VMM thread being unable
to process events while serial input is happening and vice versa.
The change also makes future work flushing the serial buffer on PTY
connections easier.
Signed-off-by: William Douglas <william.douglas@intel.com>
This change adds a SerialManager with its own epoll handling that
should be created and run by the DeviceManager when creating an
appropriately configured console (serial tty or pty).
Both stdin and pty input are handled by the SerialManager. The stdin
and pty specific methods used by the VMM should be removed in a future
commit.
Signed-off-by: William Douglas <william.douglas@intel.com>
The clone method for PtyPair should have been an impl of the Clone
trait but the method ended up not being used. Future work will make
use of the trait however so correct the missing trait implementation.
Signed-off-by: William Douglas <william.douglas@intel.com>
libc::getrandom need to be called inside unsafe and it is not
cross-platform friendly.
Change it to getrandom::getrandom that is safe and cross-platform
friendly.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Updating kvm-ioctls from 0.9.0 to 0.10.0 now that Cloud Hypervisor
relies on kvm-bindings 0.5.0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For most use cases, there is no need to create multiple VFIO containers
as it causes unwanted behaviors. Especially when passing multiple
devices from the same IOMMU group, we need to use the same container so
that it can properly list the groups that have been already opened. The
correct logic was already there in vfio-ioctls, but it was incorrectly
used from our VMM implementation.
For the special case where we put a VFIO device behind a vIOMMU, we must
create one container per device, as we need to control the DMA mappings
per device, which is performed at the container level. Because we must
keep one container per device, the vIOMMU use case prevents multiple
devices attached to the same IOMMU group to be passed through the VM.
But this is a limitation that we are fine with, especially since the
vIOMMU doesn't let us group multiple devices in the same group from a
guest perspective.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allows Cloud Hypervisor to be run under `perf` as some of the
signals will already be blocked in the child process.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the kvm-bindings dependency so that Cloud Hypervisor now depends
on the version 0.5.0, which is based on Linux kernel v5.13.0. We still
have to rely on a forked version to be able to serialize all the KVM
structures we need.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Validate the size of I/O reads and check that no request is made to an
out of bounds index (which would cause a panic.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check the size of data buffer for reading on the ApciPmTimer device to
avoid a potential panic if the guest uses non-DWORD access.
Simplify the zeroring of the buffer for AcpiShutdownDevice.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a pty is resized (using the TIOCSWINSZ ioctl -- see ioctl_tty(2)),
the kernel will send a SIGWINCH signal to the pty's foreground process
group to notify it of the resize. This is the only way to be notified
by the kernel of a pty resize.
We can't just make the cloud-hypervisor process's process group the
foreground process group though, because a process can only set the
foreground process group of its controlling terminal, and
cloud-hypervisor's controlling terminal will often be the terminal the
user is running it in. To work around this, we fork a subprocess in a
new process group, and set its process group to be the foreground
process group of the pty. The subprocess additionally must be running
in a new session so that it can have a different controlling
terminal. This subprocess writes a byte to a pipe every time the pty
is resized, and the virtio-console device can listen for this in its
epoll loop.
Alternatives I considered were to have the subprocess just send
SIGWINCH to its parent, and to use an eventfd instead of a pipe.
I decided against the signal approach because re-purposing a signal
that has a very specific meaning (even if this use was only slightly
different to its normal meaning) felt unclean, and because it would
have required using pidfds to avoid race conditions if
cloud-hypervisor had terminated, which added complexity. I decided
against using an eventfd because using a pipe instead allows the child
to be notified (via poll(2)) when nothing is reading from the pipe any
more, meaning it can be reliably notified of parent death and
terminate itself immediately.
I used clone3(2) instead of fork(2) because without
CLONE_CLEAR_SIGHAND the subprocess would inherit signal-hook's signal
handlers, and there's no other straightforward way to restore all signal
handlers to their defaults in the child process. The only way to do
it would be to iterate through all possible signals, or maintain a
global list of monitored signals ourselves (vmm:vm::HANDLED_SIGNALS is
insufficient because it doesn't take into account e.g. the SIGSYS
signal handler that catches seccomp violations).
Signed-off-by: Alyssa Ross <hi@alyssa.is>
This prepares us to be able to handle console resizes in the console
device's epoll loop, which we'll have to do if the output is a pty,
since we won't get SIGWINCH from it.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Musl often uses mmap to allocate memory where Glibc would use brk.
This has caused seccomp violations for me on the API and signal
handling threads.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
As well as reducing the amount of code this also improves the binary
size slightly:
cargo bloat --release -n 2000 --bin cloud-hypervisor | grep virtio_devices::seccomp_filters::get_seccomp_rules
Before:
0.1% 0.2% 7.8KiB virtio_devices virtio_devices::seccomp_filters::get_seccomp_rules
After:
0.0% 0.1% 3.0KiB virtio_devices virtio_devices::seccomp_filters::get_seccomp_rules
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch adds a separate function to launch two guest VMs and ensure
they are connected through ovs-dpdk, so that we can reuse this function
in other tests, e.g. the test for live-migration with ovs-dpdk.
Signed-off-by: Bo Chen <chen.bo@intel.com>
error: all if blocks contain the same code at the end
--> vmm/src/memory_manager.rs:884:9
|
884 | / Ok(mm)
885 | | }
| |_________^
Signed-off-by: Bo Chen <chen.bo@intel.com>
This concept ends up being broken with multiple types on input connected
e.g. console on TTY and serial on PTY. Already the code for checking for
injecting into the serial device checks that the serial is configured.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce a common solution for spawning the virtio threads which will
make it easier to add the panic handling.
During this effort I discovered that there were no seccomp filters
registered for the vhost-user-net thread nor the vhost-user-block
thread. This change also incorporates basic seccomp filters for those as
part of the refactoring.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently we need to test both device tree and ACPI on AArch64. As
the number of ACPI test cases is gradually increasing and expected
to increase in the future, it is better to extract all ACPI test
cases on AArch64 to a single module.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Current AArch64 power button is only for device tree using a PL061
GPIO controller device. Since AArch64 now supports ACPI, this
commit extend the power button on AArch64 to:
- Using GED for ACPI+UEFI boot.
- Using PL061 for device tree boot.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This message only occurs sporadically and so it should be included at
info!() level. Enhance the output to also include the BAR number.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These statements are useful for understanding the cause of reset or
shutdown of the VM and are not spammy so should be included at info!()
level.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch adds a separate function to perform common numa checks, so
that we can reuse this function in other tests, e.g. the test for
live-migration with numa.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Despite setting up a dedicated thread for signal handling, we weren't
making sure that the signals we were listening for there were actually
dispatched to the right thread. While the signal-hook provides an
iterator API, so we can know that we're only processing the signals
coming out of the iterator on our signal handling thread, the actual
signal handling code from signal-hook, which pushes the signals onto
the iterator, can run on any thread. This can lead to seccomp
violations when the signal-hook signal handler does something that
isn't allowed on that thread by our seccomp policy.
To reproduce, resize a terminal running cloud-hypervisor continuously
for a few minutes. Eventually, the kernel will deliver a SIGWINCH to
a thread with a restrictive seccomp policy, and a seccomp violation
will trigger.
As part of this change, it's also necessary to allow rt_sigreturn(2)
on the signal handling thread, so signal handlers are actually allowed
to run on it. The fact that this didn't seem to be needed before
makes me think that signal handlers were almost _never_ actually
running on the signal handling thread.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Move the processing of the input from stdin, PTY or file from the VMM
thread to the existing virtio-console thread. The handling of the resize
of a virtio-console has not changed but the name of the struct used to
support that has been renamed to reflect its usage.
Fixes: #3060
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Downcasting of GicDevice trait might fail. Therefore we try to
downcast the trait first and only if the downcasting succeeded we
can then use the object to call methods. Otherwise, do nothing and
log the failure.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit implements the GIC (including both GICv3 and GICv3ITS)
Pausable trait. The pause of device manager will trigger a "pause"
of GIC, where we flush GIC pending tables and ITS tables to the
guest RAM.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This test exercises the local live-migration between two Cloud
Hypervisor VMs on the same host. It ensures the following behaviors:
1. The source VM is up and functional (including various virtio-devices
are working properly);
2. The 'send-migration' and 'receive-migration' command finished
successfully;
3. The source VM terminated gracefully after live migration;
4. The destination VM is functional (including various virtio-devices
are working properly) after live migration.
Note: This test does not use vsock as we can't create two identical
vsock on the same host.
Fixes: #2965
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch adds a dedicate function to include the common checks on the
virtio-devices from the 'test_snapshot_restore' test, which will also be
reused for the upcoming 'test_live_migration' test.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This prevents the boot of the guest kernel from being blocked by
blocking I/O on the serial output since the data will be buffered into
the SerialBuffer.
Fixes: #3004
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce a dynamic buffer for storing output from the serial port. The
SerialBuffer implements std::io::Write and can be used in place of the
direct output for the serial device.
The internals of the buffer is a vector that grows dynamically based on
demand up to a fixed size at which point old data will be overwritten.
Currently the buffer is only flushed upon writes.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The rust-vmm crates we're pulling from git have renamed their main
branches. We need to update the branch names we're giving to Cargo,
or people who don't have these dependencies cached will get errors
like this when trying to build:
error: failed to get `vm-fdt` as a dependency of package `arch v0.1.0 (/home/src/cloud-hypervisor/arch)`
Caused by:
failed to load source for dependency `vm-fdt`
Caused by:
Unable to update https://github.com/rust-vmm/vm-fdt?branch=master#031572a6
Caused by:
object not found - no match for id (031572a6edc2f566a7278f1e17088fc5308d27ab); class=Odb (9); code=NotFound (-3)
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Remove the indirection of a dispatch table and simply use the enum as
the event data for the events.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Definition of kvm_tdx_init_vm used by INIT_VM has been updated in latest
kernel, needing an update on the Cloud Hypervisor side as well.
Update structure TdxInitVm to fit this change and avoid -EINVAL to be
returned by the kernel.
Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Moving to the latest kernel 5.14 simplifies greatly the list of patches
that we need to carry. Since virtio-iommu has been merged as part of the
5.14 release, the only remaining patches are virtio-watchdog and the EFI
reboot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently the latest cloudhypervisor/dev docker container is the
multi-arch image. We can pull the arm image directly.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
To enable live-migration for vhost-user devices, we now always shutdown
the source VM when the migration is completed. This patch reflects this
change in the documentation.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This resolves an issue with hotplug -> removal -> hotplug of a vfio-user
device as the allocator was not updated with the now unused entries.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use two separate events for the console and serial PTY and then drive
the handling of the inputs on the PTY separately. This results in the
correct behaviour when both console and serial are attached to the PTY
as they are triggered separately on the epoll so events are not lost.
Fixes: #3012
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check the config to find out which device is attached to the tty and
then send the input from the user into that device (serial or
virtio-console.)
Fixes: #3005
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The type is to change from 32-bit to 64-bit. See
https://github.com/rust-lang/libc/issues/1848.
The change is announced via a deprecation warning. Cloud Hypervisor's
code does not need changing. Simply suppress these warnings.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The Jenkins master is now known as the controller and the agent it provides is
called "built-in".
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The fuzzer needs to take a larger input for the whole disk image to
be most useful. Since the file is small we can test by reading and
writing over the whole file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Fazla Mehrab <akm.fazla.mehrab@intel.com>
Two tests for booting Linux cloud image from the different VHDx files:
fixed and dynamic. Another test for testing the dynamic expansion of a
generated VHDx file.
Signed-off-by: Fazla Mehrab <akm.fazla.mehrab@intel.com>
vhdx_sync.rs in block_util implements traits to represent the vhdx
crate as a supported block device in the cloud hypervisor. The vhdx
is added to the block device list in device_manager.rs at the vmm
crate so that it can automatically detect a vhdx disk and invoke the
corresponding crate.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Fazla Mehrab <akm.fazla.mehrab@intel.com>
Microsoft’s VHDx block device format specification is implemented
here as a crate. This commit includes the implementation for the
fixed and dynamic formats, where the other format is known as
differencing. The vhdx_header.rs, vhdx_bat.rs, vhdx_metadata.rs
implements parser and manipulators for the VHDx header, Block
Allocation Table, and metadata, respectively, for the VHDx file.
The vhdx_io.rs implements read and write routines for the VHDx file.
The vhdx.rs implements the Vhdx structure, which provides the wrapper
functions for standard I/O operations like read, write, and seek.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Fazla Mehrab <akm.fazla.mehrab@intel.com>
This commit adds a github action to release statically-linked arm64
binaries - ch-remote and cloud-hypervisor.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
We are relying on applying empty 'seccomp' filters to support the
'--seccomp false' option, which will be treated as an error with the
updated 'seccompiler' crate. This patch fixes this issue by explicitly
checking whether the 'seccomp' filter is empty before applying the
filter.
Signed-off-by: Bo Chen <chen.bo@intel.com>
The current code was expecting a reply with only the header, which
wasn't reading the rest of payload that was provided. This was causing
the following replies to be completely wrong as they were shifted by the
previous payload that wasn't read.
The simple way to fix this issue is by reading the header and the
expected payload when getting a reply from a DmaUnmap request.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Change the block size to the standard 512 byte sector size to that
disk images can be used (since their partition tables will be specified
in terms of 512 byte sectors.)
Also remove the hugepages=on option from the command line as it is not
necessary.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When mapping the region into the guest ensure that all the fields are
updated correctly as the unmap code path checks that they are set.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds an AArch64-only integration test case called
`test_guest_numa_nodes_dt` so that it is possible to test the
NUMA for the FDT on AArch64 platform.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Based on `--memory-zone` and `--numa` param in the Cloud Hypervisor
cmdline, the NUMA memory configuration is described. This commit
adds such NUMA memory configuration to the FDT memory node.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
It is forbidden that the same memory zone belongs to more than one
NUMA node. This commit adds related validation to the `--numa`
parameter to prevent the user from specifying such configuration.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
For the purpose of identification, each NUMA node is associated
with a unique token known as a `numa-node-id`. For the purpose of
device tree binding, a `numa-node-id` is a 32-bit integer.
The CPU node is associated with a NUMA node by the presence of a
`numa-node-id` property which contains the node id of the device.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The optional device tree node distance-map describes the relative
distance (memory latency) between all NUMA nodes.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This is to make sure the NUMA node data structures can be accessed
both from the `vmm` crate and `arch` crate.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The AArch64 platform provides a NUMA binding for the device tree,
which means on AArch64 platform, the NUMA setup can be extended to
more than the ACPI feature.
Based on above, this commit extends the NUMA setup and data
structures to following scenarios:
- All AArch64 platform
- x86_64 platform with ACPI feature enabled
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Signed-off-by: Michael Zhao <Michael.Zhao@arm.com>
Introducing a new structure VhostUserCommon allowing to factorize a lot
of the code shared between the vhost-user devices (block, fs and net).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of panicking with an expect() function, the QcowDiskSync::new
function now propagates the error properly. This ensures the VMM will
not panic, which might be the source of weird errors if only one thread
exits while the VMM continues to run.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of running the generic block fuzzer with QCOW, it's better to
use a RAW file since it's less complex and it will focus on virtqueues.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We cannot let vhost-user devices connect to the backend when the Block,
Fs or Net object is being created during a restore/migration. The reason
is we can't have two VMs (source and destination) connected to the same
backend at the same time. That's why we must delay the connection with
the vhost-user backend until the restoration is performed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing a new function to factorize a small part of the
initialization that is shared between a full reinitialization and a
restoration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The code wasn't doing what it was expected to. The '?' was simply
returning the error to the top level function, meaning the Err() case in
the match was never hit. Moving the whole logic to a dedicated function
allows to identify when something got wrong without propagating to the
calling function, so that we can still stop the dirty logging and
unpause the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the migration succeeds, the destination VM will be correctly
running, with potential vhost-user backends attached to it. We can't let
the source VM trying to reconnect to the same backends, which is why
it's safer to shutdown the source VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to prevent the vhost-user devices from reconnecting to the
backend after the migration has been successfully performed, we make
sure to kill the thread in charge of handling the reconnection
mechanism.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a way to let every Migratable object know when the migration is
complete, so they can take appropriate actions.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
During a migration, the vhost-user device talks to the backend to
retrieve the dirty pages. Once done with this, a snapshot will be taken,
meaning there's no need to communicate with the backend anymore. Closing
the communication is needed to let the destination VM being able to
connect to the same backend.
That's why we shutdown the communication with the backend in case a
migration has been started and we're asked for a snapshot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This anticipates the need for creating a new Blk, Fs or Net object
without having performed the connection with the vhost-user backend yet.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for creating vhost-user devices in a different way when
being restored compared to a fresh start, this commit introduces a new
boolean created by the Vm depending on the use case, and passed down to
the DeviceManager. In the future, the DeviceManager will use this flag
to assess how vhost-user devices should be created.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It was incorrect to call Vec::from_raw_parts() on the address pointing
to the shared memory log region since Vec is a Rust specific structure
that doesn't directly translate into bytes. That's why we use the same
function from std::slice in order to create a proper slice out of the
memory region, which is then copied into a Vec.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the common vhost-user code can handle logging dirty pages
through shared memory, we need to advertise it to the vhost-user
backends with the protocol feature VHOST_USER_PROTOCOL_F_LOG_SHMFD.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Correct operation of user devices (vfio-user) requires shared memory so
flag this to prevent it from failing in strange ways.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Create the vfio-user / user devices from the config. Currently hotplug
of the devices is not supported nor can they be placed behind the
(virt-)iommu.
Removal of the coldplugged device is however supported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Taking advantage of the refactored VFIO code implement a new
VfioUserPciDevice that wraps the client for vfio-user and exposes the
BusDevice and PciDevice into the VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to specify devices that are running in a different
userspace process and communicated with vfio-user.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement (most) of the client side (i.e. VMM side) of the vfio-user
protocol:
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst
Items that are not implemented (because they are optimisations or unused
due to alternative solutions:
* VFIO_USER_DMA_READ/WRITE - this is a way for the server to read guest
memory if the guest memory is not shared by fd where the client
doesn't support it. However since we do support sharing the memory by
fd this is not required.
* VFIO_USER_GET_REGION_IO_FDS - an optimisation to bypass the VMM by
having KVM talk directly to the backend using ioregionfd
* VFIO_USER_DIRTY_PAGES - for the implementation of live migration
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This doesn't really affect the build as we ship a Cargo.lock with fixed
versions in. However for clarity it makes sense to use fixed versions
throughout and let dependabot update them.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
AArch64 CPU topology can be described using either device tree or
ACPI. Therefore, the integration test should also cover the AArch64
ACPI CPU topology tests.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The optional Processor Properties Topology Table (PPTT) table is
used to describe the topological structure of processors controlled
by the OSPM, and their shared resources, such as caches. The table
can also describe additional information such as which nodes in the
processor topology constitute a physical package.
The ACPI PPTT table supports topology descriptions for ACPI guests.
Therefore, this commit adds the PPTT table for AArch64 to enable
CPU topology feature for ACPI.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The Arm CPU topology is defined within the `cpu-map` node, which is
a direct child of the cpus node and provides a container where the
actual topology nodes are listed.
This commit adds an optional cpu-map node in device tree, based on
the Cloud Hypervisor command line vCPU topology information.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
In an Arm system, the hierarchy of CPUs is defined through three
entities that are used to describe the layout of physical CPUs in
the system:
- cluster
- core
- thread
All these three entities have their own FDT node field. Therefore,
This commit adds an AArch64-specific helper to pass the config from
the Cloud Hypervisor command line to the `configure_system`, where
eventually the `create_fdt` is called.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Make sure the DeviceManager is triggered for all migration operations.
The dirty pages are merged from MemoryManager and DeviceManager before
to be sent up to the Vmm in lib.rs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that Migratable provides the methods for starting, stopping and
retrieving the dirty pages, we move the existing code to these new
functions.
No functional change intended.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch connects the dots between the vm.rs code and each Migratable
device, in order to make sure Migratable methods are correctly invoked
when migration happens.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding the common vhost-user code for starting logging dirty pages when
the migration is started, and its counterpart for stopping, as well as
the code in charge of retrieving the bitmap of the dirty pages that have
been logged.
All these functions are meant to be leveraged from vhost-user devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a simple field `migration_support` to VhostUserHandle in order to
store the information about the device supporting migration or not. The
value of this flag depends on the feature set negotiated with the
backend. It's considered as supporting migration if VHOST_F_LOG_ALL is
present in the virtio features and if VHOST_USER_PROTOCOL_F_LOG_SHMFD is
present in the vhost-user protocol features.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add new methods to the Migratable trait so that each device implementing
this trait can be notified when the migration starts/stops, as well as
when a dirty bitmap must be returned.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting the merge of multiple dirty pages coming
from multiple devices, this patch factorizes the creation of a
MemoryRangeTable from a bitmap, as well as providing a simple method for
merging the dirty pages regions under a single MemoryRangeTable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Replacing QCOW mentions with other viable options. There is still
docs/custom-image.md which talks about creating QCOW, however it
might be more convenient to be touched when the actual changes
are indeed there.
Related to #1985.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
These crates are written to produce log messages using the error!
macro, but their logs didn't actually go anywhere, which made it very
difficult to debug when they're not working.
I've used env_logger here because it's the same log implementation
that the hypervisor crate uses.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
I don't think the min_values(1) was doing anything, since
takes_value(true) was already set, and after this change I still get
an error if I try to supply the argument with no value.
But the unwrap below would fail if the argument wasn't supplied at
all. By setting required(true) we get a nice message from clap
instead.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
After the refactoring to split the common VFIO code out for vfio-user
there were some inconsistencies in the error handling. Correct this so
that the error is independent of the transport (hardware vs user) VFIO
and migrate to anyhow/thiserror in the process. Some unused errors from
earlier refactoring have also been removed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Returning a reference is not possible for the vfio-user code as it is
constructed for the function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rename the wrapper trait and structs since this will be used for more
than reading the PCI config.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the code to be used from a different module in the same
crate for vfio-user support.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split data that will need to be common between VfioPciDevice and
VfioUserPciDevice into a common struct. Currently this has no methods
but they will be added soon.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By splitting this into a trait with common code extracted then this
will allow extensive reuse of logic in the vfio-user version.
This commit also changed the order of parameters on
::write_config_dword() to place offset first to match the other
functions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Moving the MSHV crate form Cloud Hypervisor to rust-vmm
is done. This patch update the MSHV referent to rust-vmm
Signed-off-by: Muminul Islam <muislam@microsoft.com>
backend like SPDK required to know how many virt queues to be handled
before gets VHOST_USER_SET_INFLIGHT_FD message.
fix dpdk core dump while processing vhost_user_set_inflight_fd:
#0 0x00007fffef47c347 in vhost_user_set_inflight_fd (pdev=0x7fffe2895998, msg=0x7fffe28956f0, main_fd=545) at ../lib/librte_vhost/vhost_user.c:1570
#1 0x00007fffef47e7b9 in vhost_user_msg_handler (vid=0, fd=545) at ../lib/librte_vhost/vhost_user.c:2735
#2 0x00007fffef46bac0 in vhost_user_read_cb (connfd=545, dat=0x7fffdc0008c0, remove=0x7fffe2895a64) at ../lib/librte_vhost/socket.c:309
#3 0x00007fffef45b3f6 in fdset_event_dispatch (arg=0x7fffef6dc2e0 <vhost_user+8192>) at ../lib/librte_vhost/fd_man.c:286
#4 0x00007ffff09926f3 in rte_thread_init (arg=0x15ee180) at ../lib/librte_eal/common/eal_common_thread.c:175
Signed-off-by: Arafatms <arafatms@outlook.com>
This patch adds all the seccomp rules missing for MSHV.
With this patch MSFT internal CI runs with seccomp enabled.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This patch adds a fallback path for sending live migration, where it
ensures the following behavior of source VM post live-migration:
1. The source VM will be paused only when the migration is completed
successfully, or otherwise it will keep running;
2. The source VM will always stop dirty pages logging.
Fixes: #2895
Signed-off-by: Bo Chen <chen.bo@intel.com>
The vhost-user-net backend needs to prepare all queues before enabling vring.
For example, DPVGW will report 'the RX queue can't find' error, if we enable
vring immediately after kicking it out.
Signed-off-by: Arafatms <arafatms@outlook.com>
This rule is needed to boot windows guest.
This bug was introduced while we tried to boot
windows guest on MSHV.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This patch modify the existing live migration code
to support MSHV. Adds couple of new functions to enable
and disable dirty page tracking. Add missing IOCTL
to the seccomp rules for live migration.
Adds necessary flags for MSHV.
This changes don't affect KVM functionality at all.
In order to get better performance it is good to
enable dirty page tracking when we start live migration
and disable it when the migration is done.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Right now, get_dirty_log API has two parameters,
slot and memory_size.
MSHV needs gpa to retrieve the page states. GPA is
needed as MSHV returns the state base on PFN.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Now that vhost-user supports being snapshot and restored, we extend the
existing test_ovs_dpdk to validate snapshot/restore feature works as
expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding the support for snapshot/restore feature for all supported
vhost-user devices.
The complexity of vhost-user-fs device makes it only partially
compatible with the feature. When using the DAX feature, there's no way
to store and remap what was previously mapped in the DAX region. And
when not using the cache region, if the filesystem is mounted, it fails
to be properly restored as this would require a special command to let
the backend know that it must remount what was already mounted before.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch moves all vhost-user common functions behind a new structure
VhostUserHandle. There is no functional changes intended, the only goal
being to prepare for storing information through this new structure,
limiting the amount of parameters that are needed for each function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We didn't implement the vhost-user-net backend to support EVENT_IDX
feature, hence it is incorrect to advertise VIRTIO_RING_F_EVENT_IDX to
the VMM and the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's totally acceptable to snapshot and restore a virtio-fs device that
has no cache region, since this is a valid mode of functioning for
virtio-fs itself.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the new beta version, clippy complains about redundant allocation
when using Arc<Box<dyn T>>, and suggests replacing it simply with
Arc<dyn T>.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As we are now using an global control to start/stop dirty pages log from
the `hypervisor` crate, we need to explicitly tell the hypervisor (KVM)
whether a region needs dirty page tracking when it is created.
This reverts commit f063346de3.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Following KVM interfaces, the `hypervisor` crate now provides interfaces
to start/stop the dirty pages logging on a per region basis, and asks
its users (e.g. the `vmm` crate) to iterate over the regions that needs
dirty pages log. MSHV only has a global control to start/stop dirty
pages log on all regions at once.
This patch refactors related APIs from the `hypervisor` crate to provide
a global control to start/stop dirty pages log (following MSHV's
behaviors), and keeps tracking the regions need dirty pages log for
KVM. It avoids leaking hypervisor-specific behaviors out of the
`hypervisor` crate.
Signed-off-by: Bo Chen <chen.bo@intel.com>
To support different CPUID entry semantics, we now allow to
specify the compatible condition for each feature entry. Most entries
are considered compatible when they are "bitwise subset", with few
exceptions: 1. "equal", e.g. EBX/ECX/EDX of leaf `0x4000_0000` (KVM
CPUID SIGNATURE); 2. "smaller or equal as a number", e.g. EAX of leaf
`0x7` and leaf `0x4000_0000`;
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch adds a common function "Vmm::vm_check_cpuid_compatibility()"
to be shared by both live-migration and snapshot/restore.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We now send not only the 'VmConfig' at the 'Command::Config' step of
live migration, but also send the 'common CPUID'. In this way, we can
check the compatibility of CPUID features between the source and
destination VMs, and abort live migration early if needed.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Add missing option of hugepage_size in argument of memory,
and add some details about lacking huge pages.
Signed-off-by: Yu Li <liyu.yukiteru@bytedance.com>
With the support of dynamically turning on/off dirty-pages-log during
live-migration (only for guest RAM regions), we now can create guest
memory regions without dirty-pages-log by default both for guest RAM
regions and other regions backed by file/device.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch extends slightly the current live-migration code path with
the ability to dynamically start and stop logging dirty-pages, which
relies on two new methods added to the `hypervisor::vm::Vm` Trait. This
patch also contains a complete implementation of the two new methods
based on `kvm` and placeholders for `mshv` in the `hypervisor` crate.
Fixes: #2858
Signed-off-by: Bo Chen <chen.bo@intel.com>
The existing test_macvtap is factorized to be able to support both
coldplug and hotplug of a macvtap interface through virtio-net. Adding
the new test_macvtap_hotplug test allows for validating that sending a
TAP file descriptor through control message along with the add-net
command works.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To help with readability, we rely on exec_host_command_status() from the
macvtap test, which replaces the former "bash -c ..." syntax.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Enable specifically for the add-net command the ability to send file
descriptors along with the HTTP request. This is useful to hotplug a
macvtap interface after the VMM has already been started.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever a file descriptor is sent through the control message, it
requires fcntl() syscall to handle it, meaning we must allow it through
the list of syscalls authorized for the HTTP thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When running TDX guest, the Guest Physical Address space is limited by
a shared bit that is located on bit 47 for 4 level paging, and on bit 51
for 5 level paging (when GPAW bit is 1). In order to keep things simple,
and since a 47 bits address space is 128TiB large, we ensure to limit
the physical addressable space to 47 bits when runnning TDX.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When running a TDX guest, we need the virtio drivers to use the DMA API
to share specific memory pages with the VMM on the host. The point is to
let the VMM get access to the pages related to the buffers pointed by
the virtqueues.
The way to force the virtio drivers to use the DMA API is by exposing
the virtio devices with the feature VIRTIO_F_IOMMU_PLATFORM. This is a
feature indicating the device will require some address translation, as
it will not deal directly with physical addresses.
Cloud Hypervisor takes care of this requirement by adding a generic
parameter called "force_iommu". This parameter value is decided based on
the "tdx" feature gate, and then passed to the DeviceManager. It's up to
the DeviceManager to use this parameter on every virtio device creation,
which will imply setting the VIRTIO_F_IOMMU_PLATFORM feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This refactoring ensures all CPUID related operations are centralized in
`arch::x86_64` module, and exposes only two related public functions to
the vmm crate, e.g. `generate_common_cpuid` and `configure_vcpu`.
Signed-off-by: Bo Chen <chen.bo@intel.com>
In order to let a separate process open a TAP device and pass the file
descriptor through the control message mechanism, this patch adds the
support for sending a file descriptor over to the Cloud Hypervisor
process along with the add-net HTTP API command.
The implementation uses the NetConfig structure mutably to update the
list of fds with the one passed through control message. The list should
always be empty prior to this, as it makes no sense to provide a list of
fds once the Cloud Hypervisor process has already been started.
It is important to note that reboot is supported since the file
descriptor is duplicated upon receival, letting the VM only use the
duplicated one. The original file descriptor is kept open in order to
support a potential reboot.
Fixes#2525
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There are some seccomp rules needed for MSHV
in virtio-devices but not for KVM. We only want to
add those rules based on MSHV feature guard.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The micro-http crate now uses recvmsg() syscall in order to receive file
descriptors through control messages. This means the syscall must be
part of the authorized list in the seccomp filters.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The vfio-ioctls crate now contains a KVM feature gate. Make use of it in
Cloud Hypervisor.
That crate has two users. For the vmm crate is it straight-forward. For
the vm-device crate, we introduce a KVM feature gate as well so that the
vmm crate can pass on the configuration.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The help of arguments `memory` and `memory-zone` missing a comma.
Before adding, these parts are as follows:
> hugepage_size=<hugepage_size>hotplug_method=acpi|virtio-mem
After adding, these parts will be:
> hugepage_size=<hugepage_size>,hotplug_method=acpi|virtio-mem
Signed-off-by: Yukiteru Lee <wfly1998@sina.com>
The edk2 upstream has already suppoorted AArch64 Cloud Hypervisor,
and hence we can use upstream edk2 in CI and doc.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This new option allows the user to define a list of SGX EPC sections
attached to a specific NUMA node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to uniquely identify each SGX EPC section, we introduce a
mandatory option `id` to the `--sgx-epc` parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Enabling CONFIG_HYPERV in the Linux Kernel allows
guest to use hyperv clock source. This changes
improves guest performance. Without this changes
we saw slowness in the guest on MSHV.
Signed-off-by: Vineeth Pillai <viremana@linux.microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The guest can see that SGX supports provisioning as it is exposed
through the CPUID. This patch enables the proper backing of this
feature by having the host open the provisioning device and enable
this capability through the hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need a dedicated function to enable the SGX attribute capability
through the Hypervisor abstraction.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The cloud hypervisor tells the VM and the backend to support the PACKED_RING feature,
but it actually processes various variables according to the split ring logic, such
as last_avail_index. Eventually it will cause the following error (SPDK as an example):
vhost.c: 516:vhost_vq_packed_ring_enqueue: *ERROR*: descriptor has been used before
vhost_blk.c: 596:process_blk_task: *ERROR*: ====== Task 0x200113784640 req_idx 0 failed ======
vhost.c: 629:vhost_vring_desc_payload_to_iov: *ERROR*: gpa_to_vva((nil)) == NULL
Signed-off-by: Arafatms <arafatms@outlook.com>
This patch fixes a few things to support TDVF correctly.
The HOB memory resources must contain EFI_RESOURCE_ATTRIBUTE_ENCRYPTED
attribute.
Any section with a base address within the already allocated guest RAM
must not be allocated.
The list of TD_HOB memory resources should contain both TempMem and
TdHob sections as well.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If GICR_CTLR is restored before GICR_PROPBASER and GICR_PENDBASER,
the restoring of the latter registers will fail, as the LPI enable
bit is already set in GICR_CTLR. Therefore, in this commit, the
order of restoring GICR registers is changed.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
This commit implements the GicV3Its Snapshottable trait, including:
- GicV3Its state: GIC registers and ITS registers
- Save/restore logic of GicV3Its state
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit implements two helper functions `gicv3_its_attr_access`
and `gicv3_its_tables_access` to access ITS device attributes and
ITS tables.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
In current code, the ITS device fd of GICv3 will be lost after the
creation of GIC. This commit adds a new `its_device` field for the
`GicV3Its` struct, which will be useful to save the ITS device fd.
This fd will be used in restoring the ITS device.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The BAR calculation code was incorrect for calculating I/O BARs but also
has misleading comments (mixing bits and bytes, first and least
significant, etc).
This change adjusts the algorithm to more closely match the version
described in the PCI specification and takes advantage of Rust's binary
literals for ease of reading. Although this is slightly longer by
calculating the 64-bit and 32-bit paths separately I think this is
easier to read.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previously the same function was used to both create and remove regions.
This worked on KVM because it uses size 0 to indicate removal.
MSHV has two calls -- one for creation and one for removal. It also
requires having the size field available because it is not slot based.
Split set_user_memory_region to {create/remove}_user_memory_region. For
KVM they still use set_user_memory_region underneath, but for MSHV they
map to different functions.
This fixes user memory region removal on MSHV.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The to-be-introduced MSHV rules don't need to contain KVM rules and vice
versa.
Put KVM constants into to a module. This avoids the warnings about
dead code in the future.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The TX rate limiter didn't correctly handle the `-EAGAIN` situation
where it should replenish the rate-limiter tokens. This patch simplifies
the TX rate-limiter code path by following the similar trade-off made in
the TX rate limiter, where we always let the `last` descriptor
go-through (even if it was over the rate limit. We simply stop
processing the oncoming `descriptors` if any.
Fixes: #2817
Signed-off-by: Bo Chen <chen.bo@intel.com>
This warning isn't present on on the read case and we now have better
handling of the -EAGAIN situation including retries.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If writing to the TAP returns EAGAIN then listen for the TAP to be
writable. When the TAP becomes writable attempt to process the TX queue
again.
Fixes: #2807
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The output from getty ("login:") does not always appear. This can be
observed interactively when booting the VM. (Mashing return will bring
it up.) Instead of checking for that string to ensure the VM has booted
instead check for a message from systemd to say it has started the SSH
daemon.
Fixes: #2799
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: use of `expect` followed by a function call
--> test_infra/src/lib.rs:598:10
|
598 | .expect(&format!("Expected '{}' to run", command))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try this: `unwrap_or_else(|_| panic!("Expected '{}' to run", command))`
|
= note: `#[warn(clippy::expect_fun_call)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#expect_fun_call
warning: use of `expect` followed by a function call
--> test_infra/src/lib.rs:605:10
|
605 | .expect(&format!("Expected '{}' to run", command))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try this: `unwrap_or_else(|_| panic!("Expected '{}' to run", command))`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#expect_fun_call
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces the `ProcessorGiccAffinity` struct for the
AArch64 platform. This struct will be created and included into
the SRAT table to enable AArch64 NUMA setup.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This vcpu API is necessary for MSHV related debugging.
These two registers controls the vcpu_run in the
/dev/mshv driver code.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
It ensures all handlers for `ApiRequest` in `control_loop` are
consistent and minimum and should read better.
No functional changes.
Signed-off-by: Bo Chen <chen.bo@intel.com>
It simplifies a bit the `Vmm::control_loop` and reads better to be
consistent with other `ApiRequest` handlers. Also, it removes the
repetitive `ApiError::VmAlreadyCreated` and makes `ApiError::VmCreate`
useful.
No functional changes.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Simplified the build instruction on arm64.
And the binary built with default option can be used for both UEFI and
direct-kernel test.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
AArch64 tests were divided into 2 steps:
- Build and test with 'acpi' feature
- Build and test without 'acpi'
This can be optimized. We need only to build and test once with default
features ('acpi' is enabled).
On AArch64, ACPI only works with UEFI. If UEFI is not available, guest
kernel fall back to use FDT. Most AArch64 test cases boot from direct
kernel, the guest will keep using FDT even if ACPI is enabled. So
nothing is broken.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
We have been building Cloud Hypervisor with command like:
`cargo build --no-default-features --features ...`.
After implementing ACPI, we donot have to use specify all features
explicitly. Default build command `cargo build` can work.
This commit fixed some build warnings with default build option and
changed github workflow correspondingly.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
UEFI need to be loaded to a flash area at the beginning of guest memory
address space. To simulate the flash, we take a piece of RAM and hide
it to the guest. As this is a temporary solution, the hiden RAM for UEFI
should be as little as possible. The size was 64 MiB, that's too much,
4 MiB is enough.
The down side of such simulation is that there is a gap (4 MiB) between
the memory size in VMM's view and that in guest's view. This is to be
fixed by implementing a flash device in future.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This dependency bump needed some manual handling since the API changed
quite a lot regarding some RawFd being changed into either File or
AsRawFd traits.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
error: avoid using `collect()` when not needed
--> vmm/src/vm.rs:630:86
|
630 | let node_id_list: Vec<u32> = configs.iter().map(|cfg| cfg.guest_numa_id).collect();
| ^^^^^^^
...
664 | if !node_id_list.contains(&dest) {
| ---------------------------- the iterator could be used here instead
|
= note: `-D clippy::needless-collect` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_collect
Signed-off-by: Bo Chen <chen.bo@intel.com>
Issue from beta verion of clippy:
Error: --> vm-virtio/src/queue.rs:700:59
|
700 | if let Some(used_event) = self.get_used_event(&mem) {
| ^^^^ help: change this to: `mem`
|
= note: `-D clippy::needless-borrow` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
Signed-off-by: Bo Chen <chen.bo@intel.com>
Issue from beta verion of clippy:
error: field is never read: `type`
--> vmm/src/cpu.rs:235:5
|
235 | pub r#type: u8,
| ^^^^^^^^^^^^^^
|
= note: `-D dead-code` implied by `-D warnings`
Signed-off-by: Bo Chen <chen.bo@intel.com>
Initialize MTRR defType register the same way the KVM code does - WB caching by default.
Tested with latest mshv code.
Without this patch, these lines are present in guest serial log:
[ 0.000032] x86/PAT: MTRRs disabled, skipping PAT initialization too.
[ 0.000036] CPU MTRRs all blank - virtualized system.
This indicates the guest is detecting the set MTRR.
Signed-off-by: Nuno Das Neves <nudasnev@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The Linux kernel expects that any PCI devices that advertise I/O bars
have use an address that is within the range advertised by the bus
itself. Unfortunately we were not advertising any I/O ports associated
with the PCI bus in the ACPI tables.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch addresses this issue https://github.com/rust-lang/rust-bindgen/pull/2064.
While we access field of packed struct the compiler can generate the
correct code to create a temporary variable to access the packed struct
field. Access withing {} ensures that.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The virtio_balloon test is a bit flaky since we can't really know how
much the balloon is gonna be deflated when the guest is under memory
pressure. That's why it's safer to simply check that the balloon is not
the initial size anymore.
One small detail, but we don't need to check for the balloon size to be
higher than 0 since the returned value is a u64.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since we ran into issues while using the Azure credentials plugin for
Jenkins, let's rely directly on the Azure CLI to download the Windows
guest image along with the modified OVMF firmware.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To debug the FDT (Flattened Device Tree), we usually need to modify
source code to save the generted DTB data to disk, and use 'dtc' command
to decode the binary file into a text file to analyze.
It would be ideal if the FDT content can be seen in log.
This commit makes it real by:
- Introducing 'fdt' crate for parsing FDT.
- Printing the content of the FDT in tree view.
The parsing and printing only happen when Debug level logging enabled.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Additionally, he disk creation routine is extended to support NTFS and
variable image size.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
In order to allow a hotplugged vCPU to be assigned to the correct NUMA
node in the guest, the DSDT table must expose the _PXM method for each
vCPU. This method defines the proximity domain to which each vCPU should
be attached to.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The _PXM method always return 0, which is wrong since the SRAT might
tell differently. The point of the _PXM method is to be evaluated by the
guest OS when some new memory slot is being plugged, but this will never
happen for Cloud Hypervisor since using NUMA nodes along with memory
hotplug only works for virtio-mem.
Memory hotplug through ACPI will only happen when there's only one NUMA
node exposed to the guest, which means the _PXM method won't be needed
at all.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make sure the unique PCI bus is tied to the default NUMA node 0, and
update the documentation to let the users know about this special case.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Fixed wrong MPIDR value setting for VCPUs in FDT.
The wrong setting made only 16 VCPUs can be enabled at most, all other
VCPUs were showing off-line.
The issue was introduced when we were migrating FDT-generating code to
vmm-fdt crate.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Now that vhost crate allows the caller to set the header flags, we can
set NEED_REPLY whenever the REPLY_ACK protocol feature is supported from
both ends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The new test_virtio_balloon() is to verify if the 'deflate_on_oom'
parameter works. Its testing result is as follows:
1. Start a 4G guest with 2G balloon, check memory once starts up.
total_mem is 4294967296 bytes
actual_mem is 2147483648 bytes
orig_balloon is 2147483648 bytes
total used free shared buff/cache available
Mem: 3.8Gi 2.1Gi 1.6Gi 0.0Ki 140Mi 1.6Gi
Swap: 0B 0B 0B
2. Run a command in guest to eat up 25G memory, and check again.
total_mem is 4294967296 bytes
actual_mem is 3121610752 bytes
deflated_balloon is 1173356544 bytes
total used free shared buff/cache available
Mem: 3.8Gi 1.2Gi 2.6Gi 0.0Ki 49Mi 2.5Gi
Swap: 0B 0B 0B
From above, we can notice the balloon memory indeed deflates from
2147483648 bytes to 1173356544 bytes once an oom is going to be
triggered.
Signed-off-by: Fei Li <lifei.shirley@bytedance.com>
For vhost-user devices, memory should be shared between CLH and
vhost-user backend. However, madvise DONTNEED doesn't working in
this case. So, let's use fallocate PUNCH_HOLE to discard those
memory regions instead.
Signed-off-by: Li Hangjing <lihangjing@bytedance.com>
Sometimes we need balloon deflate automatically to give memory
back to guest, especially for some low priority guest processes
under memory pressure. Enable deflate_on_oom to support this.
Usage: --balloon "size=0,deflate_on_oom=on" \
Signed-off-by: Fei Li <lifei.shirley@bytedance.com>
Since using the VIRTIO configuration to expose the virtual IOMMU
topology has been deprecated, the virtio-iommu implementation must be
updated.
In order to follow the latest patchset that is about to be merged in the
upstream Linux kernel, it must rely on ACPI, and in particular the newly
introduced VIOT table to expose the information about the list of PCI
devices attached to the virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the reconnection thread took on the responsibility to handle
backend initiated requests as well, the variable naming should reflect
this by avoiding the "reconnect" prefix.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A new method has been introduced for WindowsGuest, that would poll on
the SSH connection and try to execute a test command. When successfull,
the polling stops and the guest is considered finished boot. The
downside of this method is that the network setup is needed always even
if the test doesn't use network, however it complies with the behavior
of other tests. I also observe a bit more stability on the local test
run, however it still appears to be a resource issue so some sporadic
fails are possible on a slower machine.
The hardcoded timeouts for guest boot and DHCP setup have been removed.
The dnsmasq invocation uses `--bind-dynamic` so then the daemon can be
started while the target interface could be down yet.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Co-authored-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Vhost user INFLIGHT_SHMFD protocol feature supports inflight I/O
tracking, this commit implement the vhost-user device (master) support
of the feature. Till this commit, specific vhost-user devices (blk, fs,
or net) have not enable this feature.
Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
Cloud Hypervisor is an open source project licensed under the [Apache v2 License](https://opensource.org/licenses/Apache-2.0) and the [BSD 3 Clause](https://opensource.org/licenses/BSD-3-Clause) license.
Cloud Hypervisor is an open source project licensed under the [Apache v2
License](https://opensource.org/licenses/Apache-2.0) and the [BSD 3
$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-arm64.img focal-server-cloudimg-arm64.raw
$ popd
```
## Containerized build
## UEFI booting
If you want to build and test Cloud Hypervisor without having to install all the required dependencies, you can also turn to the development script: dev_cli.sh.
This part introduces how to build EDK2 firmware and boot Cloud Hypervisor with it.
Currently Cloud Hypervisor only support hot plugging of CPU devices.
Currently Cloud Hypervisor supports hot plugging of CPUs devices (x86 only), PCI devices and memory resizing.
## Kernel support
@@ -9,7 +9,7 @@ or by using this kernel patch (available in 5.5-rc1 and later): https://git.kern
## CPU Hot Plug
Extra vCPUs can be added and removed from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
Extra vCPUs can be added and removed from a running `cloud-hypervisor` instance. This is controlled by two mechanisms:
1. Specifying a number of maximum potential vCPUs that is greater than the number of default (boot) vCPUs.
2. Making a HTTP API request to the VMM to ask for the additional vCPUs to be added.
@@ -65,7 +65,7 @@ As per adding CPUs to the guest, after a reboot the VM will be running with the
### ACPI method
Extra memory can be added from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
Extra memory can be added from a running `cloud-hypervisor` instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM. In the case of expanding the memory for the VM the new memory will be hotplugged into the running VM, if reducing the size of the memory then change will take effect after the next reboot.
@@ -157,7 +157,9 @@ The same API can also be used to reduce the desired RAM for a VM. It is importan
## PCI Device Hot Plug
Extra PCI devices can be added and removed from a running Cloud Hypervisor instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
Extra PCI devices can be added and removed from a running `cloud-hypervisor` instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
Note: On AArch64 platform, PCI device hotplug can only be achieved using ACPI. Please refer to the [documentation](arm64.md#uefi-booting) for more information.
To use PCI device hotplug start the VM with the HTTP server.
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size. If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the specified size for the VMM to use. Failure to do so may result in strange VMM behaviour.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `hotplug_method`
Selects the way of adding and/or removing memory to/from a booted VM.
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size.
If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the
specified size for the VMM to use. Failure to do so may result in strange VMM
behaviour, e.g. error with `ReadKernelImage` is common. If there is a strange
error with `hugepages` enabled, just disable it or check whether there are enough
huge pages.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `prefault`
Specifies if the memory must be `mmap(2)` with `MAP_POPULATE` flag.
By triggering prefault, one can allocate all required physical memory and create
its page tables while calling `mmap`. With physical memory allocated, the number
of page faults will decrease during running, and performance will also improve.
Note that boot of VM will be slower with `prefault` enabled because of allocating
physical memory and creating page tables in advance, and physical memory of the
specified size will be consumed quickly.
This option only takes effect at boot of VM. There is also a `prefault` option in
restore and its choice will overwrite `prefault` in memory.
By default this option is turned off.
_Example_
```
--memory size=1G,prefault=on
```
## Advanced Parameters
`MemoryZoneConfig` or what is known as `--memory-zone` from the CLI perspective
@@ -155,14 +184,16 @@ struct MemoryZoneConfig {
file: Option<PathBuf>,
shared: bool,
hugepages: bool,
hugepage_size: Option<u64>,
host_numa_node: Option<u32>,
hotplug_size: Option<u64>,
hotplugged_size: Option<u64>,
prefault: bool,
}
```
```
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>"
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off"
```
This parameter expects one or more occurences, allowing for a list of memory
@@ -250,24 +281,30 @@ _Example_
--memory-zone id=mem0,size=1G,shared=on
```
### `hugepages`
### `hugepages` and `hugepage_size`
Specifies if the memory zone must be `mmap(2)` with `MAP_HUGETLB` and
`MAP_HUGE_2MB`flags. This performs a memory zone mapping relying on 2MiB
pages instead of the default 4kiB pages.
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size.
If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the
specified size for the VMM to use. Failure to do so may result in strange VMM
behaviour, e.g. error with `ReadKernelImage` is common. If there is a strange
error with `hugepages` enabled, just disable it or check whether there are enough
Specifies if the memory must be `mmap(2)` with `MAP_POPULATE` flag.
By triggering prefault, one can allocate all required physical memory and create
its page tables while calling `mmap`. With physical memory allocated, the number
of page faults will decrease during running, and performance will also improve.
Note that boot of VM will be slower with `prefault` enabled because of allocating
physical memory and creating page tables in advance, and physical memory of the
specified size will be consumed quickly.
This option only takes effect at boot of VM. There is also a `prefault` option in
restore and its choice will overwrite `prefault` in memory.
By default this option is turned off.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,prefault=on
```
## NUMA settings
`NumaConfig` or what is known as `--numa` from the CLI perspective has been
@@ -343,15 +404,16 @@ it allows for specifying the distance between each NUMA node.
```rust
structNumaConfig{
id: u32,
guest_numa_id: u32,
cpus: Option<Vec<u8>>,
distances: Option<Vec<NumaDistance>>,
memory_zones: Option<Vec<String>>,
sgx_epc_sections: Option<Vec<String>>,
}
```
```
--numa <numa> Settings related to a given NUMA node "id=<node_id>,cpus=<cpus_id>,distances=<list_of_distances_to_destination_nodes>,memory_zones=<list_of_memory_zones>"
--numa <numa> Settings related to a given NUMA node "guest_numa_id=<node_id>,cpus=<cpus_id>,distances=<list_of_distances_to_destination_nodes>,memory_zones=<list_of_memory_zones>,sgx_epc_sections=<list_of_sgx_epc_sections>"
```
### `guest_numa_id`
@@ -384,18 +446,20 @@ integer of 8 bits.
For instance, if one needs to attach all CPUs from 0 to 4 to a specific node,
the syntax using `-` will help define a contiguous range with `cpus=0-4`. The
same example could also be described with `cpus=0:1:2:3:4`.
same example could also be described with `cpus=[0,1,2,3,4]`.
A combination of both `-` and `:` separators is useful when one might need to
A combination of both `-` and `,` separators is useful when one might need to
describe a list containing all CPUs from 0 to 99 and the CPU 255, as it could
simply be described with `cpus=0-99:255`.
simply be described with `cpus=[0-99,255]`.
As soon as one tries to describe a list of values, `[` and `]` must be used to
After the successful build, the resulting firmware binaries are available under `Build/OvmfCh/DEBUG_GCC5/FV` underneath the edk2 checkout.
After the successful build, the resulting firmware binaries are available under `Build/CloudHvX64/DEBUG_GCC5/FV` underneath the edk2 checkout.
## Using OVMF Binaries
Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there.
To make Cloud Hypervisor use UEFI boot, pass the `OVMF.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
To make Cloud Hypervisor use UEFI boot, pass the `CLOUDHV.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
The same firmware can be used with Cloud Hypervisor or with QEMU. This is particularly useful if using QEMU for the preparation phase.
## Building UEFI Firmware with Compatibility Support Module (CSM)
CSM is a module that allows to boot legacy operating systems using the OVMF firmware. OVMF can embed a CSM build of SeaBIOS. To build the SeaBIOS with CSM support, add `CONFIG_CSM=y` to `.config` before the build. The outcome `out/Csm16.bin` is to be moved into `OvmfPkg/Csm/Csm16/Csm16.bin` before OVMF is built. Then, the OVMF build will have to be passed the `-D CSM_ENABLE` option in order to generate a legacy aware UEFI firmware. At the current stage, all the necessary patches are included in the Cloud Hypervisor specific [SeaBIOS branch](https://github.com/cloud-hypervisor/seabios/tree/ch). Taking into account the previous instructions, the modified command sequence to compile an OVMF binary with CSM support is the following one:
Please note, that the CSM support has currently only been tested with Linux guests. There are no plans to provide legacy support for other OSes (e.g. Windows).
VFIO-user is an *experimental* protocol for allowing devices to be implemented in another process and communicate over a socket; ie.e VFIO-user is to VFIO as virtio is to vhost-user.
The protocol is documented here: https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst
The Cloud Hypervisor support for such devices is *experimental*. Not all Cloud Hypervisor functionality is supported in particular: virtio-mem and iommu are not supported.
## Usage
The `--user-device socket=<path>` parameter is used to create a vfio-user device when creating the VM specifying the socket to connect to. The device can also be hotplugged with `ch-remote add-user-device socket=<path>`.
## Example (GPIO device)
There is a simple GPIO device included in the libvfio-user repository: https://github.com/nutanix/libvfio-user#gpio
// It is safe to implement BytesValued; all members are simple numbers and any value is valid.
// SAFETY: All members are simple numbers and any value is valid.
unsafeimplByteValuedforTestCap{}
implPciCapabilityforTestCap{
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
